Projects per year
Abstract
Safety is still one of the major research challenges in reinforcement learning (RL). In this paper, we address the problem of how to avoid safety violations of RL agents during exploration in probabilistic and partially unknown environments. Our approach combines automata learning for Markov Decision Processes (MDPs) and shield synthesis in an iterative approach. Initially, the MDP representing the environment is unknown. The agent starts exploring the environment and collects traces. From the collected traces, we passively learn MDPs that abstractly represent the safety-relevant aspects of the environment. Given a learned MDP and a safety specification, we construct a shield. For each state-action pair within a learned MDP, the shield computes exact probabilities on how likely it is that executing the action results in violating the specification from the current state within the next k steps. After the shield is constructed, the shield is used during runtime and blocks any actions that induce a too large risk from the agent. The shielded agent continues to explore the environment and collects new data on the environment. Iteratively, we use the collected data to learn new MDPs with higher accuracy, resulting in turn in shields able to prevent more safety violations. We implemented our approach and present a detailed case study of a Q-learning agent exploring slippery Gridworlds. In our experiments, we show that as the agent explores more and more of the environment during training, the improved learned models lead to shields that are able to prevent many safety violations.
Original language | English |
---|---|
Title of host publication | Leveraging Applications of Formal Methods, Verification and Validation. Verification Principles - 11th International Symposium, ISoLA 2022, Proceedings |
Subtitle of host publication | ISoLA 2022 |
Editors | Tiziana Margaria, Bernhard Steffen |
Place of Publication | Cham |
Publisher | Springer |
Pages | 335-359 |
Number of pages | 25 |
ISBN (Electronic) | 978-3-031-19849-6 |
ISBN (Print) | 978-3-031-19848-9 |
DOIs | |
Publication status | Published - 2022 |
Event | ISOLA 2022: 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation - Rhodos, Greece Duration: 22 Oct 2022 → 30 Oct 2022 https://2022.isola-conference.org/ |
Publication series
Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|
Volume | 13701 LNCS |
ISSN (Print) | 0302-9743 |
ISSN (Electronic) | 1611-3349 |
Conference
Conference | ISOLA 2022 |
---|---|
Abbreviated title | ISOLA 2022 |
Country/Territory | Greece |
City | Rhodos |
Period | 22/10/22 → 30/10/22 |
Internet address |
Keywords
- Automata learning
- Markov Decision Processes
- Shielding
ASJC Scopus subject areas
- Theoretical Computer Science
- General Computer Science
Fingerprint
Dive into the research topics of 'Automata Learning meets Shielding'. Together they form a unique fingerprint.Projects
- 1 Finished
-
EU - FOCETA - Foundations for continuous engineering of trustworthy autonomy
Bloem, R. (Co-Investigator (CoI))
1/10/20 → 31/10/23
Project: Research project