Importance-driven Safety and Performance Testing for Deep Reinforcement Learning

TY - JOUR

T1 - Importance-driven Safety and Performance Testing for Deep Reinforcement Learning

AU - Pranger, Stefan

AU - Tappler, Martin

AU - Chockler, Hana

AU - Konighofer, Bettina

PY - 2026/9/7

Y1 - 2026/9/7

N2 - In many Deep Reinforcement Learning (RL) problems, the decisions of a trained policy differ in their impact on the policy’s expected safety and performance. Testing should therefore focus on states in which the agent’s decisions have the greatest influence on these outcomes. We extend a recently proposed approach for testing RL agents in states that are most relevant to safety to the joint evaluation of safety and performance. Our method rigorously computes a ranking of state importance with respect to both objectives across the state space. At each iteration, the testing framework derives optimistic and pessimistic estimates of safety and performance, which provide lower and upper bounds on the expected outcomes of policy execution across all modeled states. Upon convergence, the approach partitions the state space into regions according to whether the specified safety or performance properties are satisfied, thereby providing insights into the policy’s weaknesses. Ourapproach is characterized by two key properties. (1) Optimal Test-Case Selection: at any point during the testing process, the policy is evaluated in the states that are most critical for assessing safety and performance. (2) Formal Guarantees: by evaluating the policy in only a fraction of the state space, our approach nevertheless provides formal verification guarantees over the entire modeled state space. We provide a detailed evaluation on several examples, demonstrating that our method identifies unsafe and suboptimal policy behavior with low testing effort.

AB - In many Deep Reinforcement Learning (RL) problems, the decisions of a trained policy differ in their impact on the policy’s expected safety and performance. Testing should therefore focus on states in which the agent’s decisions have the greatest influence on these outcomes. We extend a recently proposed approach for testing RL agents in states that are most relevant to safety to the joint evaluation of safety and performance. Our method rigorously computes a ranking of state importance with respect to both objectives across the state space. At each iteration, the testing framework derives optimistic and pessimistic estimates of safety and performance, which provide lower and upper bounds on the expected outcomes of policy execution across all modeled states. Upon convergence, the approach partitions the state space into regions according to whether the specified safety or performance properties are satisfied, thereby providing insights into the policy’s weaknesses. Ourapproach is characterized by two key properties. (1) Optimal Test-Case Selection: at any point during the testing process, the policy is evaluated in the states that are most critical for assessing safety and performance. (2) Formal Guarantees: by evaluating the policy in only a fraction of the state space, our approach nevertheless provides formal verification guarantees over the entire modeled state space. We provide a detailed evaluation on several examples, demonstrating that our method identifies unsafe and suboptimal policy behavior with low testing effort.

M3 - Article

SN - 1364-503X

JO - PHILOSOPHICAL TRANSACTIONS- ROYAL SOCIETY OF LONDON SERIES A MATHEMATICAL PHYSICAL AND ENGINEERING SCIENCES

JF - PHILOSOPHICAL TRANSACTIONS- ROYAL SOCIETY OF LONDON SERIES A MATHEMATICAL PHYSICAL AND ENGINEERING SCIENCES

ER -