cs.LGNov 18, 2026

VESPA: Verifiable Symbolic Policy Adaptation for Safe Reinforcement Learning

Authors: Paul König, Sascha Marton, Stefan Lüdtke, Christian Bartelt

Organizations: COgnitive SoftwaRE (CORE), Clausthal University of Technology · Institute for Visual and Analytic Computing, University of Rostock

Abstract

Safe reinforcement learning (RL) aims to prevent agents from taking actions that are dangerous or destructive to their environment. This is especially important for safety-critical domains, such as robotics or autonomous driving. Formal verification can be used to prove agent safety through safety constraints. However, in practice, many policies do violate safety constraints, and thus are verified to be unsafe. Since the goal is to obtain a safe policy, we would need to retrain a new policy from scratch and then verify it again until a candidate happens to pass. Both training and verification are usually not cheap, which makes this process inefficient. We introduce VESPA (VE\textbf{VE}rifiable S\textbf{S}ymbolic P\textbf{P}olicy A\textbf{A}daptation), a framework for targeted fine-tuning of decision tree policies. We show that using counterexamples for policy adaptation can make existing unsafe policies safe, or at least make considerable progress towards safety. Targeted fine-tuning of the existing policy on those counterexamples enables it to learn how to avoid violating safety constraints. We thus use the result of a formal verification as a training signal for fine-tuning. On CartPole, VESPA raises the mean number of verified safe steps from 63.8 to at least 92.7, reaching after two iterations what the same loop without counterexamples reaches after ten. On MountainCarContinuous, VESPA turns an unsafe policy into a safe policy in 53 of 60 runs. We conclude that counterexamples from a verifier are an effective training signal, which can be utilized to systematically improve the safety of policies. Our implementation is available at: https://anonymous.4open.science/r/VESPA-Submission/

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SafeAdapt: Provably Safe Policy Updates in Deep Reinforcement Learning

    Apr 10, 2026Maksim Anisimov, Francesco Belardinelli, Matthew WickerSafe RLContinual RL

  2. RePolicy: Learning to Select and Apply Safety Policies for Agent Safeguards

    Aug 25, 2026Houcheng Jiang, Boxuan Zhang, Qiyong Zhong +3LLM Agent SafetyRL for LLM Agents