Temporal Difference Learning of Probabilistic Reach-Avoid Value Functions

Elias Fontanari; Pietro Noah Crestaz; Matteo Saveriano; Andrea Del Prete

cs.ROOct 20, 2026

Temporal Difference Learning of Probabilistic Reach-Avoid Value Functions

Authors: Elias Fontanari, Pietro Noah Crestaz, Matteo Saveriano, Andrea Del Prete

Organizations: Industrial Engineering Department, University of Trento, Trento, Italy. · LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France.

Abstract

Reach-avoid sets, defining the states from which a task can be completed while avoiding unsafe regions, are fundamental tools for deploying robots in environments subject to disturbances and uncertainty, as well as for runtime task-feasibility monitoring, allowing prediction of whether a state is safe and will lead to successful task completion. Classical Hamilton-Jacobi Reachability (HJR) and its recent neural extensions (e.g., Lipschitz Continuous Reachability Learning, DeepReach) treat disturbances adversarially, computing the reach-avoid set under worst-case disturbances from a bounded set. This guarantees safety but can yield overly conservative sets when disturbances are stochastic rather than adversarial, and additionally requires the disturbance set to be known or specified in advance. We propose Probabilistic reach-avoid Value Function (PVF), a self-supervised method, applicable to black-box systems, for computing reach-avoid sets as the states with a certain reach-avoid probability under stochastic, potentially unbounded disturbances. We evaluate our approach against HJR-based baselines on systems of increasing complexity, up to a Hopper locomotion task, and show that our probabilistic sets are less conservative and more accurate than adversarial HJR sets when disturbances are non-adversarial.

Figures & tables

Fig. 1: Overview of the PVF pipeline. Data collection: Trajectories are collected by running a nominal policy $\pi(x)$ on black-box systems (e.g., legged robots, manipulators, quadrotors) subject to stochastic disturbances $w$ , and stored as transition pairs $(x,x^{\prime})$ in a replay buffer. Self-supervision: Piecewise target labels $V_{\text{target}}(x)$ are assigned based on constraint violation ( $0$ ), task completion ( $1$ ), or bootstrapped value evaluation ( $V_{\theta}(x^{\prime})$ ). Training: A neural network parameterizing $V_{\theta}(x)$ is optimized via temporal-difference loss to output the continuous reach-avoid success probability.

Fig. 2: Reach-avoid value functions on the scalar toy problem. Comparison between the proposed Probabilistic Value Function ( $V_{\text{PVF}}$ , left) and the adversarial Lipschitz Continuous Reachability Learning baseline ( $V_{\text{LCRL}}$ , right). Green regions indicate states identified as safe (i.e., able to reach the target set without violating constraints).

Fig. 3: Visual comparison between PVF (left) and LCRL (right) in terms of accuracy, for Drones racing experiment. $x$ and $y$ indicate the initial position of the ego drone in the horizontal plane. The opposing drone starts at $x=0.4$ m and $y=0.0$ m

Fig. 4: Comparison between PVF and LCRL in terms of accuracy, for Drones racing experiment. Thresholds for fail detection reported near the medians.

Fig. 5: Snapshot of the Hopper legged system.

Fig. 6: Comparison between PVF and LCRL in terms of accuracy, for Hopper experiment. Thresholds for fail detection reported near the medians.

	Drones Racing	Hopper
LCRL	3175 s	361 s
PVF data collection	677 s	7145 s
PVF training	9367 s	23502 s
PVF total	10044 s	30647 s

TABLE I: Mean computation time across the different runs.

Metric	Value (%)
True Positive	74.6
False Positive	23.6
True Negative	0.0
False Negative	0.0
Unclear Cases	1.8

TABLE II: Failures-only Network results for Hopper.

	Noise Level ( $4\sigma$ )
	$0.10$	$0.15$	$0.20$
Shielded	$99.1\pm 1.6$	$97.2\pm 4.2$	$91.0\pm 8.3$
Unshielded	$97.1\pm 5.2$	$90.8\pm 7.7$	$83.1\pm 10.0$

TABLE III: Mean

\pm

standard deviation of success counts (out of 100 trials) across 11 seeds, by noise level.

Reachability Analysis

Reachability analysis determines which states a system can enter from given initial conditions, actions, and disturbances.

See all 78 papers

Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning
May 12, 2026Jingduo Pan, Taoran Wu, Yiling Xue +1Constrained RL Safe RL
Key Laboratory of System Software (Chinese Academy of Sciences), Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China
Gradient-Free Neural Hamilton-Jacobi Reachability for Scalable Safety-Critical Control
Sep 12, 2026Zeyuan Feng, Ali Fuat Sahin, Santiago Thorup +1Reachability Analysis Robotic Control
Department of Mechanical Engineering, École Polytechnique Fédérale de Lausanne, Switzerland
Winning a Won Game: Strict Reach-Avoid-Stay Control Barrier Functions for High-Dimensional Black-Box Systems
Sep 16, 2026Donggeon David Oh, Duy P. Nguyen, Gongkai Yuan +3Robotic Control Control Barrier Functions
Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218, USA

May 12, 2026J/K move · Enter open · S save

Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning

Jingduo Pan, Taoran Wu, Yiling Xue, Bai Xue

Key Laboratory of System Software (Chinese Academy of Sciences), Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China

Constrained RL Safe RL

Open paper AskPDF arXiv

We study stochastic minimum-cost reach-avoid reinforcement learning, where an agent must satisfy a reach-avoid specification with probability at least

p

while minimizing expected cumulative costs in stochastic environments. Existing safe and constrained reinforcement learning methods typically fail to jointly enforce probabilistic reach-avoid constraints and optimize cost in the learning setting in stochastic environments. To address this challenge, we introduce reach-avoid probability certificates (RAPCs), which identify states from which stochastic reach-avoid constraints are satisfiable. Building on RAPCs, we develop a contraction-based Bellman formulation that serves as a principled surrogate for integrating reach-avoid considerations into reinforcement learning, enabling cost optimization under probabilistic constraints. We establish almost sure convergence of the proposed algorithms to locally optimal policies with respect to the resulting objective. Experiments in the MuJoCo simulator demonstrate improved cost performance and consistently higher reach-avoid satisfaction rates.

Abstract

Reachability Analysis

Explore similar work

Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning

Gradient-Free Neural Hamilton-Jacobi Reachability for Scalable Safety-Critical Control

Winning a Won Game: Strict Reach-Avoid-Stay Control Barrier Functions for High-Dimensional Black-Box Systems