Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields

Siwei Ju; Jan Peters; Oleg Arenz

cs.ROOct 14, 2026

Safe Reinforcement Learning with Receding-Horizon Safety Critic and Differentiable Action Shields

Authors: Siwei Ju, Jan Peters, Oleg Arenz

Organizations: Department of Computer Science, Technical University of Darmstadt, Germany. · Hessian.AI. · German Research Center for AI (DFKI), Research Department: Systems AI for Robot Learning. · Robotics Institute Germany (RIG).

Abstract

Despite the Markov assumption, safety failures in robot reinforcement learning rarely occur instantaneously; they typically emerge over several steps as the policy enters unfamiliar states under disturbances and domain shift. We address this limitation by learning a receding-horizon safety critic that predicts whether a state will remain non-terminating over the next receding horizon, enabling anticipatory detection of impending locomotion failures. The critic is trained from rollout-derived horizon labels with a composite objective that combines horizon classification, control-barrier-function-inspired temporal consistency, and smoothness regularization, and we aggregate an ensemble of critics with a conservative Conditional Value at Risk estimate to account for uncertainty. Building on the learned critic, we derive a closed-form, differentiable CBF-style safety filter that minimally modifies the policy action to improve predicted safety, and integrate it into PPO with an anticipation loss to reduce policy-filter mismatch during training. In simulated quadruped locomotion, our method improves robustness under action noise, external perturbations, and terrain shifts, reducing catastrophic failures while maintaining competitive returns.

Figures & tables

Fig. 1: Overview of the training pipeline. Rollouts are collected in simulation under domain randomization and disturbances, and states are labeled by horizon- $H$ safety to populate separate safe/unsafe buffers (with discounted weighting for delayed violations). A predictive safety critic is trained with a composite objective (labeling, temporal consistency, and smoothness), and an ensemble is aggregated via CVaR to obtain a conservative safety estimate and uncertainty $\sigma_{t}$ . At each step, a differentiable closed-form safety layer projects the raw policy action to a minimally modified safe action, which is used both during PPO training (with an anticipation loss to reduce interventions) and at deployment for safety-shielded control.

Fig. 2: Top row: success rate; bottom row: average return. Columns correspond to four evaluation suites (action-noise and external-perturbation, each on flat terrain and heightfield terrain). Curves show mean performance over independent runs for PPO-RL, PPO-RL-NCBF, and our method, with shaded bands indicating standard error across different seeds. The x-axis denotes the perturbation/noise sampling probability.

Fig. 3: Example rollout diagnostics of the predictive safety shield. Top: ensemble safety predictions (gray, “single prediction”), their CVaR-aggregated safety estimate

\hat{p}_{\mathrm{CVaR}}(x_{t})

(black), and the ensemble standard deviation

\sigma_{t}

(red, right axis). Peaks in

\sigma_{t}

indicate increased epistemic uncertainty and coincide with sharp drops in the predicted safety. Bottom: action magnitudes before and after shielding, showing the raw action norm

\|u_{t}^{0}\|

(orange), the shielded action norm

\|\tilde{u}_{t}\|

(blue), and the correction magnitude

\|\tilde{u}_{t}-u_{t}^{0}\|

(green). Non-zero corrections occur primarily during high-uncertainty and low-safety intervals, illustrating successful interventions.

Fig. 4: Ablation on prediction horizon H for selecting the safety-critic/shield horizon under sim-to-sim domain shift. Experiment settings are similar to Fig. 2 . Curves show mean performance, with shaded bands indicating standard error across different seeds

Hyperparameter	Symbol	Value
Safety prediction horizon	$H$	5
Safety margin (threshold)	$\gamma_{c}$	0.5
Unsafe-label discount	$\gamma_{\text{label}}$	0.95
CVaR tail probability	$\alpha_{\text{cvar}}$	0.4
Ensemble size	$M$	5
CBF gain (for $\tilde{\alpha}(s)=\eta_{\text{CBF}}s$ )	$\eta_{\text{CBF}}$	0.5

TABLE I: Hyperparameters.

Fig. 5: Comparison of state-only and state-action safety critics. The state-action safety critic demonstrates better generalizability and robustness, particularly under perturbations and domain shifts. The results are shown for success rate (top) and average return (bottom) across different perturbation settings. The result is aggregated over 5 seeds, and the shaded area shows the standard error. The state-action model outperforms the state-only model under external perturbation and domain shift.

Fig. 6: Prediction of the state-based safety critic under domain shift. Top: ensemble safety predictions (gray), CVaR-aggregated safety prediction (black), and ensemble standard deviation

\sigma_{t}

(red dashed, right axis). Middle: action norms before and after shielding,

\|u_{t}^{0}\|

(orange dashed),

\|\tilde{u}_{t}\|

(blue), and correction magnitude

\|\tilde{u}_{t}-u_{t}^{0}\|

(green). Bottom: nominal-model-based one-step safety estimate

h_{\phi}(\hat{f}_{\mathrm{nom}}(x_{t},u_{t}))

(orange) versus the corresponding value evaluated on the realized next state

h(x_{t+1})

(black dashed). Under heightfield terrain with enlarged perturbations, the nominal estimate can diverge substantially from the next-state reference (e.g., around

t\!\approx\!175

–

190

), leading to spurious risk alarms and unnecessary interventions.

Metric	Experiment	State-Action Safety Critic	SafeFall
Accuracy	Perturbation - Flat Terrain	0.6160 $\pm$ 0.0200	0.6200 $\pm$ 0.0265
Accuracy	Perturbation - Heightfield Terrain	0.5800 $\pm$ 0.0254	0.5480 $\pm$ 0.0232
False Positives	Perturbation - Flat Terrain	0.0800 $\pm$ 0.0086	0.1680 $\pm$ 0.0171
False Positives	Perturbation - Heightfield Terrain	0.1760 $\pm$ 0.0197	0.2480 $\pm$ 0.0113
False Negatives	Perturbation - Flat Terrain	0.0736 $\pm$ 0.0053	0.0160 $\pm$ 0.0020
False Negatives	Perturbation - Heightfield Terrain	0.0612 $\pm$ 0.0071	0.0129 $\pm$ 0.0014

TABLE II: Comparison of performance metrics between our state-action safety critic and SafeFall predictor under perturbations on flat and heightfield terrains.

Fig. 7: Safety predictions from the SafeFall predictor across several random episodes. The trajectory length is set to 250 timesteps. SafeFall’s predictions tend to be more binary, predicting failure only shortly before the event. This contrasts with our receding-horizon method, which provides earlier and more gradual safety estimates.

Fig. 8: Comparison of SafeFall and our method in Safe RL training using each as a safety critic. The plot compares success rate (top row) and average return (bottom row) across different levels of action noise and external perturbations. Both methods show similar robustness under action noise, but our method outperforms SafeFall when faced with perturbations, demonstrating its superior generalizability.

Fig. 9: Complete ablation on prediction horizon

H

. Success rate (top row) and average return (bottom row) under action noise and external perturbations on flat and heightfield terrain. The x-axis represents the action noise or disturbance sampling probability. Curves show mean performance over 5 seeds; shaded bands indicate standard error.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 217 papers

SAFTER: Safety-Aware Filtering Through Explicit Recoverability for Robust Reinforcement Learning under Distribution Shift
Oct 21, 2026Benedictus C. G. Cinun, Bin HuSafe RL Reinforcement Learning
Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA · Department of Engineering Technology, University of Houston, Houston, TX 77004, USA
Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering
Jun 5, 2026Aditya Shirwatkar, Sebastian Sanokowski, Shishir Kolathaya +2Robot Safety Safety Filtering
Robert Bosch Center for Cyber Physical Systems, Indian Institute of Science, Bangalore, India · Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, Munich, Germany · Department of Computer Science & Automation, Indian Institute of Science, Bangalore, India +2
Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning
May 26, 2026Dhruv S. Kushwaha, Zoleikha A. BironKoopman Operator Learning Control Barrier Functions

Oct 21, 2026J/K move · Enter open · S save

SAFTER: Safety-Aware Filtering Through Explicit Recoverability for Robust Reinforcement Learning under Distribution Shift

Benedictus C. G. Cinun, Bin Hu

Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77004, USA · Department of Engineering Technology, University of Houston, Houston, TX 77004, USA

Safe RL Reinforcement Learning Robust RL Control Barrier Functions Sample-Efficient RL Robot Navigation

Open paper AskPDF arXiv

Learning safe policies that generalize to unseen environments is difficult under bounded actuation, where an unsafe decision may become unrecoverable. Conventional control barrier function (CBF) filters enforce only instantaneous safety and ignore future recoverability. We propose SAFTER (Safety-Aware Filtering Through Explicit Recoverability), which embeds a differentiable, closed-form backup-CBF filter into Soft Actor-Critic training. SAFTER unifies current safety, finite-horizon backup safety, terminal recoverability, and input admissibility into a single differentiable certificate. Its CBF condition yields an exact minimum-intervention projection without iterative online optimization. In quadruped navigation, SAFTER reaches 90% training success with 30.6% fewer transitions than Vanilla SAC and 34.1-35.1% fewer than current-state CBF-filtered SAC baselines. Across unseen simulated environments, SAFTER improves success by up to 42.4 and 61.4 percentage points over the state-of-the-art methods CBF-RL and SEA-Nav, respectively, and achieves 73.3% and 76.7% success on two real-world navigation maps. Ablations identify accurate differentiation through the safety filter as a key contributor to robustness, in which replacing the projection Jacobian with the identity approximation reduces success rate in a High-Shift environment from 61.4% to 6.2%. These results show that recoverability-aware filtering serves as both a runtime safety mechanism and a learning mechanism for sample-efficient, robust policies.

Abstract

Safe RL

Explore similar work

SAFTER: Safety-Aware Filtering Through Explicit Recoverability for Robust Reinforcement Learning under Distribution Shift

Shield-Loco: Shielding Locomotion Policies with Predictive Safety Filtering

Robust Koopman Control Barrier Filters for Safe Actor-Critic Reinforcement Learning