Safety Guarantees in Zero-Shot Reinforcement Learning for Cascade Dynamical Systems

Shima Rabiei; Sandipan Mishra; Santiago Paternain

cs.AIApr 12, 2026

Safety Guarantees in Zero-Shot Reinforcement Learning for Cascade Dynamical Systems

Authors: Shima Rabiei, Sandipan Mishra, Santiago Paternain

Organizations: The authors are with the Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute. · The author is with the Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute.

Abstract

This paper considers the problem of zero-shot safety guarantees for cascade dynamical systems. These are systems where a subset of the states (the inner states) affects the dynamics of the remaining states (the outer states) but not vice-versa. We define safety as remaining on a set deemed safe for all times with high probability. We propose to train a safe RL policy on a reduced-order model, which ignores the dynamics of the inner states, but it treats it as an action that influences the outer state. Thus, reducing the complexity of the training. When deployed in the full system the trained policy is combined with a low-level controller whose task is to track the reference provided by the RL policy. Our main theoretical contribution is a bound on the safe probability in the full-order system. In particular, we establish the interplay between the probability of remaining safe after the zero-shot deployment and the quality of the tracking of the inner states. We validate our theoretical findings on a quadrotor navigation task, demonstrating that the preservation of the safety guarantees is tied to the bandwidth and tracking capabilities of the low-level controller.

Figures & tables

Fig. 1: Empirical failure probability $\hat{p}_{\mathrm{fail}}$ of the transferred policy across inner-loop natural frequencies $\omega_{n}\in[2,12]$ rad/s and damping ratios $\zeta\in[0.2,1.0]$ . Safety violations occur in the region where $\omega_{n}\leq 5$ rad/s and $\zeta\leq 0.6$ . Outside this parameter region, the failure probability evaluates to 0.0.

Fig. 2: Mean attitude tracking error $\bar{e}_{\theta}=\frac{1}{T}\sum_{t=0}^{T-1}|\theta_{t}-\theta_{t}^{\star}|$ across inner-loop natural frequencies $\omega_{n}\in[2,12]$ rad/s and damping ratios $\zeta\in[0.2,1.0]$ . The elevated tracking error corresponds directly to the safety degradation observed in Figure 1 .

Zero-Shot RL

See all 17 papers

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance
May 11, 2026Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming +1Policy Learning Lyapunov-Based Control
Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory, Washington, D.C., USA · Department of Weapons, Robotics, and Control Engineering, United States Naval Academy, Annapolis, MD, USA
Safe Learning Control with Optimality and Stability Guarantees
Jan 26, 2025Xinyang Wang, Hongwei Zhang, Shimin Wang +2Control Barrier Functions Robust Control
Shenzhen Key Lab for Advanced Motion Control and Modern Automation Equipments, School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen, Guangdong 518055, China · School of Data Science, Lingnan University, Hong Kong · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, with M3S, SMART, and with MIT CSAIL +1
Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
Apr 28, 2026Artur Eisele, Bernd Frauenknecht, Friedrich Solowjow +1Robot Safety Safety Filtering
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany

May 11, 2026J/K move · Enter open · S save

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Adam Haroon, Erick J. Rodríguez-Seda, Cody Fleming, Tristan Schuler

Department of Mechanical Engineering, Iowa State University, Ames, IA, USA · Navy Center for Applied Research in Artificial Intelligence (NCARAI), U.S. Naval Research Laboratory, Washington, D.C., USA · Department of Weapons, Robotics, and Control Engineering, United States Naval Academy, Annapolis, MD, USA

Policy Learning Lyapunov-Based Control RL Control Robot Safety Feedback Control Runtime Assurance+2

Open paper AskPDF arXiv

Safe reinforcement learning (RL) typically asks

\textit{what}

an agent should do. We ask

\textit{when}

it needs to act, and show that a single policy can jointly learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield. We focus on stabilization around a known equilibrium, where CARE-based LQR backups, Lyapunov certificates, and classical Lyapunov-STC are well defined, enabling clean comparison against analytical baselines. A run-time assurance (RTA) layer overrides the policy via a one-step-ahead Lyapunov prediction and a precomputed LQR backup, providing a strictly stronger guarantee than constrained MDP methods that enforce safety only in expectation. On an inverted pendulum, cart--pole, and planar quadrotor, the learned policy achieves

1.91\times

1.45\times

, and

3.51\times

higher mean inter-sample interval (MSI) than a Lyapunov-triggered baseline; a fixed LQR controller at the same average rate is unstable on all three plants, showing that adaptive timing, not a lower average rate, makes sparsity safe. A CARE-derived Lyapunov reward transfers across environments without redesign, with a single weight

w_c

controlling the stability--communication tradeoff; ablations confirm the RTA shield is essential, with its removal reducing MSI by

1.27

1.84\times

and degrading state norms. A preference-conditioned extension recovers the full tradeoff frontier from one model at

\tfrac{2}{11}

of training compute, and SAC experiments show the results are algorithm-agnostic across discrete and continuous domains. A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC is intractable, and robustness to

\pm30\%

mass variation and disturbances shows graceful degradation, with the RTA absorbing what the learned policy cannot.

Abstract

Zero-Shot RL

Explore similar work

Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance

Safe Learning Control with Optimality and Stability Guarantees

Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty