C-STEP: Continuous Space-Time Empowerment for Physics-informed Safe Reinforcement Learning of Mobile Agents

Guihlerme Daubt; Adrian Redder

eess.SYMar 25, 2026

C-STEP: Continuous Space-Time Empowerment for Physics-informed Safe Reinforcement Learning of Mobile Agents

Authors: Guihlerme Daubt, Adrian Redder

Organizations: Faculty of Technology, Bielefeld University, Germany · Department of Electrical Engineering and Information Technology, Paderborn University, Germany

Abstract

Safe navigation in complex environments remains a central challenge for reinforcement learning (RL) in robotics. This paper introduces Continuous Space-Time Empowerment for Physics-informed (C-STEP) safe RL, a novel measure of agent-centric safety tailored to deterministic, continuous domains. This measure can be used to design physics-informed intrinsic rewards by augmenting positive navigation reward functions. The reward incorporates the agents internal states (e.g., initial velocity) and forward dynamics to differentiate safe from risky behavior. By integrating C-STEP with navigation rewards, we obtain an intrinsic reward function that jointly optimizes task completion and collision avoidance. Numerical results demonstrate fewer collisions, reduced proximity to obstacles, and only marginal increases in travel time. Overall, C-STEP offers an interpretable, physics-informed approach to reward shaping in RL, contributing to safety for agentic mobile robotic systems.

Figures & tables

Symbol	Definition
$\mathcal{X}_{\text{free}}$	The collision-free configuration space.
$\mathcal{R}_{T}(x)$	Set of reachable states in $\mathbb{R}^{n}$ from state $x$ within time $T$ without state constraints.
$\mathcal{R}_{T,\text{free}}(x)$	Set of reachable states where the state trajectory remains in $\mathcal{X}_{\text{free}}$ .
$\mathcal{T}_{T}(x)$	The terminal set, $\mathcal{R}_{T}(x)\setminus\mathcal{R}_{T,\text{free}}(x)$ .
$c$	A positive safety coefficient scaling the empowerment reward.

Table 1: Key Definitions and Notation

Panel of Figure 1 — Figure 1: Point maze navigation. Green spheres are agents, red is the goal, and yellow is the start. The empowered agent prefers the safer, wider path.

Panel of Figure 2 — Figure 2: Visualization of the sampling-based reachable set approximation in 2D for different initial velocities. The blue areas indicate total reachable areas; red areas represent the approximated terminal set volume.

Figure 3: Top-down view of the PyBullet simulation environment. The agent starts in the yellow region and navigates to the green goal region. The maroon-colored obstacle’s width and position are randomized in each episode, indicated by the red arrows, requiring adaptive navigation.

Figure 4: Average reward per episode for the empowered agent (

c=1

). (Evaluated every

10^{3}

training steps.)

Agent	Success rate	Avg. clear time (s)
unempowered	82%	6.56
Empowered $c=10$	89%	6.91
Empowered $c=1$	92%	7.77
Empowered $c=0.5$	99%	7.63

Table 2: Success rate and time to clear the obstacle.

Figure 5: Average time spent under different distance thresholds for empowered (blue lines) and unempowered (orange line) agents.

Distance Threshold (m)	0.1	0.2	0.3	0.4	0.5
Empowered $c=10$	31.05%	10.70%	18.99%	14.14%	9.25%
Empowered $c=1$	72.61%	49.93%	43.34%	27.59%	3.01%
Empowered $c=0.5$	85.76%	64.66%	53.10%	33.98%	19.07%

Table 3: Percentage decrease in time spent near obstacles for empowered agents relative to unempowered baseline.

Hyperparameters	Values
$T$	1 second
Trajectories	150
Points in trajectory	100

Table 4: CST Empowerment hyperparameters used in experiments.

Hyperparameters	Values
Horizon	2048
Batch size	64
Discount $(\gamma)$	0.99
GAE Parameter $(\lambda)$	0.95
Clip range	0.2
Learning rate	$3\times 10^{-5}$

Table 5: PPO parameters for the PyBullet environment.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 215 papers

Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation
May 13, 2026Qisong He, Xinmiao Huang, Jinwei Hu +4Constrained RL Risk-Sensitive RL
University of Liverpool, Liverpool, UK. · Universit´e Grenoble Alpes, Grenoble, France.
SEA-Nav: Efficient Policy Learning for Safe and Agile Quadruped Navigation in Cluttered Environments
Mar 10, 2026Shiyi Chen, Mingye Yang, Haiyi Liu +6Quadruped Robot Navigation Robot Navigation
Tsinghua University, Beijing, China. · Imperial College London, London, UK.
DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles
Sep 30, 2026Sanghyuk Park, Kwanwoo Lee, Taekyung Kim +2RL for Robotics Control Barrier Functions
Department of Intelligence and Information, Seoul National University, Republic of Korea · Department of Robotics, University of Michigan, Ann Arbor, MI, USA · Department of Mechanical Engineering, Yonsei University, Republic of Korea +1

May 13, 2026J/K move · Enter open · S save

Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation

Qisong He, Xinmiao Huang, Jinwei Hu, Zhuoyun Li+3

University of Liverpool, Liverpool, UK. · Universit´e Grenoble Alpes, Grenoble, France.

Constrained RL Risk-Sensitive RL Safe Robot Navigation Robot Safety

Open paper AskPDF arXiv

Safe navigation for mobile robots demands policies that remain reliable under the high-consequence perception uncertainty of cluttered environments. Yet most existing safe reinforcement learning (RL) methods assess safety through average cumulative cost. Such metrics can mask dangerous tail-risk behaviors. To address this, we propose a framework that trains risk-sensitive policies through Conditional Value-at-Risk (CVaR) constrained optimization on an off-policy TD3 backbone and evaluates their safety margins post-training through neural network reachability verification. During training, the policy is optimized under CVaR constraints on cumulative costs, promoting sensitivity to high-cost tail outcomes rather than average behavior alone. After training, we compute action reachable sets under bounded observation uncertainty using Taylor Model analysis, yielding a safety rate metric that quantifies the proportion of evaluated states at which the policy's reachable action set remains within prescribed safety margins. A key finding is that policies trained with CVaR constraints maintain larger safety margins from obstacles across evaluated states. This makes them significantly more amenable to formal reachability verification. Experiments across ten navigation scenarios and six baselines show that our method achieves a 98.3% success rate, the highest safety verification rate among all compared methods, while revealing that average cost rankings and reachability-based safety rankings can diverge. This indicates that reachability verification captures risks which are missed by empirical cost metrics alone. We further validate our approach on a physical Clearpath Jackal robot, demonstrating successful sim-to-real transfer.

Abstract

Safe RL

Explore similar work

Safety-Constrained Reinforcement Learning with Post-Training Reachability Verification for Robot Navigation

SEA-Nav: Efficient Policy Learning for Safe and Agile Quadruped Navigation in Cluttered Environments

DODGER: Safety-Guided Reinforcement Learning for Robot Navigation Among Dynamic Obstacles