Model-Informed Safe Reinforcement Learning for Bipedal Locomotion via Step-to-Step Prediction

Victor Paredes; Ayonga Hereid

cs.ROSep 28, 2026

Model-Informed Safe Reinforcement Learning for Bipedal Locomotion via Step-to-Step Prediction

Authors: Victor Paredes, Ayonga Hereid

Organizations: Mechanical and Aerospace Engineering, The Ohio State University, Columbus, OH, USA.

Abstract

Humanoid robots promise versatile mobility in cluttered, human-centric environments, but real deployment demands principled safety. Classical model-based gait generators yield interpretable motions but often lack the robustness and adaptability of modern reinforcement learning (RL) based approaches. We propose a model-informed reinforcement learning framework anchored to the analytical Angular Momentum Linear Inverted Pendulum (ALIP) template. We provide a step-to-step safety certificate for ALIP stepping via a discrete exponential control barrier function (DECBF) and use it as (i) a training-time shaping signal and (ii) a runtime action filter that minimally adjusts swing-foot placement to satisfy template-level constraints. Full-order safety is evaluated empirically on the Digit humanoid in MuJoCo with a whole-body controller stack. Compared to an unconstrained baseline, our approach reduces safety-violation events in the reported external-disturbance trial, while larger lateral-velocity transients reveal a safety-tracking tradeoff.

Figures & tables

Fig. 1: Overview of the ALIP-informed safe RL framework on the Digit humanoid. The policy commands swing-foot placement and torso pitch. DECBFs derived from the ALIP model constrain foot-placement actions during training (safety shaping) and execution (projection filter).

Fig. 2: Training vs. execution pipeline. During training, the policy outputs action $\mathbf{a}_{k}=(u_{k},\theta_{k})$ ; the foot-placement component $u_{k}$ is evaluated through the ALIP step-to-step map to compute the safety certificate $\mathbf{s}_{k}$ , yielding a shaping reward $r_{\text{safe}}$ . During execution, the shaping reward is replaced by a projection that finds the closest $u_{k}$ satisfying $\mathbf{s}_{k}\geq 0$ before passing the command to the whole-body controller.

Fig. 3: Orbital energy and the orbital-energy safe regions used in this work. Left: orbital-energy phase portrait with iso-energetic lines. Middle: lateral-walking safe region ( $p_{y}\in[-0.5,0.5]$ m and $E\in[-0.464,-0.012]$ ). Right: sagittal safe region ( $p_{x}\in[-0.7,0.7]$ m and $E<1.125$ ).

Quantity	Value	Rationale
$p_{x,\min},p_{x,\max}$	$(-0.70,\ 0.70)$ m	stance reach
$p_{y,\max}$	$0.50$ m	lateral reach
$W_{\min}$	$0.08$ m	no leg crossing
$v_{x,\max}$	$1.5$ m/s	energy surrogate
$T$	$0.35$ s	nominal step duration

TABLE I: Safety bounds for Digit

Fig. 4: Lateral-plane phase portrait showing the relationship between orbital energy and leg collision. During stable walking, the system remains in the safe region ( $E<0$ ). After a disturbance drives the orbital energy positive (red states), the CoM crosses the stance foot and a leg collision occurs.

Fig. 5: Velocity tracking (left) and lateral foot distance (right) under periodic external disturbances. Left: desired vs. average CoM velocity for (a) unguided, filter off; (b) unguided, filter on; (c) DECBF-guided, filter off; (d) DECBF-guided, filter on. Right: lateral distance from stance foot to swing foot at touchdown compared against the safety bound $W_{\min}=0.08$ m. Subplots (a)–(d) follow the same ordering.

Fig. 6: Sagittal phase portrait under periodic disturbances ( 25 ). (a) The unguided policy violates the orbital-energy and CoM-extension bounds. (b) The DECBF-guided policy remains within the safe region. The red star marks the initial state.

Explore similar work

MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds
Jun 9, 2026Codrin Crismariu, Ryan K. CosnerAgile Locomotion Scalable Robot Learning
Department of Mechanical Engineering Tufts University, Medford, MA, United States
SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics
May 16, 2025Lizhi Yang, Blake Werner, Ryan K. Cosner +3Control Barrier Functions Full-Body Humanoid Control
Mechanical and Civil Engineering, California Institute of Technology · Aerospace Engineering and Engineering Mechanics, UT Austin · Computer Science, Cornell University
ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids
Sep 14, 2026Gechen Qu, Tong Zhang, Bike Zhang +4Safety Filters Full-Body Humanoid Control
University of California, Berkeley · University of California, Los Angeles

Jun 9, 2026, cs.ROJ/K move · Enter open · S save

MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds

Codrin Crismariu, Ryan K. Cosner

Department of Mechanical Engineering Tufts University, Medford, MA, United States

Agile Locomotion Scalable Robot Learning Offline Reinforcement Learning Terrain Humanoid

Open paper AskPDF arXiv

Perceptive bipedal locomotion over sparse terrain remains a difficult challenge: model-based methods are precise but brittle to uncertainty, while model-free methods are robust but struggle to discover the precise, constrained motions required for safety-critical locomotion where small errors can cause catastrophic failures. We propose a model-assisted reinforcement learning (RL) framework that combines both perspectives in three steps: (1) generate a safe reference trajectory using simplified models; (2) train a privileged teacher policy guided by a control Lyapunov function (CLF) reward built around the safe reference trajectory; and (3) distill the teacher into a vision-based student policy. We show that this model-assistance procedure produces physically grounded locomotion, improving sample efficiency, reducing the need for a complex learning curriculum, and achieving smoother locomotion behavior alongside stepping stone performance comparable to model-free baselines. We validate our approach in simulation and demonstrate successful deployment on a Unitree G1 humanoid robot navigating sparse footholds with lateral constraints.

Abstract

Explore similar work

MARCH: Model-Assisted Reinforcement Learning for the Perceptive Control of Humanoids over Sparse Footholds

SHIELD: Safety on Humanoids via CBFs In Expectation on Learned Dynamics

ResSafe: Learning Safety Filtering with Residual Reinforcement Learning for Humanoids