RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion

Andrea Patrizi; Carlo Rizzardo; Arturo Laurenzi; Francesco Ruscelli; Luca Rossini; Nikos G. Tsagarakis

cs.ROMar 11, 2026

RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion

Authors: Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi, Francesco Ruscelli, Luca Rossini, Nikos G. Tsagarakis

Organizations: All authors are with the Humanoids and Human-Centred Mechatronics (HHCM) lab, Istituto Italiano di Tecnologia (IIT), Via San Quirico 19d, 16163 Genova · Andrea Patrizi is also with the Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova, Via All’Opera Pia 13, 16145 Genova.

Abstract

We propose a contact-explicit hierarchical architecture coupling Reinforcement Learning (RL) and Model Predictive Control (MPC), where a high-level RL agent provides gait and navigation commands to a low-level locomotion MPC. This offloads the combinatorial burden of contact timing from the MPC by learning acyclic gaits through trial and error in simulation. We show that only a minimal set of rewards and limited tuning are required to obtain effective policies. We validate the architecture in simulation across robotic platforms spanning 50 kg to 120 kg and different MPC implementations, observing the emergence of acyclic gaits and timing adaptations in flat-terrain legged and hybrid locomotion, and further demonstrating extensibility to non-flat terrains. Across all platforms, we achieve zero-shot sim-to-sim transfer without domain randomization, and we further demonstrate zero-shot sim-to-real transfer without domain randomization on Centauro, our 120 kg wheeled-legged humanoid robot. We make our software framework and evaluation results publicly available at https://github.com/AndrePatri/AugMPC.

Figures & tables

Fig. 1: Our hierarchical RL-MPC architecture has been trained and evaluated on several robotic platforms with different morphologies and weight distributions ( $50~\mathrm{kg}$ to $120~\mathrm{kg}$ ), successfully transferred zero-shot across different domains without any domain randomization, and applied to both hybrid and standard locomotion tasks. Supplementary videos are available at https://youtu.be/I08UywVVhN4 .

Fig. 2: The proposed hierarchical architecture uses an RL policy to generate both contact schedules and navigation commands for the underlying MPC.

Name	Descr.	Nominal Expr.	Dim.	Span	Support
$h_{\mathrm{int}}$	integrator	$\bm{f}_{i}\,\ominus\,\bm{x}_{i+1}=0$	$n_{q}{+}n_{v}$	$N$	$0{:}N{-}1$
$h_{\mathrm{dyn}}$	inv. dynamics	$[\bm{\tau}_{i}]_{\mathrm{fb}}=0$	6	$N$	$0{:}N{-}1$
$h_{x_{\mathrm{init}}}$	init. state	$[\bm{x}_{0}\,-\,\bm{x}_{\mathrm{IS}}]_{a:b}=0$	$n_{\mathrm{init}}$	$1$	$0$
$h_{\mathrm{liftoff}}^{(j)}$	force vanishing	$\bm{\lambda}^{(i)}_{j}=0$	$3$	variable	variable
$h_{\mathrm{pc}}^{(j)}$	no-slip contact	$\bm{v}_{c,i}^{(j)}=0$	$3$	variable	variable
$h_{\mathrm{rc}}^{(j)}$	no-slip rolling contact	$\bm{v}^{(j)}_{w,i}{+}R_{i}{\cdot}\bm{\omega}_{i}^{(j)}{\times}\hat{\bm{r}}_{i}^{(j)}{=}0$	$3$	variable	variable

TABLE I: Problem costs/constraints formulation: Grey-shaded costs are specific to the Centauro platform, whereas red and blue-shaded terms are used to define contact and flight phases, respectively.

Name	Descr.	Type	Range	Dim.	Obs. subspace	Ref. frame
$\bm{v}$	linear velocity (prediction)	real	$[-\infty,\infty]$	$3$	$\mathcal{O}_{\mathrm{MPC}}$	base loc.
$\bm{\lambda}$	contact forces (prediction)	real	$[-\infty,\infty]$	$3\cdot n_{c}$	$\mathcal{O}_{\mathrm{MPC}}$	base loc.
$\delta_{\mathrm{MPC}}$	MPC health $\propto(l+\kappa\cdot\\|\bm{h}\\|_{1})$	real	$[0,\infty]$	$1$	$\mathcal{O}_{\mathrm{MPC}}$	$\cdot$
$d_{\mathrm{fl}}^{(:,0)}$	$\mathcal{P}_{\mathrm{fl}}^{(:,0)}$ current position	integer	$[0,i^{*}]$	$n_{c}$	$\mathcal{O}_{\mathrm{MPC}}$	$\cdot$
$s_{\mathrm{fl}}^{(:,0)}$	$\mathcal{P}_{\mathrm{fl}}^{(:,0)}$ current node span	integer	$[0,N]$	$n_{c}$	$\mathcal{O}_{\mathrm{MPC}}$	$\cdot$

TABLE II: Observations from MPC: The agent observes linear velocity and contact forces estimates from the MPC, a minimal representation of its convergence status and the current contact schedule.

Fig. 3: High-level software architecture. The system is modular, with three main components: world interface, MPC cluster and training environment. Green modules operate on GPU-resident data; orange modules run on CPU.

Fig. 4: Task tracking episode sub-reward (first row) and CoT metric (second row) during training for all the flat terrain case studies, averaged across 800 environments. Dark shaded areas indicate the first and third quartiles of the distribution across environments; lighter regions denote the $5\text{-th}$ and $95\text{-th}$ percentiles. Action rate sub-rewards are omitted for brevity.

Fig. 5: Deterministic evaluation of a legged locomotion policy for Centauro, shown over a $50\,\mathrm{s}$ window. The injection requests actions $\bm{\chi}_{\mathrm{MPC}}$ chosen by the policy (b) generate the contact schedule and motion (a), revealing completely acyclic contact patterns and timing adaptations. The associated tracking performance is shown in (c). Flight phases are injected within the MPC horizon, for each foot $j$ , only when ${\chi}_{\mathrm{MPC}}^{(j)}<0$ .

Fig. 6: Influence of the action repeat parameter $n_{\mathrm{rep}}$ (see Sec. V-B ) on the episode total reward (averaged across 800 environments) during training, shown across both legged and hybrid locomotion tasks. Shaded areas indicate the first and third quartiles of the distribution across environments.

Fig. 7: Zero-shot sim-to-real transfer of a hybrid locomotion policy on Centauro.

Fig. 8: Evaluation of a perception-augmented hybrid locomotion policy for Centauro ascending a stepped pyramid. The policy also controls the flight phase parameters. Top row: ascent while facing forward. Bottom row: ascent while facing backward.

Robot Locomotion

Robot locomotion studies how robots move efficiently and reliably across varied terrains, environments, and movement modes.

See all 233 papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Jun 4, 2026Junheng Li, Liang Wu, Sergio A. Esteban +3RL Control Humanoid Robot Locomotion
California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA
ReCo: Response-Consistent Locomotion with Policy-Aware MPC for Legged Manipulation
Oct 1, 2026Kuankuan Sima, Yichao Gao, Chenxi Gu +2Model Predictive Control RL for Legged Locomotion
Department of Electrical and Computer Engineering, National University of Singapore, Singapore 119077, Singapore · Department of Mechanical Engineering, National University of Singapore, Singapore 119077, Singapore
MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
Jun 24, 2026Roy Xing, Seyoung Ree, Brian PlancherReinforcement Learning RL for Legged Locomotion
Dartmouth College · Harvard University

Jun 4, 2026J/K move · Enter open · S save

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang+2

California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA

RL Control Humanoid Robot Locomotion Reinforcement Learning Model Predictive Control Robotic Manipulation

Open paper AskPDF arXiv

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop

π^n

MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

Abstract

Robot Locomotion

Explore similar work

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

ReCo: Response-Consistent Locomotion with Policy-Aware MPC for Legged Manipulation

MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins