RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion
Authors: Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi, Francesco Ruscelli, Luca Rossini, Nikos G. Tsagarakis
Organizations: All authors are with the Humanoids and Human-Centred Mechatronics (HHCM) lab, Istituto Italiano di Tecnologia (IIT), Via San Quirico 19d, 16163 Genova · Andrea Patrizi is also with the Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova, Via All’Opera Pia 13, 16145 Genova.
We propose a contact-explicit hierarchical architecture coupling Reinforcement Learning (RL) and Model Predictive Control (MPC), where a high-level RL agent provides gait and navigation commands to a low-level locomotion MPC. This offloads the combinatorial burden of contact timing from the MPC by learning acyclic gaits through trial and error in simulation. We show that only a minimal set of rewards and limited tuning are required to obtain effective policies. We validate the architecture in simulation across robotic platforms spanning 50 kg to 120 kg and different MPC implementations, observing the emergence of acyclic gaits and timing adaptations in flat-terrain legged and hybrid locomotion, and further demonstrating extensibility to non-flat terrains. Across all platforms, we achieve zero-shot sim-to-sim transfer without domain randomization, and we further demonstrate zero-shot sim-to-real transfer without domain randomization on Centauro, our 120 kg wheeled-legged humanoid robot. We make our software framework and evaluation results publicly available at https://github.com/AndrePatri/AugMPC.
Figures & tables
Fig. 1: Our hierarchical RL-MPC architecture has been trained and evaluated on several robotic platforms with different morphologies and weight distributions ( 50kg to 120kg ), successfully transferred zero-shot across different domains without any domain randomization, and applied to both hybrid and standard locomotion tasks. Supplementary videos are available at https://youtu.be/I08UywVVhN4 .
Fig. 2: The proposed hierarchical architecture uses an RL policy to generate both contact schedules and navigation commands for the underlying MPC.
Name
Descr.
Nominal Expr.
Dim.
Span
Support
hint
integrator
fi⊖xi+1=0
nq+nv
N
0:N−1
hdyn
inv. dynamics
[τi]fb=0
6
N
0:N−1
hxinit
init. state
[x0−xIS]a:b=0
ninit
1
0
hliftoff(j)
force vanishing
λj(i)=0
3
variable
variable
hpc(j)
no-slip contact
vc,i(j)=0
3
variable
variable
hrc(j)
no-slip rolling contact
vw,i(j)+Ri⋅ωi(j)×r^i(j)=0
3
variable
variable
TABLE I: Problem costs/constraints formulation: Grey-shaded costs are specific to the Centauro platform, whereas red and blue-shaded terms are used to define contact and flight phases, respectively.
Name
Descr.
Type
Range
Dim.
Obs. subspace
Ref. frame
v
linear velocity (prediction)
real
[−∞,∞]
3
OMPC
base loc.
λ
contact forces (prediction)
real
[−∞,∞]
3⋅nc
OMPC
base loc.
δMPC
MPC health ∝(l+κ⋅∥h∥1)
real
[0,∞]
1
OMPC
⋅
dfl(:,0)
Pfl(:,0) current position
integer
[0,i∗]
nc
OMPC
⋅
sfl(:,0)
Pfl(:,0) current node span
integer
[0,N]
nc
OMPC
⋅
TABLE II: Observations from MPC: The agent observes linear velocity and contact forces estimates from the MPC, a minimal representation of its convergence status and the current contact schedule.
Fig. 3: High-level software architecture. The system is modular, with three main components: world interface, MPC cluster and training environment. Green modules operate on GPU-resident data; orange modules run on CPU.
Fig. 4: Task tracking episode sub-reward (first row) and CoT metric (second row) during training for all the flat terrain case studies, averaged across 800 environments. Dark shaded areas indicate the first and third quartiles of the distribution across environments; lighter regions denote the 5-th and 95-th percentiles. Action rate sub-rewards are omitted for brevity.
Fig. 5: Deterministic evaluation of a legged locomotion policy for Centauro, shown over a 50s window. The injection requests actions χMPC chosen by the policy (b) generate the contact schedule and motion (a), revealing completely acyclic contact patterns and timing adaptations. The associated tracking performance is shown in (c). Flight phases are injected within the MPC horizon, for each foot j , only when χMPC(j)<0 .
Fig. 6: Influence of the action repeat parameter nrep (see Sec. V-B ) on the episode total reward (averaged across 800 environments) during training, shown across both legged and hybrid locomotion tasks. Shaded areas indicate the first and third quartiles of the distribution across environments.
Fig. 7: Zero-shot sim-to-real transfer of a hybrid locomotion policy on Centauro.
Fig. 8: Evaluation of a perception-augmented hybrid locomotion policy for Centauro ascending a stepped pyramid. The policy also controls the flight phase parameters. Top row: ascent while facing forward. Bottom row: ascent while facing backward.
In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop πnMPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.
Junheng Li, Liang Wu, Sergio A. Esteban +3
California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA
Continuous legged manipulation requires accurate end-effector tracking while the base keeps walking. Combining reinforcement learning (RL) with model predictive control (MPC) suits this task: the learned policy provides robust locomotion, while MPC coordinates the base and arm to compensate for tracking errors. However, MPC can compensate only for base motion that it can predict, and a learned policy's command response varies with gait phase, contact, and payload. We present ReCo, a framework that couples response-consistent locomotion with policy-aware MPC for legged manipulation. Response shaping trains the policy to respond to commands consistently and repeatably across randomized dynamics. An identified closed-loop response model then lets MPC jointly plan locomotion commands and arm motion. On the simulation benchmark, ReCo reduces position and orientation root-mean-square error (RMSE) by 28.7% and 27.4% relative to the best baseline for each metric. Real-world experiments demonstrate onboard continuous legged manipulation with coordinated base and arm motion.
Kuankuan Sima, Yichao Gao, Chenxi Gu +2
Department of Electrical and Computer Engineering, National University of Singapore, Singapore 119077, Singapore · Department of Mechanical Engineering, National University of Singapore, Singapore 119077, Singapore
Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred behavior by inserting transitions generated in the same environment by a model predictive controller (MPC). Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. We analyze how the injected transitions bias the learning, allowing the policy to converge to behaviors that pure RL may fail to reach under simple reward functions. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection produces gaits qualitatively comparable to those of reward shaping and adversarial motion priors. We also show that MPC-Injection can complete a barrel roll that pure RL fails to achieve under the same simple reward and can select between trotting and bounding gaits only through changing the injected MPC data.