Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Wenqi Cai; Kyriakos G. Vamvoudakis; Sébastien Gros; Anthony Tzes

cs.ROMar 30, 2026

Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Authors: Wenqi Cai, Kyriakos G. Vamvoudakis, Sébastien Gros, Anthony Tzes

Organizations: Electrical Engineering Program, New York University Abu Dhabi (NYUAD), 129188, UAE. · School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA 30332, United States. · Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU), Trondheim, Norway.

Abstract

In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.

Figures & tables

Fig. 1 : Cost-Matching MPC-RL framework for humanoids.

Fig. 2 : Closed-loop validation of the value mismatch $\mathrm{MSE}\!\left(Q^{\mathrm{MPC}}_{\boldsymbol{\mathrm{\theta}}}-Q^{\mathrm{meas}}\right)$ across training rounds (shaded region: $10-90$ % across seeds). At each training round, $50$ trajectories are collected ( $600$ steps per trajectory), and the cost-matching objective is optimized using $2000$ gradient updates with discount factor $\gamma=0.985$ . The inset summarizes the normalized block-wise evolution of $\boldsymbol{\mathrm{\theta}}$ , illustrating how learning reshapes the MPC objective.

Panel of Figure 3 — Fig. 3 : Value-matching diagnostics on a fixed evaluation set: density plot of $Q^{\mathrm{MPC}}_{\boldsymbol{\mathrm{\theta}}}$ versus $Q^{\mathrm{meas}}$ .

Fig. 4 : Simulation snapshots of the humanoid during locomotion under a time-scheduled disturbance benchmark consisting of lateral force pulses $f_{\mathrm{ext},y}$ and a yaw-torque pulse $m_{\mathrm{ext},z}$ .

Fig. 5 : Representative closed-loop response under the disturbance benchmark. Left: full-horizon signals. Right: zoom around the push interval. The gray bands indicate push windows. The dashed horizontal line indicates the settle threshold computed from pre-disturbance statistics, and the “settled” marker indicates the first time the error remains below this threshold.

Overall Tracking
Metric	Baseline	CM-MPC	Impr. (%)
RMS Error ( $\\|\boldsymbol{\mathrm{e}}\\|_{2}$ )	1.022 $\pm$ 0.012	1.009 $\pm$ 0.016	+1.29
Peak Error ( $\\|\boldsymbol{\mathrm{e}}\\|_{\infty}$ )	1.620 $\pm$ 0.021	1.597 $\pm$ 0.045	+1.41
Control Effort & Smoothness
Max Effort ( $\\|\boldsymbol{\mathrm{u}}\\|_{99\%}$ )	472.7 $\pm$ 4.3	447.8 $\pm$ 7.1	+ 5.26
Max Jitter ( $\\|\dot{\boldsymbol{\mathrm{u}}}\\|_{99\%}$ ) [ $10^{4}$ ]	5.66 $\pm$ 0.00	5.60 $\pm$ 0.03	+1.10

TABLE I : Comprehensive Evaluation of Metrics (Mean

\pm

Std) over Six Trials.

Humanoid Robot Locomotion

Humanoid robot locomotion studies how bipedal robots walk, run, recover from disturbances, and traverse varied terrain.

See all 145 papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Jun 4, 2026Junheng Li, Liang Wu, Sergio A. Esteban +3RL Control Humanoid Robot Locomotion
California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA
RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion
Mar 11, 2026Andrea Patrizi, Carlo Rizzardo, Arturo Laurenzi +3Robot Locomotion Legged Robotics
All authors are with the Humanoids and Human-Centred Mechatronics (HHCM) lab, Istituto Italiano di Tecnologia (IIT), Via San Quirico 19d, 16163 Genova · Andrea Patrizi is also with the Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genova, Via All’Opera Pia 13, 16145 Genova.
Optimal Control with Learned Critics under Unmodeled State Dependencies
Oct 4, 2026Philipp Schoch, Markus RyllResidual Dynamics Learning Robot Dynamics Learning
Autonomous Aerial Systems Lab Technical University of Munich

Jun 4, 2026J/K move · Enter open · S save

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang+2

California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA

RL Control Humanoid Robot Locomotion Reinforcement Learning Model Predictive Control Robotic Manipulation

Open paper AskPDF arXiv

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop

π^n

MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

Abstract

Humanoid Robot Locomotion

Explore similar work

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion

Optimal Control with Learned Critics under Unmodeled State Dependencies