MPBRS: Multi-Potential-Based Reward Shaping for Unified Reward Configuration in Humanoid Locomotion

Van Chung Nguyen; Hung Manh La

cs.ROOct 21, 2026

MPBRS: Multi-Potential-Based Reward Shaping for Unified Reward Configuration in Humanoid Locomotion

Authors: Van Chung Nguyen, Hung Manh La

Organizations: are with the Advanced Robotics and Automation (ARA) Lab, Department of Computer Science and Engineering, University of Nevada, Reno, NV 89557, USA.

Abstract

Designing and tuning reward functions remains one of the main challenges in building effective reinforcement learning (RL) pipelines for humanoid locomotion. While shaping rewards can accelerate learning, naively formulated terms often conflict with the desired behavior, overfit to a specific robot, or become unstable under small weight changes. Potential-Based Reward Shaping (PBRS) offers an alternative solution: shaping the reward via differences of a potential function to guide exploration without changing the optimal policy. In this paper, we study Multi-PBRS for humanoid locomotion and propose a unified reward configuration. We then evaluate the same reward structure and weights across nine humanoid robots, requiring only robot-specific metadata (joint/link mappings and default poses) rather than per-robot reward redesign. Our experiments show that, compared to standard shaping, MPBRS provides more consistent scaling behavior and reduced tuning sensitivity across robots, making it easier to tune without altering the optimal policy.

Figures & tables

Fig. 1: The baseline (left), DRS (middle), and MPBRS (right) locomotion policies are shown. MPBRS succeeds in achieving an optimal policy similar to the carefully tuned DRS, while the baseline leans the base backward during locomotion.

Fig. 2: A snapshot of nine humanoid robots used in this paper in IsaacLab. From right to left: Fourier N1, Tienkung2, Unitree G1, Unitree H1, Booster T1, Fourier GR3V2, Kuavo S45, Hightorque HI, and Berkeley Humanoid Lite.

Name	Joints Dims	Actions Dims	Waist Dims
Fourier N1	23	13	1
Unitree H1	19	11	1
Unitree G1	29	13	3
Booster T1	23	13	1
Fourier GR3V2	31	13	3
HighTorque HI	25	13	3

TABLE I: Humanoid Robots Overview.

Observation term	Dim.	Noise / corruption
Commands $(v_{x},v_{y},\omega_{z})$	3	$0.02$
Base linear velocity (body) $\mathbf{v}_{b}$ [m/s]	3	$0.04$
Base angular velocity (body) $\boldsymbol{\omega}_{b}$ [rad/s]	3	$0.025$
Projected gravity (body) $\hat{\mathbf{g}}_{b}$	3	$0.015$
Joint position offsets $\mathbf{q}-\mathbf{q}_{0}$ [rad]	$N_{a}$	$0.004$
Joint velocities $\dot{\mathbf{q}}$ [rad/s]	$N_{a}$	$0.02$

TABLE II: Agent Observations.

Baseline	Weight	Function
Linear velocity tracking	$0.5$	$\exp\!\left(-\\|{\bf v}^{cmd}_{xy}-{\bf v}^{yaw}_{xy}\\|^{2}/\sigma_{v}^{2}\right)$
Yaw rate tracking	$0.3$	$\exp\!\left(-(\omega^{cmd}_{z}-\omega_{z})^{2}/\sigma_{\omega}^{2}\right)$
Waist default	$0.25$	$\exp\!\left(-8\,\|q_{waist}-q_{waist,0}\|\right)$
Action magnitude penalty	$-0.005$	$-\sum_{i}a_{i}^{2}$
Action rate penalty	$-0.05$	$-\sum_{i}(a_{i}-a_{i,prev})^{2}$
Termination	$-200.0$	$\begin{cases}1,&h<h_{\min},\\ 1,&\theta_{\text{tilt}}>\theta_{\max},\\ 1,&\text{self-collision},\\ 1,&\text{time-out},\\ 0,&\text{otherwise},\end{cases}$

TABLE III: Training Rewards.

Hyperparameter	Value
Initial action noise std	1.0 (log)
Value loss coefficient	1.0
Clipping $\epsilon$	0.2
Entropy coefficient	0.001
Learning rate	$3\times 10^{-4}$ (adaptive)
Discount factor $\gamma$	0.99

TABLE IV: Training Environment Hyperparameters.

Panel of Figure 7 — Fig. 3: Benchmark comparison of the Baseline, DRS, and Multi-Potential PBRS in terms of total reward, fall rate, tracking reward in the $x$ – $y$ velocity, and tracking reward in angular velocity. The first column shows (a) Baseline, (b) Direct Reward Shaping (DRS), and (c) Multi-Potential PBRS. From top to bottom, the rows show total reward, fall rate, tracking reward for $x$ – $y$ velocity, and tracking reward for angular velocity.

Panel of Figure 8 — Fig. 4: Velocity tracking plots for the policy running on the Fourier N1 robot over a 20-second interval. The robot was respawned every 10 seconds, as reflected by the jumps in the tracking error.

Panel of Figure 9 — Fig. 5: To compare the robustness of potential-based shaping against DRS, we train policies using DRS and MPBRS while varying the shaping weight from 0.1 to 500. The first row shows DRS, and the second row shows MPBRS. The comparison is based on (a) total reward, (b) tracking reward in the $x$ – $y$ direction, and (c) tracking reward for yaw rate.

Panel of Figure 10 — Fig. 6: Reward distributions for DRS and the proposed method in the sensitivity-analysis scenario. The first row (a) corresponds to DRS, and the second row (b) corresponds to MPBRS. From left to right, the columns show the distributions of the height, uprightness, and joint-regularization rewards.

Panel of Figure 11 — Fig. 7: Tracking error of our method versus DRS with a shaping weight of 500. From top to bottom, the rows show tracking velocity in $x$ , tracking velocity in $y$ , and the mean error. While our method successfully tracks the commanded velocity, DRS with this weight causes the robot to stand still.

Reward Shaping

Reward shaping designs intermediate or modified feedback to make reinforcement learning more informative, efficient, and aligned with task goals.

See all 142 papers

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation
Jun 4, 2026Junheng Li, Liang Wu, Sergio A. Esteban +3RL Control Humanoid Robot Locomotion
California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Jun 25, 2026Henrik Müller, Daniel KudenkoReward Shaping Sparse-Reward RL
L3S Research Center Leibniz University Hannover Hannover, 30167, Germany
Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation
Jun 10, 2026Mehmet Turan YardımcıMulti-Objective Reinforcement Learning Humanoid Loco-Manipulation
Department of Computer Engineering C¸ukurova University Adana, Türkiye

Jun 4, 2026J/K move · Enter open · S save

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang+2

California Institute of Technology, CA 91106, USA · Johns Hopkins University, MD 21218, USA

RL Control Humanoid Robot Locomotion Reinforcement Learning Model Predictive Control Robotic Manipulation

Open paper AskPDF arXiv

In humanoid motion control, model predictive control (MPC) offers physically grounded prediction and constraint handling, while reinforcement learning (RL) enables robust whole-body skills through large-scale simulation. However, using MPC inside RL often requires time-consuming problem construction or excessive training overhead, making such frameworks difficult to justify in practice. This work studies efficient training-time MPC guidance for humanoid locomotion and manipulation, termed MPC-RL. We introduce a centroidal-dynamics MPC reward formulation that leverages guidance from MPC trajectories in training time. To make this practical in massively parallel RL, we develop

π^n

MPC, a parallel-in-horizon and construction-free batched GPU MPC solver that operates directly on time-varying dynamics to avoid high memory usage and pre-compilation. Through a variety of comparative studies and hardware validations, we have found that MPC-RL achieves superior performance in locomotion and manipulation skills. The code base is available at https://github.com/junhengl/mpc-rl.

Abstract

Reward Shaping

Explore similar work

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

Automating Potential-based Reward Shaping with Vision Language Model Guidance

Critic Architecture Matters: Dual vs. Unified Critics for Humanoid Loco-Manipulation