Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal

Tianchen Huang; Sisheng Chen; Wei Zhou; Haopeng Zhang; Jiarong Sun; Ya Wang; Yumin Wang; Deguang Lyu

cs.ROAug 28, 2025

Physics-Guided Residual Reinforcement Learning for Humanoid Narrow-Path Traversal

Authors: Tianchen Huang, Sisheng Chen, Wei Zhou, Haopeng Zhang, Jiarong Sun, Ya Wang, Yumin Wang, Deguang Lyu, +3 more

Organizations: Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China

Abstract

Traversing narrow paths is challenging for humanoid robots due to the sparse and safety-critical footholds required. Purely template-based or end-to-end reinforcement learning-based methods suffer from such harsh terrains. This paper proposes a two stage training framework for such narrow path traversing tasks, coupling a template-based foothold planner with a low-level foothold tracker from Stage-I training and a lightweight perception aided foothold modifier from Stage-II training. With the curriculum setup from flat ground to narrow paths across stages, the resulted controller in turn learns to robustly track and safely modify foothold targets to ensure precise foot placement over narrow paths. This framework preserves the interpretability from the physics-based template and takes advantage of the generalization capability from reinforcement learning, resulting in easy sim-to-real transfer. The learned policies outperform purely template-based or reinforcement learning-based baselines in terms of success rate, centerline adherence and safety margins. Validation on a Unitree G1 humanoid robot yields successful traversal of a 0.2m wide and 3m long beam for 20 trials without any failure.

Figures & tables

Fig. 1: Overview. The proposed framework uses 3D-LIPM as foothold planner, low-level policy from Stage-I training as foothold tracker and high-level policy from Stage-II training as foothold modifier for robust and safe narrow path walking. Successful experimental validation on the Unitree G1 humanoid robot is performed.

Fig. 2: The proposed framework for humanoid robot traversing narrow paths. A two-stage training curriculum is designed for the low-level foothold tracker and high-level foothold modifier. Different background colors indicate different operational frequencies

Fig. 3: Demonstration of foothold modification in the Stage-II training. The dashed polygon shows the initial foothold target from the foothold planner for the left leg, $u_{\mathrm{init}}^{(L)}$ , while the solid polygons denote the final foothold targets from the foothold modifier for both legs, $u_{\mathrm{final}}^{(L)}$ (red) and $u_{\mathrm{final}}^{(R)}$ (blue). A smooth Gaussian penalty is given to the foothold’s lateral offset from the beam centerline, $|\,y_{w}-y_{w,\mathrm{center}}\,|$ , to encourage staying near the centerline

Method	Success rate (%)	Centerline dev. (m)	FP-RMSE (m)
No-Modifier	15	0.04690 $\pm$ 0.00057	0.01962 $\pm$ 0.00079
RL-Only	10	0.18192 $\pm$ 0.07075	— a
Ours	100	0.01639 $\pm$ 0.00117	0.02633 $\pm$ 0.00083

TABLE I: Baseline comparison results from walking on beams in simulation (mean

\pm

std over 20 runs)

Configuration	Success rate (%)	Centerline dev. (m)	FP-RMSE (m)
w/o Stage-I disturbances	50	0.09696 $\pm$ 0.07876	0.05467 $\pm$ 0.05467
Ours	100	0.01639 $\pm$ 0.00117	0.02633 $\pm$ 0.00083

TABLE II: Ablation study results from walking on a

0.20\,m

-wide beam in simulation (mean

\pm

std over 20 runs)

Fig. 4: Top view of the anterior terrain height map based on the robot’s LiDAR sensor. The map spans $x\in[0.1,1.1]\,m$ and $y\in[-0.8,0.8]\,m$ with a resolution of $0.1\,m$ uniformly, providing a real-time terrain representation for foothold modification

Setting	Success rate (%)	Traversal rate (%)
BeamDojo [ 3 ] (G1, real beam)	80	88.16
Ours (G1, real beam)	100	100

TABLE III: Hardware comparison results from walking on a

3\,m\,\times\,0.2\,m

beam in reality (mean

\pm

std). Ours:

N{=}20

trials; BeamDojo:

N{=}5

trials as reported

Term	Weight	Equation
step_tracking	$3.0$	$\begin{aligned} &\big(\mathbb{I}_{R}-\mathbb{I}_{L}\big)\,s\;\times\;\phi_{1}(\\|\delta_{p}\\|_{2};\ a_{p}\!=\!1)\\ &\times\;\phi_{1}(\|\delta_{\psi}\|;\ a_{r}\!=\!1)\end{aligned}$
tracking_lin_ vel_world	$4.0$	$\begin{aligned} &\phi_{2}\!\Big(\!\big\\|\,(v^{c}_{xy}-v_{xy})\odot{}\\ &\hskip 17.00024pt(1+\|v^{c}_{xy}\|)^{-1}\big\\|_{2};\ a_{v}\!=\!1\Big)\end{aligned}$
base_heading	$3.0$	$\displaystyle\phi_{1}\!\big(\,\|\mathrm{wrap}(\psi^{c}-\psi)\|;\ a_{\psi}\!=\!\tfrac{\pi}{2}\big)$
base_z_ orientation	$1.0$	$\displaystyle\phi_{2}\!\big(\,\\|g_{xy}\\|_{2};\ a_{g}\!=\!0.2\big)$
base_height	$1.0$	$\displaystyle\phi_{2}\!\big(\,h-h^{\ast};\ a_{h}\!=\!1\big)$
joint_ regularization	$1.0$	$\displaystyle\tfrac{1}{\|\mathcal{J}\|}\!\sum_{j\in\mathcal{J}}\phi_{2}(q_{j};\ a_{q}\!=\!1)$

TABLE IV: Stage-I rewards

Term	Weight	Equation
foothold_safety	$1.0$	$\displaystyle-\,5\!\!\sum_{f\in\{\mathrm{L},\mathrm{R}\}}\!\mathbb{I}\!\{\,h(\mathbf{p}_{t}^{\,f})<\!-0.20\,\}\,m^{\,f}_{\mathrm{swing}}$
beam_balance	$1.0$	$\displaystyle\exp\!\big(\!-\,(\|y-y_{c}\|/\sigma_{y})^{2}\big)-1$
feet_proximity	$0.1$	$\displaystyle-\,\frac{(d_{\min}-\|x_{R}-x_{L}\|)_{+}}{d_{\min}}$
forward_progress	$1.0$	$\displaystyle\max\!\big(0,\ x_{t}-x_{t-1}\big)$
face_forward	$0.1$	$\displaystyle\max\!\big(0,\,1-\|\mathrm{wrap}(\psi)\|/\pi\big)$
contact_schedule	$1.0$	$\begin{aligned} &\big(\mathbb{I}_{R}-\mathbb{I}_{L}\big)\,s\;\times\;\phi_{1}(\\|\delta_{p}\\|_{2};\ a_{p}\!=\!1)\\ &\times\;\phi_{1}(\|\delta_{\psi}\|;\ a_{r}\!=\!1)\end{aligned}$

TABLE V: Stage-II rewards

Symbol	Definition / Value
$\phi_{1}(e;a)$	$\exp\!\big(-\|e\|/(a\,\sigma)\big)$
$\phi_{2}(e;a)$	$\exp\!\big(-(e/a)^{2}/\sigma\big)$
$\sigma$	tracking shape scale $=0.25$
$h^{\ast}$	base height target $=0.78$ m
$\mathcal{J}$	joints $\{0,1,5,6\}$ (hip/waist yaw/ab-ad soft centering)
$\delta_{p},\ \delta_{\psi}$	swing-foot pos./yaw error at touchdown (to target)

TABLE VI: Used symbols and constants for reward tables

Observations
Term	Value
angular velocity noise	$\mathcal{U}(-0.2,\;0.2)\ \text{rad/s}$
projected gravity noise	$\mathcal{U}(-0.05,\;0.05)$
joint position noise	$\mathcal{U}(-0.01,\;0.01)\ \text{rad}$
joint velocity noise	$\mathcal{U}(-0.01,\;0.01)\ \text{rad/s}$
height measurement noise	$\mathcal{U}(-0.10,\;0.10)\ \text{m}$

TABLE VII: Domain Randomization Setting

Term	Value
Rollout / Runner
parallel envs	$4096$
steps per env	$24$
rollout size / update	$4096\times 24$ samples
max iterations	$5000$
save interval	$100$

TABLE VIII: Stage-I Training Hyperparameters

Rollout / Runner
Term	Value
parallel envs	1024
steps per env	inherited (not overridden)
max iterations	$10000$
save interval	$100$
policy init noise	$1.0$

TABLE IX: Stage-II Training Hyperparameters

Explore similar work

Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking
Jun 6, 2026Alessandro Montenegro, Shihao Li, Puze Liu +2Robotic Manipulation Policies Control System Operator Learning
Politecnico di Milano · Tongji University · Technische Universität Darmstadt +2
SSR: Scaling Surefooted and Symmetric Humanoid Traversal to the Open World
May 29, 2026Ruiqi Yu, Yiwen Wang, Yuan Hao +2Adaptive Locomotion Adversarial Motion Priors
Zhejiang University
Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomy
May 15, 2026William D. Compton, Zachary Olkin, Aaron D. AmesPerceptive Locomotion Policies Autonomous Navigation
Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA 91125

Jun 6, 2026, cs.ROJ/K move · Enter open · S save

Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking

Alessandro Montenegro, Shihao Li, Puze Liu, Alberto Maria Metelli+1

Politecnico di Milano · Tongji University · Technische Universität Darmstadt+2

Robotic Manipulation Policies Control System Operator Learning Unified Framework

Open paper AskPDF arXiv

Enabling humanoid robots to operate in complex, dynamic environments remains a critical challenge, fundamentally limited by the ability to navigate robustly, safely, and accurately. While reinforcement learning with velocity-commanded policies has achieved remarkable robustness in humanoid locomotion, this approach lacks explicit control of the foothold placement, leading to unsafe behavior, such as stepping onto human feet, or imprecise navigation, hindering the following manipulation task. Conversely, explicit foothold-tracking policies offer a promising alternative by directly being commanded with target foot poses. However, existing approaches are often limited by unrealistic state assumptions, compromising real-world deployment, or they are part of staged pipelines, making them tied to specific downstream tasks. In this work, we introduce a novel, lightweight framework for training general-purpose 3D foothold-tracking policies. By dynamically providing footstep support through a goal sampler, this method enables the learned policy to be agnostic to specific terrains. Our new target representation effectively mitigates challenges arising in the real world, such as noisy and inaccurate pose estimation and foot contact estimation. Designed for direct real-world transfer, our policy acts as a standalone low-level controller that can be seamlessly paired with various high-level foothold generators. We demonstrate the effectiveness of our framework through extensive experiments in simulation and in the real world. By coupling our policy with different upstream planners, we achieve natural and accurate locomotion in challenging settings, paving the way for loco-manipulation tasks in complex environments.

Abstract

Explore similar work

Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking

SSR: Scaling Surefooted and Symmetric Humanoid Traversal to the Open World

Terrain Consistent Reference-Guided RL for Humanoid Navigation Autonomy