SteadyTray: Learning Object Balancing Tasks in Humanoid Tray Transport via Residual Reinforcement Learning

Anlun Huang; Zhenyu Wu; Soofiyan Atar; Yuheng Zhi; Michael Yip

cs.ROMar 11, 2026

SteadyTray: Learning Object Balancing Tasks in Humanoid Tray Transport via Residual Reinforcement Learning

Authors: Anlun Huang, Zhenyu Wu, Soofiyan Atar, Yuheng Zhi, Michael Yip

Organizations: UC San Diego

Abstract

Stabilizing unsecured payloads against the inherent oscillations of dynamic bipedal locomotion remains a critical engineering bottleneck for humanoids in unstructured environments. To solve this, we introduce ReST-RL, a hierarchical reinforcement learning architecture that explicitly decouples locomotion from payload stabilization, evaluated via the SteadyTray benchmark. Rather than relying on monolithic end-to-end learning, our framework integrates a robust base locomotion policy with a dynamic residual module engineered to actively cancel gait-induced perturbations at the end-effector. This architectural separation ensures steady tray transport without degrading the underlying bipedal stability. In simulation, the residual design significantly outperforms end-to-end baselines in gait smoothness and orientation accuracy, achieving a 96.9% success rate in variable velocity tracking and 74.5% robustness against external force disturbances. Successfully deployed on the Unitree G1 humanoid hardware, this modular approach demonstrates highly reliable zero-shot sim-to-real generalization across various objects and external force disturbances.

Figures & tables

Fig. 2 : Overview of the ReST-RL framework. Base Policy Training: A locomotion policy is first trained to carry a tray while maintaining a stable gait. Residual Module Training: using privileged observations, a residual module learns whole-body corrective adjustments on top of the frozen base policy to stabilize the payload under disturbances. Two residual designs are considered: (a) Residual Action Adapter , which adds corrective residual actions to the base action, and (b) Residual FiLM Adapter , which modulates intermediate activations of the frozen base policy via layer-wise FiLM residuals. The student encoder distillation process is shown in Fig. 3.

Fig. 3 : Residual module distillation. The teacher encoder uses privileged observations, whereas the student encoder uses object-centric inputs; both feed into a frozen residual adapter for latent alignment.

Term	Equation
Object Upright	$\exp(-\lambda_{\text{upright}}\big\\|\mathbf{P}_{xy}\big(\mathbf{g}^{\text{obj}}_{t})\big\\|^{2}_{2})$
Object-tray Contact	$\mathbf{1}_{\{\text{object\_tray\_contact}\}}$
Linear XY Vel Tracking	$\exp(-\lambda_{v_{xy}}\big\\|\mathbf{v}^{r_{x,y}}_{t}-\mathbf{\hat{v}}^{r_{x,y}}_{t}\big\\|^{2}_{2})$
Angular Vel Tracking	$\exp(-\lambda_{\boldsymbol{\omega}_{z}}\big\\|\boldsymbol{\omega}^{r_{\text{yaw}}}_{t}-\boldsymbol{\hat{\omega}}^{r_{\text{yaw}}}_{t}\big\\|^{2}_{2})$
Torso Vertical Vel	$\exp(-\lambda_{v}v^{r_{z}\ 2}_{t})$
Torso Angular Vel	$\exp(-\lambda_{\omega}\big\\|\boldsymbol{\omega}^{r_{x,y}}_{t}\big\\|^{2}_{2})$

TABLE I : Key Rewards Functions

Simulation Results (Isaac Lab)		TrackLinErr (m/s) $\mathbf{\downarrow}$		TrackAngErr (rad/s) $\mathbf{\downarrow}$		Grav-XY $\mathbf{\downarrow}$		Success Rate (%) $\mathbf{\uparrow}$
Task	Method	mean	std	mean	std	mean	std	Success Rate (%) $\mathbf{\uparrow}$
Command Track	Base Policy (WB)	0.110	0.107	0.078	0.065	0.179	0.064	47.4
	End2End	0.116	0.111	0.096	0.078	0.046	0.029	89.1
	ReST-RL (Action WB)	0.093	0.100	0.081	0.070	0.029	0.022	95.9
	ReST-RL (Action JT)	0.160	0.102	0.135	0.106	0.031	0.019	96.7
	ReST-RL (FiLM WB)	0.106	0.098	0.069	0.058	0.046	0.023	96.9

TABLE II : Simulation results in Isaac Lab. Object stability of baselines and instantiations of ReST-RL with different Residual Module architectures is evaluated across various scenarios. In all three tasks, the target locomotion commands are identical.

Panel of Figure 5 — Fig. 4 : Training reward comparison between End2End and ReST-RL .

Fig. 5 : Success rate of ReST-RL trained with and without observation delay under increasing perception latency in Push Robot task.

Fig. 6 : Velocity tracking and disturbance recovery of ReST-RL . Torso and object velocities closely follow the commanded velocity under command changes and external pushes.

Panel of Figure 8 — Fig. 7 : Success rate of ReST-RL in ( a ) Push Robot task and ( b ) Push Object task under varying push directions and magnitudes.

Fig. 8 : Success rate of ReST-RL across different object body scales in Push Robot task.

Fig. 9 : ReST-RL deployed on a Unitree G1 humanoid demonstrating payload stabilization capabilities. ( a ) Stabilizing the payload when the robot is kicked; ( b ) stabilizing the payload when the object is pushed; ( c , d ) stably transporting various payloads.

	Term	Dim
Proprioception ( $s^{\text{prop}}$ )	Joint position	$\mathbb{R}^{29}$
	Joint velocity	$\mathbb{R}^{29}$
	Last action	$\mathbb{R}^{29}$
	Base angular velocity	$\mathbb{R}^{3}$
	Base projected gravity	$\mathbb{R}^{3}$
Object ( $s^{\text{obj}}$ )	Object position (camera frame)	$\mathbb{R}^{3}$

TABLE III : Observation Terms

Training Stage	Term	Expression	Weight
Stage 1: Pre-train Locomotion	Locomotion tracking rewards
	Track linear vel	$\exp\!\bigl(-\big\\|\mathbf{v}^{r_{x,y}}_{t}-\mathbf{\hat{v}}^{r_{x,y}}_{t}\big\\|^{2}_{2}/\sigma^{2}\bigr)$ , $\sigma{=}\sqrt{0.25}$	1.0
	Track angular vel	$\exp\!\bigl(-\big\\|\boldsymbol{\omega}^{r_{\text{yaw}}}_{t}-\boldsymbol{\hat{\omega}}^{r_{\text{yaw}}}_{t}\big\\|^{2}_{2}/\sigma^{2}\bigr)$ , $\sigma{=}\sqrt{0.25}$	1.5
	Survival & body stability rewards
	Robot Alive bonus	$\mathbf{1}_{\text{[alive]}}$	1.0
	Torso vertical lin vel	$\exp\!\bigl(-\lambda\,v^{r_{z}\ 2}_{t}\bigr)$ , $\lambda{=}2$	0.2

TABLE IV : Reward and termination terms across three training stages. Terms marked with † are delayed-timeout termination conditions: they do not immediately reset the episode, but will trigger a timeout (truncation) after the configured delay has elapsed. Terms marked with ‡ indicate a change from the previous stage.

Parameter	Range / Value
Robot static friction	$[0.3,1.0]$
Robot dynamic friction	$[0.3,1.0]$
End-effector static friction	$[2.0,3.0]$
End-effector dynamic friction	$[1.5,2.5]$
End-effector restitution	$[0.0,0.05]$
End-effector mass	$[0.05,0.3]\,$ kg

TABLE V : Domain randomization parameters

Runner
Hyperparameters	Values
Number of environments	8 192 (4 096 $\times$ 2 GPUs)
Number of steps per environment	24
Base policy
MLP size (actor & critic)	[512, 256, 128]
Transformer encoder

TABLE VI : Hyperparameters for residual teacher policy (PPO). Both adapter variants share the same base MLP, encoder, and PPO configuration.

Runner
Hyperparameters	Values
Number of environments	8 192 (4 096 $\times$ 2 GPUs)
Number of steps per environment	24
Algorithm (Distillation)
Learning rate	$5\times 10^{-5}$
Num learning epochs	1

TABLE VII : Hyperparameters for encoder distillation.

Explore similar work

Mar 9, 2026

Embedding Classical Balance Control Principles in Reinforcement Learning for Humanoid Recovery

Humanoid robots remain vulnerable to falls and unrecoverable failure states, limiting their practical utility in unstructured environments. While reinforcement learning has demonstrated stand-up behaviors, existing approaches treat recovery as a pure task-reward problem without an explicit representation of the balance state. We present a unified RL policy that addresses this limitation by embedding classical balance metrics: capture point, center-of-mass state, and centroidal momentum, as privileged critic inputs and shaping rewards directly around these quantities during training, while the actor relies solely on proprioception for zero-shot hardware transfer. Without reference trajectories or scripted contacts, a single policy spans the full recovery spectrum: ankle and hip strategies for small disturbances, corrective stepping under large pushes, and compliant falling with multi-contact stand-up using the hands, elbows, and knees. Trained on the Unitree H1-2 in Isaac Lab, the policy achieves a 93.4% recovery rate across randomized initial poses and unscripted fall configurations. An ablation study shows that removing the balance-informed structure causes stand-up learning to fail entirely, confirming that these metrics provide a meaningful learning signal rather than incidental structure. Sim-to-sim transfer to MuJoCo and preliminary hardware experiments further demonstrate cross-environment generalization. These results show that embedding interpretable balance structure into the learning framework substantially reduces time spent in failure states and broadens the envelope of autonomous recovery.

Nehar Poddar, Stephen McCrory, Luigi Penco +3

Florida Institute for Human and Machine Cognition, Pensacola, FL, USA. · University of West Florida, Pensacola, FL, USA.

Humanoid Robot Control Humanoid Robotics Reinforcement Learning

Ask PDF

Oct 5, 2026

Transporting Unsecured Stacked Payloads with a Quadrupedal Robot via Multi-Objective Reinforcement Learning

Transporting unsecured payloads with legged robots over uneven terrain requires balancing locomotion performance and payload stability, since aggressive motion can destabilize the payload even when the robot remains stable. We study quadrupedal transportation of unsecured stacked boxes on an edgeless torso-mounted board without dedicated payload sensors or active carrier mechanisms. To address this trade-off, we propose Payload-Adaptive Multi-Objective Reinforcement learning for Transportation (PAMORT). PAMORT trains a multi-objective base policy conditioned on a preference vector that weights locomotion and payload-stability reward groups, then trains a weight adjuster on the frozen policy to adapt this preference online from proprioception. In simulation, PAMORT achieves comparable or better overall transportation success than a corresponding single-objective baseline across different payload configurations, including an unseen three-box stack, despite training only with two boxes. Real-world experiments on a Unitree Go2 demonstrate zero-shot transfer to slopes and steps at or beyond the training difficulty, with mean success rates of 0.850 for PAMORT and 0.675 for the baseline across eight tasks. These results demonstrate robust unsecured-payload transportation with online adaptation of the locomotion--payload trade-off from proprioceptive information.

Nobuo Namura, Masayuki Hiromoto, Kento Uemura +2

Fujitsu Limited, Kawasaki, Japan

Multi-Objective Reinforcement Learning Robot Policy Adaptation Robotics RL for Legged Locomotion Quadruped Locomotion

Ask PDF

Jun 15, 2026

ADAPT: Analytical Disturbance-Aware Policy Training for Humanoid Locomotion

Humanoids deployed in human-centered environments must handle force-interactive tasks, where external contacts introduce unexpected disturbances that disrupt locomotion accuracy and stability. Existing learning-based approaches rely on broad domain randomization, task-specific force objectives, or learning-based force estimators from motion history, each of which compromises accuracy, task transferability, or out-of-distribution (OOD) robustness. We present Analytical Disturbance-Aware Policy Training (ADAPT), a framework that equips humanoid policies with a physically grounded disturbance observer. The core of ADAPT is an analytical whole-body disturbance observer that estimates residual force/torque online with the accessible robot dynamics, without requiring force/torque sensors. Fed directly into the policy, the estimated disturbances give the humanoid an explicit, physics-derived sense of external force/torque that can generalize across diverse unseen scenes. Experiments on a Unitree G1 humanoid show that ADAPT achieves accurate disturbance prediction and stronger robustness than a proprioception-only baseline under torso perturbations, standing pushes, and asymmetric hand payloads, with improved velocity tracking even on OOD disturbances. Moreover, ADAPT enables penalizing inferred disturbances at lower-body joints to encourage lighter locomotion.

Bofan Lyu, Jindou Jia, Kuangji Zuo +7

1MARS Lab, Nanyang Technological University

Robotic Control Humanoid Robot Locomotion Robot Policy Learning Robot State Estimation Robust RL

Ask PDF