cs.ROMar 3, 2026

ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation

Authors: Xialin He, Sirui Xu, Xinyao Li, Runpei Dong, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui

Organizations: University of Illinois Urbana-Champaign

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Achieving autonomous and versatile whole-body loco-manipulation remains a central barrier to making humanoids practically useful. Yet existing approaches are fundamentally constrained: retargeted data are often scarce or low-quality; methods struggle to scale to large skill repertoires; and, most importantly, they rely on tracking predefined motion references rather than generating behavior from perception and high-level task specifications. To address these limitations, we propose ULTRA, a unified framework with two key components. First, we introduce a physics-driven neural retargeting algorithm that translates large-scale motion capture to humanoid embodiments while preserving physical plausibility for contact-rich interactions. Second, we learn a unified multimodal controller that supports both dense references and sparse task specifications, under sensing ranging from accurate motion-capture state to noisy egocentric visual inputs. We distill a universal tracking policy into this controller, compress motor skills into a compact latent space, and apply reinforcement learning finetuning to expand coverage and improve robustness under out-of-distribution scenarios. This enables coordinated whole-body behavior from sparse intent without test-time reference motions. We evaluate ULTRA in simulation and on a real Unitree G1 humanoid. Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception, consistently outperforming tracking-only baselines with limited skills.

Figures & tables

Fig. 2 : ULTRA follows four stages: ( i ) Neural Retargeting: an RL policy converts MoCap data into physically feasible G1 rollouts with augmentation; ( ii ) Tracking: a privileged teacher tracks these rollouts using full state and references; ( iii ) Distillation: we distill the teacher into a multimodal student for realistic sensing and sparse goals, with additional RL finetuning; ( iv ) Deployment: the student runs under real sensing, supporting depth input or MoCap-based state estimation.
Fig. 3 : Qualitative comparison of our retargeting and OmniRetarget [ 2 ] at the same frame/sequence. Top: final frame; the baseline shows undesired standing foot placement. Bottom: a contact frame; ours yields more stable contacts.
 In-Distribution (ID)Out-of-Distribution (OOD)
 Succ↑\text{Succ}\uparrowSucc↑\text{Succ}\uparrowHumanoidObjectSucc↑\text{Succ}\uparrowSucc↑\text{Succ}\uparrowHumanoidObject
Method(Humanoid)(+Object)Eg-mpjpe↓E_{\text{g-mpjpe}}\downarrowEmpjpe↓E_{\text{mpjpe}}\downarrowEjitter↓E_{\text{jitter}}\downarrowEpos↓E_{\text{pos}}\downarrowErot↓E_{\text{rot}}\downarrow(Humanoid)(+Object)Eg-mpjpe↓E_{\text{g-mpjpe}}\downarrowEmpjpe↓E_{\text{mpjpe}}\downarrowEjitter↓E_{\text{jitter}}\downarrowEpos↓E_{\text{pos}}\downarrowErot↓E_{\text{rot}}\downarrow
(a) Unified Multimodal Controller
ULTRA ( Ours )67.30 ±\pm 0.1257.44 ±\pm 0.4013.49 ±\pm 0.140 5.89 ±\pm 0.020 6.27 ±\pm 0.0053.42 ±\pm 0.2165.44 ±\pm 0.3770.57 ±\pm 0.5452.00 ±\pm 0.4435.55 ±\pm 0.2314.67 ±\pm 0.080 6.81 ±\pm 0.0156.52 ±\pm 0.3567.60 ±\pm 0.58
(b) Privileged Teacher
ULTRA Teacher97.57 ±\pm 0.0589.79 ±\pm 0.1112.98 ±\pm 0.300 5.64 ±\pm 0.0514.81 ±\pm 0.0817.15 ±\pm 0.0323.28 ±\pm 0.3397.12 ±\pm 0.4381.33 ±\pm 0.7819.14 ±\pm 0.480 7.94 ±\pm 0.1115.91 ±\pm 0.0825.57 ±\pm 0.2833.49 ±\pm 0.37
TABLE I : Motion-tracking evaluation in IsaacGym. All methods are trained/evaluated on our data unless noted. Green highlights our primary tracking controller.
Fig. 4 : Zero-shot augmentation with the retargeting policy. Left: trajectory scaling. Right: object scaling. Motions remain plausible, enabling scalable data augmentation.
Table 5
Fig. 6 : Sim-to-sim comparison for egocentric goal following. Blue/green: point cloud without/with noise; yellow: object goal. Top: without finetuning. Bottom: with finetuning.
RL finetuningID GoalsOOD Goals
PointsPositionPointsPosition
✗16 / 2014 / 205 / 204 / 20
✓19 / 2016 / 209 / 2012 / 20
Δ\Delta (RL gain)+18.8%+14.3%+80.0%+200.0%
TABLE III : Sim-to-sim success rate on Mujoco across goal types with in-distributional (ID) goals from training and out-of-distribution (OOD) goals with random offsets, and across perception with egocentric point clouds or object position with no shape. Policies are trained in IsaacGym and evaluated in MuJoCo with 20 selected motion per setting.
SettingVerticalLateral
Dense Reference Tracking73% (19/26)
Sparse Goal Following (MoCap)80% (8/10)90% (9/10)
Sparse Goal Following (Egocentric)50% (5/10)60% (6/10)
TABLE IV : Real-world success rates on the OMOMO subset using a Unitree G1 humanoid. Each task is evaluated over two trials. MoCap provides object pose tracking for non-egocentric control modes, while the egocentric setting relies only on onboard sensing. MoCap is used for success evaluation in all settings. Dense reference tracking is direction-agnostic and thus reported as a single success rate.
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Unitree G1 LinkSMPL-X IndexSMPL-X Body Part
left_hip_yaw_link1Left hip
left_knee_link2Left knee
left_ankle_roll_link3Left ankle
right_hip_yaw_link5Right hip
right_knee_link6Right knee
right_ankle_roll_link7Right ankle
Appendix
TABLE A: Correspondence between Unitree G1 key links and SMPL-X body indices (52-body model) used for motion retargeting.
TermExpressionWeight
Body Tracking:
Joint positionexp⁡(−kp⋅mean(∥pl−p^l∥22))\exp\left(-k_{p}\cdot\text{mean}(\|\mathbf{p}_{l}-\hat{\mathbf{p}}_{l}\|_{2}^{2})\right)kp=10.0k_{p}=10.0
Joint rotationexp⁡(−kr⋅mean(∥q−q^∥22))\exp\left(-k_{r}\cdot\text{mean}(\|\mathbf{q}-\hat{\mathbf{q}}\|_{2}^{2})\right)kr=5.0k_{r}=5.0
Body velocityexp⁡(−kpv⋅mean(∥p˙l−p˙^l∥22))\exp\left(-k_{pv}\cdot\text{mean}(\|\dot{\mathbf{p}}_{l}-\hat{\dot{\mathbf{p}}}_{l}\|_{2}^{2})\right)kpv=0.1k_{pv}=0.1
Joint velocityexp⁡(−krv⋅mean(∥q˙−q˙^∥22))\exp\left(-k_{rv}\cdot\text{mean}(\|\dot{\mathbf{q}}-\hat{\dot{\mathbf{q}}}\|_{2}^{2})\right)krv=0.001k_{rv}=0.001
Object Tracking:
Appendix
TABLE B: Tracking reward components for the teacher policy. pl,q,p˙l,q˙\mathbf{p}_{l},\mathbf{q},\dot{\mathbf{p}}_{l},\dot{\mathbf{q}} denote the simulated body joint positions, joint rotation, and their velocities; p^l,q^,p˙^l,q˙^\hat{\mathbf{p}}_{l},\hat{\mathbf{q}},\hat{\dot{\mathbf{p}}}_{l},\hat{\dot{\mathbf{q}}} are the corresponding reference quantities. po,qo,p˙o\mathbf{p}_{o},\mathbf{q}_{o},\dot{\mathbf{p}}_{o} denote the simulated object position, rotation (quaternion), and linear velocity; p^o,q^o,p˙^o\hat{\mathbf{p}}_{o},\hat{\mathbf{q}}_{o},\hat{\dot{\mathbf{p}}}_{o} are the corresponding references. ∠(qo,q^o)\angle(\mathbf{q}_{o},\hat{\mathbf{q}}_{o}) is the relative rotation angle and huber(⋅)\mathrm{huber}(\cdot) is the Huber loss. δij\boldsymbol{\delta}_{ij} denotes the palm-to-surface distance between palm point ii and object surface sample jj , with weights wijw_{ij} ; hats denote reference values. cl∈{0,1}c_{l}\in\{0,1\} is a binary contact indicator for link ll . The scalars k⋅k_{\cdot} are temperature coefficients. All reward terms are multiplied together.
TermExpressionWeight
Velocity Penalties:
Base linear velocity∥vbase∥2\|\mathbf{v}_{\text{base}}\|_{2}−0.1-0.1
Base angular velocity∥ωbase∥22\|\boldsymbol{\omega}_{\text{base}}\|_{2}^{2}−0.01-0.01
Joint velocity∥q˙∥22\|\dot{\mathbf{q}}\|_{2}^{2}−0.0004-0.0004
Smoothness Penalties:
Action rate∥at−at−1∥2\|\mathbf{a}_{t}-\mathbf{a}_{t-1}\|_{2}−0.1-0.1
Appendix
TABLE C: Smoothness and regularization rewards for the teacher policy. All terms are penalties with negative weights wi<0w_{i}<0 . vbase\mathbf{v}_{\text{base}} and ωbase\boldsymbol{\omega}_{\text{base}} are the base linear and angular velocities. at\mathbf{a}_{t} is the action at time tt ; q˙t\dot{\mathbf{q}}_{t} is the joint velocity at time tt ; and ωt\boldsymbol{\omega}_{t} is the base angular velocity at time tt . τ\boldsymbol{\tau} denotes joint torques and ⊙\odot is elementwise multiplication. qlim\mathbf{q}_{\text{lim}} and τlim\boldsymbol{\tau}_{\text{lim}} are per-joint position and torque limits. \mathds1contact\mathds{1}_{\text{contact}} is an indicator for foot contact; fxy\mathbf{f}_{xy} and fz\mathbf{f}_{z} are the horizontal and vertical components of the contact force. dfeetd_{\text{feet}} and dkneed_{\text{knee}} are the distances between the two feet and the two knees, and clamp(⋅)\mathrm{clamp}(\cdot) clips the distance to the specified interval. g⊥feet\mathbf{g}_{\perp}^{\text{feet}} measures foot tilt relative to gravity. ref_stand and sim_stand indicate standing phases in the reference and simulation. htermh_{\text{term}} is a swing-foot height term, cc is a clearance/contact-related term, and gswingg_{\text{swing}} gates the swing penalty, which is active only during reference indicating swing.
CategoryFeatureDimensionDescription
osim\boldsymbol{o}^{\text{sim}}Root height1Root height above ground
Local body positions p\mathbf{p}11439 bodies ×\times 3 (root removed)
Local body rotations R\mathbf{R}23439 bodies ×\times 6 (tan-norm)
Local body velocities p˙\dot{\mathbf{p}}11739 bodies ×\times 3
Local body angular vel. ω\boldsymbol{\omega}11739 bodies ×\times 3
Contact indicators cc39Binary contact flags
Appendix
TABLE D: Teacher policy observation space. At time tt , ot=[otsim,otref,otΔ,otig]\boldsymbol{o}_{t}=[\boldsymbol{o}_{t}^{\text{sim}},\boldsymbol{o}_{t}^{\text{ref}},\boldsymbol{o}_{t}^{\Delta},\boldsymbol{o}_{t}^{\text{ig}}] . otsim\boldsymbol{o}_{t}^{\text{sim}} contains simulated quantities: root height; per-body local positions p\mathbf{p} , rotations R\mathbf{R} in 6D tan-norm, linear velocities p˙\dot{\mathbf{p}} , angular velocities ω\boldsymbol{\omega} , contact flags c∈{0,1}c\in\{0,1\} ; and joint states (positions q\mathbf{q} , velocities q˙\dot{\mathbf{q}} , actions a\mathbf{a} , torques τ\boldsymbol{\tau} , and history). otref\boldsymbol{o}_{t}^{\text{ref}} contains corresponding reference targets (hats), e.g., body pose (p^,R^)(\hat{\mathbf{p}},\hat{\mathbf{R}}) and object pose/velocity (x^o,q^o,v^o,ω^o)(\hat{\mathbf{x}}_{o},\hat{\mathbf{q}}_{o},\hat{\mathbf{v}}_{o},\hat{\boldsymbol{\omega}}_{o}) . otΔ\boldsymbol{o}_{t}^{\Delta} stores residuals between simulation and reference ( e.g., p−p^\mathbf{p}-\hat{\mathbf{p}} and velocity errors), and otig\boldsymbol{o}_{t}^{\text{ig}} stores interaction-graph features based on SDF distances between body points and the object surface and their residuals. Two observations based on next 1-frame and next 16-frame reference are concatenated, yielding total dimension 4052.
HyperparameterValue
Learning rate2×10−52\times 10^{-5}
Clip ratio ϵ\epsilon0.2
GAE λ\lambda (tau)0.95
Discount γ\gamma0.99
Horizon length32
Mini-batch size16384
Appendix
TABLE E: PPO hyperparameters.
TermDefinitionWeight / schedule
Total lossL=λELE+λKLLKL+λSLS+λALA+λGLG+λPLP\mathcal{L}=\lambda_{E}\mathcal{L}_{E}+\lambda_{\text{KL}}\mathcal{L}_{\text{KL}}+\lambda_{S}\mathcal{L}_{S}+\lambda_{A}\mathcal{L}_{A}+\lambda_{G}\mathcal{L}_{G}+\lambda_{P}\mathcal{L}_{P}see below
Expert imitationLE=∥μ−aexp∥22\mathcal{L}_{E}=\big\|\boldsymbol{\mu}-\boldsymbol{a}^{\text{exp}}\big\|_{2}^{2}λE=1.0\lambda_{E}=1.0
Latent KL alignmentLKL=DKL ⁣(qϕ(z) ∥ pθ(z))\mathcal{L}_{\text{KL}}=D_{\text{KL}}\!\left(q_{\phi}(z)\ \|\ p_{\theta}(z)\right)λKL=0.001+(0.1−0.001) g(s)\lambda_{\text{KL}}=0.001+(0.1-0.001)\,g(s)
Latent smoothnessLS=∥(μp+μe)t−(μp+μe)t−1∥22\mathcal{L}_{S}=\big\|\big(\boldsymbol{\mu}_{p}+\boldsymbol{\mu}_{e}\big)_{t}-\big(\boldsymbol{\mu}_{p}+\boldsymbol{\mu}_{e}\big)_{t-1}\big\|_{2}^{2}λS=0.0001+(0.001−0.0001) g(s)\lambda_{S}=0.0001+(0.001-0.0001)\,g(s)
Auxiliary predictionLA=MSE(y^aux,yaux)\mathcal{L}_{A}=\mathrm{MSE}\big(\hat{\boldsymbol{y}}_{\text{aux}},\boldsymbol{y}_{\text{aux}}\big) with mask-weightingλA=1.0\lambda_{A}=1.0
Local-goal predictionLG=MSE(g^local,glocal)\mathcal{L}_{G}=\mathrm{MSE}\big(\hat{\boldsymbol{g}}_{\text{local}},\boldsymbol{g}_{\text{local}}\big) with mask-weightingλG=1.0\lambda_{G}=1.0
Appendix
TABLE F: Distillation-stage loss terms. μ\boldsymbol{\mu} is the student action mean from the full model forward pass, and μprior\boldsymbol{\mu}^{\text{prior}} is the action mean from a prior-only forward pass (encoder disabled). aexp\boldsymbol{a}^{\text{exp}} denotes the expert action mean (teacher target) used for supervision. The prior and privileged encoder output diagonal Gaussians over the latent zz , denoted by pθ(z∣o)=N(μp,σp2)p_{\theta}(z\mid\boldsymbol{o})=\mathcal{N}(\boldsymbol{\mu}_{p},\boldsymbol{\sigma}_{p}^{2}) and qϕ(z∣o,opriv)=N(μe,σe2)q_{\phi}(z\mid\boldsymbol{o},\boldsymbol{o}^{\text{priv}})=\mathcal{N}(\boldsymbol{\mu}_{e},\boldsymbol{\sigma}_{e}^{2}) . Let ep be the epoch index and s=clip ⁣(ep−5003000, 0, 1)s=\mathrm{clip}\!\left(\frac{\text{ep}-500}{3000},\,0,\,1\right) . The cosine ramp is g(s)=1−cos⁡(πs)2g(s)=\frac{1-\cos(\pi s)}{2} .
TermActive whenDescription
Goal-reachingtarget visibleReward proximity to the sampled goal (dense).
Progresstarget visibleReward step-to-step improvement toward the goal (clipped).
Terminal bonussuccess on visible targetsAdd a bonus when all visible thresholds are met.
TerminationalwaysSame as in Table C .
SmoothnessalwaysMostly same as in Table C without reference related terms
Appendix
TABLE G: RL finetuning reward components . We mask-gate goal-related terms: if a target is unobserved, we set its reward to zero and exclude it from success checks.
CategoryFeatureDimensionDescription
oglobal\boldsymbol{o}^{\text{global}}Root position residual (xy)2Horizontal position error
Heading residual (yaw)1Yaw angle error
ocmd\boldsymbol{o}^{\text{cmd}}End-of-episode flag1Near episode end indicator
Approaching flag1Moving toward object
Leaving flag1Moving away from object
Time-to-go1Normalized remaining time
Appendix
TABLE H: Student policy observation space.
CategoryParameterRange
HumanoidAdded base mass (kg)[−3.0,3.0][-3.0,3.0]
CoM offset (m)[−0.05,0.05][-0.05,0.05]
Motor strength scale[0.8,1.2][0.8,1.2]
EnvironmentGround friction[0.5,2.0][0.5,2.0]
Gravity perturbation (m/s 2 )[−0.1,0.1][-0.1,0.1]
 Gravity randomization interval4 s
Appendix
TABLE I: Domain randomization ranges.
ParameterRange
Object mass (kg)[0.15,1.5][0.15,1.5]
Object mass rare (kg)[0.05,0.15][0.05,0.15] (every 10 episodes)
Object CoM offset (m)[−0.05,0.05][-0.05,0.05]
Object inertia scale[0.5,2.0][0.5,2.0]
Object friction[0.2,1.2][0.2,1.2]
Object restitution[0.0,0.3][0.0,0.3]
Appendix
TABLE J: Object domain randomization ranges.
ParameterValue
Gaussian noise std (m)0.02
Point dropout probability0.15
Outlier probability0.05
Outlier max distance (m)0.5
Depth noise scale0.01
Density range[0.5, 1.0]
Appendix
TABLE K: Point cloud domain randomization.
ObservationNoise Scale
Joint positions (rad)0.01
Joint velocities (rad/s)0.1
Angular velocity (rad/s)0.05
IMU0.05
Root position (m)0.05
Appendix
TABLE L: Observation noise scales.
SimulationParameterValue
PhysicsSimulation substeps1
Control frequency inverse17 ( ≈\approx 59 Hz)
Number of parallel envs4096
PhysX solver type1 (TGS)
Position iterations4
Velocity iterations1
Appendix
TABLE M: Simulation configuration.
Fig. A: ( I ) The OptiTrack camera setup used to support ULTRA control in MoCap mode. ( II ) Markers attached to the humanoid root (front and back).
Fig. B: We visualize the objects used in our real-world deployment.

Explore similar work

CardsList

  1. OpenHLM: An Empirical Recipe for Whole-Body Humanoid Loco-Manipulation

    Jun 20, 2026Yingdong Hu, Haodong Zhu, Boyuan Zheng +6Humanoid Loco-ManipulationLong-Horizon Robotic Manipulation

    Tsinghua University · Shanghai Qi Zhi Institute · Spirit AI

  2. HULK: Learning Whole-Body Forceful Loco-Manipulation for Humanoids

    Oct 6, 2026An Dang, Arturo Flores Alvarez, Yu-Ming Chen +5Humanoid Loco-ManipulationWhole-Body Control

    Amazon · University of Michigan · University of California, Los Angeles +1

  3. Make Tracking Easy: Neural Motion Retargeting for Humanoid Whole-body Control

    Mar 23, 2026Qingrui Zhao, Kaiyue Yang, Xiyu Wang +6Motion RetargetingRobot Motion Generation

    Nanjing University · Huawei Technologies