NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation

Xiangyu Miao; Junsong Wu; Jiyuan Shi; Weiji Xie; Jinrui Han; Xingyi Wang; Weinan Zhang; Chenjia Bai

cs.ROSep 30, 2026

NEXUS: Perceptive Whole-Body Control for Terrain-Adaptive Teleoperation

Authors: Xiangyu Miao, Junsong Wu, Jiyuan Shi, Weiji Xie, Jinrui Han, Xingyi Wang, Weinan Zhang, Chenjia Bai, +1 more

Organizations: The Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University · The University of Hong Kong · Zhejiang University · γ-Robotics

Abstract

Whole-body teleoperation requires a humanoid robot to reproduce a human operator's behavior even when their terrains differ. This demands that the robot perceive local terrain and adapt its posture and contacts accordingly, rather than copy the operator's motion frame by frame. However, paired motion data linking the same behaviors across flat ground and different terrains remain scarce, limiting supervision for learning terrain-adaptive control. To enable whole-body teleoperation across mismatched terrains, we introduce NEXUS, a perceptive whole-body control framework that combines human motion commands with onboard sensory feedback. We first develop a scalable terrain-aware adaptation algorithm that efficiently generates high-quality motion pairs across motions and terrains without per-motion or per-terrain tuning. Using a paired motion corpus totaling nearly 1,000 hours, we train a perceptive whole-body controller through teacher-student learning to reproduce commanded behaviors across terrains. Experiments demonstrate efficient, scalable generation of high-quality motion data and show that NEXUS combines broad behavioral coverage with terrain adaptability and tracking fidelity, outperforming existing whole-body controllers on the evaluated benchmarks. Zero-shot real-world deployment enables real-time whole-body teleoperation on diverse unseen terrains, further validating the generalization of our method. Project website: https://nexus-humanoid.github.io/

Figures & tables

Fig. 2: NEXUS overview. (a) Contact-guided motion adaptation: contact detection, pelvis adjustment, IK with penetration correction, and trajectory reconstruction from contact keyframes. Contact markers are illustrative. (b) A privileged teacher trained with PPO supervises a student through DAgger-style behavior cloning (BC) using paired adapted and flat-ground motion commands. (c) Zero-shot whole-body teleoperation with onboard depth sensing and policy inference.

Motions without hand contact
Method	VTR (%) $\uparrow$	Penetration (cm) $\downarrow$	Floating (cm) $\downarrow$	CP (%) $\uparrow$	Deviation (rad) $\downarrow$	Generation RTF $\downarrow$
NEXUS (ours)	92.08	0.814	0.880	79.12	0.00984	1.646
TCRS (in PMT)	89.69	0.606	2.965	29.63	0.08884	8.038
Root-only	63.37	2.865	0.817	54.25	0.00000	0.260
Motions with hand contact
NEXUS (ours)	80.53	1.630	0.946	71.21	0.01473	1.864

TABLE I: Kinematic motion adaptation quality and efficiency.

Fig. 3: Baseline failures and NEXUS improvements at matching frames: (a) foot floating in both baselines; (b) foot penetration in Root-only and floating in TCRS; (c) hand penetration in both baselines; (d) TCRS distorts the non-contact pose by retracting the extended free leg. NEXUS improves contact while preserving the leg’s extension. Insets magnify circled regions; dashed circles mark the support foot.

Fig. 4: Whole-body control generalization compared with four open-source controllers. (a) NEXUS achieves the highest motion completion rates across the three dataset subsets, including ties. (b) NEXUS maintains the highest completion rates across terrain difficulty levels. (c) NEXUS attains the lowest body orientation error and highest foot-contact F1 on both flat ground and terrain. Tracking metrics use each controller’s own pre-failure window.

Metric	NEXUS		w/o vision		w/o T–S
SR (%) $\uparrow$	$\mathbf{91.072}$ $\pm$	$\mathbf{14.959}$	$87.771$ $\pm$	$19.362$	$90.964$ $\pm$	$15.938$
Root Pos (cm) $\downarrow$	$\mathbf{19.596}$ $\pm$	$\mathbf{\phantom{0}5.398}$	$20.089$ $\pm$	$\phantom{0}6.651$	$31.951$ $\pm$	$\phantom{0}4.472$
MPKPE (cm) $\downarrow$	$\mathbf{7.066}$ $\pm$	$\mathbf{\phantom{0}2.949}$	$7.603$ $\pm$	$\phantom{0}3.753$	$9.482$ $\pm$	$\phantom{0}3.533$
MPKRE (rad) $\downarrow$	$\mathbf{0.127}$ $\pm$	$\mathbf{\phantom{0}0.010}$	$0.128$ $\pm$	$\phantom{0}0.010$	$0.161$ $\pm$	$\phantom{0}0.015$
Contact F1 (%) $\uparrow$	$\mathbf{80.662}$ $\pm$	$\mathbf{\phantom{0}2.545}$	$80.523$ $\pm$	$\phantom{0}2.876$	$79.637$ $\pm$	$\phantom{0}2.859$

TABLE II: Vision and teacher–student ablations.

Contact detection: entry / exit
Parameter	Value
Link-origin height [m]	$0.18\;/\;0.25$
Horizontal speed multiplier	$0.20\;/\;0.30$
Absolute vertical speed multiplier	$0.15\;/\;0.25$
Root horizontal speed floor [m/s]	$1.0$
Terrain queries and IK

TABLE III: Motion adaptation parameters.

Fig. 5: Motion adaptation schematics. (a) Height-based contact hysteresis, assuming both speed conditions remain satisfied. Shading denotes active contact. (b) PCHIP reconstruction of a scalar correction from contact keyframes, with constant extension beyond their span. Curves are illustrative, not measured results.

Reward term	Expression	Weight
Tracking
Root position	$K_{0.3}(\\|\Delta\mathbf{p}\\|^{2})$	$0.5$
Root orientation	$K_{0.4}(d_{R}(R,R^{\mathrm{adapt}})^{2})$	$0.5$
Root linear velocity	$K_{1.0}(\\|\Delta\mathbf{v}\\|^{2})$	$1.0$
Root angular velocity	$K_{3.0}(\\|\Delta\boldsymbol{\omega}\\|^{2})$	$1.0$
Body position	$K_{0.3}(\langle\\|\Delta\mathbf{p}_{b}^{\mathrm{rel}}\\|^{2}\rangle_{b})$	$2.0$

TABLE IV: Reward function details.

Startup: teacher and student
Randomized quantity	Distribution
Foot friction coefficient	$\mathcal{U}(0.3,1.2)$
Torso CoM offset $x$ [m]	$\mathcal{U}(-0.025,0.025)$
Torso CoM offset $y,z$ [m]	$\mathcal{U}(-0.05,0.05)$
Joint encoder bias [rad]	$\mathcal{U}(-0.01,0.01)$
Reset and pushes: teacher and student

TABLE V: Domain randomization applied during training.

Training hyperparameters
Setting	Teacher (PPO)	Student (DAgger)
Physics / control step [s]	0.005 / 0.020	0.005 / 0.020
Parallel environments	$4\times 16{,}384$	$4\times 16{,}384$
Initial learning rate	$10^{-3}$	$10^{-3}$
Learning-rate schedule	Adaptive (target KL $0.01$ )	Constant
Rollout steps per environment	24	16

TABLE VI: Training hyperparameters and network architecture.

Explore similar work

Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain
Jun 6, 2026Zifan Wang, Yizhao Li, Teli Ma +5Full-Body Humanoid Control Human Motion
1Mondo Robotics · 2The Hong Kong University of Science and Technology (Guangzhou) · 4Artificial General Intelligence Institute, University of Science and Technology of China +2
Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking
Apr 19, 2026Zewei Zhang, Kehan Wen, Michael Xu +7Full-Body Humanoid Control Whole-Body
Robotic Systems Lab, ETH Zurich · EPFL · Simon Fraser University +2
GigaBrain-WBC-0.5: A Behavior World Model for Robust Humanoid Whole-Body Tracking with Environment Interaction
Aug 18, 2026Ziyang Cheng, Tianshu Tang, Jinxin Lan +19Whole-Body Control World Models
Tsinghua University · GigaAI · University of Shanghai for Science and Technology +3

Jun 6, 2026, cs.ROJ/K move · Enter open · S save

Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain

Zifan Wang, Yizhao Li, Teli Ma, Qiang Zhang+4

1Mondo Robotics · 2The Hong Kong University of Science and Technology (Guangzhou) · 4Artificial General Intelligence Institute, University of Science and Technology of China+2

Full-Body Humanoid Control Human Motion Motion Spatial Supervision

Open paper AskPDF arXiv

Humanoid behavior foundation models aim to acquire reusable whole-body control policies from broad human motion priors, enabling a single controller to produce diverse and expressive behaviors. However, existing motion-centric foundation policies largely assume that the reference motion is already physically compatible with the robot's surroundings. This assumption breaks when the demonstrator, operator, and robot inhabit different environments: a human motion may specify the intended behavior, but not the footholds, clearance, body height, or contact timing required by the robot's local terrain. We introduce \emph{Perceptive Behavior Foundation Model} (Perceptive BFM), a terrain-aware humanoid control framework that grounds human motion priors in robot-centric perception. The model preserves raw kinematic motion references as the behavioral interface, while using local terrain observations to adapt contacts, posture, and timing. To provide scalable terrain supervision, we develop \emph{terrain-conformal reference synthesis} (TCRS), which converts locomotion-oriented human motion clips into terrain-consistent references through contact-aware foothold construction, foot-geometry-aware swing optimization, support-aware root reconstruction, collision repair, and multi-point inverse kinematics. We then train a blind adapted-reference teacher and transfer its terrain-conformal behavior to a deployed raw-reference student through target-frame action alignment. The student is an identity-gated Transformer tracker whose terrain features enter through residual pathways initialized to preserve the motion-tracking prior and trained to produce local corrections only when needed.

Abstract

Explore similar work

Perceptive Behavior Foundation Model: Adapting Human Motion Priors to Robot-Centric Terrain

Learning Whole-Body Humanoid Locomotion via Motion Generation and Motion Tracking

GigaBrain-WBC-0.5: A Behavior World Model for Robust Humanoid Whole-Body Tracking with Environment Interaction