Real-time whole-body humanoid teleoperation remains challenging due to the computational latency introduced by optimization-based retargeting pipelines and future-frame conditioning. Existing motion tracking policies typically require dense reference targets including full joint positions, body-part transformations, and even future frames, which tightly couples the controller to expensive upstream modules. In this work, we present HuMiT, a whole-body teleoperation system built on a unified tracking policy that requires only a compact reference target, consisting of root height, root velocity, and the SE(3) poses of four sparse keypoints at the current frame, yet achieves competitive tracking fidelity compared to methods relying on more diverse reference states including complete joint positions, dense body-part transformations, and even future frames. Experiments show that our policy achieves a communication-to-inference latency of ∼20 ms measured between headset streaming and policy output and a video-based optical flow latency of 65 ms (median) across different motion types. We validate the system on a Unitree G1 humanoid, showcasing split-second reactive responses such as catching a tossed ball, as well as coordinated loco-manipulation capabilities. Project page: https://xiaohu-art.github.io/Track/.
Figures & tables
Fig. 1: Real-world teleoperation with HuMiT. By removing optimization-based retargeting and future-frame conditions, HuMiT supports a wide range of dynamic whole-body skills with a compact-reference tracking policy: (a) Accurate whole-body tracking of operator motion, including front kick, side kick and boxing; (b) reactive ball catching enabled by low end-to-end latency; (c) carrying a water jug across the workspace; (d) pushing a wheeled chair while walking; (e) walking up to a chair, turning around and sitting down; (f) reaching overhead to hang a coat.
Fig. 2: Method overview. Human motion passes through scale calibration to produce a compact reference target ( gcpt ). Stages required by traditional pipelines (inverse kinematics, future-frame buffering) are eliminated. During training, the Actor π receives only the compact reference gcpt and proprioception sprop , and outputs actions at . The Critic additionally observes privileged states spriv and full retargeted references gfull , along with dense reward rt , enabling accurate value estimation. Both networks are updated jointly via PPO.
Reward Term
Error Term
Weight
Anchor position
xt,rootr−xt,root
1.0
Anchor orientation
Rt,rootr⊖Rt,root
1.0
Body position
pt,jr−pt,j
2.0
Body orientation
Rt,jr⊖Rt,j
2.0
Body linear velocity
p˙t,jr−p˙t,j
2.0
Body angular velocity
ωt,jr−ωt,j
2.0
TABLE I: Reward Terms. Rotations are compared using the axis–angle residual ( ⊖ ).
Method
Buffering (ms)
Online retarget.
End-to-end (ms)
SONIC [ 4 ]
200
×
245
SONIC-LL
80
×
145
TWIST2 [ 13 ]
0
✓
120
HuMiT (ours)
0
×
65
TABLE II: End-to-end teleoperation latency, measured with the external optical-flow protocol.
Fig. 3: End-to-end latency across motion types. For each motion, the flow-overlay frames of the operator (blue) and the robot (orange) are shown above. White boxes give per-pair delays in ms.
AMASS
LAFAN1
BONES-SEED
Method
Ekp↓
Ejpe↓
Elin↓
Eang↓
SR ↑
Ekp↓
Ejpe↓
Elin↓
Eang↓
SR ↑
Ekp↓
Ejpe↓
Elin↓
Eang↓
SR ↑
SONIC [ 4 ]
316.0
0.116
0.141
0.312
79.8
1492.6
0.097
0.221
0.394
20.0
143.1
0.098
0.112
0.295
98.2
SONIC-LL
400.5
0.113
0.177
0.397
59.5
1524.5
0.110
0.245
0.537
15.0
229.9
0.098
0.136
0.384
96.5
ScaleBFM-M [ 28 ]
73.2
0.123
0.145
0.444
92.6
82.1
0.102
0.226
0.699
27.5
60.2
0.079
0.136
0.435
97.7
ScaleBFM-XL [ 28 ]
71.8
0.125
0.137
0.407
94.0
82.8
0.097
0.224
0.621
32.5
58.5
0.081
0.128
0.403
98.7
ExtremControl [ 20 ]
329.5
0.246
0.259
0.675
60.1
191.0
0.203
0.244
0.781
0.0
220.2
0.138
0.203
0.638
66.3
TABLE III: Whole-body motion tracking evaluation. Ekp (mm), Ejpe (rad), Elin (m/s), Eang (rad/s), SR (%). A compact reference target, without joint-level targets, dense per-link states, or future frames, is sufficient as the deployment-time input of a whole-body tracking policy with competitive fidelity.
Task
Succ./trials
Dominant failure mode
Reactive catching
2 / 10
Arm workspace differs from operator’s
Carrying a water jug
9 / 10
Jug tips out of the grasp
Pushing a wheeled chair
5 / 10
Chair heading hard to steer
Sitting on a chair
7 / 10
Chair out of view after turning
Hanging a coat
6 / 10
Hanger orientation uncontrolled
TABLE IV: Real-world teleoperation reported as successful trials over total trials with a single operator. SONIC-LL and TWIST2 never catch the ball (0/10).
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
Humanoid whole-body teleoperation translates human motion into stable robot behavior in real time. Existing systems typically rely on online motion retargeting to bridge human--robot morphological differences, but this process adds latency and can produce physically infeasible targets. Meanwhile, diverse, noisy, and partial human-motion observations often fall outside the training distribution, potentially causing unstable robot behavior. We propose a retargeting-free policy that maps raw human motion directly to robot joint commands in a single forward pass, eliminating online kinematic adaptation. To improve robustness, we learn a codebook of full-body motion primitives that projects out-of-distribution observations onto plausible motion prototypes and recovers full-body motion from partial inputs. Experiments on a Unitree~G1 in simulation and on hardware, using virtual reality, optical mocap, text-to-motion generation, and monocular video inputs, show that our method outperforms baselines in latency and robustness.
Xiayan Xu, Jiyu Yu, Xingzhou Chen +5
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China +1
Stable, low-latency whole-body teleoperation of humanoid robots is an open research challenge, complicated by kinematic mismatches between human and robot morphologies, accumulated inertial sensor noise, non-trivial control latency, and persistent sim-to-real transfer gaps. This paper presents a complete real-time whole-body teleoperation system that maps human motion, recorded with a Virdyn IMU-based full-body motion capture suit, directly onto a Unitree G1 humanoid robot. We introduce a custom motion-processing, kinematic retargeting, and control pipeline engineered for continuous, low-latency operation without any offline buffering or learning-based components. The system is first validated in simulation using the MuJoCo physics model of the Unitree G1 (sim2sim), and then deployed without modification on the physical platform (sim2real). Experimental results demonstrate stable, synchronized reproduction of a broad motion repertoire, including walking, standing, sitting, turning, bowing, and coordinated expressive full-body gestures. This work establishes a practical, scalable framework for whole-body humanoid teleoperation using commodity wearable motion capture hardware.
Building a low-latency humanoid teleoperation system is essential for collecting diverse reactive and dynamic demonstrations. However, end-to-end latencies in existing approaches consistently hover around 200ms, roughly equal to human reaction time, that severely limits responsiveness and prevents tasks requiring rapid feedback. We identify position-only PD control as the root cause of this latency floor and propose ExtremControl, a low-latency whole-body control framework that: (1) incorporates scaled velocity feedforward in low-level control, reducing tracking delay by approximately 70ms with a provably optimal feedforward ratio; (2) operates directly on SE(3) poses of selected rigid links, primarily humanoid extremities, to avoid joint-space retargeting in the control loop. To minimize retargeting time, we derive a closed-form Cartesian-space mapping that uses a single anthropomorphic calibration step to translate human motion into humanoid link targets, accommodating operators with varying body proportions. Furthermore, we provide a theoretical framework for selecting control hyperparameters, and validate the effectiveness of velocity feedforward control in the reinforcement learning for locomotion setting through experiments in both simulation and the real world. Building on ExtremControl, we implement a teleoperation system supporting both optical motion capture and VR-based tracking, achieving end-to-end latency as low as 50ms --- well beneath the threshold at which system delay adds a full human reaction cycle to the control loop. This enables a qualitatively new class of reactive behaviors, including ping-pong ball balancing, juggling, and real-time return, that are beyond the capability of existing teleoperation systems.
Ziyan Xiong, Lixing Fang, Junyun Huang +3
UMass Amherst · Carnegie Mellon University · MIT-IBM Watson AI Lab