HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Liu Cao; Botian Xu; Ruoqu Chen; Jiajun Liu; Mingzhi Pei; Zihao Wang; Zhongyu Li; Mengdi Xu

cs.ROOct 21, 2026

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Authors: Liu Cao, Botian Xu, Ruoqu Chen, Jiajun Liu, Mingzhi Pei, Zihao Wang, Zhongyu Li, Mengdi Xu

Organizations: Tsinghua IIIS · The Chinese University of Hong Kong · Renmin University of China · Tsinghua College AI ·

Abstract

Real-time whole-body humanoid teleoperation remains challenging due to the computational latency introduced by optimization-based retargeting pipelines and future-frame conditioning. Existing motion tracking policies typically require dense reference targets including full joint positions, body-part transformations, and even future frames, which tightly couples the controller to expensive upstream modules. In this work, we present HuMiT, a whole-body teleoperation system built on a unified tracking policy that requires only a compact reference target, consisting of root height, root velocity, and the SE(3) poses of four sparse keypoints at the current frame, yet achieves competitive tracking fidelity compared to methods relying on more diverse reference states including complete joint positions, dense body-part transformations, and even future frames. Experiments show that our policy achieves a communication-to-inference latency of $\sim$ 20 ms measured between headset streaming and policy output and a video-based optical flow latency of 65 ms (median) across different motion types. We validate the system on a Unitree G1 humanoid, showcasing split-second reactive responses such as catching a tossed ball, as well as coordinated loco-manipulation capabilities. Project page: https://xiaohu-art.github.io/Track/.

Figures & tables

Fig. 1: Real-world teleoperation with HuMiT. By removing optimization-based retargeting and future-frame conditions, HuMiT supports a wide range of dynamic whole-body skills with a compact-reference tracking policy: (a) Accurate whole-body tracking of operator motion, including front kick, side kick and boxing; (b) reactive ball catching enabled by low end-to-end latency; (c) carrying a water jug across the workspace; (d) pushing a wheeled chair while walking; (e) walking up to a chair, turning around and sitting down; (f) reaching overhead to hang a coat.

Fig. 2: Method overview. Human motion passes through scale calibration to produce a compact reference target ( $g_{cpt}$ ). Stages required by traditional pipelines (inverse kinematics, future-frame buffering) are eliminated. During training, the Actor $\pi$ receives only the compact reference $g_{cpt}$ and proprioception $s_{prop}$ , and outputs actions $a_{t}$ . The Critic additionally observes privileged states $s_{priv}$ and full retargeted references $g_{full}$ , along with dense reward $r_{t}$ , enabling accurate value estimation. Both networks are updated jointly via PPO.

Reward Term	Error Term	Weight
Anchor position	$\mathbf{x}^{r}_{t,root}-\mathbf{x}_{t,root}$	$1.0$
Anchor orientation	$\mathbf{R}^{r}_{t,root}\ominus\mathbf{R}_{t,root}$	$1.0$
Body position	$\mathbf{p}^{r}_{t,j}-\mathbf{p}_{t,j}$	$2.0$
Body orientation	$\mathbf{R}^{r}_{t,j}\ominus\mathbf{R}_{t,j}$	$2.0$
Body linear velocity	$\dot{\mathbf{p}}^{r}_{t,j}-\dot{\mathbf{p}}_{t,j}$	$2.0$
Body angular velocity	$\boldsymbol{\omega}^{r}_{t,j}-\boldsymbol{\omega}_{t,j}$	$2.0$

TABLE I: Reward Terms. Rotations are compared using the axis–angle residual (

\ominus

Method	Buffering (ms)	Online retarget.	End-to-end (ms)
SONIC [ 4 ]	200	$\times$	245
SONIC-LL	80	$\times$	145
TWIST2 [ 13 ]	0	$\checkmark$	120
HuMiT (ours)	0	$\times$	65

TABLE II: End-to-end teleoperation latency, measured with the external optical-flow protocol.

Fig. 3: End-to-end latency across motion types. For each motion, the flow-overlay frames of the operator (blue) and the robot (orange) are shown above. White boxes give per-pair delays in ms.

	AMASS					LAFAN1					BONES-SEED
Method	$E_{kp}\!\downarrow$	$E_{jpe}\!\downarrow$	$E_{lin}\!\downarrow$	$E_{ang}\!\downarrow$	SR $\uparrow$	$E_{kp}\!\downarrow$	$E_{jpe}\!\downarrow$	$E_{lin}\!\downarrow$	$E_{ang}\!\downarrow$	SR $\uparrow$	$E_{kp}\!\downarrow$	$E_{jpe}\!\downarrow$	$E_{lin}\!\downarrow$	$E_{ang}\!\downarrow$	SR $\uparrow$
SONIC [ 4 ]	316.0	0.116	0.141	0.312	79.8	1492.6	0.097	0.221	0.394	20.0	143.1	0.098	0.112	0.295	98.2
SONIC-LL	400.5	0.113	0.177	0.397	59.5	1524.5	0.110	0.245	0.537	15.0	229.9	0.098	0.136	0.384	96.5
ScaleBFM-M [ 28 ]	73.2	0.123	0.145	0.444	92.6	82.1	0.102	0.226	0.699	27.5	60.2	0.079	0.136	0.435	97.7
ScaleBFM-XL [ 28 ]	71.8	0.125	0.137	0.407	94.0	82.8	0.097	0.224	0.621	32.5	58.5	0.081	0.128	0.403	98.7
ExtremControl [ 20 ]	329.5	0.246	0.259	0.675	60.1	191.0	0.203	0.244	0.781	0.0	220.2	0.138	0.203	0.638	66.3

TABLE III: Whole-body motion tracking evaluation.

E_{kp}

(mm),

E_{jpe}

(rad),

E_{lin}

(m/s),

E_{ang}

(rad/s), SR (%). A compact reference target, without joint-level targets, dense per-link states, or future frames, is sufficient as the deployment-time input of a whole-body tracking policy with competitive fidelity.

Task	Succ./trials	Dominant failure mode
Reactive catching	2 / 10	Arm workspace differs from operator’s
Carrying a water jug	9 / 10	Jug tips out of the grasp
Pushing a wheeled chair	5 / 10	Chair heading hard to steer
Sitting on a chair	7 / 10	Chair out of view after turning
Hanging a coat	6 / 10	Hanger orientation uncontrolled

TABLE IV: Real-world teleoperation reported as successful trials over total trials with a single operator. SONIC-LL and TWIST2 never catch the ball (0/10).

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives
Oct 6, 2026Xiayan Xu, Jiyu Yu, Xingzhou Chen +5Robot Teleoperation Humanoid Robot Control
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China +1
Real-Time Whole-Body Teleoperation of a Humanoid Robot Using IMU-Based Motion Capture with Sim2Sim and Sim2Real Validation
May 12, 2026Hamza Ahmed Durrani, Suleman KhanRobot Teleoperation Humanoid Robot Control
WeGo Robotics
ExtremControl: Low-Latency Humanoid Teleoperation with Scaled Velocity Feedforward
Feb 11, 2026Ziyan Xiong, Lixing Fang, Junyun Huang +3Robot Teleoperation Humanoid Robot Control
UMass Amherst · Carnegie Mellon University · MIT-IBM Watson AI Lab

Oct 6, 2026J/K move · Enter open · S save

Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives

Xiayan Xu, Jiyu Yu, Xingzhou Chen, Siyi Qian+4

The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China+1

Robot Teleoperation Humanoid Robot Control Motion Representation Learning

Open paper AskPDF arXiv

Humanoid whole-body teleoperation translates human motion into stable robot behavior in real time. Existing systems typically rely on online motion retargeting to bridge human--robot morphological differences, but this process adds latency and can produce physically infeasible targets. Meanwhile, diverse, noisy, and partial human-motion observations often fall outside the training distribution, potentially causing unstable robot behavior. We propose a retargeting-free policy that maps raw human motion directly to robot joint commands in a single forward pass, eliminating online kinematic adaptation. To improve robustness, we learn a codebook of full-body motion primitives that projects out-of-distribution observations onto plausible motion prototypes and recovers full-body motion from partial inputs. Experiments on a Unitree~G1 in simulation and on hardware, using virtual reality, optical mocap, text-to-motion generation, and monocular video inputs, show that our method outperforms baselines in latency and robustness.

Abstract

Robot Teleoperation

Explore similar work

Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives

Real-Time Whole-Body Teleoperation of a Humanoid Robot Using IMU-Based Motion Capture with Sim2Sim and Sim2Real Validation

ExtremControl: Low-Latency Humanoid Teleoperation with Scaled Velocity Feedforward