ExtremControl: Low-Latency Humanoid Teleoperation with Scaled Velocity Feedforward

Ziyan Xiong; Lixing Fang; Junyun Huang; Kashu Yamazaki; Hao Zhang; Chuang Gan

cs.ROFeb 11, 2026

ExtremControl: Low-Latency Humanoid Teleoperation with Scaled Velocity Feedforward

Authors: Ziyan Xiong, Lixing Fang, Junyun Huang, Kashu Yamazaki, Hao Zhang, Chuang Gan

Organizations: UMass Amherst · Carnegie Mellon University · MIT-IBM Watson AI Lab

Abstract

Building a low-latency humanoid teleoperation system is essential for collecting diverse reactive and dynamic demonstrations. However, end-to-end latencies in existing approaches consistently hover around 200ms, roughly equal to human reaction time, that severely limits responsiveness and prevents tasks requiring rapid feedback. We identify position-only PD control as the root cause of this latency floor and propose ExtremControl, a low-latency whole-body control framework that: (1) incorporates scaled velocity feedforward in low-level control, reducing tracking delay by approximately 70ms with a provably optimal feedforward ratio; (2) operates directly on SE(3) poses of selected rigid links, primarily humanoid extremities, to avoid joint-space retargeting in the control loop. To minimize retargeting time, we derive a closed-form Cartesian-space mapping that uses a single anthropomorphic calibration step to translate human motion into humanoid link targets, accommodating operators with varying body proportions. Furthermore, we provide a theoretical framework for selecting control hyperparameters, and validate the effectiveness of velocity feedforward control in the reinforcement learning for locomotion setting through experiments in both simulation and the real world. Building on ExtremControl, we implement a teleoperation system supporting both optical motion capture and VR-based tracking, achieving end-to-end latency as low as 50ms --- well beneath the threshold at which system delay adds a full human reaction cycle to the control loop. This enables a qualitatively new class of reactive behaviors, including ping-pong ball balancing, juggling, and real-time return, that are beyond the capability of existing teleoperation systems.

Figures & tables

Teleoperation System	Control Interface	Wrist Control	Foot Control	Task-Space Retargeting	End-to-End Latency
HOMIE [ 4 ]	Exoskeleton	$\checkmark$	$\times$	$\checkmark$	$\sim 454$ ms
HumanPlus [ 14 ]	RGB Camera	$\checkmark$	$\checkmark$	$\times$	$\sim 340$ ms
OmniH2O [ 19 ]	VR	$\checkmark$	$\times$	$\checkmark$	$\sim$ 185 ms
H2O [ 20 ]	RGB Camera	$\times$	$\checkmark$	$\times$	$\sim 373$ ms
AMO [ 28 ]	VR	$\checkmark$	$\times$	$\times$	$\sim 380$ ms
CLONE [ 31 ]	VR	$\checkmark$	$\times$	$\checkmark$	$\sim$ 178 ms

Table 1: Existing humanoid teleoperation systems with latencies estimated by optical flow on released videos.

Figure 2: Tracking objectives for humans and humanoids under VR and MoCap settings.

Figure 3: Optical flow visualization of hand movements. Upward and downward optical flow extracted from video clips are visualized in the two rightmost images. Separate tracking regions are defined for the human and robot hands to isolate their respective motion. The average optical flow magnitude within each region serves as a proxy for hand velocity, as it is approximately proportional to the hand’s image-plane speed.

Figure 4: Human and robot optical flow trajectories , along with the corresponding system latency estimates derived from selected video clips of exsisting systems.

Figure 5: Analysis of control characteristics under different velocity feedforward ratios. We train motion tracking and teleoperation policies with varying velocity feedforward ratios and evaluate their control behavior across six metrics.

Figure 6: Policy tracking performance with respect to different velocity feedforward ratio. Each dot represents a policy.

Figure 7: Different tracking delays with respect to velocity feedforward ratios.

Alignment	GMR	Ours
$R_{x}$	0.78 $\pm$ 0.21	0.91 $\pm$ 0.11
$R_{y}$	0.62 $\pm$ 0.24	0.76 $\pm$ 0.19
$R_{z}$	0.20 $\pm$ 0.26	0.41 $\pm$ 0.15

Table 2: Pearson correlation between hand trajectories before and after retargeting , averaged across our collected MoCap dataset and ACCAD dataset, weighted by the number of samples in each dataset.

Observation	${\mathbf{T}^{r}}$	$({\mathbf{T}^{r}},{q^{*}})$	${q^{*}}$
Tracking Pos Error (mm)	49 $\pm$ 7	43 $\pm$ 8	105 $\pm$ 23
Tracking Rot Error (rad)	0.19 $\pm$ 0.03	0.17 $\pm$ 0.02	0.48 $\pm$ 0.07

Table 3: Policy tracking errors in simulation,

\eta=0.9

Latency	Ping-pong Return $\uparrow$	Ping-pong Balance $\uparrow$	Cube Stack $\downarrow$
$\sim 50$ ms	80%	7.3 $\pm$ 3.1 s	7.6 $\pm$ 2.0 s
$\sim 100$ ms	60%	3.6 $\pm$ 1.7 s	8.3 $\pm$ 2.7 s
$\sim 150$ ms	30%	1.3 $\pm$ 0.4 s	9.5 $\pm$ 2.1 s
$\sim 200$ ms	20%	1.5 $\pm$ 0.6 s	10.1 $\pm 3.8$ s

Table 4: Performance of ExtremControl over 10 trials per task.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Teleoperation system setup. Solid arrows denote wired connections and dashed arrows denote wireless connections.

Component	MoCap (ms)	VR (ms)
Pose streaming (average sample age)	4	8
Wireless transmission	–	8
Inter-desktop communication	0.4	0.4
Direct extremity mapping	0.3	0.3
Policy inference	0.06	0.06
Robot communication	$<$ 1	$<$ 1

Appendix

Table 5: System-level latency breakdown. Closed-loop motion tracking is measured in simulation with

\eta=0.9

and includes the average 10 ms wait induced by the 50 Hz policy update.

Figure 9: Whole-body impedance calibration of Unitree G1 elbow joint. Solid lines correspond to proportional gains $k_{p}$ , dashed lines depict the effective impedance.

Figure 10: Measured tracking delay in simulation and real world as a function of velocity feedforward ratio. Dashed line represents the theoretical value $\frac{2(1-\eta)}{\omega_{n}}$ .

Figure 11: Experimental setup for latency validation, with two cameras and an optical motion capture system simultaneously recording.

Figure 12: Normalized optical-flow projections and the corresponding position trajectories of the human and robot are shown on the right.

Future pelvis translation	$\mathbf{p}^{r}_{\mathrm{pelvis,t:t+H}}-\mathbf{p}^{r}_{\mathrm{pelvis,t-1}}$
Future pelvis rotation	$\mathbf{r}^{r,\top}_{\mathrm{pelvis,t:t+H}}\cdot\mathbf{r}^{r}_{\mathrm{pelvis,t-1}}$
Link poses	$\mathbf{T}^{r^{\prime}}_{t:t+H}$
Link velocities (privilege)	$\mathbf{V}^{r}_{t:t+H}$
Retargeted joint configuration (privilege)	$q_{t},\dot{q}_{t}$
Foot contact probability	$\mathbb{P}_{\mathrm{k\_foot}}$

Appendix

Table 17

Last action	$a_{t-1}$
Joint configuration	$q,\dot{q}$
Pelvis rotation	$\mathbf{r}_{\mathrm{pelvis}}$
Pelvis linear velocity (privilege)	$\dot{\mathbf{p}}_{\mathrm{pelvis}}$
Pelvis angular velocity	$\dot{\mathbf{r}}_{\mathrm{pelvis}}$

Appendix

Table 18

Domain Randomization Parameters	—
Residual joint configuration	$\Delta q,\Delta\dot{q}$
Residual link poses	$\Delta\mathbf{T}^{r}$
Foot contact force	$\mathbb{F}^{\mathrm{target}}_{\mathrm{k\_foot}},\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}$

Appendix

Table 19

Reward Term	Expression	Scale
Global tracking link poses	$-\|\|(\mathbf{U}^{r})^{-1}\mathbf{T}^{r}\|\|_{2}^{2}$	30
Local tracking link poses	$-\|\|(\mathbf{U}^{r^{\prime}})^{-1}\mathbf{T}^{r^{\prime}}\|\|_{2}^{2}$	20
Retargeted DoF position	$-\|\|q_{t}-q\|\|_{2}^{2}$	3
Retargeted DoF velocity	$-\|\|\dot{q}_{t}-\dot{q}\|\|_{2}^{2}$	0.02
Foot contact reward	$-\left(\mathbb{F}^{\mathrm{target}}_{\mathrm{k\_foot}}-\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}\right)^{2}$	3
Foot contact penalty	$-\left(\mathbb{I}[\mathbb{P_{\mathrm{k\_foot}}}<0.2]\cdot\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}\right)^{2}$	10

Appendix

Table 20

$k_{p}$ ratio $\alpha_{k_{p}}$	$\mathcal{U}(0.8,1.2)$
$k_{d}$ ratio $\alpha_{k_{d}}$	$\mathcal{U}(0.8,1.2)$
Motor strength $\alpha_{\mathrm{strength}}$	$\mathcal{U}(0.8,1.2)$
Motor offset $\beta_{\mathrm{offset}}$	$\mathcal{U}(-0.1,0.1)$
Friction ratio	$\mathcal{U}(0.3,1.0)$
Torso added mass (kg)	$\mathcal{U}(-2,5)$
Torso CoM displacement (m)	$\mathcal{U}(-0.05,0.05)^{3}$

Appendix

Table 21

$\gamma$	0.99
$\lambda_{\mathrm{GAE}}$	0.95
Entropy coefficient	0.003
Value loss coefficient	1.0
Rollout length	24
Optimizer	ADAM
$\#$ epoch	5

Appendix

Table 22

learning rate $\eta$	0.0003
Batch size	256
Rollout length	24
Optimizer	ADAM
$\#$ epoch	10
$\#$ iteration	1500

Appendix

Table 23

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking
Oct 21, 2026Liu Cao, Botian Xu, Ruoqu Chen +5Robot Teleoperation Humanoid Robot Control
Tsinghua IIIS · The Chinese University of Hong Kong · Renmin University of China +2
Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives
Oct 6, 2026Xiayan Xu, Jiyu Yu, Xingzhou Chen +5Robot Teleoperation Humanoid Robot Control
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China +1
AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction
Jul 16, 2026Seok Joon Kim, Junho Lee, Federica Spinola +2Robot Teleoperation Intent Classification
Georgia Institute of Technology. · Neuromeka Ltd. · INRIA. +1

Oct 21, 2026J/K move · Enter open · S save

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Liu Cao, Botian Xu, Ruoqu Chen, Jiajun Liu+4

Tsinghua IIIS · The Chinese University of Hong Kong · Renmin University of China+2

Robot Teleoperation Humanoid Robot Control Humanoid Motion Tracking Whole-Body Robotic Manipulation

Open paper AskPDF arXiv

Real-time whole-body humanoid teleoperation remains challenging due to the computational latency introduced by optimization-based retargeting pipelines and future-frame conditioning. Existing motion tracking policies typically require dense reference targets including full joint positions, body-part transformations, and even future frames, which tightly couples the controller to expensive upstream modules. In this work, we present HuMiT, a whole-body teleoperation system built on a unified tracking policy that requires only a compact reference target, consisting of root height, root velocity, and the SE(3) poses of four sparse keypoints at the current frame, yet achieves competitive tracking fidelity compared to methods relying on more diverse reference states including complete joint positions, dense body-part transformations, and even future frames. Experiments show that our policy achieves a communication-to-inference latency of

\sim

20 ms measured between headset streaming and policy output and a video-based optical flow latency of 65 ms (median) across different motion types. We validate the system on a Unitree G1 humanoid, showcasing split-second reactive responses such as catching a tossed ball, as well as coordinated loco-manipulation capabilities. Project page: https://xiaohu-art.github.io/Track/.

Abstract

Robot Teleoperation

Explore similar work

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives

AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction