cs.ROFeb 11, 2026

ExtremControl: Low-Latency Humanoid Teleoperation with Scaled Velocity Feedforward

Authors: Ziyan Xiong, Lixing Fang, Junyun Huang, Kashu Yamazaki, Hao Zhang, Chuang Gan

Organizations: UMass Amherst · Carnegie Mellon University · MIT-IBM Watson AI Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Building a low-latency humanoid teleoperation system is essential for collecting diverse reactive and dynamic demonstrations. However, end-to-end latencies in existing approaches consistently hover around 200ms, roughly equal to human reaction time, that severely limits responsiveness and prevents tasks requiring rapid feedback. We identify position-only PD control as the root cause of this latency floor and propose ExtremControl, a low-latency whole-body control framework that: (1) incorporates scaled velocity feedforward in low-level control, reducing tracking delay by approximately 70ms with a provably optimal feedforward ratio; (2) operates directly on SE(3) poses of selected rigid links, primarily humanoid extremities, to avoid joint-space retargeting in the control loop. To minimize retargeting time, we derive a closed-form Cartesian-space mapping that uses a single anthropomorphic calibration step to translate human motion into humanoid link targets, accommodating operators with varying body proportions. Furthermore, we provide a theoretical framework for selecting control hyperparameters, and validate the effectiveness of velocity feedforward control in the reinforcement learning for locomotion setting through experiments in both simulation and the real world. Building on ExtremControl, we implement a teleoperation system supporting both optical motion capture and VR-based tracking, achieving end-to-end latency as low as 50ms --- well beneath the threshold at which system delay adds a full human reaction cycle to the control loop. This enables a qualitatively new class of reactive behaviors, including ping-pong ball balancing, juggling, and real-time return, that are beyond the capability of existing teleoperation systems.

Figures & tables

Teleoperation SystemControl InterfaceWrist ControlFoot ControlTask-Space RetargetingEnd-to-End Latency
HOMIE [ 4 ]Exoskeleton✓\checkmark×\times✓\checkmark∼454\sim 454 ms
HumanPlus [ 14 ]RGB Camera✓\checkmark✓\checkmark×\times∼340\sim 340 ms
OmniH2O [ 19 ]VR✓\checkmark×\times✓\checkmark∼\sim 185 ms
H2O [ 20 ]RGB Camera×\times✓\checkmark×\times∼373\sim 373 ms
AMO [ 28 ]VR✓\checkmark×\times×\times∼380\sim 380 ms
CLONE [ 31 ]VR✓\checkmark×\times✓\checkmark∼\sim 178 ms
Table 1: Existing humanoid teleoperation systems with latencies estimated by optical flow on released videos.
Figure 2: Tracking objectives for humans and humanoids under VR and MoCap settings.
Figure 3: Optical flow visualization of hand movements. Upward and downward optical flow extracted from video clips are visualized in the two rightmost images. Separate tracking regions are defined for the human and robot hands to isolate their respective motion. The average optical flow magnitude within each region serves as a proxy for hand velocity, as it is approximately proportional to the hand’s image-plane speed.
Figure 4: Human and robot optical flow trajectories , along with the corresponding system latency estimates derived from selected video clips of exsisting systems.
Figure 5: Analysis of control characteristics under different velocity feedforward ratios. We train motion tracking and teleoperation policies with varying velocity feedforward ratios and evaluate their control behavior across six metrics.
Figure 6: Policy tracking performance with respect to different velocity feedforward ratio. Each dot represents a policy.
Figure 7: Different tracking delays with respect to velocity feedforward ratios.
AlignmentGMROurs
RxR_{x}0.78 ±\pm 0.210.91 ±\pm 0.11
RyR_{y}0.62 ±\pm 0.240.76 ±\pm 0.19
RzR_{z}0.20 ±\pm 0.260.41 ±\pm 0.15
Table 2: Pearson correlation between hand trajectories before and after retargeting , averaged across our collected MoCap dataset and ACCAD dataset, weighted by the number of samples in each dataset.
ObservationTr{\mathbf{T}^{r}}(Tr,q∗)({\mathbf{T}^{r}},{q^{*}})q∗{q^{*}}
Tracking Pos Error (mm)49 ±\pm 743 ±\pm 8105 ±\pm 23
Tracking Rot Error (rad)0.19 ±\pm 0.030.17 ±\pm 0.020.48 ±\pm 0.07
Table 3: Policy tracking errors in simulation, η=0.9\eta=0.9 .
LatencyPing-pong Return ↑\uparrowPing-pong Balance ↑\uparrowCube Stack ↓\downarrow
∼50\sim 50 ms80%7.3 ±\pm 3.1 s7.6 ±\pm 2.0 s
∼100\sim 100 ms60%3.6 ±\pm 1.7 s8.3 ±\pm 2.7 s
∼150\sim 150 ms30%1.3 ±\pm 0.4 s9.5 ±\pm 2.1 s
∼200\sim 200 ms20%1.5 ±\pm 0.6 s10.1 ±3.8\pm 3.8 s
Table 4: Performance of ExtremControl over 10 trials per task.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Teleoperation system setup. Solid arrows denote wired connections and dashed arrows denote wireless connections.
ComponentMoCap (ms)VR (ms)
Pose streaming (average sample age)48
Wireless transmission–8
Inter-desktop communication0.40.4
Direct extremity mapping0.30.3
Policy inference0.060.06
Robot communication<< 1<< 1
Appendix
Table 5: System-level latency breakdown. Closed-loop motion tracking is measured in simulation with η=0.9\eta=0.9 and includes the average 10 ms wait induced by the 50 Hz policy update.
Figure 9: Whole-body impedance calibration of Unitree G1 elbow joint. Solid lines correspond to proportional gains kpk_{p} , dashed lines depict the effective impedance.
Figure 10: Measured tracking delay in simulation and real world as a function of velocity feedforward ratio. Dashed line represents the theoretical value 2(1−η)ωn\frac{2(1-\eta)}{\omega_{n}} .
Figure 11: Experimental setup for latency validation, with two cameras and an optical motion capture system simultaneously recording.
Figure 12: Normalized optical-flow projections and the corresponding position trajectories of the human and robot are shown on the right.
Future pelvis translationppelvis,t:t+Hr−ppelvis,t−1r\mathbf{p}^{r}_{\mathrm{pelvis,t:t+H}}-\mathbf{p}^{r}_{\mathrm{pelvis,t-1}}
Future pelvis rotationrpelvis,t:t+Hr,⊤⋅rpelvis,t−1r\mathbf{r}^{r,\top}_{\mathrm{pelvis,t:t+H}}\cdot\mathbf{r}^{r}_{\mathrm{pelvis,t-1}}
Link posesTt:t+Hr′\mathbf{T}^{r^{\prime}}_{t:t+H}
Link velocities (privilege)Vt:t+Hr\mathbf{V}^{r}_{t:t+H}
Retargeted joint configuration (privilege)qt,q˙tq_{t},\dot{q}_{t}
Foot contact probabilityPk_foot\mathbb{P}_{\mathrm{k\_foot}}
Appendix
Table 17
Last actionat−1a_{t-1}
Joint configurationq,q˙q,\dot{q}
Pelvis rotationrpelvis\mathbf{r}_{\mathrm{pelvis}}
Pelvis linear velocity (privilege)p˙pelvis\dot{\mathbf{p}}_{\mathrm{pelvis}}
Pelvis angular velocityr˙pelvis\dot{\mathbf{r}}_{\mathrm{pelvis}}
Appendix
Table 18
Domain Randomization Parameters—
Residual joint configurationΔq,Δq˙\Delta q,\Delta\dot{q}
Residual link posesΔTr\Delta\mathbf{T}^{r}
Foot contact forceFk_foottarget,Fk_footmeasure\mathbb{F}^{\mathrm{target}}_{\mathrm{k\_foot}},\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}
Appendix
Table 19
Reward TermExpressionScale
Global tracking link poses−∣∣(Ur)−1Tr∣∣22-||(\mathbf{U}^{r})^{-1}\mathbf{T}^{r}||_{2}^{2}30
Local tracking link poses−∣∣(Ur′)−1Tr′∣∣22-||(\mathbf{U}^{r^{\prime}})^{-1}\mathbf{T}^{r^{\prime}}||_{2}^{2}20
Retargeted DoF position−∣∣qt−q∣∣22-||q_{t}-q||_{2}^{2}3
Retargeted DoF velocity−∣∣q˙t−q˙∣∣22-||\dot{q}_{t}-\dot{q}||_{2}^{2}0.02
Foot contact reward−(Fk_foottarget−Fk_footmeasure)2-\left(\mathbb{F}^{\mathrm{target}}_{\mathrm{k\_foot}}-\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}\right)^{2}3
Foot contact penalty−(I[Pk_foot<0.2]⋅Fk_footmeasure)2-\left(\mathbb{I}[\mathbb{P_{\mathrm{k\_foot}}}<0.2]\cdot\mathbb{F}^{\mathrm{measure}}_{\mathrm{k\_foot}}\right)^{2}10
Appendix
Table 20
kpk_{p} ratio αkp\alpha_{k_{p}}U(0.8,1.2)\mathcal{U}(0.8,1.2)
kdk_{d} ratio αkd\alpha_{k_{d}}U(0.8,1.2)\mathcal{U}(0.8,1.2)
Motor strength αstrength\alpha_{\mathrm{strength}}U(0.8,1.2)\mathcal{U}(0.8,1.2)
Motor offset βoffset\beta_{\mathrm{offset}}U(−0.1,0.1)\mathcal{U}(-0.1,0.1)
Friction ratioU(0.3,1.0)\mathcal{U}(0.3,1.0)
Torso added mass (kg)U(−2,5)\mathcal{U}(-2,5)
Torso CoM displacement (m)U(−0.05,0.05)3\mathcal{U}(-0.05,0.05)^{3}
Appendix
Table 21
γ\gamma0.99
λGAE\lambda_{\mathrm{GAE}}0.95
Entropy coefficient0.003
Value loss coefficient1.0
Rollout length24
OptimizerADAM
#\# epoch5
Appendix
Table 22
learning rate η\eta0.0003
Batch size256
Rollout length24
OptimizerADAM
#\# epoch10
#\# iteration1500
Appendix
Table 23
More in this topic

Robot Teleoperation

Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.

See all 206 papers

Explore similar work

CardsList
Oct 21, 2026

HuMiT: Low-Latency Whole-Body Humanoid Teleoperation via Compact Reference Tracking

Real-time whole-body humanoid teleoperation remains challenging due to the computational latency introduced by optimization-based retargeting pipelines and future-frame conditioning. Existing motion tracking policies typically require dense reference targets including full joint positions, body-part transformations, and even future frames, which tightly couples the controller to expensive upstream modules. In this work, we present HuMiT, a whole-body teleoperation system built on a unified tracking policy that requires only a compact reference target, consisting of root height, root velocity, and the SE(3) poses of four sparse keypoints at the current frame, yet achieves competitive tracking fidelity compared to methods relying on more diverse reference states including complete joint positions, dense body-part transformations, and even future frames. Experiments show that our policy achieves a communication-to-inference latency of ∼\sim20 ms measured between headset streaming and policy output and a video-based optical flow latency of 65 ms (median) across different motion types. We validate the system on a Unitree G1 humanoid, showcasing split-second reactive responses such as catching a tossed ball, as well as coordinated loco-manipulation capabilities. Project page: https://xiaohu-art.github.io/Track/.

Liu Cao, Botian Xu, Ruoqu Chen +5

Tsinghua IIIS · The Chinese University of Hong Kong · Renmin University of China +2

Robot TeleoperationHumanoid Robot ControlHumanoid Motion TrackingWhole-Body Robotic Manipulation
AskPDF
Oct 6, 2026

Beyond Retargeting: Low-Latency and Robust Humanoid Whole-Body Teleoperation with Learned Atomic Motion Primitives

Humanoid whole-body teleoperation translates human motion into stable robot behavior in real time. Existing systems typically rely on online motion retargeting to bridge human--robot morphological differences, but this process adds latency and can produce physically infeasible targets. Meanwhile, diverse, noisy, and partial human-motion observations often fall outside the training distribution, potentially causing unstable robot behavior. We propose a retargeting-free policy that maps raw human motion directly to robot joint commands in a single forward pass, eliminating online kinematic adaptation. To improve robustness, we learn a codebook of full-body motion primitives that projects out-of-distribution observations onto plausible motion prototypes and recovers full-body motion from partial inputs. Experiments on a Unitree~G1 in simulation and on hardware, using virtual reality, optical mocap, text-to-motion generation, and monocular video inputs, show that our method outperforms baselines in latency and robustness.

Xiayan Xu, Jiyu Yu, Xingzhou Chen +5

The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China +1

Robot TeleoperationHumanoid Robot ControlMotion Representation Learning
AskPDF
Jul 16, 2026

AHEAD: Anticipatory Hand-Driven Teleoperation via Human Intent Prediction

Direct hand-driven teleoperation maps an operator's hand motion to robot end-effector commands at every frame, enabling precise control, but it requires constant monitoring and correction during approach, grasp, and placement, which can be slow and fatiguing. For repetitive pick-and-place tasks, supervisory (goal-based) teleoperation simplifies this process: the operator specifies goals/waypoints, and the robot executes the motion using planning algorithms. Yet, this introduces latency, as the robot must wait for the next command before it can plan and act. "How can we reduce robot reaction time while lowering operator workload?" To tackle this question, we present AHEAD, a real-time VR teleoperation system that anticipates operator intent to enable proactive, hand-driven control. In a digital twin, the operator performs pick-and-place naturally, using hand motion to convey high-level commands rather than a continuous robot trajectory. AHEAD processes a short window of 3D hand and head signals together with scene context through an attention-based classifier to predict the intended grasp object and placement slot. A state machine converts intent predictions into stable robot goals, enabling early motion while remaining stable under noisy predictions and corrective hand movements. AHEAD's intent prediction module achieves Top1 accuracy: 76% for grasp objects and 76% for target slots. Moreover, our user study shows AHEAD reduces robot reaction latency by 0.6 s (object) and 1.4 s (slot) relative to baselines. Participants also reported lower operator load, indicating faster robot responses while maintaining low operator effort in practice.

Seok Joon Kim, Junho Lee, Federica Spinola +2

Georgia Institute of Technology. · Neuromeka Ltd. · INRIA. +1

Robot TeleoperationIntent ClassificationRobotic Manipulation
AskPDF