Whole-body humanoid teleoperation enables humans to remotely control humanoid robots, serving as both a real-time operational tool and a scalable engine for collecting demonstrations for autonomous learning. Despite recent advances, existing systems are validated using aggregate metrics that conflate distinct motion regimes, masking critical failure modes. This lack of diagnostic granularity, compounded by tightly coupled and labor-intensive system configurations, hinders robust real-world deployment. A key open challenge is building a teleoperation system that is simultaneously robust, versatile, and affordable for practical use. Here we present OmniClone, a whole-body humanoid teleoperation system that achieves high-fidelity, multi-skill control on a single consumer GPU with modest data requirements. Central to our approach is OmniBench, a diagnostic benchmark that evaluates policies across stratified motion categories and difficulty levels on unseen motions, exposing the narrow specialization of prior systems. Guided by these diagnostics, we identify an optimized training data recipe and integrate system-level improvements: subject-agnostic retargeting and robust communication, that collectively reduce Mean Per-Joint Position Error (MPJPE) by over 66% while requiring orders-of-magnitude fewer computational resources than comparable methods. Crucially, OmniClone is control-source-agnostic: a single unified policy supports real-time teleoperation, generated motion playback, and Vision-Language-Action (VLA) models, while generalizing across operators of vastly different body proportions. By uniting diagnostic evaluation with practical engineering, OmniClone provides an accessible foundation for scalable humanoid teleoperation and autonomous learning.
Figures & tables
Fig. 2: sr comparison on OmniBench reveals narrow skill specialization of existing methods. While GMT and Twist2 suffer notable drops in lower-body agility tasks (deep squatting, low-altitude loco-manipulation) and high-dynamic maneuvers (jumping), OmniClone maintains near-perfect sr across all 18 stratified categories. Complementing the mpjpe radar map in Fig. 1 , these results confirm that prior systems’ aggregate scores mask significant per-category failures.
Fig. 3: Overview of the OmniClone framework, comprising model training (top) and system infrastructure (bottom). Top: a teacher–student distillation pipeline trained on an optimized data recipe, whose composition is iteratively refined via OmniBench diagnostics. Bottom: the deployment infrastructure features subject-agnostic retargeting and robust wireless communication, yielding a control-source-agnostic interface compatible with real-time teleoperation, generative motion models, and downstream vla planners such as GR00T N1.6.
TABLE I: Quantitative comparison with baselines and ablation on training data recipes, evaluated on OmniBench . All results are stratified by motion category and difficulty level, reporting sr (%) and mpjpe (mm). OmniClone achieves the best overall balance across all categories, while the ablation variants illustrate how data composition steers policy behavior. Best mpjpe per category is in bold.
Fig. 4: Ablations on training data recipes show how data composition steers policy behavior. OmniClone Dynamic excels at agile motions but collapses on manipulation and squatting; OmniClone Stable recovers stability at the cost of dynamic agility; OmniClone Balance improves breadth across both regimes. The final OmniClone recipe, obtained by filtering biased sequences from OmniClone Balance , achieves the best overall trade-off in both (a) sr and (b) mpjpe .
Fig. 5: OmniClone generalizes reliably across teleoperators of vastly different heights. Six participants ( 1.47 m– 1.94 m) each perform a composite loco-manipulation task (walk, stabilize, pick-and-place). Despite a 47 cm height span, the system maintains consistent stability and task success throughout, with all novice operators completing the task within 5 – 7 practice trials.
Fig. 6: Zero-shot playback of generated motions establishes the performance upper bound of the control policy. By replaying synthetic motions from HY-Motion-1.0 in the absence of system-induced noise (retargeting artifacts, network latency), we isolate the policy’s intrinsic tracking fidelity. Together with the sprint and long-jump results in the left panels of Fig. 1 , these results confirm that the end-to-end system’s residual errors are primarily driven by upstream infrastructure bottlenecks rather than policy capacity.
Fig. 7: A single unified policy handles the full spectrum of real-world teleoperation, from precise quasi-static manipulation to highly dynamic locomotion. (a) The humanoid is teleoperated to open a cabinet, retrieve a tissue, and place it on a table, exercising articulated object interaction and coordinated whole-body motion. (b) Stable liquid transport with fine-grained force control is demonstrated by serving tea without spillage. (c) High-speed running further validates the policy’s dynamic agility. All tasks are executed via real-time teleoperation without policy switching.
Fig. 8: Enabling Autonomy via GR00T N1.6 [ 28 ] Post-Training. We utilize high-fidelity data collected by our system to post-train the GR00T N 1.6 [ 28 ] foundation model, demonstrating superior full-body control capability.
Task
Trials
sr
Pick-and-Place
21
85.71%
Squat to Pick-and-Place
5
80.00%
TABLE II: Autonomous policy trained on OmniClone -collected demonstrations achieves high sr with only 50 demonstrations. We post-train GR00T N1.6 on full-body data collected via OmniClone and evaluate on two whole-body manipulation tasks. The results validate that OmniClone produces kinetically feasible, high-fidelity demonstrations suitable for effective downstream policy learning in low-data regimes.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Motion
Min P5
Max P95
Mean
Walk Slow
0.618
1.704
1.026
Walk Med
0.793
2.193
1.364
Walk Fast
0.665
2.279
1.584
Run Slow
0.667
2.883
1.932
Run Med
0.644
4.151
2.386
Run Fast
0.685
5.046
2.956
Appendix
TABLE A.1: Statistical overview of the generated evaluation corpus. For locomotion tasks, we report the mean velocity alongside the 5 th and 95 th percentiles (P 5 /P 95 ) to exclude transient phases such as startup and stopping transitions. For manipulation tasks, where metrics focus on root or hand height, we report the mean, min, and max values to illustrate the full extent of the reachable workspace. ( Med : Medium; Loco : Loco-manipulation; Manip : Manipulation.)
Category
Reward Term
Weight
Regularization
Action Rate Penalty
−8.0
Contact Air Time Penalty
−100.0
Joint Acceleration Penalty
−1.0×10−7
Joint Position Limits
−10.0
Velocity/Action Limits
−1.0
Tracking
Torso Global Pos. / Rot.
0.5
Appendix
TABLE A.2: Reward function configuration. Summary of reward components and their respective weights used for training. ( Pos. : Position; Rot. : Rotation; Lin. : Linear; Ang. : Angular; Vel. : Velocity. The end-effectors consist of the ‘wrist yaw link’ and ‘ankle roll link.’)
TABLE A.3: Domain randomization. Specifications of the primary randomization parameters and their respective ranges applied during training. ( CoM : Center of Mass; RFI : Random Force Injection; nominal : default values defined in the URDF. U(a,b) denotes a uniform distribution; δx,y,z for the CoM uses specific ranges per axis.)
Fig. A.1 : Autonomous whole-body pick-and-place tasks executed by GR00T N 1.6 trained on OmniClone -collected demonstrations. The policy successfully performs full-body pick-and-place from a standing pose, validating the data quality, the fidelity of the control policy, and the stability of the system infrastructure.
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
This paper investigates humanoid whole-body dexterous manipulation, where the efficient collection of high-quality demonstration data remains a central bottleneck. Existing teleoperation systems often suffer from limited portability, occlusion, or insufficient precision, which hinders their applicability to complex whole-body tasks. To address these challenges, we introduce HumDex, a portable teleoperation system designed for humanoid whole-body dexterous manipulation. Our system leverages IMU-based motion tracking to address the portability-precision trade-off, enabling accurate full-body tracking while remaining easy to deploy. For dexterous hand control, we further introduce a learning-based retargeting method that generates smooth and natural hand motions without manual parameter tuning. Beyond teleoperation, HumDex enables efficient collection of human motion data. Building on this capability, we propose a two-stage imitation learning framework that first pre-trains on diverse human motion data to learn generalizable priors, and then fine-tunes on robot data to bridge the embodiment gap for precise execution. We demonstrate that this approach significantly improves generalization to new configurations, objects, and backgrounds with minimal data acquisition costs. The entire system is fully reproducible and open-sourced at https://github.com/physical-superintelligence-lab/humdex.
Liang Heng, Yihe Tang, Jiajun Xu +3
USC Physical Superintelligence (PSI) Lab · WorldEngine AI
Humanoid whole-body teleoperation translates human motion into stable robot behavior in real time. Existing systems typically rely on online motion retargeting to bridge human--robot morphological differences, but this process adds latency and can produce physically infeasible targets. Meanwhile, diverse, noisy, and partial human-motion observations often fall outside the training distribution, potentially causing unstable robot behavior. We propose a retargeting-free policy that maps raw human motion directly to robot joint commands in a single forward pass, eliminating online kinematic adaptation. To improve robustness, we learn a codebook of full-body motion primitives that projects out-of-distribution observations onto plausible motion prototypes and recovers full-body motion from partial inputs. Experiments on a Unitree~G1 in simulation and on hardware, using virtual reality, optical mocap, text-to-motion generation, and monocular video inputs, show that our method outperforms baselines in latency and robustness.
Xiayan Xu, Jiyu Yu, Xingzhou Chen +5
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Zhejiang University, Hangzhou, China · Tencent Robotics X, Shenzhen, China +1
Whole-body teleoperation is essential for scalable robot data collection in loco-manipulation tasks, yet existing approaches relying on exoskeleton suits or multi-camera setups impose prohibitive cost, complexity, and environmental constraints. Recent methods using a single extended reality (XR) device with end-to-end reinforcement learning policies partially address these limitations but require robot-specific retraining, suffer from out-of-distribution failures, and rely on motion retargeting that neglects dynamic feasibility. We propose a hierarchical whole-body teleoperation framework driven by a single XR device that generalizes across diverse robot morphologies without retraining robot-specific policies. A Model Predictive Control (MPC)-based motion retargeter jointly optimizes alignment with the operator's intent and the robot's dynamic feasibility, generating optimal commands for existing low-level controllers. To ensure robust online execution, we introduce a state synchronization method that resets the simulator state at each MPC step to handle noisy real-world measurements and contact sensitivity, and integrate SLAM-based global pose feedback to mitigate long-term drift. Simulation results show higher success rates on whole-body control tasks for both a humanoid (over 30% lower completion time and 20% lower power consumption) and a mobile manipulator (zero collisions) compared to baselines. Real-world experiments further validate the effectiveness and flexibility of our method, demonstrating the successful deployment of the proposed retargeter on both platforms for whole-body control tasks and the ease of allowing users to adjust teleoperation behavior based on their preferences. This plug-and-play framework offers a scalable, morphology-agnostic solution for whole-body robot teleoperation, enabling real-time behavioral customization and broad applicability across platforms.
Jen-Wei Wang, Sarthak Kaingade, Andrea Tagliabue +1