Enabling humanoid robots to perform agile and adaptive interactions has long been a core challenge in robotics. Current approaches are bottlenecked by either the scarcity of realistic interaction data or the need for meticulous, task-specific reward engineering, which limits their scalability. To narrow this gap, we present HumanX, a unified framework that compiles human video into generalizable, real-world interaction skills for humanoids, without per-task design. HumanX integrates two co-designed components: XGen, an anchor-centric data generation pipeline that yields physically plausible robot interaction data from monocular video and augments it to create a diverse, scalable training dataset; and XMimic, an interaction-prioritized learning framework to achieve generalizable interaction skills from seconds of demonstration. Evaluated across five distinct domains--basketball, football, badminton, cargo pickup, and reactive fighting--HumanX successfully acquires 9 different skills and transfers them zero-shot to a physical Unitree G1 humanoid. The learned capabilities include complex maneuvers such as pump-fake turnaround fadeaway jumpshots without any external perception, as well as interactive tasks like sustained human-robot passing sequences over 10 consecutive cycles--learned from a single video demonstration. Our experiments show that HumanX achieves higher generalization success than prior methods, demonstrating a scalable pathway within a class of anchorable interactions for learning versatile, real-world robot interactive skills.
Figures & tables
Figure 2: Overview of XGen . The pipeline begins by estimating SMPL‑based human motion from video and retargeting it to the humanoid’s morphology. The video is segmented into contact and non‑contact phases. For the contact phase, a body anchor (e.g., the midpoint between the two palms) is predefined. The object trajectory is then synthesized by transforming the object according to the anchor, followed by contact optimization to refine the robot poses. During the non‑contact phases, diverse object trajectories are generated via physical simulation. Complete interaction trajectories are obtained by concatenating and smoothly interpolating the phases.
Figure 2
Figure 5: XMimic is a unified framework for learning diverse interaction skills. To reinforce interaction and prevent overfitting to isolated body motions, it employs rewards that prioritize interaction and introduces interaction perturbations and interaction-vector termination during simulation. The training follows a two-stage pipeline: a privileged teacher policy first learns the skill using full state information, which then guides the training of a deployable student policy.
Method
Basketball Catch-Shot
Badminton Hitting
Cargo Pickup
SR ↑
GSR ↑
Eo↓ (m)
Eh↓ (cm)
SR ↑
GSR ↑
Eo↓ (m)
Eh↓ (m)
SR ↑
GSR ↑
Eo↓ (cm)
Eh↓ (cm)
SkillMimic [ 59 ]
0.0%
0.0%
1.25 ( ± 0.26)
10.3 ( ± 1.37)
68.2%
30.9%
2.94 ( ± 0.91)
0.48 ( ± 0.13)
0.4%
0.0%
25.1 ( ± 0.06)
28.3 ( ± 2.13)
OmniRetarget [ 68 ]
0.0%
0.0%
1.18 ( ± 0.15)
7.83 ( ± 0.95)
75.8%
20.4%
1.25 ( ± 1.15)
0.15 ( ± 0.10)
0.0%
0.0%
10.8 ( ± 0.00)
13.3 ( ± 1.57)
HDMI [ 61 ]
53.1%
2.4%
0.70 ( ± 0.11)
10.8 ( ± 2.41)
90.0%
25.3%
1.64 ( ± 0.66)
0.07 ( ± 0.02)
95.8%
1.8%
1.79 ( ± 0.22)
13.1 ( ± 1.28)
XMimic (Base)
93.4%
4.9%
0.69 ( ± 0.14)
6.36 ( ± 1.51)
97.1%
41.6%
1.32 ( ± 0.44)
0.12 ( ± 0.07)
100%
50.9%
1.56 ( ± 0.30)
6.58 ( ± 0.76)
+ IP
86.8%
13.5%
0.70 ( ± 0.08)
7.46 ( ± 0.46)
94.0%
60.5%
1.03 ( ± 0.57)
0.10 ( ± 0.05)
100%
94.5%
1.62 ( ± 0.03)
10.5 ( ± 1.23)
Table 1: Main Simulation Results . SR, Eo , and Eh measure the success rate on the original data, the object position tracking error, and the key-body position tracking error, respectively, while GSR measures the success rate of skill generalization.
Figure 5
Figure 8: Real Robot Experiment on Blind Basketball Skills . The policy relies solely on proprioception to control objects, achieving diverse, highly dynamic, and complex interactions.
Figure 9: Real Robot Experiment on MoCap-based Interaction Skills . When utilizing MoCap system to perceive object or human motion, our method enables sustained interaction, demonstrating high precision, agility, robustness, and generalization capability. Notably, each task shown here is learned from a single demonstration video without any task-specific reward.
Basketball Catch-Pass (MoCap)
41 / 50
Reactive Fighting (MoCap)
37 / 50
Jumpshot (NEP)
8 / 10
Cargo Pickup (MoCap)
43 / 50
Dribble (NEP)
8 / 10
Basketball Pickup (NEP)
10 / 10
Football Kicking (MoCap)
42 / 50
Pump-fake (NEP)
9 / 10
Layup (NEP)
7 / 10
Table 2: Quantitative Results on Real Robot Experiments .
Figure 9
Method
Football Kicking
Badminton Hitting
w/o Tea-Stu
74.2% GSR
52.4% GSR
w/ Tea-Stu
93.1% GSR
84.3% GSR
Table 3: Evaluation on Multi-Pattern Interaction Skills in Simulation . Each skill contains three distinct interaction patterns from three human video.
Figure 12: Diverse Skill Patterns . XMimic supports learning multiple interaction patterns for a single skill, allowing the policy to autonomously select the most suitable pattern in response to object state. ( Left ): football‑kicking patterns. ( Right ): badminton‑hitting patterns.
Humanoid robots have the potential to perform dexterous manipulation in human environments, yet acquiring diverse and generalizable skills remains costly due to expensive hardware data collection and labor-intensive annotation. Recent advances in video generative models provide a promising opportunity to synthesize rich manipulation experiences from visual observations, but transferring such imagined behaviors into executable whole-body humanoid skills remains largely unexplored. In this work, we present RoboReact, a framework that automatically synthesizes whole-body humanoid manipulation skills from a single egocentric RGB-D observation. RoboReact generates human manipulation videos, extracts geometry-preserving interaction keyframes through depth-aware 3D reconstruction, and retargets them to high-DoF humanoid platforms while preserving hand-object interaction geometry. To bridge the gap between imagined plans and physical execution, RoboReact performs online object-centric re-grounding and leverages a vision-language model-guided refinement loop to adapt skills under geometric mismatch and execution deviations. The refined skills are executed through a whole-body controller, enabling coordinated whole-body manipulation and dexterous interaction. Experiments on real humanoid robots demonstrate that RoboReact generalizes across diverse object configurations and robustly recovers from execution disturbances without requiring teleoperation or human demonstrations. These results highlight the potential of combining generative models, vision-language reasoning, and closed-loop control for scalable humanoid skill acquisition.
Shuliang He, Shuai Wang, Bo Yue +3
The Chinese University of Hong Kong, Shenzhen · JD Technology · Tsinghua University
The human-like morphology of humanoid robots grants them exceptional potential for agile and versatile motor capabilities, but it also introduces significant challenges in acquiring complex skills. Traditional Learning-from-Demonstrations methods are often constrained by the high cost of collecting real-world data, the difficulty of capturing motion-specific behaviors, and the limited diversity of demonstrations across individuals. Moreover, even for the same task, humans may execute the motion in multiple distinct ways. In this paper, we propose a new framework that leverages the power of Generative AI to convert textual prompts into realistic and diverse sequences of human body movements, enabling the robot to observe multiple variations of how a single task can be performed. These synthetic demonstrations are then used as a training resource, allowing the robot to learn a broad range of task-execution styles without requiring direct human intervention. We evaluate the proposed method across four simulation scenarios. Experimental results show that the robot not only completes the tasks successfully but also demonstrates strong adaptability to complex variations in motion.
Yun-Hao Tsai, Cong-Thanh Vu, Yen-Chen Liu
Department of Mechanical Engineering, National Cheng Kung University (NCKU), Tainan 70101, Taiwan
A critical bottleneck hindering further advancement in embodied AI and robotics is the challenge of scaling robot data. To address this, the field of learning robot manipulation skills from human video data has attracted rapidly growing attention in recent years, driven by the abundance of human activity videos and advances in computer vision. This line of research promises to enable robots to acquire skills passively from the vast and readily available resource of human demonstrations, substantially favoring scalable learning for generalist robotic systems. Therefore, we present this survey to provide a comprehensive and up-to-date review of human-video-based learning techniques in robotics, focusing on both human-robot skill transfer and data foundations. We first review the policy learning foundations in robotics, and then describe the fundamental interfaces to incorporate human videos. Subsequently, we introduce a hierarchical taxonomy of transferring human videos to robot skills, covering task-, observation-, and action-oriented pathways, along with a cross-family analysis of their couplings with different data configurations and learning paradigms. In addition, we investigate the data foundations including widely-used human video datasets and video generation schemes, and provide large-scale statistical trends in dataset development and utilization. Ultimately, we emphasize the challenges and limitations intrinsic to this field, and delineate potential avenues for future research. The paper list of our survey is available at https://github.com/IRMVLab/awesome-robot-learning-from-human-videos.
Junyi Ma, Erhang Zhang, Haoran Yang +4
Shanghai Jiao Tong University, China · University of Cambridge, UK