Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis

Weisheng Xu; Jian Li; Yi Gu; Bin Yang; Haodong Chen; Shuyi Lin; Mingqian Zhou; Jing Tan

cs.ROMar 20, 2026

Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis

Authors: Weisheng Xu, Jian Li, Yi Gu, Bin Yang, Haodong Chen, Shuyi Lin, Mingqian Zhou, Jing Tan,

Organizations: Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology, Shenzhen · Shenzhen University · University of Cambridge

Abstract

Equipping humanoid robots with versatile interaction skills typically requires either extensive policy training or explicit human-to-robot motion retargeting. However, learning-based policies face prohibitive data collection costs. Meanwhile, retargeting relies on human-centric pose estimation (e.g., SMPL), introducing a morphology gap. Skeletal scale mismatches result in severe spatial misalignments when mapped to robots, compromising interaction success. In this work, we propose Dream2Act, a robot-centric framework enabling zero-shot interaction through generative video synthesis. Given a third-person image of the robot and target object, our framework leverages video generation models to envision the robot completing the task with morphology-consistent motion. We employ a high-fidelity pose extraction system to recover physically feasible, robot-native joint trajectories from these synthesized dreams, subsequently executed via a general-purpose whole-body controller. Operating strictly within the robot-native coordinate space, Dream2Act avoids retargeting errors and eliminates task-specific policy training. We evaluate Dream2Act on the Unitree G1 across four whole-body mobile interaction tasks: ball kicking, sofa sitting, bag punching, and box hugging. Dream2Act achieves a 37.5% overall success rate, compared to 0% for conventional retargeting. While retargeting fails to establish correct physical contacts due to the morphology gap (with errors compounded during locomotion), Dream2Act maintains robot-consistent spatial alignment, enabling reliable contact formation and substantially higher task completion.

Figures & tables

Figure 1: Dream2Act: Zero-shot Humanoid Interaction via Robot-Centric Video Synthesis. (A) Traditional human-centric retargeting fails due to the morphology gap, causing spatial misalignments. (B) Our Dream2Act pipeline leverages Seedance 2.0 to synthesize morphology-consistent interactions directly in the robot’s native space. (C) This visual prior translates into successful zero-shot physical execution for spatially-sensitive tasks like sitting on a sofa and kicking a ball.

Figure 2: Dream2Act System Architecture. The framework comprises three core modules: (1) Interaction Hallucination via Seedance to generate morphology-consistent visual priors; (2) Native Pose Estimation and 2D-to-3D Lifting to extract precise 3D keypoints; and (3) Morphology-Aware Kinematic Recovery employing Constrained IK and PnP scale recovery to trajectories for physical execution.

Figure 3: G1 Domain-Specific Hybrid Dataset Construction Pipeline. Our approach synergizes cost-free, high-fidelity 3D/2D geometry rendered from a 9-camera simulation environment (Isaac Lab) with Human-in-the-Loop (HITL) annotations to correct visual domain shifts, enabling robust native pose estimation.

Figure 4: Qualitative Comparison on Diverse Spatially-Sensitive Tasks. We evaluate zero-shot performance across Kicking, Hugging, Punching, and Sitting. (Top) Traditional human-centric retargeting suffers from spatial misalignments and kinematic violations due to the morphology gap. (Bottom) Dream2Act’s robot-centric hallucination ensures precise contacts and dynamic stability.

Method	Ball-kicking ( 1.64m Avg )		Box-hugging ( 0.75m )		Bag-hitting ( 1.60m )		Sofa-sitting ( 1.10m )
Method	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)
Baseline	0%	0.75	0%	0.35	0%	0.70	0%	0.65
Dream2Act (Ours)	40%	0.14	10%	0.11	30%	0.10	70%	0.28

Table 1: Quantitative comparison on four zero-shot interaction tasks. Task Length (in italics) denotes the initial distance to the target. Dream2Act significantly outperforms human-centric baselines, effectively bounding spatial alignment errors.

Figure 5: Spatial Generalization Analysis on Multi-Location Ball Kicking. To verify scale adaptability, we evaluate Dream2Act across four distinct ball positions. (Top) Human-centric retargeting fails to replicate spatial variations due to the morphology gap. (Bottom) Dream2Act successfully hallucinates and executes kicks for all targets, demonstrating robust spatial diversity.

Method	Kick Ball (a) ( 0.90m )		Kick Ball (b) ( 2.05m )		Kick Ball (c) ( 1.45m )		Kick Ball (d) ( 2.18m )
Method	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)	Succ $\uparrow$	Err $\downarrow$ (m)
Baseline	0%	0.53	0%	0.82	0%	0.84	0%	0.79
Dream2Act (Ours)	60%	0.19	20%	0.09	60%	0.09	20%	0.19

Table 2: Quantitative comparison on multi-location ball-kicking. Task lengths vary from 0.90m to 2.18m. Dream2Act maintains precise spatial alignment across all targets, avoiding the error accumulation seen in baselines.

Figure 6: Qualitative Comparison of Text-to-Motion Pipelines in Free Space. Human-centric T2M models amplify inherent artifacts during retargeting, causing dynamic constraint violations and hardware failure. Dream2Act strictly enforces physical consistency, achieving stable zero-shot deployment.

Figure 7: Qualitative Ablation on Native G1 Pose Estimation. Given a challenging real-world image: (a) Zero-shot ViTPose (COCO-only) completely misidentifies robot features; (b) A Sim-Only variant suffers from local drift due to domain shift; (c) Our Dream2Act native estimator, fine-tuned on the hybrid dataset, achieves pixel-perfect alignment.

Method	Simulation Test Set (2402 frames)		Internet Test Set (341 frames)
Method	AP $\uparrow$	2D [email protected] $\uparrow$	AP $\uparrow$	2D [email protected] $\uparrow$
Zero-shot ViTPose	0.234	64.99%	0.260	66.06%
Ours (Sim Only)	0.940	99.76%	0.497	91.61%
Ours (Hybrid)	0.785	95.78%	0.613	97.02%

Table 3: 2D Pose Estimation Performance on 12 Common Joints. We compare zero-shot generalization of standard human estimators (COCO-17) vs. our domain-specific model on shared kinematic joints.

Robot Motion Generation

Robot motion generation creates feasible, task-directed trajectories for robots to move, manipulate objects, and interact with people and environments.

See all 197 papers

RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation
Aug 4, 2026Shuliang He, Shuai Wang, Bo Yue +3Robot Skill Learning Generalization in Robotic Manipulation
The Chinese University of Hong Kong, Shenzhen · JD Technology · Tsinghua University
Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors
May 21, 2026Jiahe Chen, ZiRui Wang, Feiyu Jia +7Robot Motion Generation Robotic Manipulation
Zhejiang University · Shanghai AI Laboratory · The Chinese University of Hong Kong
Human2Humanoid: Physics-Aware Cross-Morphology Motion Retargeting for Humanoid Robots
Jun 2, 2026Tianchen Huang, Feiyang Yuan, Junchi Gu +5Cross-Embodiment Robot Learning Motion Retargeting
Institute of Humanoid Robots, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei, Anhui 230026, China

Aug 4, 2026J/K move · Enter open · S save

RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation

Shuliang He, Shuai Wang, Bo Yue, Junchi Teng+2

The Chinese University of Hong Kong, Shenzhen · JD Technology · Tsinghua University

Robot Skill Learning Generalization in Robotic Manipulation Whole-Body Robotic Manipulation Synthetic Data Generation for Robotics Robot Learning from Video Humanoid Robotics+1

Open paper AskPDF arXiv

Humanoid robots have the potential to perform dexterous manipulation in human environments, yet acquiring diverse and generalizable skills remains costly due to expensive hardware data collection and labor-intensive annotation. Recent advances in video generative models provide a promising opportunity to synthesize rich manipulation experiences from visual observations, but transferring such imagined behaviors into executable whole-body humanoid skills remains largely unexplored. In this work, we present RoboReact, a framework that automatically synthesizes whole-body humanoid manipulation skills from a single egocentric RGB-D observation. RoboReact generates human manipulation videos, extracts geometry-preserving interaction keyframes through depth-aware 3D reconstruction, and retargets them to high-DoF humanoid platforms while preserving hand-object interaction geometry. To bridge the gap between imagined plans and physical execution, RoboReact performs online object-centric re-grounding and leverages a vision-language model-guided refinement loop to adapt skills under geometric mismatch and execution deviations. The refined skills are executed through a whole-body controller, enabling coordinated whole-body manipulation and dexterous interaction. Experiments on real humanoid robots demonstrate that RoboReact generalizes across diverse object configurations and robustly recovers from execution disturbances without requiring teleoperation or human demonstrations. These results highlight the potential of combining generative models, vision-language reasoning, and closed-loop control for scalable humanoid skill acquisition.

Abstract

Robot Motion Generation

Explore similar work

RoboReact: Agentic Skill Distillation from Generated Egocentric Videos for Generalizable Whole-Body Manipulation

Imagine2Real: Towards Zero-shot Humanoid-Object Interaction via Video Generative Priors

Human2Humanoid: Physics-Aware Cross-Morphology Motion Retargeting for Humanoid Robots