Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

Seungyong Lee; Suhyeok Jang; Minyoung Kim; Jinwoo Shin

cs.RONov 3, 2026

Synthesizing Dexterous Robot Demonstrations from Human Videos via Virtual Try-On

Authors: Seungyong Lee, Suhyeok Jang, Minyoung Kim, Jinwoo Shin

Organizations: KAIST · RLWRLD · Korea University

Abstract

Robot foundation models have shown remarkable performance in robot control, yet adapting them to a new task still relies heavily on teleoperated demonstrations. Egocentric human videos offer a scalable substitute, but overlaying rendered robots on human videos or retargeting human poses into robot actions leaves significant visual or action gaps for high-DoF robot embodiments. To address this limitation, we propose RoboTryOn, which treats human-to-robot translation as virtual try-on: try-on keeps the person and swaps the garment, RoboTryOn keeps the demonstrated motion and swaps the human for the robot. Specifically, RoboTryOn converts a human video into a video of the target robot specified by a reference image, while preserving the original task, objects, and background. To maintain visual consistency, we translate selected keyframes autoregressively and use them to guide video synthesis with motion conditioned on the human hand skeleton. Additionally, we use an inverse dynamics model (IDM) that incorporates depth features to infer action labels from synthesized videos for policy learning. Our experiments across multiple humanoid embodiments show that policies trained with our synthetic data improve the average success rate on novel tasks, including challenging bimanual manipulation, from 2% to 60%, without any real robot demonstration of those tasks. Moreover, training with our synthetic data matches the performance of policies trained solely on teleoperated demonstrations at approximately a third of the collection cost, demonstrating the cost efficiency of our approach.

Figures & tables

Figure 1: RoboTryOn pipeline. (a) The image model sequentially translates keyframes of a human video into robot keyframes, the video model synthesizes the motion between them under the guidance of the hand skeleton flow, and the inverse dynamics model labels the actions. (b) The image model inpaints the masked human frame, using the previous robot keyframe as a reference. (c) The video model synthesizes the robot video, conditioned on the robot keyframes at their temporal positions and the skeleton flow.

Panel of Figure 2 — Figure 2: Examples of real-world benchmark tasks. For each of the four PnP tasks (top), both a human video and a teleoperated demonstration are available. For each of the remaining seven tasks (bottom), only a human video is available.

Table 1: Real-world results on (a) OpenArm and (b) RB-Y1. Success rate (%) over 28 trials per task. In-domain: the four PnP tasks with robot demonstrations. Out-of-domain: the seven tasks with human video only, grouped by what they add. “+” rows add the converted human videos to the teleoperated demonstrations, and the last row uses the synthesized episodes alone. Baselines are evaluated following their own protocols. Per-task results for the out-of-domain tasks are provided in Section F.2 .

Panel of Figure 3 — Figure 3: Visual gap to real robot video. (a) The same moments as recorded, as EgoMimic and Masquerade edit them, and as RoboTryOn translates them, next to similar moments from teleoperation. (b) UMAP of SigLIP features of $\pi_{0.5}$ with density contours, using equal frame counts per corpus, real robot frames from the four PnP tasks, and the other frames from all eleven tasks.

Panel of Figure 4 — Figure 4: Action gap: open-loop replay. (a) A real scene and its simulated replica. (b) The grasp moment of one human video under the three action labelings, replayed open loop. (c) Replay success over 66 episodes of three PnP tasks (per task in Table 8 ).

Panel of Figure 5 — Figure 5: Action gap: distribution and co-training. (a) Right-hand actions over all tasks. (b) Cosine between the policy gradients from teleoperated PnP data and from the out-of-domain tasks. (c) Share of hand actions within the teleoperated distribution in real robot rollouts.

Panel of Figure 6 — Figure 6: Collection cost. Success rate versus person-hours on four tasks. Marker size indicates demonstrations per task.

Method	Speed	In-dom.	Out-of-dom.	Time (s)
Teleop only	$1\times$	47.3	0 3.1	14.2
Ours (as recorded)	$1.25\times$	75.9	73.0	11.9
Ours (retimed)	$1\times$	85.7	70.4	13.8
Ours (retimed)	$2\times$	57.1	42.9	0 7.4

Table 2: Execution speed. Success rate (%) on the OpenArm and mean time from rollout start to object placement on the four PnP tasks.

Table 3: Ablation study of RoboTryOn on OpenArm. Success rates (%) are averaged over four in-domain and seven out-of-domain tasks.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Model	Output per view	Views	Steps	Time (s)
Hand reconstruction	HaWoR	3D hand pose, per frame	2	–	0.448 / frame
Keyframe translation	$f_{\theta}$	1024 $\times$ 576	2	30	34.6 / keyframe
Video synthesis, stage 1	$g_{\phi}$	512 $\times$ 288	2	8	50.0 / clip
Video synthesis, stage 2	$g_{\phi}$	1024 $\times$ 576	2	3	50.0 / clip
Action labeling	$h_{\psi}$	20-step joint chunk	2	–	0.077 / window
End-effector correction	MuJoCo IK	arm joints, per frame	–	–	0.22 / frame

Appendix

Table 5: Inference configuration and cost per unit. Time is measured on a single H100. Video stage 2 upsamples and refines the output of stage 1.

Stage	Units	Time (s)
Hand reconstruction + keyframe selection	300 frames	134
Keyframe translation	5 keyframes	173
Video synthesis, stage 1	3 clips	150
Video synthesis, stage 2	3 clips	150
Action labeling	281 windows	22
End-effector correction	300 frames	66

Appendix

Table 6: Pipeline wall-clock time for a 15 s demonstration (300 frames, one embodiment) on a single H100.

Panel of Figure 7 — Figure 7: Real-world setup. (a) Human video collection from the robot’s camera position. (b) OpenArm embodiment. (c) RB-Y1 embodiment. (d) Objects of the in-domain tasks. (e) Objects introduced for the out-of-domain tasks.

Panel of Figure 8 — Figure 8: Real-world tasks. (a) In-domain tasks with teleoperated demonstrations and human videos. (b) Out-of-domain tasks with human videos only.

Figure 9: Simulated tasks. Each row shows the human video of a task (left) and a successful episode in simulation (1–5): the right hand grasps the object, lifts it, and places it in the container.

Corpus	All eleven tasks	Four PnP tasks	Seven tasks w/o robot data
Human	125.5	120.1	133.5
EgoMimic	129.0	127.1	134.6
Masquerade	58.2	50.3	68.8
RoboTryOn (Ours)	32.7	26.6	41.9

Appendix

Table 7: Fréchet SigLIP distance to the real robot frames. Lower is closer. Real robot frames are from the four PnP tasks, and each corpus is compared on all eleven tasks (as in Figure 3 ), on the four PnP tasks alone, and on the seven tasks without robot data alone.

Labeling	PnP Bottle	PnP Box	PnP Ball	Avg.
Retargeting [ Qin et al., 2023 ]	63.6	31.8	9.1	34.8
RoboTryOn (Ours) w/o wrist correction	63.6	72.7	54.5	63.6
RoboTryOn (Ours)	77.3	63.6	68.2	69.7

Appendix

Table 8: Action gap, per task. Success (%) of replaying the action labels of the same human videos open loop, without a policy, in a simulated replica of the real scene with the objects at their recorded positions (OpenArm, 22 episodes per task). Averages are in Figure 4 (c).

Training data	$N$ / task	Person-hours	Successes / 112	SR (%)
Teleop only	10	1.05	21	18.8
	30	3.16	39	34.8
	50	5.26	53	47.3
Human video only	10	0.19	14	12.5
	30	0.56	28	25.0
	50	0.93	40	35.7

Appendix

Table 9: Success against collection cost, per condition. Data behind Figure 6 . Each policy is trained on the four PnP tasks alone with

N

demonstrations per task and evaluated over 28 trials per task (112 in total). Person-hours follow Table 10 : 38 teleoperated episodes or 214 human videos per person-hour, and the equal split uses

N/2

of each.

	Robot Teleoperation	Human Ego Video
People per session	2	1
Demonstration footage / person-hour	14 min	45 min
Episodes / person-hour	38	214
Hardware	Robot + teleoperation system	Fixed camera
Trained operator required	Yes	No
Occupies robot	Yes	No

Appendix

Table 10: Data collection cost. Teleoperation occupies two people, so one person-hour yields 14 min of robot demonstration against 45 min of egocentric human video. Because a person also performs each task faster than a teleoperated robot, the gap in episode count is 5–6

\times

Table 11: Per-task results behind Table 1 . Success rates (%) over 28 trials per task using

\pi_{0.5}

on OpenArm and RB-Y1. Robot demonstrations are available only for the PnP tasks in (a). “+” rows add the human videos converted by each method to the teleoperated demonstrations, and the RoboTryOn (Ours) only rows use the synthesized episodes alone. EgoMimic and Masquerade are deployed with their test-time mask and overlay, whereas RoboTryOn uses the raw camera view.

Panel of Figure 10 — Figure 10: One human video, three robots. Human videos (left) translated into the OpenArm, the RB-Y1, and the GR-1 by a single set of models, and only the embodiment changes.

Figure 11: Failure cases of the ablated variants. Human video and videos synthesized by the variants in Table 3(a) at the same time steps. Boxed regions are enlarged at representative failure points.

Panel of Figure 12 — Figure 12: Rollouts on all eleven tasks on the OpenArm , with the policy trained on the teleoperated and synthesized episodes: (a) the four tasks with robot demonstrations and (b) the seven with human video only.

Panel of Figure 13 — Figure 13: Rollouts on the RB-Y1 , with the policy trained on the teleoperated and synthesized episodes: (a) the four tasks with robot demonstrations and (b) the tasks with human video only. Air Fryer is omitted because no policy completes it on the RB-Y1 (0% success in Table 11 ).

Figure 14: Long-horizon synthesis. A 25-second human video (top) with its selected keyframes, and the generated robot video (bottom) with the clip boundaries at the keyframe times. Adjacent clips share the keyframe at their boundary, so robot and objects stay consistent across boundaries.

Figure 15: Failure cases. The red box in each row marks the region enlarged on the right. Generation failures (top): synthesized videos in which the generative models do not preserve the objects. On PnP Ball in Cup the transparent cup suddenly disappears from the hand as it is lifted, leaving the ball on the table. On Air Fryer the fingers pass through the handle instead of closing around it (dashed circle). IDM errors (bottom): policy rollouts that fail because the action labels of the synthesized episodes place the hand imprecisely. On PnP Coffee Pot the hand reaches the handle but closes below it, and on PnP Bottle to Shelf it closes low on the bottle and pushes it against the shelf instead of lifting it onto it.

Synthetic Data Generation for Robotics

Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.

See all 315 papers

Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis
Mar 20, 2026Weisheng Xu, Jian Li, Yi Gu +12Robot Motion Generation Humanoid Robotics
Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology, Shenzhen · Shenzhen University +1
Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand
Oct 14, 2026Nagasrikanth KallakuriCross-Embodiment Robot Learning Robotic Manipulation
Dxtr AI Co.
RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer
Oct 7, 2026Huang Huang, Wensi Ai, Ziyu Chen +6Sim-to-Real Transfer Synthetic Data Generation for Robotics
Stanford University · Nvidia · Microsoft Research

Mar 20, 2026J/K move · Enter open · S save

Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis

Weisheng Xu, Jian Li, Yi Gu, Bin Yang+11

Hong Kong University of Science and Technology (Guangzhou) · Harbin Institute of Technology, Shenzhen · Shenzhen University+1

Robot Motion Generation Humanoid Robotics Zero-Shot Robotic Manipulation Robot Learning

Open paper AskPDF arXiv

Equipping humanoid robots with versatile interaction skills typically requires either extensive policy training or explicit human-to-robot motion retargeting. However, learning-based policies face prohibitive data collection costs. Meanwhile, retargeting relies on human-centric pose estimation (e.g., SMPL), introducing a morphology gap. Skeletal scale mismatches result in severe spatial misalignments when mapped to robots, compromising interaction success. In this work, we propose Dream2Act, a robot-centric framework enabling zero-shot interaction through generative video synthesis. Given a third-person image of the robot and target object, our framework leverages video generation models to envision the robot completing the task with morphology-consistent motion. We employ a high-fidelity pose extraction system to recover physically feasible, robot-native joint trajectories from these synthesized dreams, subsequently executed via a general-purpose whole-body controller. Operating strictly within the robot-native coordinate space, Dream2Act avoids retargeting errors and eliminates task-specific policy training. We evaluate Dream2Act on the Unitree G1 across four whole-body mobile interaction tasks: ball kicking, sofa sitting, bag punching, and box hugging. Dream2Act achieves a 37.5% overall success rate, compared to 0% for conventional retargeting. While retargeting fails to establish correct physical contacts due to the morphology gap (with errors compounded during locomotion), Dream2Act maintains robot-consistent spatial alignment, enabling reliable contact formation and substantially higher task completion.

Abstract

Synthetic Data Generation for Robotics

Explore similar work

Morphology-Consistent Humanoid Interaction through Robot-Centric Video Synthesis

Human Demonstrations as Post-Training Data for Any Robot Body: Retargeting the Object, Not the Hand

RoboRender: Robot-Oriented Video Generation for Visual Sim-to-Real Transfer