Recent advancements in learning from human demonstration have shown promising results in addressing the scalability and high cost of data collection required to train robust visuomotor policies. However, existing approaches are often constrained by a reliance on multiview camera setups, depth sensors, or custom hardware and are typically limited to policy execution from third-person or egocentric cameras. In this paper, we present WARPED, a framework designed to synthesize realistic wrist-view observations from human demonstration videos to facilitate the training of visuomotor policies using only monocular RGB data. With data collected from an egocentric RGB camera, our system leverages vision foundation models to initialize the interactive scene. A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector. Lastly, photo-realistic wrist-view observations are synthesized via Gaussian Splatting to directly train a robotic policy. We demonstrate that WARPED achieves success rates comparable to policies trained on teleoperated demonstration data for five tabletop manipulation tasks, while requiring 5-8x less data collection time.
Figures & tables
Fig. 1: WARPED: A framework that warps egocentric human demonstrations into wrist-camera observations and trajectories for training robot policies.
Fig. 2: Overview of WARPED. Images of the scene are captured to build an initial Gaussian Splat representation. The user then performs a tabletop manipulation task, recording an egocentric demonstration from a GoPro attached to a helmet. The interactive scene is aligned with the Gaussian Splat, and the object pose is localized using vision foundation models. The object pose is then tracked over time via hand–object optimization. The resulting hand trajectories are retargeted to a robot gripper, and per-frame wrist-camera views of the scene are rendered and used to train a diffusion policy.
Fig. 3: Overview of hand–object optimization. The object pose is first estimated using supervised mask, depth, and DINOv2 losses. Then the hand and object pose are jointly refined using mask, depth, and interaction constraints.
Fig. 4: (a) Frequently contacted hand vertices. (b) Pose mapping from hand to end-effector using thumb and index joints. At the contact start frame (c), the closest 50 object points to the thumb and fingertips are identified (d) and used to refine the initial gripper pose and width (e) into a configuration that establishes contact with the points (f).
Fig. 5: Example demonstration frames, original wrist-view renders, and real rollout images at initial contact.
Fig. 6: Example demonstration frames and wrist-view renders illustrating object appearance, object pose, and gripper pose augmentations.
Fig. 7: (a) GoPro Hero9 attached to a helmet to record human demonstrations. (b) GoPro Hero9 with Max Lens Mod 1.0 mounted above gripper. (c) Objects used for data collection. (d) Novel objects used for evaluation.
Fig. 8: Real-world rollouts for all evaluated tasks.
Method
Rotate Box
Pour Mug
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
16/20
19/20
16/20
15/20
19/20
Alter (Heng et al. )
7/20
3/20
0/20
0/20
8/20
WARPED (no augmentation)
0/20
17/20
0/20
0/20
8/20
WARPED (background distractors)
18/20
15/20
17/20
9/20
17/20
WARPED
20/20
18/20
17/20
11/20
17/20
Teleoperation + WARPED
19/20
20/20
17/20
11/20
20/20
TABLE I: Success rates comparing teleoperation, the Alter baseline, and WARPED.
Method
Novel Obj
Rotate Box
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
Obj 1
8/10
4/10
7/10
9/10
Obj 2
2/10
2/10
4/10
9/10
WARPED
Obj 1
10/10
8/10
7/10
10/10
Obj 2
8/10
5/10
2/10
9/10
TABLE II: Success rates on novel object instances.
Method
Rotate Box
Pour Mug
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
22:51
24:59
31:49
30:16
15:20
WARPED
3:37
3:18
3:27
5.19
3.41
TABLE III: Demonstration collection time (MM:SS).
Fig. 9: Example of background distractor rollout for Can on Plate task.
Method
Object
Can on Plate
Rotate Box
UMI
Train
9/10
2/10
Novel
10/10
0/10
WARPED
Train
8/10
10/10
Novel
9/10
10/10
TABLE IV: Success rates comparing UMI and WARPED.
Task
Img-H
P-H
Act-H
Freq
Speed
V-Arch
Compute
Rotate Box
2
2
8
10
0.33
ViT-B/16
4xV100
Pour Mug
2
2
8
10
0.33
ViT-B/16
4xV100
Bottle from Rack
2
2
12
10
0.33
ViT-B/16
4xV100
Wipe Brush
2
2
12
10
0.33
ViT-B/16
4xV100
Can on Plate
2
2
8
10
0.33
ViT-B/16
4xV100
Can on Plate OOD
1
2
8
5
0.17
ViT-L/14
4xL40
TABLE V: Training hyperparameters per task. Img-H: image observation horizon; P-H: proprioception observation horizon; Act-H: action horizon; Freq: rollout frequency; Speed: rollout frequency relative to data collection; V-Arch: vision encoder architecture; Compute: GPU configuration used for training.
Hyperparameter
Value
Image Resolution
224×224
Pretrained Vision Encoder
CLIP 1
Denoising Steps
50
Optimizer
AdamW
Base Learning Rate
3×10−4
Learning Rate Scheduler
Cosine Decay
TABLE VI: Common training hyperparameters.
Fig. 10: Hand-to-end-effector pose mapping. (a) Example hand output showing thumb, index, and derived TCP and base keypoints. (b) Gripper model with corresponding TCP and base keypoints. (c) Front view of the gripper aligned to the hand using the mapped TCP and base keypoints. (d) Side view of the gripper alignment using the TCP and base keypoints.
Fig. 11: Scenes used in the out-of-distribution experiments. The top four rows show training scenes, and the bottom row shows scenes used for evaluation.
Stage
Operation
Time
Once per task
Scene Reconstruction
Structure-from-Motion
04:15
Scene Gaussian Splat (15000 steps)
04:05
Total (Scene Reconstruction)
08:20
Object Mesh Reconstruction
SAM3D Object Mesh Build
00:17
Mesh Rendering
00:24
TABLE VII: Data processing times averaged across all tasks and demonstrations on an AMD Ryzen 9 7950X CPU and an NVIDIA GeForce RTX 3090 GPU. Timings reflect a sequential, unoptimized implementation.
Dexterous manipulation is limited by the cost of collecting large-scale robot demonstrations. Egocentric human videos offer a scalable source of diverse manipulation behaviors, but directly using them for robot learning requires bridging two gaps: the visual gap between human and robot observations, and the action gap between human motion and robot-executable action. We propose EgoEngine, a scalable framework for transforming egocentric human manipulation videos into high-fidelity robot data. Given an egocentric RGB video, EgoEngine produces: (i) a high-fidelity robot observation video replacing human with robot while preserving scene context and temporal alignment, and (ii) a task-aligned, executable robot action trajectory under feasibility constraints. Experiments in simulation and on real robots show that EgoEngine enables scalable conversion of human videos into robot data and, to our knowledge, demonstrates the first zero-shot visuomotor dexterous policy learning from egocentric human videos without real-robot demonstrations. Project website: https://egoengine.github.io.
Yangcen Liu, Shuo Cheng, Xinchen Yin +6
Georgia Institute of Technology · Tsinghua University
Training a visuomotor policy calls for abundant demonstrations that closely match the target environment, yet collecting them anew remains expensive. Existing demonstration synthesis methods reduce this cost but remain constrained by high manual effort, limited visual fidelity, or heavy reliance on physics simulators. This paper introduces GaussianFactory, a high-fidelity data engine that mass-produces demonstrations with a single video scan as its only human input and no physics engine in the generation loop. Specifically, GaussianFactory reconstructs the scene as an editable 3D Gaussian Splatting (3DGS) replica and samples from the object-combination tasks the scene affords. For each task, it plans grasps and trajectories purely kinematically on the geometric reconstruction, rendering photorealistic demonstrations that visually match the target environment. Physical dynamics enter the pipeline only where contact force interactions dictate the outcome---during grasp formation, via a learned contact model pretrained once on an interaction dataset. To evaluate the downstream utility of the synthesized demonstrations, we implement an end-to-end scan-to-deployment workflow in two setups: a simulated scene that stands in for the real world to enable reproducibility, and a real-world workspace with a physical UR10e robot. In each setup, a standard diffusion policy trained solely on the synthesized demonstrations achieves 95.1% and 84.2% success rates, respectively.
Beichen Wang, Yuen-Hei Yeung, V. R. Sridhar Devarakonda +1
Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.
Feng Jiang, Shu Chen, Yuanfei Wang +7
Peking University · Hong Kong University of Science and Technology (Guangzhou) · Lightwheel