Recent advancements in learning from human demonstration have shown promising results in addressing the scalability and high cost of data collection required to train robust visuomotor policies. However, existing approaches are often constrained by a reliance on multiview camera setups, depth sensors, or custom hardware and are typically limited to policy execution from third-person or egocentric cameras. In this paper, we present WARPED, a framework designed to synthesize realistic wrist-view observations from human demonstration videos to facilitate the training of visuomotor policies using only monocular RGB data. With data collected from an egocentric RGB camera, our system leverages vision foundation models to initialize the interactive scene. A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector. Lastly, photo-realistic wrist-view observations are synthesized via Gaussian Splatting to directly train a robotic policy. We demonstrate that WARPED achieves success rates comparable to policies trained on teleoperated demonstration data for five tabletop manipulation tasks, while requiring 5-8x less data collection time.
Figures & tables
Fig. 1: WARPED: A framework that warps egocentric human demonstrations into wrist-camera observations and trajectories for training robot policies.
Fig. 2: Overview of WARPED. Images of the scene are captured to build an initial Gaussian Splat representation. The user then performs a tabletop manipulation task, recording an egocentric demonstration from a GoPro attached to a helmet. The interactive scene is aligned with the Gaussian Splat, and the object pose is localized using vision foundation models. The object pose is then tracked over time via hand–object optimization. The resulting hand trajectories are retargeted to a robot gripper, and per-frame wrist-camera views of the scene are rendered and used to train a diffusion policy.
Fig. 3: Overview of hand–object optimization. The object pose is first estimated using supervised mask, depth, and DINOv2 losses. Then the hand and object pose are jointly refined using mask, depth, and interaction constraints.
Fig. 4: (a) Frequently contacted hand vertices. (b) Pose mapping from hand to end-effector using thumb and index joints. At the contact start frame (c), the closest 50 object points to the thumb and fingertips are identified (d) and used to refine the initial gripper pose and width (e) into a configuration that establishes contact with the points (f).
Fig. 5: Example demonstration frames, original wrist-view renders, and real rollout images at initial contact.
Fig. 6: Example demonstration frames and wrist-view renders illustrating object appearance, object pose, and gripper pose augmentations.
Fig. 7: (a) GoPro Hero9 attached to a helmet to record human demonstrations. (b) GoPro Hero9 with Max Lens Mod 1.0 mounted above gripper. (c) Objects used for data collection. (d) Novel objects used for evaluation.
Fig. 8: Real-world rollouts for all evaluated tasks.
Method
Rotate Box
Pour Mug
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
16/20
19/20
16/20
15/20
19/20
Alter (Heng et al. )
7/20
3/20
0/20
0/20
8/20
WARPED (no augmentation)
0/20
17/20
0/20
0/20
8/20
WARPED (background distractors)
18/20
15/20
17/20
9/20
17/20
WARPED
20/20
18/20
17/20
11/20
17/20
Teleoperation + WARPED
19/20
20/20
17/20
11/20
20/20
TABLE I: Success rates comparing teleoperation, the Alter baseline, and WARPED.
Method
Novel Obj
Rotate Box
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
Obj 1
8/10
4/10
7/10
9/10
Obj 2
2/10
2/10
4/10
9/10
WARPED
Obj 1
10/10
8/10
7/10
10/10
Obj 2
8/10
5/10
2/10
9/10
TABLE II: Success rates on novel object instances.
Method
Rotate Box
Pour Mug
Bottle from Rack
Wipe Brush
Can on Plate
Teleoperation
22:51
24:59
31:49
30:16
15:20
WARPED
3:37
3:18
3:27
5.19
3.41
TABLE III: Demonstration collection time (MM:SS).
Fig. 9: Example of background distractor rollout for Can on Plate task.
Method
Object
Can on Plate
Rotate Box
UMI
Train
9/10
2/10
Novel
10/10
0/10
WARPED
Train
8/10
10/10
Novel
9/10
10/10
TABLE IV: Success rates comparing UMI and WARPED.
Task
Img-H
P-H
Act-H
Freq
Speed
V-Arch
Compute
Rotate Box
2
2
8
10
0.33
ViT-B/16
4xV100
Pour Mug
2
2
8
10
0.33
ViT-B/16
4xV100
Bottle from Rack
2
2
12
10
0.33
ViT-B/16
4xV100
Wipe Brush
2
2
12
10
0.33
ViT-B/16
4xV100
Can on Plate
2
2
8
10
0.33
ViT-B/16
4xV100
Can on Plate OOD
1
2
8
5
0.17
ViT-L/14
4xL40
TABLE V: Training hyperparameters per task. Img-H: image observation horizon; P-H: proprioception observation horizon; Act-H: action horizon; Freq: rollout frequency; Speed: rollout frequency relative to data collection; V-Arch: vision encoder architecture; Compute: GPU configuration used for training.
Hyperparameter
Value
Image Resolution
224×224
Pretrained Vision Encoder
CLIP 1
Denoising Steps
50
Optimizer
AdamW
Base Learning Rate
3×10−4
Learning Rate Scheduler
Cosine Decay
TABLE VI: Common training hyperparameters.
Fig. 10: Hand-to-end-effector pose mapping. (a) Example hand output showing thumb, index, and derived TCP and base keypoints. (b) Gripper model with corresponding TCP and base keypoints. (c) Front view of the gripper aligned to the hand using the mapped TCP and base keypoints. (d) Side view of the gripper alignment using the TCP and base keypoints.
Fig. 11: Scenes used in the out-of-distribution experiments. The top four rows show training scenes, and the bottom row shows scenes used for evaluation.
Stage
Operation
Time
Once per task
Scene Reconstruction
Structure-from-Motion
04:15
Scene Gaussian Splat (15000 steps)
04:05
Total (Scene Reconstruction)
08:20
Object Mesh Reconstruction
SAM3D Object Mesh Build
00:17
Mesh Rendering
00:24
TABLE VII: Data processing times averaged across all tasks and demonstrations on an AMD Ryzen 9 7950X CPU and an NVIDIA GeForce RTX 3090 GPU. Timings reflect a sequential, unoptimized implementation.