Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Figures & tables
Figure 1: Simulation pre-training for dexterity. Top: 75 hours of teleoperated simulation data spanning six scenes and hundreds of object instances, covering contact-rich low-level dexterous behavior. Bottom: the pre-trained policy adapts to the real-world dexterous tasks shown, with 1 hour of real demonstrations.
Figure 2: Diverse operator strategies on open-ended simulation tasks. Each pair of frames shows two distinct strategies the same operator pool produced on a single long-horizon task: (a) spelling a target word from a pile of letter blocks, (b) stacking blocks into a Jenga tower, and (c) arranging dominoes into a chain.
Figure 3: Policy Architecture. Our policy is a diffusion transformer that consumes a sequence of proprioception ( o ), action ( a ), multi-view images, and noised action chunks as interleaved tokens. In training, the model denoises all action chunks simultaneously with a causal mask, amortizing its sequence length. We additionally use sliding window attention in training, with a matched rolling KV cache for efficient inference.
Figure 4: Autonomous rollouts of five tasks. (A) plate racking: lifting plates and racking them in a dish rack; (B) mug hanging: hanging a mug on a mug tree after a bimanual handover; (C) jenga playing: pushing a block out of the tower with one hand, pulling it free with the other, and placing it on top; (D) cup stacking: unstacking nested cups and building a pyramid; (E) bottles in bin: tossing bottles into a bin.
Figure 5: Simulation pre-training improves real-world dexterity. We evaluate five bimanual dexterous tasks on the real robot, comparing SPD pre-trained checkpoints to those trained from scratch (BC). In (A-E), we report the fraction of trials that reach each successive manipulation stage and plot training loss curves. In (F), we plot average task progress; error bars show standard error.
Figure 6: Ablations on history conditioning and chunk size. Visualization of the average task progress in Table 1 , for policies pre-trained with SPD (left) and trained from scratch (right).
w
c
plates
mugs
jenga
cups
bottles
SPD, pre-trained
1
8
31.9
35.0
0.0
0.0
0.0
32
55.6
58.3
5.0
15.0
36.2
32
8
80.6
93.3
85.0
55.6
68.8
32
44.4
78.3
16.7
36.9
70.0
BC, from-scratch
1
8
16.2
38.3
0.0
0.0
0.0
32
36.9
60.0
18.3
17.5
33.8
Table 1: Average task progress for all variants. Progress (%) for every sliding-window ( w ) and action-chunk ( c ) combination, with the best variant per task bolded within each training regime. w=32 , c=8 is the chosen configuration.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
Task
Episodes
Minutes
Jenga
Hollow tower
92
567
Tower
87
473
Dominos
107
471
Criss-cross
103
386
Handover (L → R)
96
172
Handover (R → L)
72
109
Appendix
Table 2: spd-75h dataset statistics. Per-task episode counts and durations, grouped by scene. Spelling variants are merged into a single Spelling task, and tasks with fewer than ten episodes are omitted. Durations are reported in minutes at 30 Hz.
Task
Setting
Max Score
Scoring Rubric
bottles in bin
4 bottles, 1 bin
4
+1 for each bottle tossed into the bin. 60-second timeout.
plate racking
2 plates, 1 dish rack
4
+1 for lifting each plate; +1 for racking it
cup stacking
6 cups
8
+1 for each cup placed correctly; +1 for each subsequent destack move
jenga playing
1 tower
3
+1 for pushing a middle block out; +1 for pulling it from the other side without collapsing the tower; +1 for placing it on top
mug hanging
1 mug, 1 mug tree
3
+1 for lifting the mug; +1 for the handover; +1 for hanging it on the hook
Appendix
Table 3: Evaluation task rubrics. Max score is the maximum achievable score for the task; reported progress is the achieved score normalized by this maximum.
Hyperparameter
Value
Batch size
64
Learning rate
1×10−3
Learning rate schedule
constant
Weight decay
0.1
Optimizer
Muon (matrices), AdamW (rest)
Parameters
222M
Appendix
Table 4: Pre-training hyperparameters.
bottles
plates
cups
jenga
mugs
Training steps
6k
6k
10k
6k
6k
Dataset size (minutes)
72
70
121
48
44
Dataset size (episodes)
270
161
217
193
238
Appendix
Table 5: Real-world fine-tuning hyperparameters. Only settings that differ from pre-training ( Table 4 ) are listed; all other hyperparameters are inherited. Dataset size is reported per task.