Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Figures & tables
Figure 1: Simulation pre-training for dexterity. Top: 75 hours of teleoperated simulation data spanning six scenes and hundreds of object instances, covering contact-rich low-level dexterous behavior. Bottom: the pre-trained policy adapts to the real-world dexterous tasks shown, with 1 hour of real demonstrations.
Figure 2: Diverse operator strategies on open-ended simulation tasks. Each pair of frames shows two distinct strategies the same operator pool produced on a single long-horizon task: (a) spelling a target word from a pile of letter blocks, (b) stacking blocks into a Jenga tower, and (c) arranging dominoes into a chain.
Figure 3: Policy Architecture. Our policy is a diffusion transformer that consumes a sequence of proprioception ( o ), action ( a ), multi-view images, and noised action chunks as interleaved tokens. In training, the model denoises all action chunks simultaneously with a causal mask, amortizing its sequence length. We additionally use sliding window attention in training, with a matched rolling KV cache for efficient inference.
Figure 4: Autonomous rollouts of five tasks. (A) plate racking: lifting plates and racking them in a dish rack; (B) mug hanging: hanging a mug on a mug tree after a bimanual handover; (C) jenga playing: pushing a block out of the tower with one hand, pulling it free with the other, and placing it on top; (D) cup stacking: unstacking nested cups and building a pyramid; (E) bottles in bin: tossing bottles into a bin.
Figure 5: Simulation pre-training improves real-world dexterity. We evaluate five bimanual dexterous tasks on the real robot, comparing SPD pre-trained checkpoints to those trained from scratch (BC). In (A-E), we report the fraction of trials that reach each successive manipulation stage and plot training loss curves. In (F), we plot average task progress; error bars show standard error.
Figure 6: Ablations on history conditioning and chunk size. Visualization of the average task progress in Table 1 , for policies pre-trained with SPD (left) and trained from scratch (right).
w
c
plates
mugs
jenga
cups
bottles
SPD, pre-trained
1
8
31.9
35.0
0.0
0.0
0.0
32
55.6
58.3
5.0
15.0
36.2
32
8
80.6
93.3
85.0
55.6
68.8
32
44.4
78.3
16.7
36.9
70.0
BC, from-scratch
1
8
16.2
38.3
0.0
0.0
0.0
32
36.9
60.0
18.3
17.5
33.8
Table 1: Average task progress for all variants. Progress (%) for every sliding-window ( w ) and action-chunk ( c ) combination, with the best variant per task bolded within each training regime. w=32 , c=8 is the chosen configuration.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
Task
Episodes
Minutes
Jenga
Hollow tower
92
567
Tower
87
473
Dominos
107
471
Criss-cross
103
386
Handover (L → R)
96
172
Handover (R → L)
72
109
Appendix
Table 2: spd-75h dataset statistics. Per-task episode counts and durations, grouped by scene. Spelling variants are merged into a single Spelling task, and tasks with fewer than ten episodes are omitted. Durations are reported in minutes at 30 Hz.
Task
Setting
Max Score
Scoring Rubric
bottles in bin
4 bottles, 1 bin
4
+1 for each bottle tossed into the bin. 60-second timeout.
plate racking
2 plates, 1 dish rack
4
+1 for lifting each plate; +1 for racking it
cup stacking
6 cups
8
+1 for each cup placed correctly; +1 for each subsequent destack move
jenga playing
1 tower
3
+1 for pushing a middle block out; +1 for pulling it from the other side without collapsing the tower; +1 for placing it on top
mug hanging
1 mug, 1 mug tree
3
+1 for lifting the mug; +1 for the handover; +1 for hanging it on the hook
Appendix
Table 3: Evaluation task rubrics. Max score is the maximum achievable score for the task; reported progress is the achieved score normalized by this maximum.
Hyperparameter
Value
Batch size
64
Learning rate
1×10−3
Learning rate schedule
constant
Weight decay
0.1
Optimizer
Muon (matrices), AdamW (rest)
Parameters
222M
Appendix
Table 4: Pre-training hyperparameters.
bottles
plates
cups
jenga
mugs
Training steps
6k
6k
10k
6k
6k
Dataset size (minutes)
72
70
121
48
44
Dataset size (episodes)
270
161
217
193
238
Appendix
Table 5: Real-world fine-tuning hyperparameters. Only settings that differ from pre-training ( Table 4 ) are listed; all other hyperparameters are inherited. Dataset size is reported per task.
Large-scale dexterous grasp datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pick-and-place manipulation. We study whether such data can instead support functional dexterity in articulated tool use, where a robot must acquire a tool, maintain contact, and operate its functional moving parts. We adapt a hierarchical imitation learning framework that combines high-level hand sub-goal prediction with a low-level goal-conditioned controller. We construct a 355k-trajectory grasp-pretraining dataset from large-scale dexterous grasp annotations and use it to pretrain the low-level controller. The controller is then fine-tuned on downstream task demonstrations. To evaluate this setting, we introduce DexCraft, a simulation benchmark with six articulated tool-use tasks requiring coordinated finger motion. Across simulation and real-world experiments, our approach outperforms end-to-end diffusion policy baselines and hierarchical policies trained from scratch. In the real world, it improves full-task success by 33.3 percentage points over DP3. These results show that grasp datasets can serve not only as resources for grasp synthesis, but also as scalable pretraining data for contact-rich dexterous manipulation. Videos are shown on https://yingyuan0414.github.io/grasp2dexterity/ .
Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/
Jin Wu, Lianjie Yuan, Zeyan Sun +4
The University of Texas at Austin · University of Florida · BrainCo
Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.
Zijian Zeng, Fei Ding, Huiming Yang +2
Tsinghua University · Alibaba Group · Bengbu University +1