We introduce Accelerating Dexterity via Pre-Training (ADEPT), a large-scale reinforcement learning (RL) framework for learning sim-to-real transferable dexterity across high degree-of-freedom (DoF) robot embodiments that can solve long-horizon tasks directly from raw visuo-tactile perception. ADEPT pretrains a dexterous policy on a generic object reposing task, then post-trains downstream policies with this pretrained behavior as a prior. ADEPT enables learning new behaviors that are otherwise difficult to discover from scratch on multi-fingered robots and avoids learning the same set of skills over again for every new downstream task. The pretrained policy zero-shots the reposing phase of downstream tasks, but naïve RL fine-tuning rapidly degrades this capability during transfer. We address this with a stable post-training recipe combining behavior-cloning distillation, critic warm-up, and conservative on-policy updates. To safely exploit the full kinematic dexterity, we introduce a joint-space Geometric Fabric that mediates between the RL policy and the robot. We distill post-trained teachers into perceptive students that zero-shot sim-to-real transfer on two embodiments: a 23 DoF Kuka-Allegro with two RGB cameras, and a 29 DoF Flexiv-Sharpa with two RGB cameras and five vision-based tactile sensors, and can solve long-horizon tasks from challenging initial states with dexterity at human-level speed.
Figures & tables
Figure 1: ADEPT enables sim-to-real RL of high DoF arm-hand robots to reach, grasp, lift, reorient, transport, align, and insert. We demonstrate on a 23 DoF Kuka-Allegro from two RGB cameras and on a 29 DoF Flexiv-Sharpa from two RGB cameras and five fingertip tactile sensors.
Figure 2: ADEPT Overview. (1) Pre-train (πpre,Vpre) via PPO on a generic reposing task. (2) Post-train into (πpost,Vpost) for the downstream contact-rich task via BC distillation, frozen actor critic warm-up, and conservative PPO. (3) Distill πpost into a stereo RGB student πstudent . (4) Deploy πstudent zero-shot on the real robot.
Metric
Primitive
FMB Peg
VisDex [ 4 ] (Fixed Goal)
Number of objects
16
2
152
Kuka-Allegro SR ↑
0.73±0.003
0.76±0.003
0.77±0.011
Flexiv-Sharpa SR ↑
0.64±0.007
0.58±0.011
0.61±0.015
Table 1: Episodic success rates (SR) of pre-trained reposing teacher on unseen objects. Mean ± std across seeds at the final ADR level.
Zero-shot Success Rate (%) at ADR Level across 1024 Episodes
Task (Embodiment)
0
5
10
15
20
25
30
35
40
45
50
FMB Peg (Kuka-Allegro)
98.6
94.1
84.7
78.6
71.9
67.9
58.7
52.5
39.6
18.0
0.00
FMB Peg (Flexiv-Sharpa)
94.5
87.1
80.3
76.8
74.9
70.4
66.8
59.2
52.3
22.9
0.70
Table 2: Zero-shot success rate of the pretrained reposing teacher on FMB peg reposing across ADR.
Figure 3: Training curves on FMB peg insertion comparing (1) training from scratch, (2) direct PPO finetuning of the pretrained reposing teacher, and (3) ADEPT post-training. Blue inset: success-rate collapse for (2) during transfer.
Variant
BC
WU
LR
Clip
ADR50
Train SR (%)
Time to ADR 50
(a)
ADEPT (full)
✓
✓
10−5
.05
5/5
46.0±1.1
19.9±1.0 h
(b)
No BC
✗
✓
10−5
.05
4/5
38.5±1.2
35.2±4.9 h
(c)
No warm-up
✓
✗
10−5
.05
4/5
28.4±3.7
20.2±1.5 h
(d)
Standard PPO
✓
✓
10−3
.20
0/5
0.0±0.0
Collapse (ADR 20)
(e)
High LR only
✓
✓
10−3
.05
0/5
0.0±0.0
Collapse (ADR 20)
(f)
Loose clip
✓
✓
10−5
.20
5/5
46.8±2.0
17.6±1.8 h
Table 3: Post-training component ablations on KUKA-Allegro FMB peg insertion; 5 independent non-PBT seeds per row, all from one pretrained checkpoint. ADR50 counts seeds reaching the final ADR level; time is averaged over seeds that reached it. Train SR is the instantaneous training success rate over parallel environments.
Figure 4: Real-world setup. (A) The 23 DoF Kuka-Allegro and 29 DoF Flexiv-Sharpa platforms, each with two RealSense RGB cameras (left and center). (B) FMB pegs and boards, plates, and dish rack. (C) Diverse initial states and lighting used in real-world testing.
Modality
Robot
Task
Reach
Grasp
Lifting
Reorient
Align
Insert (SR)
Vision
Kuka-Allegro
FMB Star
10/10
9/10
8/10
8/10
7/10
5/10
Vision
Kuka-Allegro
FMB Square/Round
10/10
8/10
6/10
4/10
3/10
3/10
Vision
Flexiv-Sharpa
FMB Square/Round
10/10
7/10
5/10
3/10
3/10
3/10
Visuo-Tactile
Flexiv-Sharpa
FMB Square/Round
10/10
10/10
10/10
9/10
8/10
8/10
Vision
Kuka-Allegro
Dish
10/10
10/10
8/10
7/10
6/10
6/10
Table 4: Per-stage success rates on real-world long-horizon manipulation, trained with ADEPT. Each stage is cumulative: success at stage k requires success at all earlier stages.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Reposing Task. Visualizes the out-of-bound range in red, the goal sampling range in green, and the object spawn range in orange. (a) visualizes a frame of the initial grasp of the object and (b) visualizes a frame where the robot has completed the task by reaching the target pose frame. The object point clouds can be seen in green spheres, which are on the object.
Figure 6: 16 primitive objects used during pre-training reposing task. The scale and physical properties of the objects are randomized during pre-training.
Figure 7: Downstream FMB Task. Visualizes the peg (orange) and board (sky blue) spawn ranges, object keypoints and goal pose keypoints. Grey keypoints represent the path set by the ADR level (denoted goal path annealing). At ADR 0, the goal is set to where the light blue sphere is placed and at ADR 50, the goal is set to the insertion goal where the light green sphere is placed. (a) shows sample initialization of the scene where the peg is dropped on to the table. In (b), we show how the reposing teacher reaches the goal at ADR 20, which is above the board. In (c), we visualize a sample policy rollout that inserts the peg into the board successfully where the keypoints align by the matching colors. The object point clouds in green spheres can be seen on the peg.
Hyperparameter
Pretraining (reposing)
Post-training (FMB)
PPO Actor learning rate
1×10−3
1×10−5
Actor LR schedule
KL-adaptive
linear decay over Max epochs
BC actor learning rate
-
1×10−4
Max epochs
750,000
200,000
Critic learning rate
5×10−5
5×10−5
PPO clip ϵ
0.20
0.05
Appendix
Table 5: PPO hyperparameter comparison between reposing pretraining and FMB post-training. The actor learning rate is reduced by 100× and the PPO clip is tightened by 4× during post-training, while the critic learning rate is held constant. The actor learning-rate schedule changes from KL-adaptive (pretraining, with target KL = Actor KL threshold) to linear decay over Max epochs (post-training); the Actor KL threshold is logged but not used during post-training under the linear schedule.
Stage 1 : reposing
Stage 2 : FMB
Stage 3 : Distillation
Observation field
Actor
Critic
Actor
Critic
Teacher (actor)
Student (actor)
Proprioceptive / robot state
robot_dof_pos (noisy)
23
23
23
23
23
23
robot_dof_vel (noisy)
23
23
23
23
23
23
hand_pos (5 bodies × 3)
15
15
15
15
15
15
hand_vel (5 bodies × 3)
15
15
15
15
15
15
Appendix
Table 6: Observation fields consumed at each pipeline stage in the current Cspace-mode codebase. Stage 1 = reposing pretraining (PPO, actor + asymmetric critic); Stage 2 = FMB post-training (PPO, actor + asymmetric critic); Stage 3 = two-camera RGB vision distillation, where a student network is supervised by the Stage 2 actor (treated as the teacher) via behavior cloning. Actor and student rows read the noisy variant of each proprioceptive field; the critic reads the corresponding clean variant. Each cell shows the dimensionality of the field when it is consumed at that stage; ✗ means the field is not consumed at that stage, and “image” denotes an image-tensor input. Rows newly introduced at Stage 2 (FMB extras / nominal delta) and Stage 3 (two-camera RGB) are highlighted.
Stage 1 : reposing
Stage 2 : FMB
Stage 3 : Distillation
Observation field
Actor
Critic
Actor
Critic
Teacher (actor)
Student (actor)
Proprioceptive / robot state
proprio (29 joints + 6 FK bodies)
94
94
94
94
94
94
last_actions
29
29
29
29
29
✗
fabric_q , qd , qdd ( 3×29 )
87
87
87
87
87
87
proprio_nominal_delta (29 joint + 18 FK)
✗
✗
✗
✗
✗
✗
Appendix
Table 7: Flexiv–Sharpa (29 DoF). Observation fields consumed by the Rizon–Sharpa Cspace pipeline. Stage 1 uses the reposing PBT actor and asymmetric critic; Stage 2 uses the FMB bootstrap teacher; Stage 3 distills that state-based teacher into the stereo RGB + five-fingertip TacMap student. The entries and dimensions follow the checked-in agent configurations. “Image” denotes a tensor encoded outside the flat low-dimensional input.
Figure 8: Training curves on FMB peg insertion comparing (1) training from scratch, (2) direct PPO finetuning of the pretrained reposing teacher, and (3) ADEPT post-training. Blue inset: success-rate collapse for (2) during transfer.
Figure 9: Training curves across different seeds for training from scratch. ADEPT post-training reliably converges to a solution while most seeds for training from scratch either plateau or take a very long time to train for a fixed compute budget.
Figure 10: Real-world camera views for (a) Kuka-Allegro and (b) Flexiv-Sharpa setups.
Figure 11: A sample 16x16 grid of visual randomizations on the left camera of the FMB training scene for distillation.
Parameter
Range / Distribution
Physics (per-env at reset, ADR-annealed)
Object mass scale
(0.5,3.0)× nominal
Object static friction
(0.5,1.2)
Object dynamic friction
(0.3,1.0)
Object restitution
(0.8,1.0)
Robot joint stiffness scale
(0.5,2.0)× default
Appendix
Table 8: Domain randomization parameters used during teacher–student distillation. Per-env ranges are sampled uniformly at episode reset; ADR-annealed ranges advance from the lower bound to the upper bound over training as the running success rate exceeds the ADR threshold; per-step noise is added at every control step; per-frame randomization is applied at each camera render.
Figure 12: Comparison of grasp behaviors. (a, b, c, d) Downstream-task teacher trained from scratch at ADR level 50, with grasps that lack the natural finger gaits emerging from reposing pretraining. (e, f) Pretrained reposing teacher applied zero-shot to FMB peg insertion at ADR level 20, occasionally producing undesirable grasps (e.g., bottom-end, upside-down) that solve the reposing segment but fail at insertion. (g, h) Downstream-task teacher trained with ADEPT at ADR level 50, with natural grasps where the fingers wrap around the peg and caging.
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.
Zijian Zeng, Fei Ding, Huiming Yang +2
Tsinghua University · Alibaba Group · Bengbu University +1
Dexterous manipulation requires both large-scale task progression and precise contact-rich interaction, making it challenging to collect demonstrations that effectively support both regimes. We present SkillWeave, a heterogeneous demonstration framework for long-horizon dexterous manipulation that combines teleoperation for coarse reaching and transport with kinesthetic teaching for precise, contact-rich skills. To address the visual mismatch introduced by the demonstrator's presence during kinesthetic data collection, we propose an object-mask-conditioned diffusion policy that uses offline object segmentation for training supervision and a lightweight learned mask predictor at deployment, avoiding online segmentation and image inpainting. To mitigate distribution shift between independently trained sub-task policies, we introduce successor-aware terminal steering, which selects among actions sampled from the predecessor policy to guide the system toward states supported by the successor's demonstrated initial-state distribution. Across three real-world long-horizon tasks, SkillWeave achieves 27% average end-to-end success. Mask-conditioned kinesthetic policies improve dexterous sub-task success to an average of 65%, while successor-aware handoffs achieve an average composition efficiency of 87%. These results show that matching demonstration modality to interaction regime, explicitly addressing kinesthetic visual mismatch, and steering policy handoffs toward successor-supported states substantially improves long-horizon dexterous manipulation. Videos and code are available at skillweave-authors.github.io .
Ryosei Tamura, Xiaoxiang Dong, Uksang Yoo +4
Robotics Institute, Carnegie Mellon University, USA. · Department of Computer Science, Keio University, Japan. · College of Engineering, University of California, Berkeley, USA. +1