Sim-to-real RL has achieved impressive results in dexterous manipulation but remains limited to tasks with accurate rigid-body simulation. However, these methods cannot be applied to skills that involve deformable and particles, with which the simulation speed will become too slow for large-scale RL learning. We present DAPPT, a sim-to-real framework for end-to-end dexterous tool-use manipulation skills, e.g., sand scooping, play-dough rod-rolling, without simulating non-rigid materials. Our method learns a domain-aware RL policy by tracking procedural trajectories parameterized by domain parameters, and distills it into an end-to-end domain-aware sensorimotor policy. Furthermore, explicit domain represen-tation enables fast selection of domain parameters that can maximize downstream task performance. We evaluate DAPPT end-to-end on six dexterous manipulation tasks, including sand scooping, play-dough rod-rolling, sand scraping, mouse-wheel scrolling, dispenser pressing, and spray triggering. Our method achieves 1.5-3.8x in task metrics through the best domain parameter and transfers zero-shot to the real world.
Figures & tables
Fig. 1 : DAPPT learns dexterous skills on the 19-dof hand-arm system in rigid-body simulation and zero-shot transfer to the real-world. We evaluate on 6 dexterous manipulation tasks. The left 3 skills are scraping sand, rod-rolling play-dough, and scooping sand. The right 3 are manipulating articulated tools: mouse-wheel rolling, dispenser pressing, and spray triggering.
Fig. 2 : Illustration of domain-aware policy π(p;o;c) and an example of the domain parameters c={α,d,vi,vl} of the sand scooping task. We visualize the geometric meanings to the procedural trajectory of the scoop.
Scrape ↓
Rod-Roll ↑
Scoop ↑
Genesis Best (Ours)
17%
2.85 cm
13.9 g
SimToolReal [ 12 ]
95%
0 cm
9.2 g
Mouse-Wheel ↑
Dispenser ↑
Spray ↑
Genesis Best (Ours)
0.37 rad
# 2.26k
# 9.12k
SimToolReal [ 12 ]
0.19 rad
# 0
# 0
TABLE I : Comparison of the task metrics between Genesis Best of our domain-aware policy and the policy from SimToolReal [ 12 ] .
Fig. 3 : The percentage of difference of Genesis Best over Genesis Avg , in terms of the task metric and the average per-step tracking error (poses / joint values).
Mouse-Wheel
Dispenser
Scoop
Domain-Conditioned
0.37 rad
# 2.26k
13.9 g
Fixed-Domain
0.19 rad
# 0.96k
0.32 g
Performance Drop
48.6%
57.5%
97.7%
TABLE II : We compare the Genesis Best with our domain aware policy v.s. a policy trained on the fixed Genesis Best domain parameters. There is a significant drop in task metric.
Fig. 4 : Ablation study on without using the tele-op demo (w/o demo) or without using the contact trajectory as the referenced learning rewards (w/o contact). We show the average pose / joint tracking error during the tracking stage at the end of the RL training, clamped at a maximal of 0.2 (m/rad).
Motion Success
Task Success
Task Metric
Scooping
20/20
14/20
7.7 g
Rod-Rolling
16/20
11/20
2.7 cm
Scraping
19/20
19/20
29%
Mouse-Wheel
19/20
19/20
0.33 rad
Dispenser
18/20
14/20
N/A
Spray
16/20
10/20
N/A
TABLE III : Real-world experiment of all 6 dexterous tasks. We report motion success, task success and task metrics over task success trials.
Fig. 5 : Qualitative examples on the different between rollouts with the Genesis Best domain parameter and some other parameter (Section V-C ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Genesis Avg
Genesis Best
Tracking Err
Task Metric
Tracking Err
Task Metric
Scooping
5.1 cm
3.64 g
4.1 cm
13.9 g
Rod-Rolling
5.6 cm
1.55 cm
4.3 cm
2.85 cm
Scraping
3.8 cm
33%
3.2 cm
17%
Mouse-wheel
0.85 rad
0.23 rad
0.85 rad
0.37 rad
Dispenser
6.7 mm
# 0.97k
7.2 mm
# 2.26k
Appendix
TABLE IV : We show the Tracking Error of average pose distance (Scooping, Rod-Rolling, Scraping) / joint values (Dispenser, Mouse-Wheel, Spray) with the target procedural trajectory and the Task Metric in simulation evaluation
Fig. 6 : Testing our arm controller calibration. We compare the trajectory with calibrated Kd s, original parameters in simulation and with the real-robot trajectory, under the same open-loop command sequence.
Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.
Zijian Zeng, Fei Ding, Huiming Yang +2
Tsinghua University · Alibaba Group · Bengbu University +1
The ability to manipulate tools significantly expands the set of tasks a robot can perform. Yet, tool manipulation represents a challenging class of dexterity, requiring grasping thin objects, in-hand object rotations, and forceful interactions. Since collecting teleoperation data for these behaviors is challenging, sim-to-real reinforcement learning (RL) is a promising alternative. However, prior approaches typically require substantial engineering effort to model objects and tune reward functions for each task. In this work, we propose SimToolReal, taking a step towards generalizing sim-to-real RL policies for tool manipulation. Instead of focusing on a single object and task, we procedurally generate a large variety of tool-like object primitives in simulation and train a single RL policy with the universal goal of manipulating each object to random goal poses. This approach enables SimToolReal to perform general dexterous tool manipulation at test-time without any object or task-specific training. We demonstrate that SimToolReal outperforms prior retargeting and fixed-grasp methods by 37% while matching the performance of specialist RL policies trained on specific target objects and tasks. Finally, we show that SimToolReal generalizes across a diverse set of everyday tools, achieving strong zero-shot performance over 120 real-world rollouts spanning 24 tasks, 12 object instances, and 6 tool categories.
Learning generalist policies capable of accomplishing a wide range of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of transforming a rigid object from a current pose to a target pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework. Project page: https://dex4d.github.io/.
Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki +1