Learning generalist policies capable of accomplishing a wide range of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of transforming a rigid object from a current pose to a target pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework. Project page: https://dex4d.github.io/.
Figures & tables
Figure 1: Overview of Dex4D . We leverage video generation and 4D reconstruction to generate object-centric point tracks . Conditioned on the point tracks, we use a task-agnostic sim-to-real policy trained via Anypose-to-Anypose for task execution. Our policy trained entirely in simulation can be seamlessly deployed in the real world zero-shot and generalizes to diverse configurations.
Figure 2: Overview of our Dex4D teacher and student network architectures. (a) We first learn a teacher policy via RL with privileged states and full points sampled on the whole object, leveraging our proposed Paired Point Encoding representation. (b) Given partial observation, i.e., robot proprioception, last action, and masked paired points, we distill the teacher to learn a transformer-based student dynamics action model that jointly predicts actions and future robot states.
Figure 3: Comparison between our Paired Point Encoding with other representations. Our point features keep correspondence and permutation-invariance , boosting performance for policy learning.
Method
CL
Apple2Plate
Pour
Hammer
StackCup
RotateBox
Sponge2Bowl
Avg.
SR
TP
SR
TP
SR
TP
SR
TP
SR
TP
SR
TP
SR
TP
NovaFlow [ 44 ]
✗
0.44
0.57
0.66
0.76
0.12
0.16
0.42
0.59
0.25
0.32
0.18
0.29
0.345
0.448
LVP [ 45 ]
✗
0.26
0.64
0.39
0.42
0.00
0.00
0.10
0.61
0.00
0.16
0.11
0.41
0.143
0.373
NovaFlow-CL [ 44 ]
✓
0.58
0.76
0.73
0.85
0.27
0.30
0.44
0.73
0.33
0.60
0.27
0.41
0.437
0.608
Dex4D (Ours)
✓
0.86
0.91
0.86
0.92
0.28
0.31
0.60
0.82
0.44
0.69
0.56
0.62
0.600
0.712
Table 1: Quantitative results measured by Success Rate (SR) and Task Progress (TP) in simulation. CL stands for closed-loop. All tasks are zero-shot . We adapt NovaFlow and NovaFlow-CL to dexterous hands by using our method for first-stage dexterous grasping.
Method
SR
TP
MLP Point Encoding
0.057
0.172
Decoupled Point Encoding
0.203
0.363
w/o Self-Attention
0.490
0.675
w/o Dynamics Modeling
0.570
0.683
Dex4D (Ours)
0.600
0.712
Table 2: Ablation studies on the student policy. SR and TP are task-averaged.
Figure 4: Our point track policy is robust to different degrees of point loss.
Method
LiftToy
Broccoli2Plate
Meat2Bowl
Pour
Total
LVP [ 45 ]
9/20
6/20
7/20
0/20
22/80
NovaFlow-CL [ 44 ]
10/20
7/20
7/20
0/20
24/80
π0.5 -DROID* [ 58 ]
5/20
6/20
9/20
0/20
20/80
Dex4D (Ours)
14/20
10/20
13/20
8/20
45/80
Table 3: Quantitative results measured by Success Rate (SR) in the real world. All tasks are zero-shot , all objects are unseen and there are no real robot demonstrations. * denotes using parallel grippers.
Figure 5: Failure analysis of our method.
Figure 6: Our Dex4D can achieve a wide variety of open-world tasks in a zero-shot manner, generalizing across tasks, objects, scenes, and external perturbations.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Overview of simulated tasks. Two frames are shown in each column for each task.
Figure 8: Overview of our real-world benchmarking suites. Two frames are shown in each column for each task.
Figure 9: Qualitative comparison between our method and the NovaFlow-CL baseline. The baseline suffers from object dropping and inaccurate post-grasping movement due to the lack of hand feedback and vulnerability to few and noisy visible points , while our method performs robustly.
Figure 10: Mean reward curve of the first two stages of teacher training. Step 15k is the curriculum boundary. Our method outperforms both ablation variants.
Figure 11: Comparison between metric depth estimation and relative depth estimation. Relative depth estimation yields smoother, more spatio-temporally consistent results and fewer floater points.
Stage 1
Stage 2
Stage 3
Step
0-15k
15k-25k
25k-50k
Object Category
Bottle
Bottle
All
Too Far Reset Threshold (m)
0.3
1×106
1×106
Goal Reset Stable Ratio
0.1
0.2
0.2
Arm Speed Scale
10
1.5
1.5
Control Frequency (Hz)
30
30
5
Appendix
Table 4: Curriculum settings of our three-stage RL teacher training.
Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.
Zijian Zeng, Fei Ding, Huiming Yang +2
Tsinghua University · Alibaba Group · Bengbu University +1
Collecting real-world robot data for dexterous manipulation is costly and time-consuming. While high-fidelity physics simulators enable scalable data synthesis and policy learning, constructing deployment-ready digital twins manually remains labor-intensive, and residual visual, geometric, and dynamics gaps hinder reliable sim-to-real transfer. We present DEXTERA, an automated real-to-sim-to-real framework that transforms a single RGB image into deployable policies for dexterous manipulation across four unified stages: (1) single-image scene factorization into a static Gaussian background and interactive rigid or articulated assets with VLM-inferred physical parameters; (2) metric scene global alignment, object canonicalization, and morphology-balanced robot calibration; (3) scalable simulator task primitive construction, VR teleoperation, and object-centric trajectory synthesis; and (4) a shared multimodal policy interface supporting both imitation learning and reinforcement learning. We evaluate DEXTERA across 13 task-embodiment pairs, 2 dexterous robot platforms, and 6 policy architectures. Experimental results demonstrate that DEXTERA achieves superior visual fidelity and 3D geometric reconstruction compared to generative baselines, while cross-domain trajectory replays validate strong physical interaction consistency. Furthermore, simulation-only trained policies enable viable zero-shot real-robot deployment, while simulation-real co-training substantially improves mean physical policy success from 29.2% to 61.9% across diverse policy architectures. Project website: https://dextera-project.github.io/
Jin Wu, Lianjie Yuan, Zeyan Sun +4
The University of Texas at Austin · University of Florida · BrainCo
Human videos are an abundant source of dexterous manipulation behaviors, but they lack tactile information that is crucial for contact-rich interaction. This raises a fundamental question: can robots learn deployable visual-tactile dexterous manipulation policies from human video demonstrations without robot-side data collection? We present DEX-X, a framework for learning visual-tactile dexterous manipulation from human videos through simulation. Our key insight is that simulation can serve as a tactile completion engine. Given monocular human demonstrations, DEX-X reconstructs hand-object interactions in simulation, where physically grounded contact dynamics provide tactile supervision unavailable in the original videos. Leveraging this recovered tactile information, we train visual-tactile dexterous manipulation policies and distill them into deployable policies operating on point-cloud observations and tactile sensing. We demonstrate zero-shot sim-to-real transfer on a dexterous hand-arm platform across diverse grasping and contact-rich tool-use tasks. The teacher policy achieves 65.9% average success across six task categories in simulation, while the distilled visual-tactile policy achieves 93% success on real-world cube picking and 53% on the challenging table-cleaning task. Zero-shot generalization to unseen object geometries is also observed on object-picking tasks. Our results suggest that simulated interaction is a key bridge between human videos and deployable dexterous manipulation policies, providing the missing physical supervision needed for scalable robot skill learning from Internet-scale human video data.
Ruoqu Chen, Feixiang Ruan, Liu Cao +9
Tsinghua University · Shanghai Qizhi Institute · Sharpa +2