Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation

Yuxuan Kuang; Sungjae Park; Katerina Fragkiadaki; Shubham Tulsiani

cs.ROFeb 17, 2026

Dex4D: Task-Agnostic Point Track Policy for Zero-Shot Dexterous Manipulation

Authors: Yuxuan Kuang, Sungjae Park, Katerina Fragkiadaki, Shubham Tulsiani

Organizations: Carnegie Mellon University

Abstract

Learning generalist policies capable of accomplishing a wide range of everyday tasks remains an open challenge in dexterous manipulation. In particular, collecting large-scale manipulation data via real-world teleoperation is expensive and difficult to scale. While learning in simulation provides a feasible alternative, designing multiple task-specific environments and rewards for training is similarly challenging. We propose Dex4D, a framework that instead leverages simulation for learning task-agnostic dexterous skills that can be flexibly recomposed to perform diverse real-world manipulation tasks. Specifically, Dex4D learns a domain-agnostic 3D point track conditioned policy capable of transforming a rigid object from a current pose to a target pose. We train this 'Anypose-to-Anypose' policy in simulation across thousands of objects with diverse pose configurations, covering a broad space of robot-object interactions that can be composed at test time. At deployment, this policy can be zero-shot transferred to real-world tasks without finetuning, simply by prompting it with desired object-centric point tracks extracted from generated videos. During execution, Dex4D uses online point tracking for closed-loop perception and control. Extensive experiments in simulation and on real robots show that our method enables zero-shot deployment for diverse dexterous manipulation tasks and yields consistent improvements over prior baselines. Furthermore, we demonstrate strong generalization to novel objects, scene layouts, backgrounds, and trajectories, highlighting the robustness and scalability of the proposed framework. Project page: https://dex4d.github.io/.

Figures & tables

Figure 1: Overview of Dex4D . We leverage video generation and 4D reconstruction to generate object-centric point tracks . Conditioned on the point tracks, we use a task-agnostic sim-to-real policy trained via Anypose-to-Anypose for task execution. Our policy trained entirely in simulation can be seamlessly deployed in the real world zero-shot and generalizes to diverse configurations.

Figure 2: Overview of our Dex4D teacher and student network architectures. (a) We first learn a teacher policy via RL with privileged states and full points sampled on the whole object, leveraging our proposed Paired Point Encoding representation. (b) Given partial observation, i.e., robot proprioception, last action, and masked paired points, we distill the teacher to learn a transformer-based student dynamics action model that jointly predicts actions and future robot states.

Figure 3: Comparison between our Paired Point Encoding with other representations. Our point features keep correspondence and permutation-invariance , boosting performance for policy learning.

Method	CL	Apple2Plate		Pour		Hammer		StackCup		RotateBox		Sponge2Bowl		Avg.
Method	CL	SR	TP	SR	TP	SR	TP	SR	TP	SR	TP	SR	TP	SR	TP
NovaFlow [ 44 ]	✗	0.44	0.57	0.66	0.76	0.12	0.16	0.42	0.59	0.25	0.32	0.18	0.29	0.345	0.448
LVP [ 45 ]	✗	0.26	0.64	0.39	0.42	0.00	0.00	0.10	0.61	0.00	0.16	0.11	0.41	0.143	0.373
NovaFlow-CL [ 44 ]	✓	0.58	0.76	0.73	0.85	0.27	0.30	0.44	0.73	0.33	0.60	0.27	0.41	0.437	0.608
Dex4D (Ours)	✓	0.86	0.91	0.86	0.92	0.28	0.31	0.60	0.82	0.44	0.69	0.56	0.62	0.600	0.712

Table 1: Quantitative results measured by Success Rate (SR) and Task Progress (TP) in simulation. CL stands for closed-loop. All tasks are zero-shot . We adapt NovaFlow and NovaFlow-CL to dexterous hands by using our method for first-stage dexterous grasping.

Method	SR	TP
MLP Point Encoding	0.057	0.172
Decoupled Point Encoding	0.203	0.363
w/o Self-Attention	0.490	0.675
w/o Dynamics Modeling	0.570	0.683
Dex4D (Ours)	0.600	0.712

Table 2: Ablation studies on the student policy. SR and TP are task-averaged.

Figure 4: Our point track policy is robust to different degrees of point loss.

Method	LiftToy	Broccoli2Plate	Meat2Bowl	Pour	Total
LVP [ 45 ]	9/20	6/20	7/20	0/20	22/80
NovaFlow-CL [ 44 ]	10/20	7/20	7/20	0/20	24/80
$\pi_{0.5}$ -DROID* [ 58 ]	5/20	6/20	9/20	0/20	20/80
Dex4D (Ours)	14/20	10/20	13/20	8/20	45/80

Table 3: Quantitative results measured by Success Rate (SR) in the real world. All tasks are zero-shot , all objects are unseen and there are no real robot demonstrations. * denotes using parallel grippers.

Figure 5: Failure analysis of our method.

Figure 6: Our Dex4D can achieve a wide variety of open-world tasks in a zero-shot manner, generalizing across tasks, objects, scenes, and external perturbations.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Overview of simulated tasks. Two frames are shown in each column for each task.

Figure 8: Overview of our real-world benchmarking suites. Two frames are shown in each column for each task.

Figure 9: Qualitative comparison between our method and the NovaFlow-CL baseline. The baseline suffers from object dropping and inaccurate post-grasping movement due to the lack of hand feedback and vulnerability to few and noisy visible points , while our method performs robustly.

Figure 10: Mean reward curve of the first two stages of teacher training. Step 15k is the curriculum boundary. Our method outperforms both ablation variants.

Figure 11: Comparison between metric depth estimation and relative depth estimation. Relative depth estimation yields smoother, more spatio-temporally consistent results and fewer floater points.

	Stage 1	Stage 2	Stage 3
Step	0-15k	15k-25k	25k-50k
Object Category	Bottle	Bottle	All
Too Far Reset Threshold (m)	0.3	$1\times 10^{6}$	$1\times 10^{6}$
Goal Reset Stable Ratio	0.1	0.2	0.2
Arm Speed Scale	10	1.5	1.5
Control Frequency (Hz)	30	30	5

Appendix

Table 4: Curriculum settings of our three-stage RL teacher training.

Term	Expression	Weight (Stage 1 & 2)	Weight (Stage 3)
$r_{\mathrm{goal}}$	$\mathbbm{1}_{\mathrm{contact}}(1.4-3d_{t})$	1.0	1.0
$r_{\mathrm{f,o}}$	$-0.5\times\sum_{j=1}^{4}d(p_{j}^{\mathrm{finger}},p^{\mathrm{obj}})$	1.0	0.0
$r_{\mathrm{h,o}}$	$-0.5\times d(p^{\mathrm{hand}},p^{\mathrm{obj}})$	1.0	0.0
$r_{\mathrm{bonus}}$	$\mathbbm{1}_{\mathrm{contact}}\mathbbm{1}_{\mathrm{success}}\frac{5.0}{1+10d_{t}}$ $+\mathbbm{1}_{\mathrm{stay\_success}}\times 10$	1.0	1.0
$r_{\mathrm{curl}}$	$-0.001\lVert\bm{\theta}_{t}\rVert_{2}^{2}$	1.0	10.0
$r_{\mathrm{table}}$	$\min(10\min_{j}(h^{\mathrm{finger}}_{j}-0.62),0)$	1.0	1.0

Appendix

Table 5: Reward shaping for RL teacher training in different stages.

Term	Operation	Distribution	Range
Observation White Noise	Additive	Gaussian	$[0,0.002]$
Observation Correlated Noise	Additive	Gaussian	$[0,0.001]$
Action White Noise	Additive	Gaussian	$[0,0.05]$
Action Correlated Noise	Additive	Gaussian	$[0,0.015]$
Gravity	Additive	Gaussian	$[0,0.4]$
Joint Stiffness	Scaling	Uniform	$[0.9,1.1]$

Appendix

Table 6: Domain Randomization Setup.

Figure 12: Our Dex4D policy shows dexterity on in-hand and non-prehensile manipulation.

Figure 13: With the same simulation setting (reward, curriculum, etc.), the training can be seamlessly transferred to the XHand.

Figure 14: Our training is stable with different seeds.

Hyperparameter	Value
Algorithm	PPO
Optimizer	Adam
Number of Environments	4096
Number of Object Points	128
Discount Factor ( $\lambda$ )	0.95
GAE parameter ( $\gamma$ )	0.96

Appendix

Table 7: Hyperparameters of the Teacher Policy.

Hyperparameter	Value
Algorithm	DAgger
Optimizer	Adam
Number of Environments	3200
Number of Object Points	64
Learning Rate	$1\times 10^{-4}$
Token Dimension	128

Appendix

Table 8: Hyperparameters of the Student Policy.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
May 3, 2026Zijian Zeng, Fei Ding, Huiming Yang +2Domain Randomization Dexterous Manipulation
Tsinghua University · Alibaba Group · Bengbu University +1
DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real
Sep 17, 2026Jin Wu, Lianjie Yuan, Zeyan Sun +4Sim-to-Real Transfer Synthetic Data Generation for Robotics
The University of Texas at Austin · University of Florida · BrainCo
Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction
Sep 7, 2026Ruoqu Chen, Feixiang Ruan, Liu Cao +9Sim-to-Real Transfer Robotic Manipulation
Tsinghua University · Shanghai Qizhi Institute · Sharpa +2

May 3, 2026J/K move · Enter open · S save

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

Zijian Zeng, Fei Ding, Huiming Yang, Xianwei Li+1

Tsinghua University · Alibaba Group · Bengbu University+1

Domain Randomization Dexterous Manipulation Sim-to-Real Transfer Generalization in Robotic Manipulation Robotic RL VLMs for Robotics

Open paper AskPDF arXiv

Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually designed domain randomization or task-specific adaptation, limiting their generalizability across diverse manipulation scenarios. We present DexSim2Real, an integrated framework that leverages vision-language foundation models to bridge the sim-to-real gap for dexterous manipulation. Our system combines three components: (1) Foundation Model-Guided Domain Randomization (FM-DR), which uses a vision-language model as a visual realism critic to optimize simulation parameters via closed-loop CMA-ES, complementing text-based approaches like DrEureka with direct visual feedback; (2) a Tactile-Visual Cross-Attention Policy (TVCAP) that adapts cross-attention visuo-tactile fusion to zero-shot sim-to-real RL; and (3) a Progressive Skill Curriculum (PSC) that builds on LLM-based task decomposition with a difficulty scheduler tailored to contact-rich dexterous tasks. Extensive experiments on six challenging manipulation tasks with blinded evaluation demonstrate that DexSim2Real achieves a 78.2% average real-world success rate, outperforming DrEureka and DeXtreme while reducing the sim-to-real performance gap to only 8.3%.

Abstract

Robotic Manipulation

Explore similar work

DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation

DEXTERA: From a Single Image to Deployable Dexterous Manipulation via Real-to-Sim-to-Real

Dex-X: Learning Visual-Tactile Dexterous Manipulation From Human Videos with Simulated Interaction