AdaDexTrack: Dynamic Modulation for Adaptive and Generalizable Dexterous Manipulation Tracking

Jianibieke Adalibieke; Qianwei Han; Xueyi Liu; Yuzhe Qin; Li Yi

cs.ROOct 24, 2026

AdaDexTrack: Dynamic Modulation for Adaptive and Generalizable Dexterous Manipulation Tracking

Authors: Jianibieke Adalibieke, Qianwei Han, Xueyi Liu, Yuzhe Qin, Li Yi

Organizations: Shanghai Qi Zhi Institute · Tsinghua University · UC San Diego

Abstract

Language is a natural way to command robots, but converting a single instruction into a long-horizon, contact-rich hand-object interaction remains challenging: synthesized references are noisy, human-to-robot retargeting introduces embodiment bias, and fixed-reference tracking lets small errors snowball. We address this with AdaDexTrack, a modulator-in-the-loop framework for language-conditioned manipulation tracking. A distilled generalist tracker serves as the skill carrier, while a tightly aligned modulator performs three feedback corrections: reference modulation (continual adjustment of what to track), object-latent modulation (online adaptation of the object representation to recruit suitable skills), and positional-target modulation (small state-dependent refinements for execution). The tracker is learned via large-scale specialist to generalist distillation on a corpus of language-conditioned hand-object trajectories; the modulator is trained with RL under the same task objective, ensuring tight coupling. Across large-scale evaluations, AdaDexTrack consistently outperforms prior SOTA on unseen-trajectory and unseen-object sets in both average tracking error and success rate, demonstrating robustness and generalization. We further show zero-shot sim-to-real transfer on real hardware, where adding the modulator yields substantial gains over a tracker-only variant. AdaDexTrack reframes language-conditioned dexterous manipulation as modulated tracking, replacing the open-loop, fixed-reference tracking with in-loop modulation that adjusts the reference, object latent, and positional target, yielding drift-resistant execution from noisy text references.

Figures & tables

Figure 2: AdaDexTrack pipeline overview. Left: From a large-scale set of text-guided HOI references, we train many RL teacher trackers and distill their behaviors into a single generalist tracker. Right: On top of this tracker, we insert a modulator in the loop that performs reference modulation, object latent modulation, and positional target modulation, yielding drift-resistant, accurate long-horizon tracking.

Dataset	Method	$T_{\mathrm{o}}$ (cm, $\downarrow$ )	$R_{\mathrm{o}}$ (rad, $\downarrow$ )	$T_{\mathrm{h}}$ (cm, $\downarrow$ )	$R_{\mathrm{h}}$ (rad, $\downarrow$ )	$E_{\mathrm{finger}}$ (rad, $\downarrow$ )	Succ. (%, $\uparrow$ )
Unseen Trajectory	ObjDex	4.95	0.6950	8.02	0.6296	0.6872	51.72/44.59
	DexTrack	6.93	0.7046	9.17	0.5742	0.2823	76.13/73.89
	Vanilla RL (PPO)	12.10	0.8992	13.30	0.9142	0.3595	43.03/53.94
	Ours (General Tracker)	11.57	0.7607	10.33	0.5501	0.2791	62.20/65.01
	Ours (General Tracker+R)	10.06	0.7445	9.76	0.5408	0.2702	67.09/68.14
	Ours (General Tracker+R+T)	4.68	0.6272	7.99	0.5062	0.2413	84.45 / 80.42

Table 1: Quantitative results. Best and second-best entries are highlighted in bold red and italic blue , respectively. Ablations are denoted as “Ours (General Tracker+R)”, “Ours (General Tracker+R+T)”, “Ours (General Tracker+R+O+T)”, where R = reference modulation, O = object latent modulation, and T = positional target modulation. Success rate reported as Mean-error/Completion.

Figure 3: Qualitative results. Panels (a–d): given a noisy text-generated reference retargeted to the robot, AdaDexTrack successfully tracks the long-horizon interaction, while the tracker-only baseline fails. This is because the in-loop modulator continuously adapts the reference, object latent, and positional target, mitigating sim-to-real noise and preventing error accumulation, whereas the tracker-only baseline accumulates errors over time.

Figure 4: Reference modulation (General Tracker+R). Ref. : original object reference; Mod. Ref. : reference after modulation by the reference modulator; Track. Traj. : executed object trajectory.

Figure 5: Object latent modulation (General Tracker+O). Red : actual object and its pose. Blue : point cloud decoded from the modulated object latent and its pose. (a) Hammer; (b) Stamp.

Panel of Figure 6 — Figure 6: Effect of scaling the amount of reference data.

Methods	Unseen Traj	Unseen Obj
Ours (w/o modulator )	26.63%	22.27%
Ours	52.36%	36.71%

Table 2: Zero-shot sim-to-real experiments. Completion rate (%).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Language-guided HOI reference augmentation in 3D and top-down XY views.

Figure 8: System identification of the arm joints. For each joint, we plot three traces: (1) the commanded input, (2) the simulated response using the identified dynamics model, and (3) the measured real response. The close agreement between (2) and (3) indicates that the identified parameters faithfully capture the joint dynamics.

Parameter	Type	Distribution	Initial Range
Robot
Mass	Scaling	uniform	$[0.9,1.1]$
Friction	Scaling	uniform	$[0.9,1.1]$
Joint stiffness	Scaling	uniform	$[0.9,1.1]$
Joint damping	Scaling	uniform	$[0.9,1.1]$
Object

Appendix

Table 3: Domain randomization configuration.

Index	Description
0 – 22	arm-hand joint positions
22 – 57	the palm pose and the poses of the four fingertips
57 – 64	object pose
64 – 71	distance between the current object pose and the object reference pose at the current timestep
71 – 99	object reference poses at future timesteps $\{1,2,4,12\}$ relative to the current timestep
99 – 121	distance between the current and reference arm–hand joint positions at the current timestep

Appendix

Table 4: Observation space used for specialist policy training.

Index	Description
0 – 22	arm-hand joint positions
22 – 57	the palm pose and the poses of the four fingertips
57 – 64	object pose
64 – 71	distance between the current object pose and the object reference pose at the current timestep
71 – 141	object reference poses at future ten timesteps relative to the current timestep
141 – 147	distance between the current and reference arm joint positions at the current timestep

Appendix

Table 5: Observation space for the low-level controller in ObjDex.

Figure 9: Real-world setup. (a) Hardware. (b) Test objects. The blue box marks objects used for the real-world unseen-trajectory set; the orange box marks objects used for the real-world unseen-object set.

Figure 10: FoundationPose estimation behavior. (a) Wineglass: although the object is static, the estimate continuously spins around the blue symmetry axis due to axial symmetry. (b) Cube: despite discrete symmetry, the pose remains stable (no continuous spin). (c) Textured wineglass: asymmetric texture breaks the symmetry and yields a stable object frame.

Figure 11: Eight textured axially symmetric objects used in our real-world experiments.

Method	Completion (%, $\uparrow$ )
Ours (w/o Modulator)	29.72%
Ours	58.61%

Appendix

Table 6: Real-world results on textured axially symmetric objects.

Method	Unseen Traj.	Unseen Obj.
Ours (w/o Modulator)	36.42%	18.75%
Ours	65.71%	32.50%

Appendix

Table 7: Real-world results without grip tape on the objects.

Input	Method	Unseen Traj.	Unseen Obj.
GT Pointcloud	Ours	65.71%	32.50%
Recon. Pointcloud	Ours	59.28%	34.73%
Partial Pointcloud	Ours	49.46%	25.72%

Appendix

Table 8: Real-world results under practical object-geometry inputs. GT Pointcloud uses ground-truth object geometry; Recon. Pointcloud is reconstructed from a single RGB image using InstantMesh [ 48 ] ; Partial Pointcloud is obtained from segmented single-view depth.

Figure 12: Lipschitz-style perturbation analysis of object latent modulation. The response ratio

\rho=\|\Delta a\|_{2}/\epsilon

remains bounded under small latent perturbations, supporting a locally smooth latent-to-action mapping and meaningful interpolation between object anchors.

Figure 13: Qualitative results in simulation.

Figure 14: Additional qualitative results in real-world.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 239 papers

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking
Oct 20, 2026Yuxuan Kuang, Jiashun Wang, Yixuan Sun +5Robot Skill Learning Robotic Manipulation
Carnegie Mellon University · NVIDIA
ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control
Jun 2, 2026Yutong Liang, Quanquan Peng, Ri-Zhao Qiu +1Robot Trajectory Tracking Long-Horizon Robotic Manipulation
University of California San Diego
AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning
Aug 14, 2026Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5Robotic Manipulation Learning from Demonstration
Zhejiang University · Shanghai Innovation Institute · Fudan University +2

Oct 20, 2026J/K move · Enter open · S save

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

Yuxuan Kuang, Jiashun Wang, Yixuan Sun, Yufei Ye+4

Carnegie Mellon University · NVIDIA

Robot Skill Learning Robotic Manipulation RL for Robotics Goal-Conditioned Robotic Manipulation Human-to-Robot Motion Imitation

Open paper AskPDF arXiv

Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

Learning Adaptable Dexterous Skills via Hybrid Dexterous Mimicking

ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control

AdvDex: Learning Dexterous Manipulation from Human Demonstrations via Joint-Aligned Actions and Adversarial Learning