Language is a natural way to command robots, but converting a single instruction into a long-horizon, contact-rich hand-object interaction remains challenging: synthesized references are noisy, human-to-robot retargeting introduces embodiment bias, and fixed-reference tracking lets small errors snowball. We address this with AdaDexTrack, a modulator-in-the-loop framework for language-conditioned manipulation tracking. A distilled generalist tracker serves as the skill carrier, while a tightly aligned modulator performs three feedback corrections: reference modulation (continual adjustment of what to track), object-latent modulation (online adaptation of the object representation to recruit suitable skills), and positional-target modulation (small state-dependent refinements for execution). The tracker is learned via large-scale specialist to generalist distillation on a corpus of language-conditioned hand-object trajectories; the modulator is trained with RL under the same task objective, ensuring tight coupling. Across large-scale evaluations, AdaDexTrack consistently outperforms prior SOTA on unseen-trajectory and unseen-object sets in both average tracking error and success rate, demonstrating robustness and generalization. We further show zero-shot sim-to-real transfer on real hardware, where adding the modulator yields substantial gains over a tracker-only variant. AdaDexTrack reframes language-conditioned dexterous manipulation as modulated tracking, replacing the open-loop, fixed-reference tracking with in-loop modulation that adjusts the reference, object latent, and positional target, yielding drift-resistant execution from noisy text references.
Figures & tables
Figure 2: AdaDexTrack pipeline overview. Left: From a large-scale set of text-guided HOI references, we train many RL teacher trackers and distill their behaviors into a single generalist tracker. Right: On top of this tracker, we insert a modulator in the loop that performs reference modulation, object latent modulation, and positional target modulation, yielding drift-resistant, accurate long-horizon tracking.
Dataset
Method
To (cm, ↓ )
Ro (rad, ↓ )
Th (cm, ↓ )
Rh (rad, ↓ )
Efinger (rad, ↓ )
Succ. (%, ↑ )
Unseen Trajectory
ObjDex
4.95
0.6950
8.02
0.6296
0.6872
51.72/44.59
DexTrack
6.93
0.7046
9.17
0.5742
0.2823
76.13/73.89
Vanilla RL (PPO)
12.10
0.8992
13.30
0.9142
0.3595
43.03/53.94
Ours (General Tracker)
11.57
0.7607
10.33
0.5501
0.2791
62.20/65.01
Ours (General Tracker+R)
10.06
0.7445
9.76
0.5408
0.2702
67.09/68.14
Ours (General Tracker+R+T)
4.68
0.6272
7.99
0.5062
0.2413
84.45 / 80.42
Table 1: Quantitative results. Best and second-best entries are highlighted in bold red and italic blue , respectively. Ablations are denoted as “Ours (General Tracker+R)”, “Ours (General Tracker+R+T)”, “Ours (General Tracker+R+O+T)”, where R = reference modulation, O = object latent modulation, and T = positional target modulation. Success rate reported as Mean-error/Completion.
Figure 3: Qualitative results. Panels (a–d): given a noisy text-generated reference retargeted to the robot, AdaDexTrack successfully tracks the long-horizon interaction, while the tracker-only baseline fails. This is because the in-loop modulator continuously adapts the reference, object latent, and positional target, mitigating sim-to-real noise and preventing error accumulation, whereas the tracker-only baseline accumulates errors over time.
Figure 4: Reference modulation (General Tracker+R). Ref. : original object reference; Mod. Ref. : reference after modulation by the reference modulator; Track. Traj. : executed object trajectory.
Figure 5: Object latent modulation (General Tracker+O). Red : actual object and its pose. Blue : point cloud decoded from the modulated object latent and its pose. (a) Hammer; (b) Stamp.
Figure 6: Effect of scaling the amount of reference data.
Figure 7: Language-guided HOI reference augmentation in 3D and top-down XY views.
Figure 8: System identification of the arm joints. For each joint, we plot three traces: (1) the commanded input, (2) the simulated response using the identified dynamics model, and (3) the measured real response. The close agreement between (2) and (3) indicates that the identified parameters faithfully capture the joint dynamics.
Parameter
Type
Distribution
Initial Range
Robot
Mass
Scaling
uniform
[0.9,1.1]
Friction
Scaling
uniform
[0.9,1.1]
Joint stiffness
Scaling
uniform
[0.9,1.1]
Joint damping
Scaling
uniform
[0.9,1.1]
Object
Appendix
Table 3: Domain randomization configuration.
Index
Description
0 – 22
arm-hand joint positions
22 – 57
the palm pose and the poses of the four fingertips
57 – 64
object pose
64 – 71
distance between the current object pose and the object reference pose at the current timestep
71 – 99
object reference poses at future timesteps {1,2,4,12} relative to the current timestep
99 – 121
distance between the current and reference arm–hand joint positions at the current timestep
Appendix
Table 4: Observation space used for specialist policy training.
Index
Description
0 – 22
arm-hand joint positions
22 – 57
the palm pose and the poses of the four fingertips
57 – 64
object pose
64 – 71
distance between the current object pose and the object reference pose at the current timestep
71 – 141
object reference poses at future ten timesteps relative to the current timestep
141 – 147
distance between the current and reference arm joint positions at the current timestep
Appendix
Table 5: Observation space for the low-level controller in ObjDex.
Figure 9: Real-world setup. (a) Hardware. (b) Test objects. The blue box marks objects used for the real-world unseen-trajectory set; the orange box marks objects used for the real-world unseen-object set.
Figure 10: FoundationPose estimation behavior. (a) Wineglass: although the object is static, the estimate continuously spins around the blue symmetry axis due to axial symmetry. (b) Cube: despite discrete symmetry, the pose remains stable (no continuous spin). (c) Textured wineglass: asymmetric texture breaks the symmetry and yields a stable object frame.
Figure 11: Eight textured axially symmetric objects used in our real-world experiments.
Method
Completion (%, ↑ )
Ours (w/o Modulator)
29.72%
Ours
58.61%
Appendix
Table 6: Real-world results on textured axially symmetric objects.
Method
Unseen Traj.
Unseen Obj.
Ours (w/o Modulator)
36.42%
18.75%
Ours
65.71%
32.50%
Appendix
Table 7: Real-world results without grip tape on the objects.
Input
Method
Unseen Traj.
Unseen Obj.
GT Pointcloud
Ours
65.71%
32.50%
Recon. Pointcloud
Ours
59.28%
34.73%
Partial Pointcloud
Ours
49.46%
25.72%
Appendix
Table 8: Real-world results under practical object-geometry inputs. GT Pointcloud uses ground-truth object geometry; Recon. Pointcloud is reconstructed from a single RGB image using InstantMesh [ 48 ] ; Partial Pointcloud is obtained from segmented single-view depth.
Figure 12: Lipschitz-style perturbation analysis of object latent modulation. The response ratio ρ=∥Δa∥2/ϵ remains bounded under small latent perturbations, supporting a locally smooth latent-to-action mapping and meaningful interpolation between object anchors.
Figure 13: Qualitative results in simulation.
Figure 14: Additional qualitative results in real-world.
Dexterous multi-finger hands hold promise to unlock diverse skills like functional grasps, in-hand reorientation, and dynamic handover that the human-designed world assumes, yet existing learning recipes force a trade-off. Goal-reaching reinforcement learning generalizes to arbitrary object poses but, lacking any human prior, converges to unnatural and unreliable finger strategies; motion tracking reproduces human-like behavior from a single clip but cannot deviate from it. We present AdaDex, a goal-conditioned RL framework that turns a single hand-object trajectory of a few seconds---from mocap data or HOI reconstruction from in-the-wild RGB(D) video---into an adaptable dexterous skill. At the core of AdaDex is Hybrid Dexterous Mimicking: a shared policy is trained on two populations of massively parallel environments: motion tracking environments trained with dense motion tracking objective, which instill the human prior from human motion; and generalization environments with noised initial states and goals and a sparse object-success reward, which force the skill to work far off the reference. From seconds of human motion per skill, the resulting policies can learn diverse adaptable dexterous skills, such as functional grasping, object reposing, in-hand reorientation with finger gaiting, and dynamic behaviors like throw-and-catch. Our method outperforms baselines in terms of generalization and functionality, and we demonstrate transfer across simulators and to the real world. Project page: https://yuxuank.com/AdaDex/.
Human demonstrations provide strong priors for robot manipulation, yet it is non-trivial to transfer them to execute on real robots due to the kinematic gap. In dexterous manipulation, it remains challenging to track long-horizon, contact-rich sequences even in simulators: a reference-tracking policy must keep objects on their target trajectories while preserving demonstrated joint motion and contact timing. Existing approaches often rely on hand-crafted reward tuning that require per-sequence tuning and break under limited interaction budgets. We introduce ConTrack, a reinforcement learning (RL) framework that scales with tracking data. ConTrack treats object tracking as a constraint and allocates remaining control authority to motion fidelity, which allows it to adapt task--style trade-offs online using a dual-variable update. In addition, ConTrack also stabilizes long-horizon learning with an adaptive mid-trajectory reset library that reuses policy-reachable simulator states. Our qualitative and quantitative results in simulation tracking and real robot demonstrate that ConTrack improves success and object pose accuracy significantly over prior arts while preserving joint and contact fidelity. Website: https://www.lyt0112.com/projects/ConTrack.
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5
Zhejiang University · Shanghai Innovation Institute · Fudan University +2