Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Figures & tables
Figure 1: AdvDex. A unified Vision-Language-Action framework for scalable cross-embodiment dexterous manipulation. By aligning hardware kinematics through a shared action space and reducing embodiment-specific visual information through domain-adversarial learning, AdvDex supports zero-shot generalization to novel objects and unseen environments.
Figure 2
Figure 3: Joint-Aligned Action Space. A canonical SE(3) wrist and 15-joint representation shared across embodiments.
Figure 4: Overview of the Domain-Adversarial VLA. A domain discriminator connected to the Vision-Language Model through a Gradient Reversal Layer encourages the extracted cognition tokens to become more domain-invariant before they condition the Diffusion Transformer for joint-aligned action prediction.
Method
OmniShare-Unseen
HOI4D
dh-o↓
MPJPE ↓
MWTE ↓
dh-o↓
MPJPE ↓
MWTE ↓
VITRA [ 36 ]
20.1
16.2
14.8
16.3
13.7
11.5
w/o OmniShare
19.8
15.7
14.2
15.6
14.1
11.3
w/o Adv
6.4
4.9
4.2
13.9
13.3
10.8
Ours
3.2
2.8
2.5
10.5
7.2
6.1
Table 2: Evaluation and ablation of hand action prediction (mm).
Method
Seen
Unseen
Grasp Single Object
Multi-Object Grasp
Pour Water
Push Cube
Stack Bottle
Unseen Objects
Unseen Environment
π0.5 [ 29 ]
75.0
60.0
55.0
85.0
60.0
35.0
45.0
VITRA [ 36 ]
70.0
65.0
40.0
90.0
70.0
40.0
35.0
Ablations
w/o Pre-train
35.0
20.0
10.0
50.0
25.0
5.0
0.0
w/o OmniShare
75.0
55.0
45.0
85.0
65.0
25.0
20.0
Table 3: Success rates on real-world robot dexterous manipulation tasks (in %).
Method
Human Tasks
Box Doll
Press Button
Move Bottle
Tool Use
π0.5 [ 29 ]
30.0
50.0
30.0
0.0
VITRA [ 36 ]
40.0
45.0
25.0
20.0
Ablations
w/o Adv (Pre-train)
50.0
35.0
35.0
15.0
w/o Adv (Post-train)
25.0
30.0
15.0
5.0
Table 4: Success rates on unseen real-world dexterous manipulation tasks via Co-Training (in %).
Figure 5: Few-shot Fine-tuning Success Rates. Evaluation on the single-object grasping task across 0, 5, and 20 demonstrations.
Figure 6: t-SNE Feature Distributions. Intermediate features during pre-training (a) without and (b) with the GRL, and during post-training (c) without and (d) with the GRL.