Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence
Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.
Figures & tables
Figure 2: Wearable Hand Motion Collection System.
Figure 3: Overview Framework (a) We construct an expressive yet compact representation to capture the dynamics in dexterous manipulation. (b) METIS is pretrained on multi-source EgoAtlas dataset, where human and robot actions are align under a unified action space. (c) METIS integrates reasoning and acting within a framework, enabling effective deployment to downstream tasks.
Method
Pick and Place
Close Laptop
Open Drawer
Grasp Two Drinks into Basket
Put Cola into Basket
Open Drawer and Put Bread
SR
SR
SR
SR
PSR
SR
PSR
SR
PSR
ACT
35.0%
65.0%
95.0%
25.0%
40.0%
50.0%
53.3%
5.0%
5.0%
OpenVLA-OFT
50.0%
80.0%
10.0%
40.0%
57.5%
55.0%
56.7%
0.0%
1.0%
π0.5
60.0%
85.0%
70.0%
65.0%
72.5%
75.0%
76.7%
60.0%
65.0%
GR00T N1.5
70.0%
80.0%
80.0%
65.0%
70.0%
70.0%
73.3%
70.0%
72.5%
METIS (Ours)
85.0%
95.0%
90.0%
75.0%
85.0%
70.0%
76.7%
75.0%
82.5%
Table 1: Main results of six real-world tasks. Each experiment is evaluated with 20 trials. SR denotes Success Rate, and PSR denotes Progress Success Rate.
Figure 4Table 5Figure 6
Method
Token Count
Throughput (Hz)
Pick and Place
Open Drawer and Put Bread
Pour Drinks
METIS w/ Naive Tokenizer
1440
0.93
15.0%
10.0%
0.0%
METIS w/ Fast Tokenizer
138
9.87
50.0%
25.0%
15.0%
METIS w/ Motion-aware Dynamics
44
28.30
60.0%
35.0%
30.0%
Table 5: Ablation on motion-aware dynamics (success rate vs. inference efficiency).
Table 8
Method
Open Drawer and Put Bread
Pour Drinks
Visual-only
45.0%
50.0%
Motion-only
60.0%
55.0%
Motion-aware
75.0%
80.0%
Table 7: Ablations of visual dynamics. (a) Success rates with different dynamics components. (b) Effect of the visual-token budget on codebook usage and visual reconstruction error.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Visualization of hand motion. We use 3D keypoints to represent hand motion. To ensure consistency across embodiments, the wrist frames of human and robot hands are strictly aligned.
Figure 9: Visualization of subtask-level annotations.
Method
Trajs
Frames
Hours
Pose
Subtask
Human
Robot
In-the-wild
ARCTIC
296
214.5K
2.0
✓
✗
100%
0%
✗
H2O
109
65.3K
0.6
✓
✓
100%
0%
✗
HoloAssist
100
777.3K
7.3
✓
✓
100%
0%
✗
Oakink
134
146K
1.4
✓
✓
100%
0%
✗
EgoDex
314.8K
77.9M
721.3
✓
✗
100%
0%
✓
PH2D
1.8K
416.5K
3.9
✓
✗
66.1%
33.9%
✓
Appendix
Table 8: Statistics of EgoAtlas. We use in-the-wild to denote data collected in diverse, unconstrained real-world scenes.
Figure 10: Visualization of task progress across embodiments.
Training Dataset Mixture
H2O [ 23 ]
0.8%
OAKINK [ 58 ]
1.9%
PH2D [ 39 ]
5.4%
ARCTIC [ 8 ]
2.8%
EgoDex [ 12 ]
40.3%
Holoassist [ 52 ]
10.6%
Appendix
Table 9: The dataset name and sampling weights used in EgoAtlas pretraining dataset.
Name
Value
Steps
40000
Batch Size
4
Learning Rate
5e−4
Action Chunks
30
LoRA Rank
32
Action Head
l1regression
Appendix
Table 10: Hyperparameters for OpenVLA-OFT.
Name
Value
Steps
30000
Batch Size
32
Learning Rate
5e−5
Action Chunks
10
Ema decay
0.999
Warm-up steps
10000
Appendix
Table 11: Hyperparameters for π0.5 .
Name
Value
Steps
10000
Batch Size
32
Learning Rate
1e−4
Weight Decay
1e−5
Warmup Ratio
0.05
Action Chunks
16
Appendix
Table 12: Hyperparameters for GR00T N1.5.
Name
Value
Steps
600000
Batch Size
8
Learning Rate
1e−5
Weight Decay
1e−4
Warmup Ratio
0.05
Action Chunks
100
Appendix
Table 13: Hyperparameters for ACT.
Figure 11: (a) Task categories and distribution of the collected dataset. (b) Quantitative analysis of code usage with dynamics tokens. (c) Quantitative analysis of motion reconstruction with dynamics tokens.
Dexterous manipulation is essential for real-world robot autonomy, mirroring the central role of human hand coordination in daily activity. Humans rely on rich multimodal perception--vision, sound, and language-guided intent--to perform dexterous actions, motivating vision-based, language-conditioned manipulation systems for robots. However, training reliable vision-language-action (VLA) models for dexterous manipulation requires large-scale demonstrations across many robotic hands. In addition, as new dexterous embodiments appear rapidly, collecting data for each becomes costly and impractical, creating a need for scalable cross-embodiment learning. We introduce XL-VLA, a vision-language-action framework integrated with a unified latent action space shared across diverse dexterous hands. This embodiment-invariant latent space is directly pluggable into standard VLA architectures, enabling seamless cross-embodiment training and efficient reuse of both existing and newly collected data. Experimental results demonstrate that XL-VLA consistently outperforms baseline VLA models operating in raw joint spaces, establishing it as an effective solution for scalable cross-embodiment dexterous manipulation.
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While prior work demonstrates human to robot transfer in constrained settings, it is unclear whether large scale human data can support fine grained, high degree of freedom dexterous manipulation. We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data. We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss. This validation loss strongly correlates with downstream real robot performance, establishing large scale human data as a predictable supervision source. Beyond scale, we introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training. This enables strong long horizon dexterous manipulation and one shot task adaptation with minimal robot supervision. Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand, and transfers effectively to robots with lower DoF hands, indicating that large scale human motion provides a reusable, embodiment agnostic motor prior.
Ruijie Zheng, Dantong Niu, Yuqi Xie +12
NVIDIA · University of California, Berkeley · University of Maryland