Organizations: State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence
Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.
Figures & tables
Figure 2: Wearable Hand Motion Collection System.
Figure 3: Overview Framework (a) We construct an expressive yet compact representation to capture the dynamics in dexterous manipulation. (b) METIS is pretrained on multi-source EgoAtlas dataset, where human and robot actions are align under a unified action space. (c) METIS integrates reasoning and acting within a framework, enabling effective deployment to downstream tasks.
Method
Pick and Place
Close Laptop
Open Drawer
Grasp Two Drinks into Basket
Put Cola into Basket
Open Drawer and Put Bread
SR
SR
SR
SR
PSR
SR
PSR
SR
PSR
ACT
35.0%
65.0%
95.0%
25.0%
40.0%
50.0%
53.3%
5.0%
5.0%
OpenVLA-OFT
50.0%
80.0%
10.0%
40.0%
57.5%
55.0%
56.7%
0.0%
1.0%
π0.5
60.0%
85.0%
70.0%
65.0%
72.5%
75.0%
76.7%
60.0%
65.0%
GR00T N1.5
70.0%
80.0%
80.0%
65.0%
70.0%
70.0%
73.3%
70.0%
72.5%
METIS (Ours)
85.0%
95.0%
90.0%
75.0%
85.0%
70.0%
76.7%
75.0%
82.5%
Table 1: Main results of six real-world tasks. Each experiment is evaluated with 20 trials. SR denotes Success Rate, and PSR denotes Progress Success Rate.
Figure 4Table 5Figure 6
Method
Token Count
Throughput (Hz)
Pick and Place
Open Drawer and Put Bread
Pour Drinks
METIS w/ Naive Tokenizer
1440
0.93
15.0%
10.0%
0.0%
METIS w/ Fast Tokenizer
138
9.87
50.0%
25.0%
15.0%
METIS w/ Motion-aware Dynamics
44
28.30
60.0%
35.0%
30.0%
Table 5: Ablation on motion-aware dynamics (success rate vs. inference efficiency).
Table 8
Method
Open Drawer and Put Bread
Pour Drinks
Visual-only
45.0%
50.0%
Motion-only
60.0%
55.0%
Motion-aware
75.0%
80.0%
Table 7: Ablations of visual dynamics. (a) Success rates with different dynamics components. (b) Effect of the visual-token budget on codebook usage and visual reconstruction error.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Visualization of hand motion. We use 3D keypoints to represent hand motion. To ensure consistency across embodiments, the wrist frames of human and robot hands are strictly aligned.
Figure 9: Visualization of subtask-level annotations.
Method
Trajs
Frames
Hours
Pose
Subtask
Human
Robot
In-the-wild
ARCTIC
296
214.5K
2.0
✓
✗
100%
0%
✗
H2O
109
65.3K
0.6
✓
✓
100%
0%
✗
HoloAssist
100
777.3K
7.3
✓
✓
100%
0%
✗
Oakink
134
146K
1.4
✓
✓
100%
0%
✗
EgoDex
314.8K
77.9M
721.3
✓
✗
100%
0%
✓
PH2D
1.8K
416.5K
3.9
✓
✗
66.1%
33.9%
✓
Appendix
Table 8: Statistics of EgoAtlas. We use in-the-wild to denote data collected in diverse, unconstrained real-world scenes.
Figure 10: Visualization of task progress across embodiments.
Training Dataset Mixture
H2O [ 23 ]
0.8%
OAKINK [ 58 ]
1.9%
PH2D [ 39 ]
5.4%
ARCTIC [ 8 ]
2.8%
EgoDex [ 12 ]
40.3%
Holoassist [ 52 ]
10.6%
Appendix
Table 9: The dataset name and sampling weights used in EgoAtlas pretraining dataset.
Name
Value
Steps
40000
Batch Size
4
Learning Rate
5e−4
Action Chunks
30
LoRA Rank
32
Action Head
l1regression
Appendix
Table 10: Hyperparameters for OpenVLA-OFT.
Name
Value
Steps
30000
Batch Size
32
Learning Rate
5e−5
Action Chunks
10
Ema decay
0.999
Warm-up steps
10000
Appendix
Table 11: Hyperparameters for π0.5 .
Name
Value
Steps
10000
Batch Size
32
Learning Rate
1e−4
Weight Decay
1e−5
Warmup Ratio
0.05
Action Chunks
16
Appendix
Table 12: Hyperparameters for GR00T N1.5.
Name
Value
Steps
600000
Batch Size
8
Learning Rate
1e−5
Weight Decay
1e−4
Warmup Ratio
0.05
Action Chunks
100
Appendix
Table 13: Hyperparameters for ACT.
Figure 11: (a) Task categories and distribution of the collected dataset. (b) Quantitative analysis of code usage with dynamics tokens. (c) Quantitative analysis of motion reconstruction with dynamics tokens.