Egocentric human recordings are far cheaper than dexterous robot demonstrations, promising stronger pretraining and direct skill learning without task-specific robot data. Yet humans and robots differ in morphology and kinematics, so human recordings lack the joint positions and actuator commands that supervise robot policies, providing only retargeted end-effector (EEF) actions. This raises two questions: how human data should supervise the policy, and how a dexterous robot should read out actions at deployment. We address both in SHAPE (Supervision from Human Action tokens with Parallel Executable streams), a vision-language-action (VLA) model whose backbone predicts discrete action tokens of EEF chunks as an auxiliary task, while a shared flow-matching expert outputs two executable streams, EEF and joint. On a bimanual robot with 22-DoF hands, a matched study shows that discrete supervision helps learn from human data, driving backbone gradients and shaping action-informative representations across domains. Under standard control pipelines at deployment, direct joint readout excels at fine-grained control with robot demonstrations, scoring 0.80 across six tasks against 0.63 for π0.5, whereas IK-based EEF readout excels at human-to-robot transfer. Pretrained with human data and discrete tokens, SHAPE learns new tasks from human demonstrations without task-specific robot data.
Figures & tables
Figure 1: SHAPE learns dexterous manipulation from egocentric human data. (a) Pretraining pairs human recordings, providing only retargeted end-effector (EEF) actions, with robot teleoperation (EEF and joint actions). (b) The backbone predicts discrete EEF tokens across both domains as an auxiliary target, while a shared flow-matching expert outputs independent EEF and joint streams. (c) Adding human data and discrete tokens in pretraining improves policies under both robot-demo and human-only fine-tuning (top), where direct joint and IK-based EEF readouts respectively excel (bottom).
Figure 2: Overview of SHAPE. Human recordings (orange) provide head images and EEF actions; robot data (blue) add wrist images and joint actions. The backbone predicts discrete FAST tokens of the EEF chunk ( LFAST ) as an auxiliary task alongside the sub-task ( Ltext ) and future frames ( Lvideo ). A shared flow-matching expert reads the backbone at every layer and independently denoises an EEF and a joint chunk, isolated by the attention mask. At deployment, training-only heads (dashed) are omitted, and either stream can drive the robot: the joint stream with robot demos, and the EEF stream for human-to-robot transfer.
Source
Tasks
Episodes
Hours
EEF
Joint
FAST
Human egocentric
320
73.1k
2,811
✓
–
✓
Robot teleoperation
105
10.5k
228
✓
✓
✓
Table 1: Pretraining data and the four matched pretrained models. Left: pretraining corpora and action supervision targets. Right: the four matched models, which share the architecture, robot data, and training budget.
Figure 3: Real-robot tasks with robot demonstrations. The left panel shows top and side views of the robot, while the right panels illustrate the four stages of each task, with partial credit indicated in parentheses for Reinsert USB and Reinsert Type-C.
Method
Stream
Insert
Reinsert
Reinsert
Wipe
Open
Close
Score ↑
Battery
USB
Type-C
Bowl
Cabinet
Bottle Cap
GR00T N1.7
Joint
5/20
0.18
0.52
12/20
18/20
12/20
0.51
π0.5
Joint
13/20
0.25
0.62
13/20
17/20
15/20
0.63
SHAPE
EEF
6/20
0.15
0.54
14/20
17/20
12/20
0.52
Joint
16/20
0.55
0.64
18/20
18/20
20/20
0.80
w/o human
EEF
7/20
0.15
0.42
10/20
15/20
13/20
0.47
Table 2: Real-robot results with robot demonstrations. Each task is evaluated over 20 trials; w/o both omits both human data and LFAST in pretraining. Cells give the number of successes, or the mean partial credit for USB and Type-C, and Score is the six-task mean. The chart on the right shows the scores with 95% intervals.
Figure 4: Human-to-robot transfer with zero task-specific robot demonstrations. Top: egocentric human demonstrations, the only fine-tuning data for each task. Bottom: robot rollouts of SHAPE with the EEF stream.
Pretraining
Hum.
FAST
Egg.
Cup
Stack
Total (success rate)
w/o both
✗
✗
0
0
0
0/60 (0%)
w/o human
✗
✓
0
3
1
4/60 (6.7%)
w/o FAST
✓
✗
3
2
2
7/60 (11.7%)
SHAPE
✓
✓
10
8
3
21/60 (35%)
joint stream
✓
✓
0
0
0
0/60 (0%)
Table 3: Human-to-robot transfer results with zero task-specific robot demonstrations. Successes out of 20 trials per task on Pick Eggplant, Pick Cup, and Stack Cups; all rows use the EEF stream except the indented joint-stream row of SHAPE. The chart on the right shows the total success rates with 95% intervals.
Method
Hum.
FAST
Close
PnP
All
GR00T N1.7
–
–
69.2
60.4
62.6
π0.5
–
–
51.4
69.2
64.8
w/o both
✗
✗
61.4
62.2
62.0
w/o human
✗
✓
62.6
66.1
65.2
w/o FAST
✓
✗
57.1
73.8
69.6
SHAPE
✓
✓
66.6
74.1
72.2
Table 4: A second robot in simulation. Success rate (%) on the 24 GR-1 tabletop tasks of RoboCasa, joint stream. Close and PnP are the 6 articulated and 18 pick-and-place tasks. The chart on the right shows the overall success rates with 95% intervals.
Figure 5: Offline action error across pretraining steps. (a) Right-wrist error reduction from adding LFAST with human data (w/o FAST minus SHAPE) across splits; bands show 95% CIs. (b) Right-wrist errors of the four matched models (orange: with human data; blue: robot-only; dashed: without FAST; corner squares match split colors in (a)).
Figure 6: Discrete supervision makes human frames more action-informative and closer to matched-task robot frames. Shaded bands and error bars show 95% confidence intervals. (a) Linear probe R2 per domain and robot-to-human transfer r on last-layer prefix-mean features at 30k steps (shuffled-label baseline dashed). (b) Share of a human frame’s 10 nearest robot frames from its matched task across three task families (last-layer prefix mean, 15k and 30k steps). (c) Matched-task robot neighbor share across backbone layers at 30k steps on head-camera image tokens (layer 0: vision tower; layers 4–36: language model).
Figure 7: How losses and data sources share the backbone. (a) Share of each module’s gradient energy (squared norm of the loss-weighted mean gradient across batches) at 30k steps on human and robot batches. (b–d) Cosines between mean gradients across pretraining steps (robot batches in (c)). Error bars show ± 2 standard errors where both mean gradients exceed batch-to-batch noise.
Metric
w/o FAST
Cont. head
SHAPE
Backbone grad. norm / LFAST
–
0.98×
1.00×
Human new-ep. error (mm) ↓
62.9
59.2
60.7
Held-out robot error (mm) ↓
16.3
16.1
15.7
Human probe R2 / transfer r↑
0.089 / 0.205
0.230 / 0.437
0.143 / 0.281
Matched-task NN, L24 / last (%) ↑
42 / 37
66 / 40
61 / 57
Real-robot H2R, EEF (%) ↑
11.7
18.3
35.0
Table 5: Gradient-matched continuous head vs. SHAPE. Pretraining metrics at 30k steps (Appendix 12.6 ); NN is the matched-task neighbor share at layer 24 (L24; Fig. 6 c) and at the last layer (Fig. 6 b). H2R disables auxiliary action heads for all models (SHAPE vs. Cont.: p=0.055 ); GR-1 averages 80k–100k ( p=0.001 w/ aux, p=0.11 w/o aux vs. SHAPE). The chart on the right contrasts last-layer probe R2 , matched-task neighbors (dashed: chance), and real-robot H2R transfer with 95% intervals.
Component
EEF stream
Joint stream
Per side
wrist 3 + 6, tips 5 × 3
arm 7, hand 22
Torso and neck
7
7
Total
55
65
Supervised, robot
53
63
Supervised, human
48
0
Table S1 : Dimensions of the two action streams. States use the same layout. Per side, the EEF stream has the wrist position (3), the wrist rotation as the first two rows of its rotation matrix (6), and the positions of the five fingertips in the wrist frame (5 × 3). The joint stream is not supervised on human pretraining samples.
Pretraining
Fine-tuning
Optimizer
AdamW, β=(0.9,0.95) , weight decay 10−8
Grad. clipping
global norm 1.0
Learning rate
5.7×10−5
5×10−5
Expert LR
0.5 × learning rate
Schedule
5% warmup, cosine
5% warmup, cosine to 1%
Batch
1,152
128
Table S2 : Training hyperparameters. Fine-tuning lengths are given for the tasks with robot demonstrations and for the H2R tasks. Additional terms in the last row apply chunk temporal smoothing (0.02) and scale human EEF flow matching by 0.5 to temper higher-variance retargeted hand trajectories.
Task
Demonstrations
Train
Validation
Insert Battery
robot
243
6
Reinsert USB
robot
202
6
Reinsert Type-C
robot
221
7
Wipe Bowl
robot
196
5
Open Cabinet
robot
447
12
Close Bottle Cap
robot
413
16
Table S3 : Fine-tuning data per task. Entries give the number of episodes.
Reinsert USB
Reinsert Type-C
Method
Stream
Unplug
Insert
Unplug
Grasp
Insert
GR00T N1.7
Joint
5
2
16
16
2
π0.5
Joint
8
2
20
20
1
SHAPE
EEF
6
0
20
16
0
Joint
20
2
20
20
2
w/o human
EEF
6
0
16
12
0
Table S4 : Stage completion on Reinsert USB and Reinsert Type-C. Entries give trials (out of 20) that complete each stage for the baselines and the four matched models; a trial that completes a stage has completed the earlier ones. A trial earns the credit of the last stage it completes (USB: 0.5 unplug, 1.0 insert; Type-C: 0.3 unplug, 0.6 grasp, 1.0 insert); Table 2 gives the mean credit.
Pretraining
Stream
Eggplant
Cup
Stack
w/o both
EEF
0
0
0
Joint
0
0
0
w/o human
EEF
0
3
1
Joint
0
0
0
w/o FAST
EEF
3
2
2
Joint
0
0
0
Table S5 : Human-to-robot transfer across both streams. Entries give successes out of 20 trials for both readouts of every pretrained model.
Figure S1 : Open-loop right-wrist paths from the robot’s home posture. A human Pick Eggplant demonstration is replayed to the grasp in the torso frame. EEF stream: the demonstrated wrist motion applied at the robot’s wrist; joint stream: the demonstrated joint offsets added to the home posture.
Figure S2 : Offline action error across pretraining steps. Errors (mm; lower is better) of the four models are shown for the right wrist, right fingertips, and left wrist (rows). (a) Main test set, EEF stream (splits in Appendix 12.1 ); on human recordings, only the two models pretrained on them are shown. (b) Second test set, EEF stream and joint stream through forward kinematics. Orange: with human data; dashed: without FAST.
Figure S3 : Matched-task robot neighbors across layers. Share of matched-task robot neighbors (Fig. 6 b, c) from the vision tower (layer 0) through the language model (layers 4–36), at (a) 30k and (b) 15k pretraining steps across head-camera image tokens and all prefix positions. Shaded bands show 95% confidence intervals over human episodes.
Human R2↑
Transfer r↑
Features
SHAPE
w/o FAST
Δ↑
SHAPE
w/o FAST
Δ↑
Vision tower
0.028
0.020
+0.008
0.056
0.059
−0.003
Language model
Layer 4
0.018
0.020
−0.001
0.058
0.051
+0.007
Layer 12
0.024
0.016
+0.008
0.055
0.038
+0.017
Layer 20
0.056
0.034
+0.022
0.101
0.064
+0.037
Table S6 : Linear probes at 30k steps. Language-model rows use the head-camera image tokens, except the prefix mean of Fig. 6 a (last layer, all prefix positions); expert rows use the input projection and the middle and last layers. Δ = SHAPE − w/o FAST; bold: 95% CI excludes zero.
Language model
Vision tower
Steps
H
R
H
R
5k
79
52
21
17
10k
68
52
31
19
15k
58
61
26
50
20k
54
47
30
67
25k
89
97
46
42
Table S7 : Gradient energy share of LFAST across pretraining. Entries give the share of the expected gradient energy (%), after loss weighting, that LFAST contributes to the language model and the vision tower of SHAPE, on human (H) and robot (R) batches. The action expert receives only LFM .
Pretraining
Steps
Decoded ↑ robot / human
TF acc. ↑ robot / human
Δ↓
SHAPE
40k
0.85 / 0.84
0.45 / 0.28
+1.4 [+1.0, +1.8]
30k
0.81 / 0.74
0.45 / 0.27
+1.5 [+1.0, +1.9]
w/o human
40k
0.82 / 0.00
0.46 / 0.06
–
30k
0.84 / 0.03
0.45 / 0.07
–
Table S8 : Greedy decoding of the FAST tokens. Evaluation on the windows of Appendix 12.1 . Decoded: share of windows whose tokens form a valid chunk; TF acc.: next-token accuracy given ground-truth prefixes; Δ : right-wrist error of SHAPE’s decoded tokens minus that of its zero-noise expert, on decoded robot windows (mm; 95% CI over episodes).
Metric
SHAPE
w/o FAST
Cont. head
Δ
95% CI ( p )
Backbone gradient budget and open-loop right-wrist error at 30k steps (mm ↓ )
Weighted VLM grad. norm / LFAST
1.00×
–
0.98× (fixed 0.13× )
–
–
Training episodes (EEF)
10.62
10.84
9.47
+1.15
[+0.92, +1.41]
Human, new episodes (EEF)
60.67
62.88
59.19
+1.48
[+0.37, +2.63]
Held-out tasks (EEF)
15.70
16.33
16.10
−0.40
[ −2.40 , +1.93]
Robot data of human tasks (EEF)
23.72
24.37
23.44
+0.28
[ −0.22 , +0.81]
Table S9 : Gradient-matched continuous-head control vs. SHAPE and w/o FAST. Pretraining metrics are evaluated at 30k steps (Appendices 12.1 – 12.4 ); downstream closed-loop evaluations start from the pretrained EMA checkpoints, following Appendix 11.1 for real-robot H2R transfer (60 trials across 3 tasks; auxiliary action heads disabled for all models), Table 2 for three real-robot tasks with robot demonstrations (60 trials across Insert Battery, Reinsert USB, and Wipe Bowl; 30-epoch fine-tuning retaining each model’s pretraining auxiliary action head), and Appendix 13 for RoboCasa GR-1 (mean of 80k, 90k, and 100k checkpoints over 1,200 task–seed pairs). Δ denotes SHAPE minus Cont. head (negative favors SHAPE on error rows ↓ , positive favors SHAPE on ↑ rows; bold marks Δ and the better of the two values where the 95% CI excludes zero and p<0.05 ). On H2R, the three-task score CI excludes zero while the stratified exact CMH test gives p=0.055 ; † compares SHAPE fine-tuned w/ LFAST against Cont. head fine-tuned w/o aux.
Group
Task
GR00T N1.7
π0.5
w/o both
w/o human
w/o FAST
SHAPE
Close
Bottle → Cabinet Close
80.7
76.7
74.0
75.3
72.7
82.7
Can → Drawer Close
86.0
44.7
76.0
73.3
60.0
78.0
Cup → Drawer Close
51.3
26.7
48.0
49.3
42.0
48.7
Milk → Microwave Close
71.3
50.7
56.7
60.0
62.7
66.0
Potato → Microwave Close
70.7
56.7
66.0
70.7
60.7
74.0
Wine → Cabinet Close
55.3
53.3
48.0
46.7
44.7
50.0
Table S10 : Per-task success rates (%) on the 24 RoboCasa GR-1 tabletop tasks. Entries average 50 evaluation seeds across the three checkpoints of Appendix 13 for both external baselines and the four matched models.
π0.5
GR00T N1.7
Initialization
public checkpoint
Human data
none
20k h
Cameras
head and both wrists
Actions
65 joints, relative to the state
Chunk
25
40
Batch
256
256
Table S11 : Setups of the baselines on the six tasks with robot demonstrations. Human data: hours of egocentric human video in pretraining. Chunks are in steps at 60 Hz, and steps range over the tasks.
Human behavior is among the most scalable sources of data for learning physical intelligence, yet how to effectively leverage it for dexterous manipulation remains unclear. While prior work demonstrates human to robot transfer in constrained settings, it is unclear whether large scale human data can support fine grained, high degree of freedom dexterous manipulation. We present EgoScale, a human to dexterous manipulation transfer framework built on large scale egocentric human data. We train a Vision Language Action (VLA) model on over 20,854 hours of action labeled egocentric human video, more than 20 times larger than prior efforts, and uncover a log linear scaling law between human data scale and validation loss. This validation loss strongly correlates with downstream real robot performance, establishing large scale human data as a predictable supervision source. Beyond scale, we introduce a simple two stage transfer recipe: large scale human pretraining followed by lightweight aligned human robot mid training. This enables strong long horizon dexterous manipulation and one shot task adaptation with minimal robot supervision. Our final policy improves average success rate by 54% over a no pretraining baseline using a 22 DoF dexterous robotic hand, and transfers effectively to robots with lower DoF hands, indicating that large scale human motion provides a reusable, embodiment agnostic motor prior.
Ruijie Zheng, Dantong Niu, Yuqi Xie +12
NVIDIA · University of California, Berkeley · University of Maryland
Dexterous manipulation is a fundamental capability for embodied intelligence, but scaling it remains difficult because robot demonstrations are expensive to collect and action spaces vary across embodiments. Policies trained on heterogeneous data can also entangle task-relevant visual cues with embodiment-specific appearance, limiting cross-embodiment generalization. We present AdvDex, a unified Vision-Language-Action framework for learning dexterous manipulation from human and robot demonstrations. First, we introduce OmniShare, a large-scale multimodal dataset of human manipulation demonstrations that provides high-quality kinematic supervision and tactile measurements while reducing reliance on robot teleoperation. Second, we propose the Joint-Aligned Action Space (JAAS), a canonical action representation comprising an SE(3) wrist pose and 15 finger joints, thereby functionally aligning human hands, dexterous robot hands, and parallel grippers. Finally, we use domain-adversarial learning to reduce embodiment-specific information in the learned visual representation. Experiments on hand-action prediction and real-world dexterous manipulation show consistent improvements over baselines, effective zero-shot human-to-robot skill transfer, generalization to unseen objects and environments, and data-efficient few-shot adaptation.
Zhiyue Zhao, Jingyi Wu, Hairuo Liu +5
Zhejiang University · Shanghai Innovation Institute · Fudan University +2
Building a generalist robot that can perceive, reason, and act across diverse tasks remains an open challenge, especially for dexterous manipulation. A major bottleneck lies in the scarcity of large-scale, action-annotated data for dexterous skills, as teleoperation is difficult and costly. Human data, with its vast scale and diverse manipulation behaviors, provides rich priors for learning robotic actions. While prior works have explored leveraging human demonstrations, they are often constrained by limited scenarios and a large visual gap between human and robots. To eliminate these limitations, we propose METIS, a vision-language-action (VLA) model for dexterous manipulation pretrained on multi-source egocentric datasets. We first construct EgoAtlas, which integrates large-scale human and robotic data from multiple sources, all unified under a consistent action space. We further extract motion-aware dynamics, a compact and discretized motion representation, which provides efficient and expressive supervision for VLA training. Built upon them, METIS integrates reasoning and acting into a unified framework, enabling effective deployment to downstream dexterous manipulation tasks. Our method demonstrates exceptional dexterous manipulation capabilities, achieving highest average success rate in six real-world tasks. Experimental results also highlight the superior generalization and robustness to out-of-distribution scenarios. These findings emphasize METIS as a promising step toward a generalist model for dexterous manipulation.
Yankai Fu, Ning Chen, Junkai Zhao +5
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence