Organizations: Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China · University of Chinese Academy of Sciences, China
Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at https://github.com/wangwen-banban/EMPIRE.
Figures & tables
Figure 1: EMPIRE at a glance. Stage I learns to plan; Stage II learns to act with the planner frozen.
Figure 2: Overview of EMPIRE. Stage I (left): RGB and inferred monocular-depth features, together with the caption and camera FoV, form a multimodal prefix that conditions explicit motion-plan generation. Stage II (right): The entire Stage-I VLM is frozen, while the motion generator is trained with flow matching by conditioning on the current hand state and cross-attending to the hidden states Hplan of Stage-I prediction. Bottom: During inference, the frozen VLM generates a plan with hidden states, which directly condition the DiT to generate future motion.
Figure 3: The EMPIRE-651K construction and plan-supervision pipeline. (A) Raw ARKit skeletons are converted into wrist-aligned MANO motion targets. (B) Multi-task EgoDex videos are split into task-specific episodes, which are then segmented and resampled into aligned forecasting windows. (C) Qwen2.5-VL first captions each video segment and then generates a caption-grounded motion plan from the segment and caption; Qwen3 labels caption–plan consistency, and inconsistent samples are discarded.
Metric
Score
Caption quality
4.57
Motion-plan quality
4.66
Overall quality
4.62
Usable pairs
84.0%
Table 1: Human audit of EMPIRE-651K annotations. Final results over 100 stratified training windows. Quality scores use a 1 – 5 scale; A pair is usable when both its caption and plan means are at least 4/5 .
Method
Params
MPJPE ↓
Wrist MPJPE ↓
Finger- relative MPJPE ↓
Infer.(s/window) ↓
VITRA re-impl. ( 23 )
3.2 B
105.37
91.42
45.37
0.4
Being-H0-1B ( 27 )
1.2 B
115.61
100.23
66.87
15.6
Being-H0-8B ( 27 )
8.2 B
85.39
68.61
50.95
28.3
Being-H0-14B ( 27 )
15.4 B
84.90
66.54
51.65
83.5
EMPIRE (ours)
3.2 B
84.53
73.25
38.97
1.0
Table 2: Main results on EMPIRE-651K. EMPIRE is compared with baselines on the same held-out test set. See the Evaluation subsection for protocol details and metric definitions.
Difficulty (baseline MPJPE)
#tasks
Mean Δ
Plan helps
Easy ( <100 )
43
−4.6
25/43
Medium ( 100 – 175 )
49
−26.2
47/49
Hard ( ≥175 )
19
−49.7
17/19
All
111
−22.3
89/111
Table 3: Planning benefit by task difficulty. Δ means that ours minus baseline; Negative means improvement.
Figure 4: Qualitative comparison of motion predictions against baselines. Task: peel_place_sticker .
Configuration
Overall ↓
Seen ↓
Unseen ↓
VITRA re-impl. (caption only)
124.52
92.71
129.73
+ motion plan
102.26
90.87
104.80
+ depth (cross-attention)
104.71
110.10
103.51
+ depth (concatenation)
95.42
99.33
94.55
Table 4: Model-component ablation. All results in the table are reported using MPJPE.
Setting
VLM grad.?
LM loss
MPJPE ↓
Single-stage
Yes
–
124.52
Two-stage, trainable
Yes
0.5
118.44
Two-stage, frozen (ours)
No
0
94.67
Table 5: Motion-gradient isolation in Stage II. The plan loss coefficient is set to 0.5.
Action head
Params
Overall ↓
Seen ↓
Unseen ↓
DiT-M
46 M
107.93
114.15
106.55
DiT-B (default)
182 M
95.42
99.33
94.55
DiT-L
637 M
100.08
95.37
101.13
Table 6: DiT-capacity ablation. All results in the table are reported using MPJPE
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Target
Criterion
Definition
Score
Caption
Grounding
Described actions and objects are consistent with the visual evidence in the video.
4.38/5
Caption
Hand attribution
Left/right roles, hand activities, and None assignments are correctly identified.
4.55/5
Plan
Grounding
Described actions, objects, and contact states are supported by the visual evidence.
4.69/5
Plan
Coverage
Key sub-actions and all visibly active hands are covered.
4.68/5
Plan
Temporal coordination
Step order, simultaneous motion, and bimanual coordination are correct.
4.60/5
Plan
Conditioning utility
The plan is specific, non-redundant, complete, and useful for motion generation.
4.35/5
Appendix
Table 7: Human-audit criteria and final scores.
Partition
Caption
Plan
Overall
Usable ( % )
Part 1
4.45
4.51
4.48
75.0%
Part 2
4.63
4.72
4.68
85.0%
Part 3
4.80
4.71
4.76
95.0%
Part 4
4.35
4.56
4.46
75.0%
Part 5
4.63
4.80
4.71
90.0%
Overall
4.57
4.66
4.62
84.0%
Appendix
Table 8: Human-audit quality scores across training partitions. Each partition contains 20 cases sampled from distinct episodes. The overall row reports the aggregate results over all 100 audited cases.
Figure 5: Some annotation examples from the human audit. Each row shows four uniformly spaced ground-truth frames from an audited five-second window, followed by its caption and temporally ordered motion-plan steps.
Configuration
Stage I: Learn to Plan
Stage II: Learn to Act
Trainable modules
Gemma-2 decoder, multimodal projector, depth mapper, and FoV encoder; SigLIP and DA3 encoders are frozen
DiT-B and conditioning projectors; the Planner and all condition encoders are frozen
Training objective
Autoregressive plan-language cross-entropy
Flow matching with t∼Beta(1.5,1.0) on [0,1]
Training length
1 epoch ( 10,171 optimizer steps)
4 epochs ( 40,684 optimizer steps)
Optimizer
AdamW, (β1,β2)=(0.9,0.999) , weight decay 0.1
Same as Stage I
Learning rate
1×10−5
1×10−4
Schedule / clipping
Constant learning rate, no warmup; gradient-norm clipping at 1.0
Same as Stage I
Appendix
Table 9: Training details. Stage-specific settings from the final Stage I and Stage II checkpoint configurations.
Run
Trainable
Epoch
Steps
s/step
Hours
Baseline (single stage)
VLM + DiT
4
40,684
10.26
116
Ours Stage-I (plan SFT)
VLM
1
10,171
12.14
34
Ours Stage-II (DiT)
DiT
4
40,684
3.23
37
Ours total
—
5
50,855
—
71
Appendix
Table 10: Gradient-optimization cost on EMPIRE-651K.
Model
AR tokens
s / window
Baseline (no plan)
0
0.4
EMPIRE (plan + depth)
≤96
1.0
Being-H0-1B
∼10,400
15.6
Being-H0-8B
∼10,400
28.3
Being-H0-14B
∼10,400
83.5
Appendix
Table 11: End-to-end inference cost.
Stage-II plan condition
26-task subset
Train Part 1–5
Ground-truth plan
94.67
90.22
Stage-I predicted plan (ours)
93.40
84.53
Appendix
Table 12: Stage II training with predicted versus ground-truth plans.
Figure 6: Qualitative comparison of motion predictions against baselines. Task: insert_remove_bookshelf .
Figure 7: Qualitative comparison of motion predictions against baselines. Task: dry_hands .
Beijing University of Posts and Telecommunications, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · North China Electric Power University, Beijing, China +2