Organizations: School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai, China.
Forecasting future hand movements from egocentric videos plays an important role in understanding human intentions and hand-object interactions. However, different manipulation intentions can lead to substantially different hand trajectories even when interacting with the same object. Existing methods have not sufficiently explored the intrinsic relationships between articulated hand movements and object semantics to capture such operation-dependent motion patterns. Therefore, we propose TOMAS, a topology-aware operation modeling and adaptive staging framework for egocentric hand forecasting. Specifically, we develop a topology-aware hand-object reasoning mechanism that associates hand skeletal structures and articulation dynamics with object semantics and spatial relationships to jointly infer potential interaction targets and latent manipulation operations. The resulting object-operation representations provide interaction conditions for future hand trajectory prediction, with supervision adapted to the annotations available in each dataset. Furthermore, we introduce an adaptive staging mechanism that organizes future hand movements into ordered latent stages with learnable temporal boundaries. Stage-specific geometric subgoals are then incorporated to guide trajectory refinement according to the underlying manipulation patterns. Experimental results on multiple publicly available benchmarks demonstrate that TOMAS achieves state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting.
Figures & tables
Fig. 1: Operation-dependent hand forecasting. The same object can support different operations with distinct future hand trajectories.
Fig. 2: System overview of TOMAS. Given observed egocentric video and historical hand trajectories, TOMAS employs topology-aware hand–object reasoning to jointly model candidate-object semantics and latent operation representations. The resulting interaction-aware representations guide ordered latent motion stages and stage-wise geometric subgoals for future hand trajectory forecasting.
Fig. 3: Visualization of articulated hand motion and skeletal topology.
Fig. 4: Topology-aware object–operation joint reasoning. Observed object candidates and articulated hand configurations are encoded as candidate-specific interaction representations. Cross-modal memory retrieval supports joint object–operation scoring and the formation of interaction-conditioning tokens.
Fig. 5: Interaction-conditioned adaptive staging and geometry-guided trajectory generation. Interaction-conditioned motion features determine ordered stage allocations and stage-level operation mixtures. The resulting candidate-relative geometric subgoals guide continuous trajectory decoding and refinement.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Method
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
CVH [ 9 ]
1.100
1.278
0.952
1.018
0.146
0.148
OCT [ 1 ]
0.370
0.524
0.309
0.397
0.103
0.126
USST [ 2 ]
0.183
0.341
0.120
0.185
0.031
0.052
S-Mamba [ 38 ]
0.185
0.355
0.138
0.207
0.038
0.074
Diff-IP3D [ 9 ]
0.199
0.377
0.156
0.229
0.049
0.081
TABLE I: 3D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Best and second-best scores are in bold and green, respectively.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Method
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
CVH [ 9 ]
0.180
0.230
0.188
0.221
0.206
0.208
OCT [ 1 ]
0.108
0.122
0.091
0.147
0.387
0.381
USST [ 2 ]
0.082
0.118
0.060
0.087
0.040
0.068
S-Mamba [ 38 ]
0.084
0.141
0.071
0.118
0.051
0.094
Diff-IP2D [ 9 ]
0.080
0.130
0.066
0.087
0.042
0.074
TABLE II: 2D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Best and second-best scores are in bold and green, respectively.
Method
ADE ↓
FDE ↓
OCT [ 1 ]
0.160
0.160
USST [ 2 ]
0.200
0.233
Uni-Hand [ 4 ]
0.181
0.208
TOMAS (ours)
0.140
0.165
TABLE III: Short-horizon comparison on EGTEA Gaze+.
Fig. 6: Qualitative comparison of 2D hand-center forecasting on EgoPAT3D-DT and H2O-PT. Green, blue, and red denote observed trajectories, future ground truth, and predicted trajectories, respectively.
Fig. 7: Qualitative comparison of 3D hand-center forecasting on EgoPAT3D-DT and H2O-PT. Green, blue, and red denote observed trajectories, future ground truth, and predicted trajectories, respectively. Coordinates are measured in meters.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Variant
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Full (TOMAS)
0.158
0.324
0.103
0.174
0.025
0.050
A1 (w/o Hand Joints)
0.161
0.326
0.106
0.175
0.027
0.051
A2 (w/o Graph Prop.)
0.161
0.329
0.105
0.182
0.025
0.051
A3 (w/o Rel. Geometry)
0.160
0.326
0.103
0.178
0.025
0.051
A4 (Decoupled Obj.-Op.)
0.161
0.326
0.107
0.175
0.025
0.051
TABLE IV: Ablation results for 3D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Full reports the three-seed average, while A1–A9 report single-seed results.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Variant
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Full (TOMAS)
0.053
0.092
0.042
0.075
0.035
0.069
A1 (w/o Hand Joints)
0.056
0.094
0.044
0.080
0.037
0.072
A2 (w/o Graph Prop.)
0.054
0.094
0.043
0.077
0.036
0.072
A3 (w/o Rel. Geometry)
0.054
0.093
0.044
0.077
0.036
0.071
A4 (Decoupled Obj.-Op.)
0.054
0.094
0.043
0.077
0.035
0.069
TABLE V: Ablation results for 2D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Full reports the three-seed average, while A1–A9 report single-seed results.
Variant
ADE ↓
FDE ↓
Full (TOMAS)
0.140
0.165
A1 (w/o Hand Joints)
0.159
0.185
A2 (w/o Graph Prop.)
0.145
0.173
A3 (w/o Rel. Geometry)
0.145
0.173
A4 (Decoupled Obj.-Op.)
0.145
0.173
A5 (w/o Global Cond.)
0.147
0.175
TABLE VI: Ablation results for short-horizon hand trajectory forecasting on EGTEA Gaze+. Full reports the three-seed average, while A1–A9 report single-seed results.
Variant
Top-1(%)
2D Full
72.82
2D A1 (w/o Hand Joints)
63.68
3D Full
77.61
3D A1 (w/o Hand Joints)
74.63
TABLE VII: Native H2O-PT target ranking. Each model has 1,735 windows; 1,608 have derived interaction targets. Coverage is 100% under the candidate association rule.
Fig. 8: Visualization of learned operation and candidate-interaction representations. PCA projects high-dimensional features into two dimensions. Candidate names are inferred from CLIP, and colors indicate dominant operation components in the learned representation.
Fig. 9: Candidate-conditioned historical evidence retrieval. The heat maps show final-layer, head-averaged attention from valid candidate queries and null to hand-center, hand-topology, egomotion, and RGB memory. Bars report native object-selection probabilities. Candidate names are inferred from CLIP.
Fig. 10: Learned ordered motion stages. Stage weights vary over normalized future time, while dashed vertical lines indicate predicted transition positions.
Fig. 11: Local effect of subgoal-guided refinement. The full prediction is shown in red, and the prediction without final-pass relative-subgoal-coordinate input is gray and dashed. Green indicates the last observed hand position.
Forecasting dexterous hand motions from egocentric observations is fundamental to intelligent interactive systems. Existing VLM-based methods typically map observations directly to future motions, overlooking the underlying manipulation process that governs hand-object interactions. Moreover, end-to-end optimization couples manipulation learning with motion synthesis, causing motion-generation gradients to interfere with the pre-learned manipulation-aware representations. To overcome these limitations, we propose EMPIRE, a two-stage framework that introduces Explicit Manipulation Planning as an Intermediate Representation for Egocentric hand-motion forecasting. Stage I: Learn to Plan. EMPIRE first learns explicit manipulation plans from multimodal context to capture the progression of hand-object interactions. Stage II: Learn to Act. A motion generator synthesizes future bimanual hand motions conditioned on frozen planner representations, preventing motion-generation gradients from affecting manipulation planning. To support our method, we further construct EMPIRE-651K, a bimanual hand-motion forecasting dataset comprising 650,910 training windows across 111 tasks, each paired with an explicit per-hand manipulation plan. Under identical training and evaluation protocols, EMPIRE achieves state-of-the-art forecasting accuracy, with an MPJPE of 84.53 mm and a finger-relative error of 38.97mm. We release the code and dataset at https://github.com/wangwen-banban/EMPIRE.
Wen Wang, Ruibing Hou, Hong Chang +2
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China · University of Chinese Academy of Sciences, China
Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand
Learning embodied manipulation from egocentric human data offers a scalable path, reducing the cost of collecting robot-specific demonstrations. However, existing representation learning methods largely stop at perceptual understanding, such as extracting what a scene contains or what an action is. A capable embodied agent is more than understanding: it predicts the next motion. We propose that forecasting 3D human hand motion is one suitable pretraining task, as the representation jointly learns how to manipulate objects and how the motion will unfold next. Building on this hypothesis, we introduce SFHand, the first representation learning method via streaming egocentric 3D hand forecasting. SFHand learns by autoregressively predicting future 3D hand states, including hand type, 2D bounding box, 3D pose, and trajectory, from a continuous stream of video and language instructions. Its autoregressive architecture is equipped with an ROI-enhanced memory layer, capturing temporal context while focusing on salient hand-centric regions. To enable this research, we also introduce SFHand, the first open-source large-scale dataset pairing synchronized 3D hand poses with fine-grained language instructions (247K clips, 3.95M frames). Transferred to downstream embodied control, the learned representation shows state-of-the-art performance on both simulation and real-world manipulation benchmarks, improving success rate by up to 13.4%. Our method also achieves leading 3D hand forecasting accuracy, with 35.8% lower ADE than baseline methods. These results suggest that forecasting hand motion is a better pretraining task for representation learning in embodied manipulation. Projec page: https://ruicongliu.github.io/sfhand/.