Organizations: School of Information and Control Engineering, China University of Mining and Technology, Xuzhou, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai, China.
Forecasting future hand movements from egocentric videos plays an important role in understanding human intentions and hand-object interactions. However, different manipulation intentions can lead to substantially different hand trajectories even when interacting with the same object. Existing methods have not sufficiently explored the intrinsic relationships between articulated hand movements and object semantics to capture such operation-dependent motion patterns. Therefore, we propose TOMAS, a topology-aware operation modeling and adaptive staging framework for egocentric hand forecasting. Specifically, we develop a topology-aware hand-object reasoning mechanism that associates hand skeletal structures and articulation dynamics with object semantics and spatial relationships to jointly infer potential interaction targets and latent manipulation operations. The resulting object-operation representations provide interaction conditions for future hand trajectory prediction, with supervision adapted to the annotations available in each dataset. Furthermore, we introduce an adaptive staging mechanism that organizes future hand movements into ordered latent stages with learnable temporal boundaries. Stage-specific geometric subgoals are then incorporated to guide trajectory refinement according to the underlying manipulation patterns. Experimental results on multiple publicly available benchmarks demonstrate that TOMAS achieves state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting.
Figures & tables
Fig. 1: Operation-dependent hand forecasting. The same object can support different operations with distinct future hand trajectories.
Fig. 2: System overview of TOMAS. Given observed egocentric video and historical hand trajectories, TOMAS employs topology-aware hand–object reasoning to jointly model candidate-object semantics and latent operation representations. The resulting interaction-aware representations guide ordered latent motion stages and stage-wise geometric subgoals for future hand trajectory forecasting.
Fig. 3: Visualization of articulated hand motion and skeletal topology.
Fig. 4: Topology-aware object–operation joint reasoning. Observed object candidates and articulated hand configurations are encoded as candidate-specific interaction representations. Cross-modal memory retrieval supports joint object–operation scoring and the formation of interaction-conditioning tokens.
Fig. 5: Interaction-conditioned adaptive staging and geometry-guided trajectory generation. Interaction-conditioned motion features determine ordered stage allocations and stage-level operation mixtures. The resulting candidate-relative geometric subgoals guide continuous trajectory decoding and refinement.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Method
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
CVH [ 9 ]
1.100
1.278
0.952
1.018
0.146
0.148
OCT [ 1 ]
0.370
0.524
0.309
0.397
0.103
0.126
USST [ 2 ]
0.183
0.341
0.120
0.185
0.031
0.052
S-Mamba [ 38 ]
0.185
0.355
0.138
0.207
0.038
0.074
Diff-IP3D [ 9 ]
0.199
0.377
0.156
0.229
0.049
0.081
TABLE I: 3D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Best and second-best scores are in bold and green, respectively.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Method
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
CVH [ 9 ]
0.180
0.230
0.188
0.221
0.206
0.208
OCT [ 1 ]
0.108
0.122
0.091
0.147
0.387
0.381
USST [ 2 ]
0.082
0.118
0.060
0.087
0.040
0.068
S-Mamba [ 38 ]
0.084
0.141
0.071
0.118
0.051
0.094
Diff-IP2D [ 9 ]
0.080
0.130
0.066
0.087
0.042
0.074
TABLE II: 2D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Best and second-best scores are in bold and green, respectively.
Method
ADE ↓
FDE ↓
OCT [ 1 ]
0.160
0.160
USST [ 2 ]
0.200
0.233
Uni-Hand [ 4 ]
0.181
0.208
TOMAS (ours)
0.140
0.165
TABLE III: Short-horizon comparison on EGTEA Gaze+.
Fig. 6: Qualitative comparison of 2D hand-center forecasting on EgoPAT3D-DT and H2O-PT. Green, blue, and red denote observed trajectories, future ground truth, and predicted trajectories, respectively.
Fig. 7: Qualitative comparison of 3D hand-center forecasting on EgoPAT3D-DT and H2O-PT. Green, blue, and red denote observed trajectories, future ground truth, and predicted trajectories, respectively. Coordinates are measured in meters.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Variant
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Full (TOMAS)
0.158
0.324
0.103
0.174
0.025
0.050
A1 (w/o Hand Joints)
0.161
0.326
0.106
0.175
0.027
0.051
A2 (w/o Graph Prop.)
0.161
0.329
0.105
0.182
0.025
0.051
A3 (w/o Rel. Geometry)
0.160
0.326
0.103
0.178
0.025
0.051
A4 (Decoupled Obj.-Op.)
0.161
0.326
0.107
0.175
0.025
0.051
TABLE IV: Ablation results for 3D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Full reports the three-seed average, while A1–A9 report single-seed results.
EgoPAT3D-DT (seen)
EgoPAT3D-DT (unseen)
H2O-PT
Variant
ADE ↓
FDE ↓
ADE ↓
FDE ↓
ADE ↓
FDE ↓
Full (TOMAS)
0.053
0.092
0.042
0.075
0.035
0.069
A1 (w/o Hand Joints)
0.056
0.094
0.044
0.080
0.037
0.072
A2 (w/o Graph Prop.)
0.054
0.094
0.043
0.077
0.036
0.072
A3 (w/o Rel. Geometry)
0.054
0.093
0.044
0.077
0.036
0.071
A4 (Decoupled Obj.-Op.)
0.054
0.094
0.043
0.077
0.035
0.069
TABLE V: Ablation results for 2D hand trajectory forecasting on EgoPAT3D-DT and H2O-PT. Full reports the three-seed average, while A1–A9 report single-seed results.
Variant
ADE ↓
FDE ↓
Full (TOMAS)
0.140
0.165
A1 (w/o Hand Joints)
0.159
0.185
A2 (w/o Graph Prop.)
0.145
0.173
A3 (w/o Rel. Geometry)
0.145
0.173
A4 (Decoupled Obj.-Op.)
0.145
0.173
A5 (w/o Global Cond.)
0.147
0.175
TABLE VI: Ablation results for short-horizon hand trajectory forecasting on EGTEA Gaze+. Full reports the three-seed average, while A1–A9 report single-seed results.
Variant
Top-1(%)
2D Full
72.82
2D A1 (w/o Hand Joints)
63.68
3D Full
77.61
3D A1 (w/o Hand Joints)
74.63
TABLE VII: Native H2O-PT target ranking. Each model has 1,735 windows; 1,608 have derived interaction targets. Coverage is 100% under the candidate association rule.
Fig. 8: Visualization of learned operation and candidate-interaction representations. PCA projects high-dimensional features into two dimensions. Candidate names are inferred from CLIP, and colors indicate dominant operation components in the learned representation.
Fig. 9: Candidate-conditioned historical evidence retrieval. The heat maps show final-layer, head-averaged attention from valid candidate queries and null to hand-center, hand-topology, egomotion, and RGB memory. Bars report native object-selection probabilities. Candidate names are inferred from CLIP.
Fig. 10: Learned ordered motion stages. Stage weights vary over normalized future time, while dashed vertical lines indicate predicted transition positions.
Fig. 11: Local effect of subgoal-guided refinement. The full prediction is shown in red, and the prediction without final-pass relative-subgoal-coordinate input is gray and dashed. Green indicates the last observed hand position.
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, China · University of Chinese Academy of Sciences, China