cs.CVNov 17, 2026

EgoHandWAM: Probing the Role of Hand Motion in Egocentric World Action Modeling

Authors: Kailin Wang, Haoxiang Jie, Yaoyuan Yan, Zhiyou Heng, Zhaosong Li

Organizations: AI Lab, Country Garden Services Group · School of Mechanical Science and Engineering, Huazhong University of Science and Technology · Omni AI

Abstract

How much does hand motion tell us about what a first-person camera will see next? EgoHandWAM forecasts three-dimensional wrist and fingertip trajectories and a future visual feature from the current image and recent hand states. We train on 751,065 EgoDex windows using a frozen DINOv2 encoder. On held-out development sessions, the action-only model reduces two-second wrist endpoint error from 60.26 to 56.48~mm relative to a stationary forecast. It still underestimates long reaches. Adding future-feature supervision gives nearly identical trajectory accuracy. With the recorded hand history, the visual head reduces smooth L1 error by 1.32% relative to copying the current feature. Shuffling histories across sessions erases this gain. In a separate four-task probe, adding recorded future displacements to observed context and future tracking-validity masks reduces visual-feature error by 3.93%. Masks alone give little improvement. A trajectory predicted from the same observations adds little. On two simulated RoboTwin tasks, substituting aligned human examples for an auxiliary batch of robot examples gives no consistent increase in task success across three training seeds.

Figures & tables

Explore similar work

CardsList
  1. FutureDuet: Decoupling Observation Access from Future Supervision in World Action Models

    Sep 28, 2026Jie Wu, Yuzhi Huang, Junqi Liu +5World Models for RoboticsWorld Action Models