Organizations: AI Lab, Country Garden Services Group · School of Mechanical Science and Engineering, Huazhong University of Science and Technology · Omni AI
How much does hand motion tell us about what a first-person camera will see next? EgoHandWAM forecasts three-dimensional wrist and fingertip trajectories and a future visual feature from the current image and recent hand states. We train on 751,065 EgoDex windows using a frozen DINOv2 encoder. On held-out development sessions, the action-only model reduces two-second wrist endpoint error from 60.26 to 56.48~mm relative to a stationary forecast. It still underestimates long reaches. Adding future-feature supervision gives nearly identical trajectory accuracy. With the recorded hand history, the visual head reduces smooth L1 error by 1.32% relative to copying the current feature. Shuffling histories across sessions erases this gain. In a separate four-task probe, adding recorded future displacements to observed context and future tracking-validity masks reduces visual-feature error by 3.93%. Masks alone give little improvement. A trajectory predicted from the same observations adds little. On two simulated RoboTwin tasks, substituting aligned human examples for an auxiliary batch of robot examples gives no consistent increase in task success across three training seeds.
Figures & tables
Fig. 1: EgoDex forecasting model. The action head predicts hand coordinates at all 60 query times. The visual head uses the hidden state at the final time query to predict the visual feature at two seconds. Solid arrows show the forward computation and residual connections. Dashed orange arrows indicate gradients to trainable parameters. The encoder is frozen. The photographs show recorded observations; hand symbols illustrate the output geometry.
Method
Wrist ADE
Wrist FDE
MPJPE-12
Stationary wrist and fingertips
40.685
60.262
48.056
Constant velocity
90.566
170.297
113.126
Action-only
38.638±0.056
56.480±0.197
45.413±0.035
Joint
38.631±0.057
56.460±0.189
45.413±0.041
Two-second wrist displacement, grouped by recorded distance
Wrist group
Recorded
Action-only forecast
TABLE I: Main comparison on the same EgoDex development set. Endpoint and trajectory errors (top) and wrist displacements (bottom) are in millimeters. Smaller errors indicate more accurate forecasts. Learned models use the full eligible training set. Model errors give the mean and sample standard deviation (SD) across four paired seeds. Forecast displacements give the mean. Both learned models use the same inputs, initialization and training budget.
Fig. 2: Wrist forecasts from the four-task probe in three held-out EgoDex lid episodes [ 2 ] . Cyan dots mark recorded positions and pink crosses mark forecasts made at 0 s. Rows A and B show the left wrist, and row C shows the right wrist.
Forecast
Smooth L1
Cosine distance
Copy current feature
0.1850
0.2045
Joint, aligned history
0.1825
0.2029
Joint, shuffled history
0.1884
0.2084
TABLE II: Two-second DINO feature prediction on matched EgoDex windows. Model errors are means across four training seeds. Lower is better.
Probe input
Four tasks
Lid task
Copy current feature
0.436436
0.690897
Observed context
0.389063
0.579985
Context + recorded future mask
0.389087
0.577395
Context + shuffled forecast
0.389552
0.581673
Context + forecast
0.389050
0.580010
Context + future mask and displacements
0.373793
0.537339
TABLE III: Predicting visual feature change at two seconds. Normalized mean squared error is macro-averaged over four tasks or reported for the lid task alone. Lower values indicate smaller errors. Recorded trajectories include future validity masks.
Fig. 3: Robot transfer experiment. The solid path maps robot observations to executable commands. The dashed paths add human observations and a future-feature loss during training. The EgoDex trajectory forecast in Fig. 1 is evaluated separately.
Supervision
Bottle grasping
Block handover
Robot only
16.0±2.0
39.0±8.7
Robot + aligned history
15.0±2.6
37.7±4.0
Robot + shuffled history
14.7±1.5
41.7±9.0
TABLE IV: Closed-loop success (%) on the same 100 held-out scenes per task. Entries are means and sample SDs across three training seeds. Aligned and shuffled refer to the hand history paired with each human video during training.