Organizations: AI Lab, Country Garden Services Group · School of Mechanical Science and Engineering, Huazhong University of Science and Technology · Omni AI
How much does hand motion tell us about what a first-person camera will see next? EgoHandWAM forecasts three-dimensional wrist and fingertip trajectories and a future visual feature from the current image and recent hand states. We train on 751,065 EgoDex windows using a frozen DINOv2 encoder. On held-out development sessions, the action-only model reduces two-second wrist endpoint error from 60.26 to 56.48~mm relative to a stationary forecast. It still underestimates long reaches. Adding future-feature supervision gives nearly identical trajectory accuracy. With the recorded hand history, the visual head reduces smooth L1 error by 1.32% relative to copying the current feature. Shuffling histories across sessions erases this gain. In a separate four-task probe, adding recorded future displacements to observed context and future tracking-validity masks reduces visual-feature error by 3.93%. Masks alone give little improvement. A trajectory predicted from the same observations adds little. On two simulated RoboTwin tasks, substituting aligned human examples for an auxiliary batch of robot examples gives no consistent increase in task success across three training seeds.
Figures & tables
Fig. 1: EgoDex forecasting model. The action head predicts hand coordinates at all 60 query times. The visual head uses the hidden state at the final time query to predict the visual feature at two seconds. Solid arrows show the forward computation and residual connections. Dashed orange arrows indicate gradients to trainable parameters. The encoder is frozen. The photographs show recorded observations; hand symbols illustrate the output geometry.
Method
Wrist ADE
Wrist FDE
MPJPE-12
Stationary wrist and fingertips
40.685
60.262
48.056
Constant velocity
90.566
170.297
113.126
Action-only
38.638±0.056
56.480±0.197
45.413±0.035
Joint
38.631±0.057
56.460±0.189
45.413±0.041
Two-second wrist displacement, grouped by recorded distance
Wrist group
Recorded
Action-only forecast
TABLE I: Main comparison on the same EgoDex development set. Endpoint and trajectory errors (top) and wrist displacements (bottom) are in millimeters. Smaller errors indicate more accurate forecasts. Learned models use the full eligible training set. Model errors give the mean and sample standard deviation (SD) across four paired seeds. Forecast displacements give the mean. Both learned models use the same inputs, initialization and training budget.
Fig. 2: Wrist forecasts from the four-task probe in three held-out EgoDex lid episodes [ 2 ] . Cyan dots mark recorded positions and pink crosses mark forecasts made at 0 s. Rows A and B show the left wrist, and row C shows the right wrist.
Forecast
Smooth L1
Cosine distance
Copy current feature
0.1850
0.2045
Joint, aligned history
0.1825
0.2029
Joint, shuffled history
0.1884
0.2084
TABLE II: Two-second DINO feature prediction on matched EgoDex windows. Model errors are means across four training seeds. Lower is better.
Probe input
Four tasks
Lid task
Copy current feature
0.436436
0.690897
Observed context
0.389063
0.579985
Context + recorded future mask
0.389087
0.577395
Context + shuffled forecast
0.389552
0.581673
Context + forecast
0.389050
0.580010
Context + future mask and displacements
0.373793
0.537339
TABLE III: Predicting visual feature change at two seconds. Normalized mean squared error is macro-averaged over four tasks or reported for the lid task alone. Lower values indicate smaller errors. Recorded trajectories include future validity masks.
Fig. 3: Robot transfer experiment. The solid path maps robot observations to executable commands. The dashed paths add human observations and a future-feature loss during training. The EgoDex trajectory forecast in Fig. 1 is evaluated separately.
Supervision
Bottle grasping
Block handover
Robot only
16.0±2.0
39.0±8.7
Robot + aligned history
15.0±2.6
37.7±4.0
Robot + shuffled history
14.7±1.5
41.7±9.0
TABLE IV: Closed-loop success (%) on the same 100 held-out scenes per task. Entries are means and sample SDs across three training seeds. Aligned and shuffled refer to the hand history paired with each human video during training.
Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand
We introduce P3-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/
World Action Models (WAMs) augment robot action generation with future visual supervision. Existing WAMs commonly fuse main and wrist observations into one visual stream and train both with the same future-video objective, despite their different visual dynamics. A stable main camera reveals scene-level task evolution, whereas wrist cameras move with the end effector, mixing local interaction changes with viewpoint shifts and self-occlusion. These contrasting predictive demands suggest that the two views may benefit from different future objectives. We introduce FutureDuet, which retains both views for control, while allowing each visual stream to receive a different future objective. For the main view, future RGB models task evolution, while interaction masks and robot skeletons focus supervision on task objects and robot motion. For the wrist stream, future latent prediction models short-horizon interaction changes without requiring pixel-level reconstruction. ActionDiT jointly reads the resulting Task State and Interaction State, combining scene-level progress with close-range interaction evidence. All auxiliary prediction modules are training-only, adding no inference overhead. FutureDuet achieves 94.2% clean and 94.1% randomized success on RoboTwin50 and 99.2% average success on LIBERO. The improvements are most pronounced on six RoboTwin50 tasks that require precise interaction, averaging gains of 9.2% and 12.8% over Fast-WAM in clean and randomized settings. Controlled studies further show complementary gains from separating the wrist pathway and designing future supervision separately for the two views.
Jie Wu, Yuzhi Huang, Junqi Liu +5
Tsinghua University · TeleAI, China Telecom · Shenzhen Technology University