cs.CVNov 9, 2026
SaveGrasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos
Organizations: University of Washington · University of North Carolina at Chapel Hill · Johns Hopkins University · Skild AI
Abstract
To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present , a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model for several prediction tasks: forecasting future interactions, predicting hand motion given object motion or vice versa, generating both trajectories toward a desired final object pose, and completing partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets as training data for all tasks. We demonstrate strong performance across all tasks, outperforming released forecasters on datasets with 3D annotations. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment. Videos are available at https://grasping-the-future.github.io.
Figures & tables
Figure 1: Unified model for different types of hand-object prediction. Grasping-the-Future forecasts future motion, fills temporal gaps, predicts either entity from the other’s motion, and generates motion toward a given goal. Neutral poses show observations; teal and amber arrows indicate hand and object prediction, respectively. Fading poses show successive times.
Figure 2: Model overview. An interaction is represented as a grid of tokens, with rows indicating states and relations , and one column per frame. Observed cells are given, and unobserved cells are initialized to noise. A transformer attends to rows, columns and across the grid to predict the flow velocity of the noise cells (with the context tokens (blue) as additional keys/values). The noise is decoded into a sample by ten Euler steps, producing hand and object trajectories. Multiple samples for the same input vs. ground truth (dashed). The bottom row is used for training. An observation mask is sampled per window during training so the same model trains on all tasks (but with a different set of observed cells). Flow matching is used for velocity supervision and auxiliary losses are applied to decoded trajectories.
Figure 3: Extraction pipeline. Using four refinement steps, we combine pretrained estimates of hands, objects, depth, and camera motion into a canonical trajectory (green). We only keep clips that pass all consistency checks. The bottom row displays examples rejected by each check, and the bar displays the number of clips we discard across stages.
Figure 4: Qualitative predictions across five tasks. Each row shows one clip, its observation mask, the input frame with and without the observed motion, and three snapshots of the predictions. Mask rows correspond to the two hands and the object. Orange denotes observations and purple predictions; darker orange indicates later observations, while faded purple shows earlier predictions. Blue denotes the reference object pose in the hands-to-object row and the given goal pose.
| Future prediction | Hands obj. | Obj. hands | Temporal infilling | Goal, all given | Goal, hands given | Goal, object given | ||||||
| Entity | object | hands | object | hands | object | hands | object | hands | object | hands | object | hands |
| Error at | end | end | end | end | middle | middle | middle | middle | end | middle | middle | end |
| Stay-put | 142.6 | 145.6 | 149.0 | 156.6 | 51.7 | 69.3 | 98.8 | 127.7 | 149.0 | 127.7 | 98.8 | 156.6 |
| Constant velocity | 627.1 | 819.3 | – | – | – | – | – | – | – | – | – | – |
| Interpolation | – | – | – | – | 38.2 | 49.0 | 76.0 | 92.3 | – | 92.3 | 76.0 | – |
| Rigid carry | – | – | 108.6 | 156.4 | – | – | – | – | – | – | – | – |
Table 1: Prediction across seven tasks. Position errors are in millimeters, and hand errors average the left and right wrists. The row “Error at” shows whether a column scores the last frame or, when the final pose is given as input, the middle of the generated frames. “Motion capture only” is our model trained without the trajectories extracted from internet videos. For both models, we report the error of a single sample, the energy score of the samples, and the error of the best of the samples (Section 4.1 ). Because the energy score subtracts the distance between samples, it is not the error of any one trajectory and can be lower than the best-of- error. The best of is chosen using the reference, so it has its own block and is left out when marking the best (bold) and second-best (underlined) results. The last block shows the percentage of windows in which the reference lies within the samples, ideally .
| Inputs at inference | ADE | FDE | ||||||||||
| Method | RGB | Depth | Hands | Object | Text | L wrist | R wrist | Object | L wrist | R wrist | Object | |
| USST ( Bao et al., 2023 ) | ✓ | ✓ | 1 | 134.6 | 135.2 | – | 196.3 | 182.3 | – | |||
| ForeHand4D ( Prakash et al., 2025 ) | ✓ | 20 | 92.4 | 103.1 | – | 122.8 | 133.6 | – | ||||
| EgoH4 ( Hatano et al., 2025 ) | ✓ | 2D | 1 | 223.4 | 266.9 | – | 292.5 | 327.8 | – | |||
| Uni-Hand ( Ma et al., 2026 ) | ✓ | ✓ | ✓ | 20 | 360.8 | 344.1 | – | 359.5 | 348.7 | – | ||
| ObjectForesight ( Soraki et al., 2026 ) | ✓ | ✓ | ✓ | 20 | – | – | 77.0 | – | – | 130.4 | ||
Table 2: Comparison with different forecasting approaches. ADE and FDE in millimeters with the listed inputs. Methods with samples are scored with the energy score of all samples, which equals the error when . Our model uses estimated depth here. Bold and underline mark the best and second-best results.
Figure 5: Testing physical plausibility through retargeting. We test whether simulated robot hands can manipulate the object along the predicted trajectory from Grasping-the-Future . The left panel shows the observed frame, our s forecast, and simulated replay after SPIDER retargeting ( Pan et al., 2026 ) . The table reports success across trajectories in different scenes.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Observed in training | Observed at evaluation | Training probability |
|---|---|---|---|
| Forecasting | first frames, | first s | |
| Hands object | hands throughout, object in the first frame | same | |
| Object hands | object throughout, hands in the first frame | same | |
| Goal-conditioned † | first – and last – frames | first and last frames | |
| Temporal infilling | all except one gap of – of the window | all except the middle third | |
| Random blocks | all except one block per selected entity | training only |
Table 3: Observation patterns of each task. is the number of valid frames. † Goal-conditioned training gives goals for all entities, only the hands, or only the object, with probabilities , , and . Entities without a goal are observed only in the first frame.
| Architecture | |
| Transformer | blocks; width ; heads; feed-forward width ; GELU; dropout |
| Parameters | M trainable; M frozen |
| Training data | |
| Dataset sizes | Action100M / ; TACO / ; DexYCB / ; OakInk2 / ; SHOW3D / ; HOT3D / ; H2O / ; HO3D / ; HO-Cap / ; total / |
Table 4: Model and training settings. Dataset sizes give training trajectories / five-second windows.
| Flow matching | Decoded trajectory | Contact and samples | |||
|---|---|---|---|---|---|
| Hand translation | Wrist and object position | Fingertip distance | |||
| Wrist rotation | Wrist and object rotation | Attachment position | |||
| Finger articulation | Finger-joint rotation | Attachment rotation | |||
| Object translation | Smoothness | Attachment velocity | |||
| Object rotation | Joint limits | Contact classifier | |||
| Hand-object relations | MANO pose prior | Energy score | |||
Table 5: Training loss weights. The flow-matching weights combine the six components of , and the other weights scale the auxiliary losses.
| DexYCB | HOT3D | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Variant | wrist | depth | surface | obj. depth | wrist | depth | surface | obj. depth | drift |
| Raw extraction | |||||||||
| All denoiser corrections | |||||||||
| Moving-average smoothing | |||||||||
| Full refinement | |||||||||
Table 6: Refinement results. Median errors in millimeters, except object rotation drift in degrees. Hand metrics use the right hand, and surface is the hand vertex error after aligning the wrists. Depth errors are signed and best at zero.
| Method | Contact deviation | Relative position | Motion deviation | Acceleration | Hand joint error | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|
| (mm) | (mm) | (mm) | (m/s 2 ) | (mm) | (%) | (%) | |
| Published results | |||||||
| ArcticNet-SF ( Fan et al., 2023 ) | 94.2 | 166.7 | 70.1 | 3.9 / 5.3 | 44.8 | 16.5 | 55.9 |
| JointTransformer ( Abou Zeid, 2023 ) | 30.1 | 78.6 | 20.0 | 3.1 / 9.5 | 22.9 | 17.6 | 56.9 |
| HOPformer ( Bansal et al., 2026 ) | 20.7 | 65.8 | 11.4 | 2.5 / 4.1 | 19.9 | 29.8 | 69.7 |
| Same clips, prepared inputs and ground-truth mesh | |||||||
Table 7: EPIC-Contact reconstruction. Published results cover all test clips. We score the other rows on the clips our pipeline completes. Motion deviation and acceleration depend on frame rate, so we compute them for both methods at our frames, sampled at about Hz. Acceleration is given for the hand and the object. SR is the percentage of frames whose object pose error is below or of the object diameter. HOPformer is trained on EPIC-Contact and, like the other published methods, receives prepared hand crops and each clip’s ground-truth object mesh. In the last block, HOPformer instead receives only the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts, and our pipeline reconstructs the mesh from video given the object’s name. Bold marks the better result in the last block.
| EPIC-Contact | HOT3D | DexYCB | ||||
|---|---|---|---|---|---|---|
| Method | Hand | Object | Hand | Object | Hand | Object |
| HOPformer | 23.9 | 24.0 | 52.2 | 38.5 | 96.6 | 28.0 |
| Ours | 15.8 | 18.3 | 55.0 | 25.5 | 10.3 | 20.7 |
Table 8: HOPformer and our pipeline when every input comes from the video. HOPformer receives the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts. Hand error is the -joint error after aligning the wrist. Object error is the ADD-S error after aligning the object centers, as in the EPIC-Contact evaluator, which averages the distance from each point of the predicted mesh to the nearest point of the reference mesh. Errors are in millimeters, and both methods are scored on the same frames. EPIC-Contact is HOPformer’s training domain, HOT3D is also egocentric, and DexYCB is filmed from third-person views.
| Hands ADE | Object ADE | Hands FDE | Object FDE | Inside (%) | ||||||||||
| Model | ES | ES | ES | ES | Hands | Object | ||||||||
| Main model | 86.7 | 60.2 | 49.6 | 64.9 | 42.0 | 37.5 | 128.5 | 93.3 | 74.8 | 110.3 | 76.9 | 65.1 | 82 | 79 |
| Losses and relation tokens | ||||||||||||||
| No physical-loss group | 90.2 | 62.2 | 51.3 | 64.9 | 43.0 | 38.5 | 135.4 | 101.3 | 80.1 | 111.9 | 80.5 | 68.8 | 81 | 76 |
| No relation tokens | 91.2 | 62.4 | 51.1 | 67.9 | 43.7 | 39.3 | 132.7 | 98.8 | 76.4 | 113.4 | 78.8 | 67.7 | 81 | 76 |
| Scene conditioning and augmentation | ||||||||||||||
Table 9: Model ablations. (a) Trajectory errors in millimeters for one sample ( ), the best of ( ), and the energy score of all (ES). ADE averages the six tasks that generate each entity, and FDE the three where its final pose is not given. Inside is the percentage of windows with the reference inside the samples at the final position (Section 4.1 ). (b) Physical metrics of one sample.
| Model | Tip error (mm) | Contact F1 (%) | Held (%) | Slipping (%) | Detached (%) |
|---|---|---|---|---|---|
| Main model | 65.7 | 68.2 | 15.5 | 47.5 | 24.9 |
| Losses and relation tokens | |||||
| No physical-loss group | 72.2 | 65.3 | 14.3 | 44.5 | 29.1 |
| No relation tokens | 70.4 | 66.3 | 14.2 | 45.5 | 28.1 |
| Scene conditioning and augmentation | |||||
| No scene encoder | 67.6 | 67.3 | 15.6 | 46.6 | 26.0 |
Table 10: Interaction quality of the ablations. Means over the samples of each window and over all seven tasks. Tip error is the fingertip error during contact, and held, slipping, and detached are percentages of reference-grasp frames. For every ablation, the confidence interval of its difference from the main model excludes zero for tip error, contact F1, and detached frames.
Table 11: Energy score with matched spread. The best baseline of Table 1 , alone and with Gaussian noise matching our samples’ spread in each window, against our energy score. Rigid carry is not tested.
| Method | Inside | Apart | Error | |
|---|---|---|---|---|
| Hand forecasting | ||||
| ForeHand4D ( Prakash et al., 2025 ) | 20 | 82% | 191.8 | 224.1 |
| EgoH4 ( Hatano et al., 2025 ) | 20 | 0% | 0.2 | 310.2 |
| Uni-Hand ( Ma et al., 2026 ) | 20 | 0% | 11.2 | 359.8 |
| Ours | 20 | 76% | 125.2 | 160.5 |
| Object forecasting | ||||
Table 12: Reference inside the samples. Forecasting at the final position on the windows of Table 2 , with both wrists averaged for hand methods. Ours is scored on the same windows and with the same number of samples as the methods in its group. Inside is the percentage of windows with the reference inside the samples, ideally for and for . Apart is the mean distance between two samples and Error the mean distance from a sample to the reference, both in millimeters.
| Future prediction | Hands obj. | Obj. hands | Temporal infilling | Goal, all given | Goal, hands given | Goal, object given | ||||||
| Entity | object | hands | object | hands | object | hands | object | hands | object | hands | object | hands |
| Error at | end | end | end | end | middle | middle | middle | middle | end | middle | middle | end |
| Energy score of the samples | ||||||||||||
| Specialist | 95.2 | 101.2 | 37.5 | 62.0 | 21.7 | 28.7 | 53.6 | 60.0 | 53.0 | 61.3 | 54.7 | 61.0 |
| Ours | 89.0 | 94.4 | 35.1 | 59.6 | 22.2 | 29.8 | 51.0 | 59.3 | 71.1 | 62.0 | 53.5 | 70.5 |
| Reference inside the samples (%) | ||||||||||||
Table 13: Comparison with task specialists. In each column, the specialist is the model trained only for that task, and one goal specialist covers all three goal patterns. Bold marks the better of the two models in each block. Notation follows Table 1 .