Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos

Rustin Soraki; Matthew Wallingford; Ryan Aby; Brandon Chee; Jason Ren; Ali Farhadi; Homanga Bharadhwaj; Roozbeh Mottaghi

cs.CVNov 9, 2026

Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Videos

Authors: Rustin Soraki, Matthew Wallingford, Ryan Aby, Brandon Chee, Jason Ren, Ali Farhadi, Homanga Bharadhwaj, Roozbeh Mottaghi

Organizations: University of Washington · University of North Carolina at Chapel Hill · Johns Hopkins University · Skild AI

Abstract

To interact with an object, we need to anticipate both how it will move and how our hands should move with it. Human videos capture these relationships across everyday activities, offering a natural source for learning to predict future interactions. Learning from them requires recovering motion from completed interactions and using it to anticipate what could happen next. We present $\textit{Grasping-the-Future}$ , a unified generative model of joint 3D hand-object motion, including wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, we train a single model for several prediction tasks: forecasting future interactions, predicting hand motion given object motion or vice versa, generating both trajectories toward a desired final object pose, and completing partially observed interactions. We explicitly model hand-object relations and incorporate contact constraints to help preserve grasp relationships during motion. To obtain 3D supervision from internet videos, we develop a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct completed interactions. We combine these recovered trajectories with annotated interaction datasets as training data for all tasks. We demonstrate strong performance across all tasks, outperforming released forecasters on datasets with 3D annotations. Moreover, we show geometric and physical plausibility of the predicted trajectories by retargeting them to a robotic embodiment. Videos are available at https://grasping-the-future.github.io.

Figures & tables

Figure 1: Unified model for different types of hand-object prediction. Grasping-the-Future forecasts future motion, fills temporal gaps, predicts either entity from the other’s motion, and generates motion toward a given goal. Neutral poses show observations; teal and amber arrows indicate hand and object prediction, respectively. Fading poses show successive times.

Figure 2: Model overview. An interaction is represented as a grid of tokens, with rows indicating states $\mathbf{s}$ and relations $\mathbf{r}$ , and one column per frame. Observed cells are given, and unobserved cells are initialized to noise. A transformer attends to rows, columns and across the grid to predict the flow velocity of the noise cells (with the context tokens (blue) as additional keys/values). The noise is decoded into a sample by ten Euler steps, producing hand and object trajectories. Multiple samples for the same input vs. ground truth (dashed). The bottom row is used for training. An observation mask is sampled per window during training so the same model trains on all tasks (but with a different set of observed cells). Flow matching is used for velocity supervision and auxiliary losses are applied to decoded trajectories.

Figure 3: Extraction pipeline. Using four refinement steps, we combine pretrained estimates of hands, objects, depth, and camera motion into a canonical trajectory (green). We only keep clips that pass all consistency checks. The bottom row displays examples rejected by each check, and the bar displays the number of clips we discard across stages.

Figure 4: Qualitative predictions across five tasks. Each row shows one clip, its observation mask, the input frame with and without the observed motion, and three snapshots of the predictions. Mask rows correspond to the two hands and the object. Orange denotes observations and purple predictions; darker orange indicates later observations, while faded purple shows earlier predictions. Blue denotes the reference object pose in the hands-to-object row and the given goal pose.

	Future prediction		Hands $\to$ obj.	Obj. $\to$ hands	Temporal infilling		Goal, all given		Goal, hands given		Goal, object given
Entity	object	hands	object	hands	object	hands	object	hands	object	hands	object	hands
Error at	end	end	end	end	middle	middle	middle	middle	end	middle	middle	end
Stay-put	142.6	145.6	149.0	156.6	51.7	69.3	98.8	127.7	149.0	127.7	98.8	156.6
Constant velocity	627.1	819.3	–	–	–	–	–	–	–	–	–	–
Interpolation	–	–	–	–	38.2	49.0	76.0	92.3	–	92.3	76.0	–
Rigid carry	–	–	108.6	156.4	–	–	–	–	–	–	–	–

Table 1: Prediction across seven tasks. Position errors are in millimeters, and hand errors average the left and right wrists. The row “Error at” shows whether a column scores the last frame or, when the final pose is given as input, the middle of the generated frames. “Motion capture only” is our model trained without the trajectories extracted from internet videos. For both models, we report the error of a single sample, the energy score of the

20

samples, and the error of the best of the

20

samples (Section 4.1 ). Because the energy score subtracts the distance between samples, it is not the error of any one trajectory and can be lower than the best-of-

20

error. The best of

20

is chosen using the reference, so it has its own block and is left out when marking the best (bold) and second-best (underlined) results. The last block shows the percentage of windows in which the reference lies within the

20

samples, ideally

95\%

Inputs at inference

ADE

FDE

Method

RGB

Depth

Hands

Object

Text

N

L wrist

R wrist

Object

L wrist

R wrist

Object

USST ( Bao et al., 2023 )

✓

\times

✓

\times

\times

134.6

135.2

–

196.3

182.3

–

ForeHand4D ( Prakash et al., 2025 )

✓

\times

\times

\times

\times

92.4

103.1

–

122.8

133.6

–

EgoH4 ( Hatano et al., 2025 )

✓

\times

\times

\times

223.4

266.9

–

292.5

327.8

–

Uni-Hand ( Ma et al., 2026 )

✓

\times

\times

360.8

344.1

–

359.5

348.7

–

ObjectForesight ( Soraki et al., 2026 )

✓

\times

✓

\times

–

77.0

–

130.4

Table 2: Comparison with different forecasting approaches. ADE and FDE in millimeters with the listed inputs. Methods with

N

samples are scored with the energy score of all samples, which equals the error when

N{=}1

. Our model uses estimated depth here. Bold and underline mark the best and second-best results.

Figure 5: Testing physical plausibility through retargeting. We test whether simulated robot hands can manipulate the object along the predicted trajectory from Grasping-the-Future . The left panel shows the observed frame, our $4.5$ s forecast, and simulated replay after SPIDER retargeting ( Pan et al., 2026 ) . The table reports success across $103$ trajectories in different scenes.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Observed in training	Observed at evaluation	Training probability
Forecasting	first $K$ frames, $K\sim\mathrm{Unif}\{1,\ldots,T_{v}-1\}$	first $0.5$ s	$0.19$
Hands $\to$ object	hands throughout, object in the first frame	same	$0.19$
Object $\to$ hands	object throughout, hands in the first frame	same	$0.19$
Goal-conditioned †	first $1$ – $5$ and last $1$ – $3$ frames	first and last frames	$0.19$
Temporal infilling	all except one gap of $20$ – $50\%$ of the window	all except the middle third	$0.19$
Random blocks	all except one block per selected entity	training only	$0.05$

Appendix

Table 3: Observation patterns of each task.

T_{v}

is the number of valid frames. † Goal-conditioned training gives goals for all entities, only the hands, or only the object, with probabilities

0.5

0.25

, and

0.25

. Entities without a goal are observed only in the first frame.

Architecture

Transformer	$12$ blocks; width $512$ ; $8$ heads; feed-forward width $2048$ ; GELU; dropout $0.1$
Parameters	$103.3$ M trainable; $12.7$ M frozen
Training data

Dataset sizes	Action100M $62{,}562$ / $109{,}346$ ; TACO $4{,}089$ / $6{,}267$ ; DexYCB $2{,}280$ / $2{,}280$ ; OakInk2 $2{,}213$ / $9{,}938$ ; SHOW3D $1{,}638$ / $3{,}552$ ; HOT3D $1{,}016$ / $1{,}016$ ; H2O $191$ / $930$ ; HO3D $173$ / $867$ ; HO-Cap $150$ / $465$ ; total $74{,}312$ / $134{,}661$

Appendix

Table 4: Model and training settings. Dataset sizes give training trajectories / five-second windows.

Flow matching		Decoded trajectory		Contact and samples
Hand translation	$0.884$	Wrist and object position	$1.0$	Fingertip distance	$0.335$
Wrist rotation	$0.738$	Wrist and object rotation	$0.368$	Attachment position	$0.178$
Finger articulation	$1.112$	Finger-joint rotation	$0.422$	Attachment rotation	$0.041$
Object translation	$0.730$	Smoothness	$3.2\times 10^{-4}$	Attachment velocity	$0.121$
Object rotation	$0.306$	Joint limits	$3.35\times 10^{-3}$	Contact classifier	$0.1$
Hand-object relations	$0.185$	MANO pose prior	$5.0\times 10^{-4}$	Energy score	$1.0$

Appendix

Table 5: Training loss weights. The flow-matching weights combine the six components of

\mathcal{L}_{\mathrm{FM}}

, and the other weights scale the auxiliary losses.

	DexYCB				HOT3D
Variant	wrist	depth	surface	obj. depth	wrist	depth	surface	obj. depth	drift
Raw extraction	$106.4$	$-93.5$	$26.1$	$-56.2$	$191.2$	$153.3$	$67.4$	$69.7$	$30.1$
All denoiser corrections	$97.6$	$-4.3$	$41.9$	$-12.3$	$162.5$	$72.5$	$93.6$	$19.6$	$43.5$
Moving-average smoothing	$99.2$	$-62.4$	$29.1$	$-48.6$	$205.7$	$173.2$	$72.1$	$75.7$	$32.2$
Full refinement	$88.6$	$-4.7$	$27.3$	$-23.8$	$158.7$	$78.6$	$70.2$	$19.4$	$30.1$

Appendix

Table 6: Refinement results. Median errors in millimeters, except object rotation drift in degrees. Hand metrics use the right hand, and surface is the hand vertex error after aligning the wrists. Depth errors are signed and best at zero.

Published results
Method	Contact deviation $\downarrow$	Relative position $\downarrow$	Motion deviation $\downarrow$	Acceleration $\downarrow$	Hand joint error $\downarrow$	[email protected] $\uparrow$	[email protected] $\uparrow$
	(mm)	(mm)	(mm)	(m/s 2 )	(mm)	(%)	(%)
ArcticNet-SF ( Fan et al., 2023 )	94.2	166.7	70.1	3.9 / 5.3	44.8	16.5	55.9
JointTransformer ( Abou Zeid, 2023 )	30.1	78.6	20.0	3.1 / 9.5	22.9	17.6	56.9
HOPformer ( Bansal et al., 2026 )	20.7	65.8	11.4	2.5 / 4.1	19.9	29.8	69.7
Same $215$ clips, prepared inputs and ground-truth mesh

Appendix

Table 7: EPIC-Contact reconstruction. Published results cover all

237

test clips. We score the other rows on the

215

clips our pipeline completes. Motion deviation and acceleration depend on frame rate, so we compute them for both methods at our frames, sampled at about

10

Hz. Acceleration is given for the hand and the object. SR is the percentage of frames whose object pose error is below

5\%

10\%

of the object diameter. HOPformer is trained on EPIC-Contact and, like the other published methods, receives prepared hand crops and each clip’s ground-truth object mesh. In the last block, HOPformer instead receives only the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts, and our pipeline reconstructs the mesh from video given the object’s name. Bold marks the better result in the last block.

	EPIC-Contact		HOT3D		DexYCB
Method	Hand $\downarrow$	Object $\downarrow$	Hand $\downarrow$	Object $\downarrow$	Hand $\downarrow$	Object $\downarrow$
HOPformer	23.9	24.0	52.2	38.5	96.6	28.0
Ours	15.8	18.3	55.0	25.5	10.3	20.7

Appendix

Table 8: HOPformer and our pipeline when every input comes from the video. HOPformer receives the video frames, hand crops from the WiLoR detector, and the SAM 3D mesh of the object, aligned to the template of the class it predicts. Hand error is the

21

-joint error after aligning the wrist. Object error is the ADD-S error after aligning the object centers, as in the EPIC-Contact evaluator, which averages the distance from each point of the predicted mesh to the nearest point of the reference mesh. Errors are in millimeters, and both methods are scored on the same frames. EPIC-Contact is HOPformer’s training domain, HOT3D is also egocentric, and DexYCB is filmed from third-person views.

Losses and relation tokens
	Hands ADE $\downarrow$			Object ADE $\downarrow$			Hands FDE $\downarrow$			Object FDE $\downarrow$			Inside (%)
Model	$N{=}1$	$N{=}20$	ES	$N{=}1$	$N{=}20$	ES	$N{=}1$	$N{=}20$	ES	$N{=}1$	$N{=}20$	ES	Hands	Object
Main model	86.7	60.2	49.6	64.9	42.0	37.5	128.5	93.3	74.8	110.3	76.9	65.1	82	79
No physical-loss group	90.2	62.2	51.3	64.9	43.0	38.5	135.4	101.3	80.1	111.9	80.5	68.8	81	76
No relation tokens	91.2	62.4	51.1	67.9	43.7	39.3	132.7	98.8	76.4	113.4	78.8	67.7	81	76
Scene conditioning and augmentation

Appendix

Table 9: Model ablations. (a) Trajectory errors in millimeters for one sample (

N{=}1

), the best of

20

(

N{=}20

), and the energy score of all

20

(ES). ADE averages the six tasks that generate each entity, and FDE the three where its final pose is not given. Inside is the percentage of windows with the reference inside the samples at the final position (Section 4.1 ). (b) Physical metrics of one sample.

Losses and relation tokens
Model	Tip error (mm) $\downarrow$	Contact F1 (%) $\uparrow$	Held (%) $\uparrow$	Slipping (%) $\downarrow$	Detached (%) $\downarrow$
Main model	65.7	68.2	15.5	47.5	24.9
No physical-loss group	72.2	65.3	14.3	44.5	29.1
No relation tokens	70.4	66.3	14.2	45.5	28.1
Scene conditioning and augmentation
No scene encoder	67.6	67.3	15.6	46.6	26.0

Appendix

Table 10: Interaction quality of the ablations. Means over the

20

samples of each window and over all seven tasks. Tip error is the fingertip error during contact, and held, slipping, and detached are percentages of reference-grasp frames. For every ablation, the

95\%

confidence interval of its difference from the main model excludes zero for tip error, contact F1, and detached frames.

Table 11: Energy score with matched spread. The best baseline of Table 1 , alone and with Gaussian noise matching our samples’ spread in each window, against our energy score. Rigid carry is not tested.

Hand forecasting
Method	$N$	Inside	Apart	Error
ForeHand4D ( Prakash et al., 2025 )	20	82%	191.8	224.1
EgoH4 ( Hatano et al., 2025 )	20	0%	0.2	310.2
Uni-Hand ( Ma et al., 2026 )	20	0%	11.2	359.8
Ours	20	76%	125.2	160.5
Object forecasting

Appendix

Table 12: Reference inside the samples. Forecasting at the final position on the windows of Table 2 , with both wrists averaged for hand methods. Ours is scored on the same windows and with the same number of samples as the methods in its group. Inside is the percentage of windows with the reference inside the samples, ideally

95\%

for

N{=}20

and

91\%

for

N{=}10

. Apart is the mean distance between two samples and Error the mean distance from a sample to the reference, both in millimeters.

Energy score of the $20$ samples
	Future prediction		Hands $\to$ obj.	Obj. $\to$ hands	Temporal infilling		Goal, all given		Goal, hands given		Goal, object given
Entity	object	hands	object	hands	object	hands	object	hands	object	hands	object	hands
Error at	end	end	end	end	middle	middle	middle	middle	end	middle	middle	end
Specialist	95.2	101.2	37.5	62.0	21.7	28.7	53.6	60.0	53.0	61.3	54.7	61.0
Ours	89.0	94.4	35.1	59.6	22.2	29.8	51.0	59.3	71.1	62.0	53.5	70.5
Reference inside the $20$ samples (%)

Appendix

Table 13: Comparison with task specialists. In each column, the specialist is the model trained only for that task, and one goal specialist covers all three goal patterns. Bold marks the better of the two models in each block. Notation follows Table 1 .

Hand-Object Interaction

See all 50 papers

Jul 17, 2026

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/

Homanga Bharadhwaj, Yash Jangir

Department of Computer Science, Johns Hopkins University

Scene Flow Estimation Trajectory Prediction 3D Scene Understanding Human-Object Interaction 3D Scene Prediction

Ask PDF

Nov 22, 2025

Predictive Representation Learning for Manipulation via Streaming Egocentric 3D Hand Forecasting

Learning embodied manipulation from egocentric human data offers a scalable path, reducing the cost of collecting robot-specific demonstrations. However, existing representation learning methods largely stop at perceptual understanding, such as extracting what a scene contains or what an action is. A capable embodied agent is more than understanding: it predicts the next motion. We propose that forecasting 3D human hand motion is one suitable pretraining task, as the representation jointly learns how to manipulate objects and how the motion will unfold next. Building on this hypothesis, we introduce SFHand, the first representation learning method via streaming egocentric 3D hand forecasting. SFHand learns by autoregressively predicting future 3D hand states, including hand type, 2D bounding box, 3D pose, and trajectory, from a continuous stream of video and language instructions. Its autoregressive architecture is equipped with an ROI-enhanced memory layer, capturing temporal context while focusing on salient hand-centric regions. To enable this research, we also introduce SFHand, the first open-source large-scale dataset pairing synchronized 3D hand poses with fine-grained language instructions (247K clips, 3.95M frames). Transferred to downstream embodied control, the learned representation shows state-of-the-art performance on both simulation and real-world manipulation benchmarks, improving success rate by up to 13.4%. Our method also achieves leading 3D hand forecasting accuracy, with 35.8% lower ADE than baseline methods. These results suggest that forecasting hand motion is a better pretraining task for representation learning in embodied manipulation. Projec page: https://ruicongliu.github.io/sfhand/.

Ruicong Liu, Yifei Huang, Liangyang Ouyang +2

The University of Tokyo, Tokyo, Japan

Predictive Representation Learning Robotic Manipulation Human Motion Prediction Hand-Object Interaction

Ask PDF

May 8, 2026

EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting

Forecasting future 3D hand pose sequences from egocentric video is essential for understanding human intention and enabling embodied applications such as AR/VR assistance and human-robot interaction. However, this task remains a highly challenging problem because egocentric hand motion is driven by complex human intent, exhibits highly dexterous articulations, and is observed under drastic viewpoint shifts induced by ego-motion. In this work, we introduce EggHand, a foundation-model-based framework for egocentric hand pose forecasting that unifies multimodal semantic reasoning with dynamic motion modeling. Our approach couples an action decoder from a Vision-Language-Action (VLA) model, which captures the structured temporal dynamics of hand motion, with an egocentric video-text encoder that provides viewpoint-aware contextual information learned from large-scale first-person video. Together, these components overcome the brittleness of generic visual encoders under ego-motion and enable joint reasoning over motion, context, and high-level intent-without relying on body pose or external tracking. Experiments on the EgoExo4D dataset show that EggHand sets a new state of the art in forecasting accuracy, remains robust under severe ego-motion, and further enables controllable prediction via language-based task prompts. Project page: https://jyoun9.github.io/EggHand

Jaeyoung Choi, Hyeondong Kim, Yujin Kim +1

DGIST, Republic of Korea

3D Hand Pose Estimation Egocentric Video Understanding Human Motion Prediction

Ask PDF

Abstract

Hand-Object Interaction

Explore similar work

MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

Predictive Representation Learning for Manipulation via Streaming Egocentric 3D Hand Forecasting

EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecasting