Organizations: College Of Computer Science, Chongqing University · School of Computer Science and Technology, Zhejiang Normal University · School of Computer Science and Technology, Beijing Institute of Technology · Department of Computing, Hong Kong Polytechnic University
The complex interactive dependencies between human behavior and 3D scenes make accurately predicting human intent a major challenge for human motion prediction (HMP). While recent studies use human gaze to infer intent, gaze signals suffer from ambiguity, drift, and a reliance on eye-tracking hardware. Research in cognitive science indicates that egocentric views inherently encode potential interactions, while head motion serves as a crucial precursor in motion planning. Motivated by this, we propose EgoHMP, a novel framework that uses egocentric views and head motions as robust carriers of interaction intent. By translating this intent into an envisioning of future interactions, EgoHMP achieves precise HMP in 3D scenes. EgoHMP first employs a vision-motion alignment encoder to align the visual and motion feature spaces. Based on these aligned features, an egocentric intent predictor with a modality-aware modulation mechanism adaptively suppresses invalid interaction cues to predict an accurate interaction probability map. Furthermore, to enhance the stability and realism of the generated motions, we propose a contact-aware motion decoder. Specifically, inspired by the human cognitive process of planning trajectories prior to action execution, we first predict a global trajectory to serve as a self-prompt, effectively mitigating the optimization conflict in diffusion models. Subsequently, we introduce a multi-stage denoiser, which relieves motion artifacts by incorporating human contact modeling through a multi-stage optimization strategy. Finally, a GCN-based motion decoder is employed to synthesize physically plausible and semantically consistent motions. Extensive experiments demonstrate that EgoHMP achieves state-of-the-art performance.
Figures & tables
Figure 1: EgoHMP captures human intent through synergistic egocentric observations and head motions, and reveals interaction concepts by combining the scene and human interaction regions, thereby envisioning the interaction, which enables precise human motion prediction in 3D scenes.
Figure 2: Pipeline of EgoHMP. EgoHMP first employs a pre-trained vision-motion alignment encoder to extract aligned features by minimizing the discrepancy between visual and motion disparities (Sec. 3.2 ). Subsequently, the egocentric intent predictor uses these aligned features to query the 3D scene point clouds, inferring an interaction probability map (Sec. 3.3 ). Finally, guided by this probability map, a contact-aware motion decoder synthesizes the final human motion sequence (Sec. 3.4 ).
Method
GIMO [ 5 ]
GTA-IM [ 24 ]
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
Gaze-based
BiFu [ 5 ]
661
727
167.8
205.0
415.8
405.3
398.7
3.73
683
1066
186.4
252.6
455.2
445.6
438.1
3.95
SIF3D [ 17 ]
590
666
156.6
195.7
345.2
336.8
330.4
4.59
626
836
164.9
227.7
385.6
378.2
371.9
4.82
GAP3DS [ 7 ]
575
623
141.2
171.4
210.5
205.1
201.3
0.95
610
799
155.0
210.1
235.4
228.6
224.5
1.12
Table 1: Quantitative Results. T-P and T-E denote Traj-Path and Traj-End. M-P and M-E denote MPJPE-Path and MPJPE-End. Intent localization is evaluated via TDE@ K for K∈{1,50,100} . Pen. denotes penetration error. All metrics are reported in millimeters (mm) except for Pen. (in % ). Lower values indicate better performance ( ↓ ). The best results are highlighted in bold .
Method
Scene-Aware
M-P (mm) ↓
M-E (mm) ↓
EgoEgo [ 15 ]
✗
315.0
401.3
EgoEgo [ 15 ]
✓
263.8
335.4
EgoCast [ 31 ]
✗
225.8
281.6
EgoCast [ 31 ]
✓
153.6
193.7
EgoAllo [ 16 ]
✗
207.4
268.8
EgoAllo [ 16 ]
✓
144.7
178.4
Table 2: Quantitative Results on Ego-Exo4D [ 25 ] .
Metric ↓
✗ FV
G
✗ τV,τM
✗ Spm
✗ Flocal
✗ Dms
✗ Dm
✗ LA
✗ Ljoint
✗ Lslide
Full
T-P
676
589
501
612
482
483
483
558
484
483
481
T-E
784
643
576
691
547
548
546
639
546
547
545
M-P
168.9
145.9
130.4
150.4
135.3
134.7
118.6
132.6
122.8
115.4
114.9
M-E
216.8
171.7
150.6
182.2
167.2
165.8
142.0
161.4
153.5
140.9
138.0
Table 3: Quantitative Ablations. FV : egocentric view signal; G : using the gaze signal instead of the egocentric view signal; τV,τM : modality-aware modulation; Spm : self-prompted mechanism; Flocal : scene-aware conditioner; Dms : multi-stage denoiser; Dm : GCN-based motion decoder; LA/Ljoint/Lslide : loss functions. ✗ means without.
Figure 3: Visualization Ablations. (a) The gradient flow of the visual ( V ) and motion ( M ) modalities across 70 sampled training epochs. (b) The refinement process of the multi-stage denoiser.
Setting
Perturbation
T-P ↓
T-E ↓
M-P ↓
M-E ↓
Pen. ↓
FS ↓
GCD ↓
HCD ↓
Clean Input
Original Data
481
545
114.9
138.0
0.81
3.70
36.4
43.5
Head-Pose Noise
Gaussian, σ=0.05
487
553
120.2
142.1
0.81
3.71
36.6
44.9
Gaussian, σ=0.10
505
572
128.6
155.7
0.83
3.74
37.0
47.3
Calibration Error
Offset (5 cm, 2∘ )
511
579
131.8
161.5
0.83
3.72
36.9
51.5
Offset (10 cm, 5∘ )
524
594
139.3
170.4
0.88
3.76
37.5
54.0
Incomplete Geo.
30% Points Dropped
490
565
119.3
144.2
0.86
3.71
36.5
44.7
Table 4: Robustness Evaluation. (1) Foot Sliding (FS) penalizes abnormal horizontal shifts of ankle and toe vertices when they fall below contact height thresholds (80mm and 40mm). (2) Ground Contact Deviation (GCD) captures floating/penetration artifacts via the absolute minimum vertical distance between supporting vertices and the local scene surface. (3) Human Contact Distance (HCD) evaluates interaction quality by measuring the average Euclidean distance from the intended contact vertices to the intended scene surface.
Figure 4: Visualization Comparison. The blue, green, and tan human meshes represent the historical, predicted, and ground-truth motion sequences, respectively. Zoom in for better visualization.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
M→V↑
V→M↑
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
R@1
R@5
R@10
R@1
R@5
R@10
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
(a) E2E
0.00
0.06
0.12
0.03
0.09
0.11
588
672
163.2
200.4
312.5
305.7
298.4
3.15
(b) Pre-train w/o kl
57.53
90.90
95.53
55.95
92.23
96.77
580
647
151.6
189.1
295.4
288.6
282.5
2.42
(c) Pre-train w/o nce
0.00
0.04
0.11
0.01
0.10
0.13
490
558
119.4
144.2
188.5
180.2
174.0
0.89
(d) Full Pre-training
66.37
94.75
97.84
63.25
95.44
98.15
481
545
114.9
138.0
161.3
153.2
148.6
0.81
Appendix
Table 5: Quantitative Results. Evaluated on the GIMO [ 5 ] . We analyze the necessity of the pre-training for the vision-motion alignment encoder and the impact of different objectives within the alignment loss. E2E denotes training the entire framework end-to-end. LA denotes the vision-motion alignment loss. kl denotes the KL divergence loss in LA . nce denotes the InfoNCE contrastive loss in LA . R@1, R@5, and R@10 denote the cross-modal retrieval recall metrics (%). M→V denotes motion-to-vision retrieval, and V→M denotes vision-to-motion retrieval.
Method
Metric
Bedroom
Classroom
Garden
Kitchen
Lab
Livingroom
Seminarroom
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
BiFu [ 5 ]
Traj
215
495
735
312
425
605
202
500
578
257
440
515
256
515
595
317
580
702
394
622
558
MPJPE
62.0
106.5
162.6
84.1
136.0
151.3
83.2
193.0
223.5
91.5
100.7
118.1
49.5
106.2
203.2
89.4
119.8
210.2
100.7
131.0
161.5
SIF3D [ 17 ]
Traj
171
415
485
145
405
482
186
425
565
165
445
395
231
505
645
241
565
698
205
415
542
MPJPE
77.5
98.4
162.6
93.3
138.0
141.3
68.5
176.2
175.1
93.1
97.4
101.5
61.4
88.1
191.9
102.2
115.3
166.2
94.7
131.1
158.6
GAP3DS [ 7 ]
Traj
168.5
350
415
179
391
413
183
422
472
163
377
368
235
465
560
173
515
576
174
375
531
Appendix
Table 6: Quantitative Results. Trajectory deviation (Traj) and pose error (MPJPE) across seven 3D scenes on the GIMO [ 5 ] . Time horizons ( 0.5 , 2.0 , 5.0 ) are reported in (s). Best results are in bold .
Figure 5: Qualitative Visualization. Given specific interaction intents, EgoHMP synthesizes diverse poses, confirming effective exploration of the valid interaction subspace without mean pose collapse.
Method
APD ↑
ADE ↓
FDE ↓
MMADE ↓
MMFDE ↓
DLow [ 45 ]
49.37
13.49
14.95
11.70
14.46
SmoothDMP [ 46 ]
57.38
11.31
13.57
9.75
12.04
BeLFusion [ 41 ]
39.46
9.56
10.04
7.91
9.30
STAB-ACB [ 47 ]
52.11
11.49
13.95
11.06
12.14
GAP3DS [ 7 ]
42.47
5.37
6.51
4.85
5.62
EgoHMP (Full)
44.10
4.98
6.17
4.12
5.08
Appendix
Table 7: Quantitative Results. ↑ : higher is better, ↓ : lower is better. The best results are highlighted in bold .
Metric
Vision Backbone (with 16384 points)
Scene Points (with DINOv2-L)
ResNet-50 [ 49 ]
CLIP [ 38 ]
DINOv2-B [ 37 ]
DINOv2-L [ 37 ]
DINOv3-L [ 50 ]
4096
8192
16384
32768
T-P ( ↓ )
572
534
512
481
479
538
511
481
480
M-P ( ↓ )
138.6
133.2
124.5
114.9
113.2
129.4
121.5
114.9
114.3
VRAM ( ↓ )
12100
12650
12680
14610
14850
7840
11120
14610
18067
Speed ( ↑ )
17.5
16.8
16.7
15.1
14.8
27.4
20.8
15.1
8.9
Appendix
Table 8: Quantitative Results. VRAM denotes the GPU memory consumption (in MB), and Speed represents the inference speed (in samples/s). The best results are highlighted in bold .
Figure 6: Qualitative Visualization. The blue, green, and tan human meshes represent the historical, predicted, and ground-truth motion sequences, respectively. Zoom in for better visualization.
Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice. To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time. Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating the effectiveness of the proposed framework. The project page is available at https://sgip2026.github.io.
Anticipating human motion from an egocentric perspective is fundamental for proactive assistance in AR/VR, human-robot collaboration, and embodied AI. While recent works incorporate language as a semantic prior to reduce the ill-posed nature of egocentric forecasting, they largely neglect the 3D spatial and semantic context that governs how motion unfolds, and treat pose and language prediction as separate inference streams. We introduce Ego3DLM, built on two core principles: accurate motion forecasting requires explicit spatial and semantic understanding of the 3D environment, and pose and language must be predicted holistically in a single pass, since motion is inherently tied to the semantic interpretation of actions being performed. Given three-point tracking, 3D scene features, and egocentric video, Ego3DLM simultaneously decodes past pose, future pose, past narration, and future narration in a single autoregressive pass, grounding predicted poses and descriptions in one another to enforce cross-modal and temporal consistency. We adopt a three-stage training scheme: (1) spatial-semantic scene awareness pretraining; (2) holistic instruction tuning over all four outputs in a single pass; and (3) GRPO-based reinforcement finetuning with intra- and inter-modal rewards that directly optimize pose-language fidelity. Experiments on the Nymeria benchmark demonstrate that Ego3DLM achieves state-of-the-art performance across future motion prediction, past motion tracking, and motion description, showing that 3D scene grounding and holistic cross-modal prediction yield physically plausible and semantically coherent motion forecasts. The project page is available at https://jaewoo97.github.io/Ego3DLM/.
Egocentric motion forecasting has primarily focused on hands and manipulated objects, leaving future human head motion comparatively underexplored. During manipulation, the head both redirects perception toward the target to acquire task-relevant evidence and coordinates with body and hand motion. We therefore formulate future six Degree of Freedom (6-DoF) head-motion prediction conditioned on observed hand motion and inferred target context, and propose HAP, a Hand-Driven Active Perception framework. HAP infers confidence for each target object from observed hand motion and object geometry. Then constructs a dynamic Predictive Target-Centric Amodal Occlusion Graph (P-TAOG) representing current and potential occlusion among candidate objects. Directed graph and causal temporal reasoning encode the evolving target conditioned perceptual state, which is fused with hand and head motion history. A horizon-wise gate then blends the learned trajectory with a constant velocity prior. We further introduce Bottle, an egocentric RGB-D dataset of object manipulation toward specified targets, with coordinated head and hand motion under changing target visibility. Experiments on the public dataset and Bottle show that HAP achieves lower head motion prediction errors than representative baselines, supporting the value of hand driven intention and dynamic occlusion reasoning for anticipating human head motion. Code will be released at https://HAP-ego.github.io/HAP.
Yunji Feng, Junyi Ma, Guanzhong Sun +2
IRMV Lab, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China · Zhongguancun Academy, Beijing 100094, China · School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China