Organizations: College Of Computer Science, Chongqing University · School of Computer Science and Technology, Zhejiang Normal University · School of Computer Science and Technology, Beijing Institute of Technology · Department of Computing, Hong Kong Polytechnic University
The complex interactive dependencies between human behavior and 3D scenes make accurately predicting human intent a major challenge for human motion prediction (HMP). While recent studies use human gaze to infer intent, gaze signals suffer from ambiguity, drift, and a reliance on eye-tracking hardware. Research in cognitive science indicates that egocentric views inherently encode potential interactions, while head motion serves as a crucial precursor in motion planning. Motivated by this, we propose EgoHMP, a novel framework that uses egocentric views and head motions as robust carriers of interaction intent. By translating this intent into an envisioning of future interactions, EgoHMP achieves precise HMP in 3D scenes. EgoHMP first employs a vision-motion alignment encoder to align the visual and motion feature spaces. Based on these aligned features, an egocentric intent predictor with a modality-aware modulation mechanism adaptively suppresses invalid interaction cues to predict an accurate interaction probability map. Furthermore, to enhance the stability and realism of the generated motions, we propose a contact-aware motion decoder. Specifically, inspired by the human cognitive process of planning trajectories prior to action execution, we first predict a global trajectory to serve as a self-prompt, effectively mitigating the optimization conflict in diffusion models. Subsequently, we introduce a multi-stage denoiser, which relieves motion artifacts by incorporating human contact modeling through a multi-stage optimization strategy. Finally, a GCN-based motion decoder is employed to synthesize physically plausible and semantically consistent motions. Extensive experiments demonstrate that EgoHMP achieves state-of-the-art performance.
Figures & tables
Figure 1: EgoHMP captures human intent through synergistic egocentric observations and head motions, and reveals interaction concepts by combining the scene and human interaction regions, thereby envisioning the interaction, which enables precise human motion prediction in 3D scenes.
Figure 2: Pipeline of EgoHMP. EgoHMP first employs a pre-trained vision-motion alignment encoder to extract aligned features by minimizing the discrepancy between visual and motion disparities (Sec. 3.2 ). Subsequently, the egocentric intent predictor uses these aligned features to query the 3D scene point clouds, inferring an interaction probability map (Sec. 3.3 ). Finally, guided by this probability map, a contact-aware motion decoder synthesizes the final human motion sequence (Sec. 3.4 ).
Method
GIMO [ 5 ]
GTA-IM [ 24 ]
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
Gaze-based
BiFu [ 5 ]
661
727
167.8
205.0
415.8
405.3
398.7
3.73
683
1066
186.4
252.6
455.2
445.6
438.1
3.95
SIF3D [ 17 ]
590
666
156.6
195.7
345.2
336.8
330.4
4.59
626
836
164.9
227.7
385.6
378.2
371.9
4.82
GAP3DS [ 7 ]
575
623
141.2
171.4
210.5
205.1
201.3
0.95
610
799
155.0
210.1
235.4
228.6
224.5
1.12
Table 1: Quantitative Results. T-P and T-E denote Traj-Path and Traj-End. M-P and M-E denote MPJPE-Path and MPJPE-End. Intent localization is evaluated via TDE@ K for K∈{1,50,100} . Pen. denotes penetration error. All metrics are reported in millimeters (mm) except for Pen. (in % ). Lower values indicate better performance ( ↓ ). The best results are highlighted in bold .
Method
Scene-Aware
M-P (mm) ↓
M-E (mm) ↓
EgoEgo [ 15 ]
✗
315.0
401.3
EgoEgo [ 15 ]
✓
263.8
335.4
EgoCast [ 31 ]
✗
225.8
281.6
EgoCast [ 31 ]
✓
153.6
193.7
EgoAllo [ 16 ]
✗
207.4
268.8
EgoAllo [ 16 ]
✓
144.7
178.4
Table 2: Quantitative Results on Ego-Exo4D [ 25 ] .
Metric ↓
✗ FV
G
✗ τV,τM
✗ Spm
✗ Flocal
✗ Dms
✗ Dm
✗ LA
✗ Ljoint
✗ Lslide
Full
T-P
676
589
501
612
482
483
483
558
484
483
481
T-E
784
643
576
691
547
548
546
639
546
547
545
M-P
168.9
145.9
130.4
150.4
135.3
134.7
118.6
132.6
122.8
115.4
114.9
M-E
216.8
171.7
150.6
182.2
167.2
165.8
142.0
161.4
153.5
140.9
138.0
Table 3: Quantitative Ablations. FV : egocentric view signal; G : using the gaze signal instead of the egocentric view signal; τV,τM : modality-aware modulation; Spm : self-prompted mechanism; Flocal : scene-aware conditioner; Dms : multi-stage denoiser; Dm : GCN-based motion decoder; LA/Ljoint/Lslide : loss functions. ✗ means without.
Figure 3: Visualization Ablations. (a) The gradient flow of the visual ( V ) and motion ( M ) modalities across 70 sampled training epochs. (b) The refinement process of the multi-stage denoiser.
Setting
Perturbation
T-P ↓
T-E ↓
M-P ↓
M-E ↓
Pen. ↓
FS ↓
GCD ↓
HCD ↓
Clean Input
Original Data
481
545
114.9
138.0
0.81
3.70
36.4
43.5
Head-Pose Noise
Gaussian, σ=0.05
487
553
120.2
142.1
0.81
3.71
36.6
44.9
Gaussian, σ=0.10
505
572
128.6
155.7
0.83
3.74
37.0
47.3
Calibration Error
Offset (5 cm, 2∘ )
511
579
131.8
161.5
0.83
3.72
36.9
51.5
Offset (10 cm, 5∘ )
524
594
139.3
170.4
0.88
3.76
37.5
54.0
Incomplete Geo.
30% Points Dropped
490
565
119.3
144.2
0.86
3.71
36.5
44.7
Table 4: Robustness Evaluation. (1) Foot Sliding (FS) penalizes abnormal horizontal shifts of ankle and toe vertices when they fall below contact height thresholds (80mm and 40mm). (2) Ground Contact Deviation (GCD) captures floating/penetration artifacts via the absolute minimum vertical distance between supporting vertices and the local scene surface. (3) Human Contact Distance (HCD) evaluates interaction quality by measuring the average Euclidean distance from the intended contact vertices to the intended scene surface.
Figure 4: Visualization Comparison. The blue, green, and tan human meshes represent the historical, predicted, and ground-truth motion sequences, respectively. Zoom in for better visualization.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
M→V↑
V→M↑
Motion Metrics ↓
Intent Metrics ↓
Phys. ↓
R@1
R@5
R@10
R@1
R@5
R@10
T-P
T-E
M-P
M-E
@1
@50
@100
Pen.
(a) E2E
0.00
0.06
0.12
0.03
0.09
0.11
588
672
163.2
200.4
312.5
305.7
298.4
3.15
(b) Pre-train w/o kl
57.53
90.90
95.53
55.95
92.23
96.77
580
647
151.6
189.1
295.4
288.6
282.5
2.42
(c) Pre-train w/o nce
0.00
0.04
0.11
0.01
0.10
0.13
490
558
119.4
144.2
188.5
180.2
174.0
0.89
(d) Full Pre-training
66.37
94.75
97.84
63.25
95.44
98.15
481
545
114.9
138.0
161.3
153.2
148.6
0.81
Appendix
Table 5: Quantitative Results. Evaluated on the GIMO [ 5 ] . We analyze the necessity of the pre-training for the vision-motion alignment encoder and the impact of different objectives within the alignment loss. E2E denotes training the entire framework end-to-end. LA denotes the vision-motion alignment loss. kl denotes the KL divergence loss in LA . nce denotes the InfoNCE contrastive loss in LA . R@1, R@5, and R@10 denote the cross-modal retrieval recall metrics (%). M→V denotes motion-to-vision retrieval, and V→M denotes vision-to-motion retrieval.
Method
Metric
Bedroom
Classroom
Garden
Kitchen
Lab
Livingroom
Seminarroom
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
0.5
2.0
5.0
BiFu [ 5 ]
Traj
215
495
735
312
425
605
202
500
578
257
440
515
256
515
595
317
580
702
394
622
558
MPJPE
62.0
106.5
162.6
84.1
136.0
151.3
83.2
193.0
223.5
91.5
100.7
118.1
49.5
106.2
203.2
89.4
119.8
210.2
100.7
131.0
161.5
SIF3D [ 17 ]
Traj
171
415
485
145
405
482
186
425
565
165
445
395
231
505
645
241
565
698
205
415
542
MPJPE
77.5
98.4
162.6
93.3
138.0
141.3
68.5
176.2
175.1
93.1
97.4
101.5
61.4
88.1
191.9
102.2
115.3
166.2
94.7
131.1
158.6
GAP3DS [ 7 ]
Traj
168.5
350
415
179
391
413
183
422
472
163
377
368
235
465
560
173
515
576
174
375
531
Appendix
Table 6: Quantitative Results. Trajectory deviation (Traj) and pose error (MPJPE) across seven 3D scenes on the GIMO [ 5 ] . Time horizons ( 0.5 , 2.0 , 5.0 ) are reported in (s). Best results are in bold .
Figure 5: Qualitative Visualization. Given specific interaction intents, EgoHMP synthesizes diverse poses, confirming effective exploration of the valid interaction subspace without mean pose collapse.
Method
APD ↑
ADE ↓
FDE ↓
MMADE ↓
MMFDE ↓
DLow [ 45 ]
49.37
13.49
14.95
11.70
14.46
SmoothDMP [ 46 ]
57.38
11.31
13.57
9.75
12.04
BeLFusion [ 41 ]
39.46
9.56
10.04
7.91
9.30
STAB-ACB [ 47 ]
52.11
11.49
13.95
11.06
12.14
GAP3DS [ 7 ]
42.47
5.37
6.51
4.85
5.62
EgoHMP (Full)
44.10
4.98
6.17
4.12
5.08
Appendix
Table 7: Quantitative Results. ↑ : higher is better, ↓ : lower is better. The best results are highlighted in bold .
Metric
Vision Backbone (with 16384 points)
Scene Points (with DINOv2-L)
ResNet-50 [ 49 ]
CLIP [ 38 ]
DINOv2-B [ 37 ]
DINOv2-L [ 37 ]
DINOv3-L [ 50 ]
4096
8192
16384
32768
T-P ( ↓ )
572
534
512
481
479
538
511
481
480
M-P ( ↓ )
138.6
133.2
124.5
114.9
113.2
129.4
121.5
114.9
114.3
VRAM ( ↓ )
12100
12650
12680
14610
14850
7840
11120
14610
18067
Speed ( ↑ )
17.5
16.8
16.7
15.1
14.8
27.4
20.8
15.1
8.9
Appendix
Table 8: Quantitative Results. VRAM denotes the GPU memory consumption (in MB), and Speed represents the inference speed (in samples/s). The best results are highlighted in bold .
Figure 6: Qualitative Visualization. The blue, green, and tan human meshes represent the historical, predicted, and ground-truth motion sequences, respectively. Zoom in for better visualization.
IRMV Lab, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China · Zhongguancun Academy, Beijing 100094, China · School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China