SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion Prediction
Organizations: DGIST · UNIVA
Abstract
Human motion prediction (HMP) aims to forecast future movement from an observed motion sequence, and recent methods have incorporated 3D scene geometry and object semantics to improve long-term plausibility. However, the destination of future motion remains ambiguous in cluttered scenes: the same observed motion can lead to different interaction targets or navigation modes. Existing methods either handle this ambiguity implicitly or rely on auxiliary signals like gaze that are hard to obtain in practice. To address this, we propose SGIP, which explicitly predicts where in the scene future motion will be grounded across both objects and open navigation regions, and uses it to condition future trajectory and full-body pose prediction. This scene-grounded interaction prior is learned via intent distillation: at training time only, a teacher model scores candidate scene regions by combining an LLM prompted with a textual description of the person's intent with geometric motion-scene features. A student model learns to reproduce these scores from only the observed motion and 3D scene, making intent and LLM unnecessary at test time. Across multiple benchmark datasets, SGIP outperforms prior scene-aware baselines on both trajectory and pose accuracy in a realistic deployment setting where neither gaze nor intent is available at inference, demonstrating the effectiveness of the proposed framework. The project page is available at https://sgip2026.github.io.
Figures & tables
| Method | Auxiliary Cue | Traj-path | Traj-dest | MPJPE-path | MPJPE-dest | |
| GIMO | Models with additional or privileged cues | |||||
| LTD [ 42 ] w/ gaze | 655 | 801 | 155.6 | 202.0 | ||
| SPGSN [ 37 ] w/ gaze | 702 | 982 | 156.0 | 202.0 | ||
| AuxFormer [ 63 ] w/ gaze | 622 | 746 | 161.5 | 201.1 | ||
| BiFu [ 73 ] | 661 | 727 | 167.8 | 205.0 | ||
| SIF3D [ 40 ] | 590 | 666 | 156.6 | 195.7 | ||
| APD | minADE@ | minFDE@ | FDE@minADE | |
|---|---|---|---|---|
| 1 | – | 0.549 | 0.681 | 0.681 |
| 2 | 0.884 | 0.517 | 0.601 | 0.605 |
| 3 | 0.927 | 0.467 | 0.523 | 0.571 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Used at | Inputs |
|---|---|---|
| VLM intent extractor (offline) | training only | training video, 3D motion visualization, intent taxonomy |
| LLM rubric scorer (offline) | training only | intent , candidate semantic labels, scene context |
| Privileged teacher | training only | , , , , cached |
| Observation-only student | training + inference | , , |
| GT interaction target | evaluation only | never input to any network |
| Prior | Rank 1 | Rank 2 | Rank 3 | |
|---|---|---|---|---|
| Rubric only ( ) | 54 | 24 | 2 | 80 |
| Geometry only ( ) | 51 | 16 | 11 | 78 |
| Teacher ( ) | 86 | 5 | 3 | 94 |
| Student | 76 | 5 | 3 | 84 |
| Student loss | T–S Agree@1 | GT Acc@1 | Traj. P/D |
|---|---|---|---|
| only | 43% | 41% | 856/896 |
| only | 70% | 76% | 574/688 |
| 70% | 76% | 549/681 |
| Single future | Multi-future ( ) | ||||
|---|---|---|---|---|---|
| Method | Traj. P/D | MPJPE P/D | minADE | minFDE | FDE@minADE |
| AffordMotion | 577/714 | 160.9/ 194.1 | 0.597 | 0.728 | 0.730 |
| SGIP | 549/681 | 153.1 /195.1 | 0.467 | 0.523 | 0.571 |
| Variant | Cue | Traj. P/D | MPJPE P/D |
|---|---|---|---|
| SGIP (deployed) | – | 549/681 | 153.1/195.1 |
| gaze | 528/635 | 149.9/190.6 | |
| intent | 485/556 | 151.2/187.8 | |
| gaze+intent | 475/539 | 149.7/180.8 |
| Path error | Pose error | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | 0.5s | 1.0s | 1.5s | 2.0s | mean | 0.5s | 1.0s | 1.5s | 2.0s | mean | |
| HIK | HUMOF [ 53 ] | ✓ | 96.9 | 231.3 | 400.3 | 546.6 | 251.9 | 73.5 | 99.3 | 116.7 | 123.7 | 90.6 |
| SGIP | ✗ | 89.2 | 130.9 | 167.7 | 205.7 | 127.1 | 81.5 | 114.9 | 130.7 | 138.8 | 103.1 | |
| HOI-M 3 | HUMOF [ 53 ] | ✓ | 104.7 | 238.0 | 385.8 | 538.4 | 253.9 | 117.2 | 163.0 | 189.3 | 210.7 | 148.9 |
| SGIP | ✗ | 134.9 | 280.7 | 437.4 | 583.0 | 291.1 | 121.1 | 159.2 | 179.1 | 200.9 | 145.0 | |
| Condition | Target recall (%) | Target Acc@1 (%) | Traj. P/D | MPJPE P/D |
|---|---|---|---|---|
| Clean | 100.00 | 75.68 | 549.0/681.0 | 153.1/195.1 |
| Gaussian, | / | / | ||
| Gaussian, | / | / | ||
| Gaussian, | / | / | ||
| Target–neighbor merge | 100.00 | 56.76 | 661.0/875.0 | 159.2/211.3 |
| Target two-way split | 100.00 | 54.05 | 644.0/908.0 | 157.1/208.8 |
| GIMO | PROX | |||||
|---|---|---|---|---|---|---|
| VLM | Agreement | Traj. P/D | MPJPE P/D | Agreement | Traj. P/D | MPJPE P/D |
| Qwen3-VL-8B (default) | – | 549/681 | 153.1/195.1 | – | 470/576 | 194.6/197.7 |
| Qwen3-VL-4B | 90.7% | 527/753 | 152.8/196.8 | 96.6% | 459/570 | 195.0/209.5 |
| Idefics3-8B | 92.1% | 584/723 | 154.9/190.6 | 97.0% | 493/564 | 181.1/202.9 |
| Stage | Frequency | Runtime |
|---|---|---|
| One-time preprocessing | ||
| OpenIns3D scene initialization | once per scene | s |
| Online student inference | ||
| Motion-dependent descriptors | per observation | ms |
| Student prior + Top-1 selection | per observation | ms |
| Host-to-device transfer | per observation | ms |
| Candidate set | Object tokens | Retention (%) |
|---|---|---|
| Before cone filtering | 100.00 | |
| After cone filtering ( ) |
| Group | Hyperparameter | Value |
| Candidate tokens | cone aperture | |
| GT–proposal matching IoU (evaluation) | ||
| Teacher prior | fusion weight , temperature (Eq. 7 ) | , |
| (Eq. 6 ) | ||
| Gaze variant | ||
| Encoders | 4-layer Transformer, , 4 heads |
Human Motion Prediction
Human motion prediction forecasts how people will move, supporting applications from animation and activity understanding to safe robot interaction.
See all 61 papers