INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving
Organizations: Nanyang Technological University, Singapore · Tongji University, Shanghai, China
Abstract
Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.
Figures & tables
| Method | NC | DAC | TTC | C | EP | PDMS |
|---|---|---|---|---|---|---|
| PDM-Closed ( Dauner et al. 2023 ) | 94.6 | 99.8 | 89.9 | 86.9 | 99.9 | 89.1 |
| Human driver ( Dauner et al. 2024 ) | 100 | 100 | 100 | 99.9 | 87.5 | 94.8 |
| UniAD ( Hu et al. 2023 ) | 97.8 | 91.9 | 92.9 | 100 | 78.8 | 83.4 |
| PARA-Drive ( Weng et al. 2024 ) | 97.9 | 92.4 | 93.0 | 99.8 | 79.3 | 84.0 |
| VAD-v2 ( Chen et al. 2024b ) | 98.1 | 94.8 | 94.3 | 100 | 80.6 | 86.2 |
| AutoVLA ( Zhou et al. 2026b ) | 98.4 | 95.6 | 98.0 | 99.9 | 81.9 | 89.1 |
| Method | NC | DAC | DDC | TL | EP | TTC | LK | HC | EC | EPDMS | EPDMS |
| ReCogDrive ( Li et al. 2025d ) | 98.3 | 95.2 | 99.5 | 99.8 | 87.1 | 97.5 | 96.6 | 98.3 | 86.5 | 83.6 | – |
| Hydra-MDP++ ( Li et al. 2025a ) | 98.4 | 98.0 | 99.4 | 99.8 | 87.5 | 97.7 | 95.3 | 98.3 | 77.4 | 85.1 | – |
| DiffusionDriveV2 ( Zou et al. 2025 ) | 97.7 | 96.6 | 99.2 | 99.8 | 88.9 | 97.2 | 96.0 | 97.8 | 91.0 | 85.5 | 87.5 |
| SparseDriveV2 ( Sun et al. 2026 ) | 98.1 | 98.1 | 99.6 | 99.8 | 91.1 | 97.3 | 96.9 | 98.2 | 78.4 | 86.7 | 90.1 |
| DriveSuprim ( Yao et al. 2026 ) | 98.4 | 98.6 | 99.6 | 99.8 | 90.5 | 97.8 | 97.0 | 98.3 | 78.6 | 87.1 | – |
| World4Drive ( Zheng et al. 2025 ) | 97.8 | 96.3 | 99.4 | 99.8 | 88.3 | 97.1 | 97.7 | 98.0 | 53.9 | 84.8 | – |
| Method | L2 Avg. (m) | Collision Avg. (%) |
|---|---|---|
| UniAD ( Hu et al. 2023 ) | 0.69 | 0.12 |
| PARA-Drive ( Weng et al. 2024 ) | 0.48 | 0.25 |
| AutoVLA ( Zhou et al. 2026b ) | 0.40 | 0.20 |
| OpenREAD ( Zhang et al. 2025 ) | 0.36 | 0.11 |
| ORION ( Fu et al. 2025 ) | 0.34 | 0.37 |
| OpenDrive-VLA ( Zhou et al. 2026a ) | 0.33 | 0.10 |
| Slot Context | Action Align. | Future-Slot Align. | NC | DAC | TTC | PDMS |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 98.9 | 97.6 | 96.4 | 90.9 |
| ✓ | ✗ | ✗ | 99.2 | 97.7 | 97.0 | 91.3 |
| ✓ | ✓ | ✗ | 99.2 | 98.0 | 97.1 | 91.6 |
| ✓ | ✗ | ✓ | 99.3 | 98.4 | 97.1 | 92.0 |
| ✓ | ✓ | ✓ | 99.3 | 98.4 | 97.3 | 92.2 |
| Action Conditioning | NC | DAC | TTC | EP | PDMS |
|---|---|---|---|---|---|
| ✗ | 99.2 | 98.3 | 97.0 | 86.4 | 91.9 |
| ✓ | 99.3 | 98.4 | 97.3 | 86.2 | 92.2 |
| Stage I-II Video Data | NC | DAC | TTC | EP | PDMS |
|---|---|---|---|---|---|
| Trajectory-only (no Stage I-II) | 98.9 | 97.6 | 96.4 | 85.6 | 90.9 |
| 20k navtrain | 99.2 | 98.0 | 97.1 | 86.2 | 91.7 |
| 50k navtrain | 99.2 | 98.2 | 97.2 | 86.1 | 91.9 |
| navtrain | 99.3 | 98.4 | 97.3 | 86.2 | 92.2 |
| navtrain + 40k PhysicalAI-AV | 99.4 | 98.5 | 97.3 | 86.3 | 92.4 |
| Stage | Loss term | Weight |
| Stage I | DINO feature reconstruction | |
| Temporal correspondence | ||
| VAE latent reconstruction | ||
| Stage II | Latent-action quantization | |
| Future-slot prediction | ||
| JEPA feature reconstruction |
| Method | Stage 1 | Stage 2 | EPDMS | ||||||||||||||||
| NC | DAC | DDC | TL | EP | TTC | LK | HC | EC | NC | DAC | DDC | TL | EP | TTC | LK | HC | EC | ||
| ReCogDrive ( Li et al. 2025d ) | 96.4 | 78.9 | 98.7 | 99.8 | 82.6 | 95.6 | 94.4 | 97.6 | 74.2 | 80.2 | 65.0 | 82.4 | 98.7 | 85.2 | 76.9 | 43.8 | 96.6 | 71.8 | 25.7 |
| DiffusionDrive ( Liao et al. 2025 ) | 96.8 | 86.0 | 98.8 | 99.3 | 84.0 | 95.8 | 96.7 | 97.6 | 79.6 | 80.1 | 72.8 | 84.4 | 98.4 | 85.9 | 76.6 | 46.4 | 96.3 | 72.8 | 27.5 |
| DriveSuprim ( Yao et al. 2026 ) | 98.9 | 95.1 | 99.2 | 99.6 | 76.1 | 99.1 | 94.7 | 97.6 | 54.2 | 87.9 | 88.8 | 89.6 | 98.8 | 80.3 | 86.0 | 53.5 | 97.1 | 56.1 | 42.1 |
| GuideFlow ( Liu et al. 2025 ) | 97.8 | 97.1 | 100 | 100 | 81.4 | 98.5 | 91.4 | 92.8 | 34.2 | 87.3 | 92.3 | 98.0 | 96.9 | 75.8 | 85.5 | 59.3 | 95.4 | 53.5 | 46.7 |
| DriveVLA-W0 ( Li et al. 2025c ) | 96.8 | 83.3 | 99.0 | 99.6 | 84.6 | 95.3 | 96.4 | 97.6 | 78.2 | 76.8 | 64.3 | 79.9 | 98.3 | 89.2 | 75.0 | 46.8 | 95.8 | 53.1 | 24.4 |
| Method | L2 (m) | Collision (%) | ||||||
|---|---|---|---|---|---|---|---|---|
| 1s | 2s | 3s | Avg. | 1s | 2s | 3s | Avg. | |
| UniAD ( Hu et al. 2023 ) | 0.44 | 0.67 | 0.96 | 0.69 | 0.04 | 0.08 | 0.23 | 0.12 |
| ParaDrive ( Weng et al. 2024 ) | 0.25 | 0.46 | 0.74 | 0.48 | 0.14 | 0.23 | 0.39 | 0.25 |
| AutoVLA ( Zhou et al. 2026b ) | 0.21 | 0.38 | 0.60 | 0.40 | 0.13 | 0.18 | 0.28 | 0.20 |
| OpenREAD ( Zhang et al. 2025 ) | 0.17 | 0.34 | 0.56 | 0.36 | 0.04 | 0.08 | 0.22 | 0.11 |
| ORION ( Fu et al. 2025 ) | 0.17 | 0.31 | 0.55 | 0.34 | 0.05 | 0.25 | 0.80 | 0.37 |
| Representation | OSC | TSC |
|---|---|---|
| Generic Learned Queries | 0.22 | 0.58 |
| Recurrent Slot Attention | 0.72 | 0.86 |
| Representation | (m) | (rad) | (m/s) |
|---|---|---|---|
| Pooled visual-transition features | 2.3 | 1.26 | 5.1 |
| Pre-VQ latents | 1.4 | 1.04 | 2.6 |
| Quantized action tokens | 0.9 | 0.95 | 2.0 |
| Future Alignment Target | PDMS |
|---|---|
| Dense future DINO features | 91.5 |
| Future slots | 92.0 |
| Scorer Supervision | Trajectory-only | Full INTERACT |
|---|---|---|
| Oracle-score | 90.9 | 92.2 |
| Imitation-based WTA confidence | 90.2 | 91.6 |
| Planner | Number of Modes | World-Model Supervision | PDMS |
|---|---|---|---|
| Trajectory-only planner | 16 | ✗ | 90.9 |
| INTERACT | 16 | ✓ | 92.2 |
| Trajectory-only planner | 64 | ✗ | 92.4 |
| INTERACT | 64 | ✓ | 93.5 |
World Models for Autonomous Driving
World models for autonomous driving predict how scenes and traffic evolve under candidate actions to support planning, control, and simulation.
See all 160 papers