Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.
Figures & tables
Figure 1: Overview of the framework. Stage I learns object-centric slot representations. Stage II learns latent ego actions and ego-conditioned future slot dynamics. Stage III distills the learned world dynamics into a trajectory planner, with future-slot and action-token predictions supervised by pseudo-labels from the frozen slot encoder and action tokenizer, respectively.
Method
NC ↑
DAC ↑
TTC ↑
C ↑
EP ↑
PDMS ↑
PDM-Closed ( Dauner et al. 2023 )
94.6
99.8
89.9
86.9
99.9
89.1
Human driver ( Dauner et al. 2024 )
100
100
100
99.9
87.5
94.8
UniAD ( Hu et al. 2023 )
97.8
91.9
92.9
100
78.8
83.4
PARA-Drive ( Weng et al. 2024 )
97.9
92.4
93.0
99.8
79.3
84.0
VAD-v2 ( Chen et al. 2024b )
98.1
94.8
94.3
100
80.6
86.2
AutoVLA ( Zhou et al. 2026b )
98.4
95.6
98.0
99.9
81.9
89.1
Table 1: NAVSIM-v1 benchmark. We compare INTERACT with published and recent methods on the test set ( navtest ). Higher is better for all metrics.
Method
NC ↑
DAC ↑
DDC ↑
TL ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
EPDMS ∗↑
EPDMS ↑
ReCogDrive ( Li et al. 2025d )
98.3
95.2
99.5
99.8
87.1
97.5
96.6
98.3
86.5
83.6
–
Hydra-MDP++ ( Li et al. 2025a )
98.4
98.0
99.4
99.8
87.5
97.7
95.3
98.3
77.4
85.1
–
DiffusionDriveV2 ( Zou et al. 2025 )
97.7
96.6
99.2
99.8
88.9
97.2
96.0
97.8
91.0
85.5
87.5
SparseDriveV2 ( Sun et al. 2026 )
98.1
98.1
99.6
99.8
91.1
97.3
96.9
98.2
78.4
86.7
90.1
DriveSuprim ( Yao et al. 2026 )
98.4
98.6
99.6
99.8
90.5
97.8
97.0
98.3
78.6
87.1
–
World4Drive ( Zheng et al. 2025 )
97.8
96.3
99.4
99.8
88.3
97.1
97.7
98.0
53.9
84.8
–
Table 2: Results on the NAVSIM-v2 navtest benchmark. EPDMS ∗ denotes results computed with the original NAVSIM-v2 evaluation code before applying the human-behavior filtering fix, while EPDMS denotes results computed with the updated official implementation. Higher values indicate better performance for all metrics.
Method
L2 Avg. (m) ↓
Collision Avg. (%) ↓
UniAD ( Hu et al. 2023 )
0.69
0.12
PARA-Drive ( Weng et al. 2024 )
0.48
0.25
AutoVLA ( Zhou et al. 2026b )
0.40
0.20
OpenREAD ( Zhang et al. 2025 )
0.36
0.11
ORION ( Fu et al. 2025 )
0.34
0.37
OpenDrive-VLA ( Zhou et al. 2026a )
0.33
0.10
Table 3: Open-loop planning results on nuScenes .
Figure 2: Visualization of object-centric slot dynamics and future prediction. The first four columns show historical slot assignments extracted by the Stage I recurrent slot encoder. The last three columns show the ground-truth future image, the future slot layout predicted by the Stage II conditional slot predictor, and the reconstructed future image, respectively.
Figure 3: Mode-wise planning and future-slot predictions in three NAVSIM scenes. The upper and lower rows show the highest- and lowest-ranked modes, respectively. Red and green denote predicted and ground-truth trajectories.
Slot Context
Action Align.
Future-Slot Align.
NC ↑
DAC ↑
TTC ↑
PDMS ↑
✗
✗
✗
98.9
97.6
96.4
90.9
✓
✗
✗
99.2
97.7
97.0
91.3
✓
✓
✗
99.2
98.0
97.1
91.6
✓
✗
✓
99.3
98.4
97.1
92.0
✓
✓
✓
99.3
98.4
97.3
92.2
Table 4: Ablation of the world-model signals transferred to Stage III on NAVSIM navtest . A common frozen Stage I-II teacher is used wherever alignment targets are required.
Action Conditioning
NC ↑
DAC ↑
TTC ↑
EP ↑
PDMS ↑
✗
99.2
98.3
97.0
86.4
91.9
✓
99.3
98.4
97.3
86.2
92.2
Table 5: Effect of ego-action conditioning in Stage II.
Stage I-II Video Data
NC ↑
DAC ↑
TTC ↑
EP ↑
PDMS ↑
Trajectory-only (no Stage I-II)
98.9
97.6
96.4
85.6
90.9
20k navtrain
99.2
98.0
97.1
86.2
91.7
50k navtrain
99.2
98.2
97.2
86.1
91.9
navtrain
99.3
98.4
97.3
86.2
92.2
navtrain + 40k PhysicalAI-AV
99.4
98.5
97.3
86.3
92.4
Table 6: Scaling self-supervised video pretraining.
Stage
Loss term
Weight
Stage I
DINO feature reconstruction Lfeat
λfeat=1.0
Temporal correspondence Ltemp
λtemp=1.0
VAE latent reconstruction Lvae
λvae=1.0
Stage II
Latent-action quantization Lact
λactII=1.0
Future-slot prediction Lobject
λobject=1.0
JEPA feature reconstruction Lvisual
λvisual=1.0
Table 7: Loss weights used in the three training stages. The scoring loss is enabled only when candidate-level oracle supervision is available.
Figure 4: Additional visualizations of object-centric dynamics and learned latent actions. Each row shows historical slot assignments, the ground-truth future image, the predicted future slots, the slot-based future reconstruction, and the three-level residual codebook indices of the inferred ego action. Dashed lines group representative driving behaviors, including car following, approaching and stopping behind a leading vehicle, accelerating from an intersection, and right-turn maneuvers. Samples within each group exhibit similar latent-action codes, with identical assignments at the first quantization level, indicating that meaningful coarse ego-motion patterns emerge without action supervision.
Method
Stage 1
Stage 2
EPDMS ↑
NC ↑
DAC ↑
DDC ↑
TL ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
NC ↑
DAC ↑
DDC ↑
TL ↑
EP ↑
TTC ↑
LK ↑
HC ↑
EC ↑
ReCogDrive ( Li et al. 2025d )
96.4
78.9
98.7
99.8
82.6
95.6
94.4
97.6
74.2
80.2
65.0
82.4
98.7
85.2
76.9
43.8
96.6
71.8
25.7
DiffusionDrive ( Liao et al. 2025 )
96.8
86.0
98.8
99.3
84.0
95.8
96.7
97.6
79.6
80.1
72.8
84.4
98.4
85.9
76.6
46.4
96.3
72.8
27.5
DriveSuprim ( Yao et al. 2026 )
98.9
95.1
99.2
99.6
76.1
99.1
94.7
97.6
54.2
87.9
88.8
89.6
98.8
80.3
86.0
53.5
97.1
56.1
42.1
GuideFlow ( Liu et al. 2025 )
97.8
97.1
100
100
81.4
98.5
91.4
92.8
34.2
87.3
92.3
98.0
96.9
75.8
85.5
59.3
95.4
53.5
46.7
DriveVLA-W0 ( Li et al. 2025c )
96.8
83.3
99.0
99.6
84.6
95.3
96.4
97.6
78.2
76.8
64.3
79.9
98.3
89.2
75.0
46.8
95.8
53.1
24.4
Table 8: Results on the NAVSIM-v2 navhard benchmark. Stage 1 evaluates real safety-critical scenes, while Stage 2 evaluates their Gaussian-Splatting-generated synthetic counterparts under perturbed ego states. EPDMS denotes the final aggregated two-stage score. Higher values indicate better performance for all metrics.
Method
L2 (m) ↓
Collision (%) ↓
1s
2s
3s
Avg.
1s
2s
3s
Avg.
UniAD ( Hu et al. 2023 )
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
ParaDrive ( Weng et al. 2024 )
0.25
0.46
0.74
0.48
0.14
0.23
0.39
0.25
AutoVLA ( Zhou et al. 2026b )
0.21
0.38
0.60
0.40
0.13
0.18
0.28
0.20
OpenREAD ( Zhang et al. 2025 )
0.17
0.34
0.56
0.36
0.04
0.08
0.22
0.11
ORION ( Fu et al. 2025 )
0.17
0.31
0.55
0.34
0.05
0.25
0.80
0.37
Table 9: Complete open-loop planning results on nuScenes. Results are highlighted in bold and underlined for the best and second-best performance among all compared methods, respectively. Lower is better for all metrics.