cs.CVNov 2, 2026

INTERACT: Learning Interaction-Aware World Dynamics from Driving Videos for Autonomous Driving

Authors: Zhan Chen, Songyan Zhang, Wenhui Huang, Zirui Li, Zhidong Wang, Qihang Huang, Shiyu Fang, Chen Lv

Organizations: Nanyang Technological University, Singapore · Tongji University, Shanghai, China

Abstract

Safe autonomous driving requires more than recognizing the current scene: the ego vehicle must anticipate how surrounding traffic participants will evolve under its own motion. However, existing end-to-end driving models typically encode scenes into dense visual or BEV features, leaving object-level interactions implicit. We propose (INTERACT INTERaction-aware Action-Conditioned Trajectory learning), a three-stage framework that learns a slot-based object-centric world model from unlabeled driving videos and transfers it to trajectory planning. INTERACT first learns recurrent slots that group dense visual features into temporally structured scene-entity representations without object annotations. It then infers latent ego actions from slot context and video transitions, and uses them to condition future slot prediction, capturing how ego motion and surrounding entities co-evolve. Finally, the learned slots, latent actions, and future slot predictions are distilled into a trajectory planner to guide trajectory generation and selection. Extensive experiments on NAVSIM and nuScenes validate the effectiveness of INTERACT. In particular, it achieves state-of-the-art performance on NAVSIM, with improvements in safety-critical collision metrics.

Figures & tables

Explore similar work

CardsList
  1. DriveVA: Video Action Models are Zero-Shot Drivers

    Apr 5, 2026Mengmeng Liu, Diankun Zhang, Jiuming Liu +7Action-Conditioned Video GenerationWorld Models

  2. ProDrive: Proactive Planning for Autonomous Driving via Ego-Environment Co-Evolution

    Apr 28, 2026Chuyao Fu, Shengzhe Gan, Zhuoli Ouyang +5End-to-End Autonomous DrivingWorld Models

  3. FlowAD: Ego-Scene Interactive Modeling for Autonomous Driving

    Mar 11, 2026Mingzhe Guo, Yixiang Yang, Chuanrong Han +4Autonomous DrivingEnd-to-End Autonomous Driving