cs.RONov 17, 2026

Making the World React Realistically: A Closed-Loop Interactive World Model for Autonomous Driving

Authors: Jia Hu, Yang Chang, Panqu Wang, Zhijun Fu, Chang Liu, Changjian Yu, Jintao Lai

Organizations: Tongji University · ZERON · Zhengzhou University of Light Industry · Budapest University of Technology and Economics · 

Abstract

World models (WMs) generate driving scenarios beyond recorded logs, enabling the training and evaluation of autonomous driving systems in unseen interactions. However, existing WMs struggle to ensure both behavioral realism and visual fidelity, since they overlook the relationship between vehicle interactions and the ego visual observations. We propose a closed-loop interactive WM that couples behavior modeling and visual rendering. First, imitation-learned background-vehicle (BV) policies are fine-tuned with multi-agent reinforcement learning in the closed loop against an end-to-end ego planner that reacts to the rendered images, so that their interaction behaviors match those in real logs. Second, the viewpoints exposed by these interactions guide a per-Gaussian diagnosis of each 3D Gaussian vehicle asset, which attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces with generated Gaussians, while preserving the observed appearance. Third, signal lights are rendered according to the driving intentions of the vehicles, by an editor that learns the light states unobserved for a vehicle through counterfactual fine-tuning with a light-state classifier. On long-tail closed-loop scenarios, our method reduces the divergence from real-world interaction behaviors by 85% and improves novel-view vehicle PSNR by 2.0dB over the state-of-the-art WM. The significance of BV light signaling in WMs is shown, as the rendered signal lights conveying BV driving intentions indirectly impact ego driving behaviors in a quarter of the scenarios.

Figures & tables

Explore similar work

CardsList
  1. Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving

    Mar 28, 2026Qiqi Liu, Huan Xu, Jingyu Li +5VLMs for Autonomous DrivingWorld Model-Based Planning

  2. WAM-Sim0.1: World Anchored Memory Meets Online RL for Consistent and Realistic Driving Simulation

    Nov 5, 2026Jiahao Cui, Jiaye Li, Wenhao Zhang +7Memory-Augmented World ModelsWorld Models for Autonomous Driving

  3. The DAWN of World-Action Interactive Models

    May 12, 2026Hongbo Lu, Liang Yao, Chenghao He +6Diffusion-Based PlanningWorld Model-Based Planning