World models (WMs) generate driving scenarios beyond recorded logs, enabling the training and evaluation of autonomous driving systems in unseen interactions. However, existing WMs struggle to ensure both behavioral realism and visual fidelity, since they overlook the relationship between vehicle interactions and the ego visual observations. We propose a closed-loop interactive WM that couples behavior modeling and visual rendering. First, imitation-learned background-vehicle (BV) policies are fine-tuned with multi-agent reinforcement learning in the closed loop against an end-to-end ego planner that reacts to the rendered images, so that their interaction behaviors match those in real logs. Second, the viewpoints exposed by these interactions guide a per-Gaussian diagnosis of each 3D Gaussian vehicle asset, which attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces with generated Gaussians, while preserving the observed appearance. Third, signal lights are rendered according to the driving intentions of the vehicles, by an editor that learns the light states unobserved for a vehicle through counterfactual fine-tuning with a light-state classifier. On long-tail closed-loop scenarios, our method reduces the divergence from real-world interaction behaviors by 85% and improves novel-view vehicle PSNR by 2.0dB over the state-of-the-art WM. The significance of BV light signaling in WMs is shown, as the rendered signal lights conveying BV driving intentions indirectly impact ego driving behaviors in a quarter of the scenarios.
Figures & tables
Figure 1: Overview of our world model. Offline: driving logs support 3DGS reconstruction, interaction-guided asset enhancement, and style-conditioned IL. Online: the BV policy is fine-tuned with the ego planner on renderings with intention-triggered signal lights.
Figure 2: Architecture of the BV behavior model. The style latent zi routes to one expert e∗ in its style cluster, which maps the scene feature f to displacements Δp . After IL, the policy (blue) is frozen and the residual head rθ (orange), trained with MARL, adds an offset δ .
Figure 3: Closed-loop interactions generated by our BV policy. Top: ego-centric bird’s-eye view (BEV) with executed trajectories; bottom: our rendering at the marked time from the camera shown in the BEV, with the interacting BV boxed.
Figure 4: A novel view exposed by interaction: the rear camera of the ego (red) sees the front of an overtaken BV (blue). MTGS smears the body, which DiffusionHarmonizer cannot restore; World Engine has frayed edges; ours is complete and keeps the observed appearance.
Method
Overlap ↑
Mean JS ↓
Collision rate ↓
Progress (m) ↑
TTCt0
amax
vt1rel
Mean
GT (log)
–
–
–
–
–
3.50%
27.66
RealEngine † [ 20 ]
0.8588
0.4324
0.6778
0.6563
0.1152
12.71%
20.30
OmniNWM ‡ [ 25 ]
0.7455
0.1492
0.8544
0.5830
0.1886
23.08%
14.65
World Engine [ 28 ]
0.9527
0.5771
0.8498
0.7932
0.0485
10.38%
25.91
Ours
0.9775
0.8719
0.9102
0.9198
0.0074
9.74%
28.10
Table 1: Interaction realism on 288 closed-loop scenarios. GT: logged reference on the same BVs and timestamps; its collisions are overlaps of logged boxes. † BV agents run in our closed loop and renderer. ‡ BV states estimated from generated videos.
Method
Vehicle region
Full image
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MTGS [ 27 ]
17.05
0.411
0.466
18.17
0.571
0.468
Difix3D+ [ 43 ]
16.91
0.396
0.415
18.18
0.539
0.383
DiffusionHarmonizer [ 55 ]
17.11
0.406
0.432
18.08
0.560
0.427
Asset Harvester [ 4 ]
15.24
0.359
0.504
18.06
0.570
0.469
World Engine [ 28 ]
15.97
0.375
0.490
18.08
0.570
0.469
Table 2: Novel-view fidelity on held-out frames of 11 logs. All methods share the MTGS background and differ in the target vehicle assets or post-processing.
Figure 5: Rendered turn signal during a closed-loop cut-in: it turns on at t=1.5 s, before visible lateral motion, and blinks at 1 Hz.
Figure 6: Brake-light editing on rendered vehicles (each pair: target ON / OFF). General editors alter the body or the scene and often fail to switch lights off; ours edits only the anchored lamp regions.
Table 4: Lamp-region similarity to real photos and body change outside lamps (same-state / target-state references).
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
Native MTGS asset
17.05
0.411
0.466
+ diagnosis-guided repair
17.59
0.421
0.464
+ calibration (i)
17.62
0.423
0.463
calib. + harmonizer only
17.55
0.413
0.430
+ fusion, fixed weights
17.80
0.431
0.458
+ fusion, learned weights (ii, full)
17.97
0.438
0.448
Table 5: Ablation of interaction-guided asset enhancement (vehicle region). Rows are cumulative except the indented control.
Variant
Brake mF1 ↑
On-recall ↑
Jump rate ↓
w/o counterfactual loss
0.626
35.4%
–
w/o 3D anchors (2D seg.)
0.845
83.7%
41.4%
w/o reference view
0.825
85.1%
–
w/o state map
0.813
84.1%
–
Full
0.839
86.1%
31.9%
Table 6: Ablation of light signaling. On-recall: vehicles observed only with signal lights off ( 2,811 items). Jump rate: lamp-center shift above 5% of the box size between consecutive steps.
Autonomous driving requires reasoning about how the environment evolves and planning actions accordingly. Existing world-model-based approaches typically predict future scenes first and plan afterwards, resulting in open-loop imagination that may drift from the actual decision process. In this paper, we present Uni-World VLA, a unified vision-language-action (VLA) model that tightly interleaves future frame prediction and trajectory planning. Instead of generating a full world rollout before planning, our model alternates between predicting future frames and ego actions step by step, allowing planning decisions to be continuously conditioned on the imagined future observations. This interleaved generation forms a closed-loop interaction between world modeling and control, enabling more adaptive decision-making in dynamic traffic scenarios. In addition, we incorporate monocular depth information into frames to provide stronger geometric cues for world modeling, improving long-horizon scene prediction. Experiments on the NAVSIM benchmark show that our approach achieves competitive closed-loop planning performance while producing high-fidelity future frame predictions. These results demonstrate that tightly coupling world prediction and planning is a promising direction for scalable VLA driving systems.
Qiqi Liu, Huan Xu, Jingyu Li +5
Fudan University · Shanghai Innovation Institute · Li Auto Inc. +1
Autoregressive driving world models systematically suffer from structural drift, identity drift, and quality decay over long rollout horizons due to transient conditioning signals that lack persistent state. We present WAM-Sim, a driving world model anchored by a persistent hierarchical memory and optimized via online flow-matching RL. WAM-Sim structures persistent context across three temporal and semantic scales: pose-aligned structural tokens encoding persistent road topology, read-only scene tokens supplying geolocated background appearance, and write-once instance tokens preserving individual traffic participant identities across occlusions and chunk boundaries. To restore visual sharpness and physical plausibility degraded by multi-scale conditioning, we introduce an online post-training framework that optimizes self-generated rollouts via flow-matching RL guided by hierarchy-aligned visual rewards and a vision-language model physical judge. The invariant memory empirically confines policy optimization within the anchored context, helping suppress semantic collapse and reward hacking, while online RL systematically enforces physical commonsense and fine-grained visual fidelity. On the nuScenes benchmark, WAM-Sim achieves a superior generation fidelity (FID=3.47, FVD=25.72) and strong downstream perception performance (24.16 mAP, 71.23 road mIoU). On minute-scale rollouts of self-collected data, WAM-Sim substantially mitigates structural and identity drift, preserving high scene similarity and subject identity coherence across extended horizons. Project page: https://fudan-generative-vision.github.io/WAM-Sim.
Jiahao Cui, Jiaye Li, Wenhao Zhang +7
Fudan University · Shenzhen Yinwang Intelligent Technology Co., Ltd
A plausible scene evolution depends on the maneuver being considered, while a good maneuver depends on how the scene may evolve. Existing World Action Models (WAMs) largely miss this reciprocity, treating world prediction and action generation as either isolated parallel branches or rigid predict-then-plan pipelines. We formalize this perspective as World-Action Interactive Models (WAIMs), and instantiate it in autonomous driving with DAWN (Denoising Actions and World iNteractive model), a simple yet strong latent generative baseline. DAWN operates in a compact semantic latent space and couples a World Predictor with a World-Conditioned Action Denoiser: the predicted world hypothesis conditions action denoising, while the denoised action hypothesis is fed back to update the world prediction, so that both are recursively refined during inference. Rather than eliminating test-time world evolution altogether or rolling out the full future in pixel space, DAWN performs a short explicit latent rollout that is sufficient to support long-horizon trajectory generation in complex interactive scenes. Experiments show that DAWN achieves strong planning performance and favorable safety-related results across multiple autonomous driving benchmarks. More broadly, our results suggest that interactive world-action generation is a principled path toward truly actionable world models.
Hongbo Lu, Liang Yao, Chenghao He +6
1COWARobot Co. Ltd · 2Shanghai Jiao Tong University · 3Hohai University