World models (WMs) generate driving scenarios beyond recorded logs, enabling the training and evaluation of autonomous driving systems in unseen interactions. However, existing WMs struggle to ensure both behavioral realism and visual fidelity, since they overlook the relationship between vehicle interactions and the ego visual observations. We propose a closed-loop interactive WM that couples behavior modeling and visual rendering. First, imitation-learned background-vehicle (BV) policies are fine-tuned with multi-agent reinforcement learning in the closed loop against an end-to-end ego planner that reacts to the rendered images, so that their interaction behaviors match those in real logs. Second, the viewpoints exposed by these interactions guide a per-Gaussian diagnosis of each 3D Gaussian vehicle asset, which attenuates view-dependent colors on under-observed Gaussians and fills never-observed surfaces with generated Gaussians, while preserving the observed appearance. Third, signal lights are rendered according to the driving intentions of the vehicles, by an editor that learns the light states unobserved for a vehicle through counterfactual fine-tuning with a light-state classifier. On long-tail closed-loop scenarios, our method reduces the divergence from real-world interaction behaviors by 85% and improves novel-view vehicle PSNR by 2.0dB over the state-of-the-art WM. The significance of BV light signaling in WMs is shown, as the rendered signal lights conveying BV driving intentions indirectly impact ego driving behaviors in a quarter of the scenarios.
Figures & tables
Figure 1: Overview of our world model. Offline: driving logs support 3DGS reconstruction, interaction-guided asset enhancement, and style-conditioned IL. Online: the BV policy is fine-tuned with the ego planner on renderings with intention-triggered signal lights.
Figure 2: Architecture of the BV behavior model. The style latent zi routes to one expert e∗ in its style cluster, which maps the scene feature f to displacements Δp . After IL, the policy (blue) is frozen and the residual head rθ (orange), trained with MARL, adds an offset δ .
Figure 3: Closed-loop interactions generated by our BV policy. Top: ego-centric bird’s-eye view (BEV) with executed trajectories; bottom: our rendering at the marked time from the camera shown in the BEV, with the interacting BV boxed.
Figure 4: A novel view exposed by interaction: the rear camera of the ego (red) sees the front of an overtaken BV (blue). MTGS smears the body, which DiffusionHarmonizer cannot restore; World Engine has frayed edges; ours is complete and keeps the observed appearance.
Method
Overlap ↑
Mean JS ↓
Collision rate ↓
Progress (m) ↑
TTCt0
amax
vt1rel
Mean
GT (log)
–
–
–
–
–
3.50%
27.66
RealEngine † [ 20 ]
0.8588
0.4324
0.6778
0.6563
0.1152
12.71%
20.30
OmniNWM ‡ [ 25 ]
0.7455
0.1492
0.8544
0.5830
0.1886
23.08%
14.65
World Engine [ 28 ]
0.9527
0.5771
0.8498
0.7932
0.0485
10.38%
25.91
Ours
0.9775
0.8719
0.9102
0.9198
0.0074
9.74%
28.10
Table 1: Interaction realism on 288 closed-loop scenarios. GT: logged reference on the same BVs and timestamps; its collisions are overlaps of logged boxes. † BV agents run in our closed loop and renderer. ‡ BV states estimated from generated videos.
Method
Vehicle region
Full image
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MTGS [ 27 ]
17.05
0.411
0.466
18.17
0.571
0.468
Difix3D+ [ 43 ]
16.91
0.396
0.415
18.18
0.539
0.383
DiffusionHarmonizer [ 55 ]
17.11
0.406
0.432
18.08
0.560
0.427
Asset Harvester [ 4 ]
15.24
0.359
0.504
18.06
0.570
0.469
World Engine [ 28 ]
15.97
0.375
0.490
18.08
0.570
0.469
Table 2: Novel-view fidelity on held-out frames of 11 logs. All methods share the MTGS background and differ in the target vehicle assets or post-processing.
Figure 5: Rendered turn signal during a closed-loop cut-in: it turns on at t=1.5 s, before visible lateral motion, and blinks at 1 Hz.
Figure 6: Brake-light editing on rendered vehicles (each pair: target ON / OFF). General editors alter the body or the scene and often fail to switch lights off; ours edits only the anchored lamp regions.
Table 4: Lamp-region similarity to real photos and body change outside lamps (same-state / target-state references).
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
Native MTGS asset
17.05
0.411
0.466
+ diagnosis-guided repair
17.59
0.421
0.464
+ calibration (i)
17.62
0.423
0.463
calib. + harmonizer only
17.55
0.413
0.430
+ fusion, fixed weights
17.80
0.431
0.458
+ fusion, learned weights (ii, full)
17.97
0.438
0.448
Table 5: Ablation of interaction-guided asset enhancement (vehicle region). Rows are cumulative except the indented control.
Variant
Brake mF1 ↑
On-recall ↑
Jump rate ↓
w/o counterfactual loss
0.626
35.4%
–
w/o 3D anchors (2D seg.)
0.845
83.7%
41.4%
w/o reference view
0.825
85.1%
–
w/o state map
0.813
84.1%
–
Full
0.839
86.1%
31.9%
Table 6: Ablation of light signaling. On-recall: vehicles observed only with signal lights off ( 2,811 items). Jump rate: lamp-center shift above 5% of the box size between consecutive steps.