Driving world action models (WAMs) learn from paired trajectories and future observations. Post-training with reinforcement learning allows the policy to explore alternative trajectories and improve planning using driving rewards. However, driving logs contain future videos only for the recorded trajectories, leaving these alternatives without corresponding visual supervision. Thus, we introduce SimVTA, a simulation-grounded post-training framework that combines trajectory optimization with video supervision. First, we sample candidate trajectories for each observation and use a real-world simulator (e.g., World Engine) to render future videos for high-reward candidates in real-world scenes reconstructed with 3D Gaussian Splatting (3DGS). This provides trajectory-matched supervision independent of the model's own predictions. Under a limited rendering budget, we prioritize video supervision for higher-error futures among these rendered candidates. Finally, we optimize trajectory planning with Group Relative Policy Optimization (GRPO) and use a flow-matching loss on the matched videos to update shared representations. At inference time, the model predicts trajectories directly, without simulator calls. On NAVSIM v1, applying SimVTA to a WAM initialization improves PDMS from 86.2 to 91.3 (+5.1). It achieves 88.2 EPDMS on NAVSIM v2 and an average HD-Score of 38.8 in closed-loop evaluation on HUGSIM. Ablations show that updating shared representations with video supervision improves planning, while prioritizing higher-error futures among high-reward trajectories further improves performance.
Figures & tables
Figure 1: From logged futures to candidate-matched supervision. (1) WAM pretraining uses the ground-truth (GT) trajectory and its logged future video. (2) Trajectory GRPO explores candidate trajectories using driving rewards, without matched video supervision. (3) World Engine renders reconstructed scenes along selected candidate paths, providing matched supervision independent of the model’s predictions.
Figure 2: Simulation-grounded video–trajectory alignment. (a) Training overview. (b) Trajectory learning combines GRPO with a behavior-cloning (BC) regularizer. (c) Video supervision selects high-reward, higher-error futures. (d) World Engine renders matched futures along executed paths.
Figure 3: Planning results and video-supervision comparisons. (a) NAVSIM v1. (b) NAVSIM v2 and HUGSIM. (c) Video-supervision configurations and rendering budget on NAVSIM v1.
Figure 4: Qualitative planning comparison. Left: planned trajectories of SimVTA (A, blue) and Epona (B, orange). Right: World Engine renderings along the corresponding paths at 1–4 s. Red boxes highlight Epona’s collision and road-departure regions.
Source
FID ↓
FVD ↓
3DGS renderings
World Engine
7.6
21.8
WAM predictions
Trajectory GRPO only (Exp0)
20.4
105.4
SimVTA (Exp2)
13.1
46.3
Table 4: Video quality.
Figure 5: Matched futures for alternative actions. Left: shared observation history and executed paths for going straight (A), changing lanes to the left (B), and changing lanes to the right (C). Right: corresponding future frames rendered by the simulator. Colors match the paths and image rows.
Table 7Table 8Table 9
Figure 6: Video–trajectory alignment before and after post-training. Left: observation history and trajectory in BEV. Right: World Engine renderings and predictions from SimVTA and Epona.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value
Sampled trajectories / rendering budget
G=32 / K=4
Batch size per GPU / default random seed
4 / 1234
Training hardware
16 NVIDIA H20 GPUs
Initial learning rate / AdamW betas
10−5 / (0.9,0.95)
Weight decay / LR decay factor
0.01 / 0.5
LR decay steps
300, 1500
Appendix
Table 11: Exp2 video-supervision settings and default optimization parameters.
Setting
K
Rendered pool
Video-training set
Video objective
PDMS
Exp0
0
None
None
None
89.1
Exp1
4
High reward
Q2
FM
89.1
Exp2
4
High reward
Q2
FM
91.3
Exp3
4
High reward
Q2, shuffled
FM
83.1
Exp4
4
2 high + 2 low
Q2 + Q3
FM
90.9
Exp5
4
High reward
Q1 + Q2
FM
91.0
Appendix
Table 12: Video-supervision configurations. K is the maximum number of rendered candidates per observation in each sampling round. For Exp9, this budget applies to offline cache construction. FM denotes flow matching.
Setting
NC
DAC
TTC
Comfort
EP
PDMS
Exp0
97.1
96.7
96.0
99.8
83.6
89.1
Exp1
97.6
96.7
96.4
99.7
83.3
89.1
Exp2
98.0
97.2
97.0
100.0
86.7
91.3
Exp3
94.1
91.3
91.8
99.6
81.2
83.1
Exp4
97.4
97.3
96.4
99.9
86.4
90.9
Exp5
98.0
97.1
97.0
99.9
86.1
91.0
Appendix
Table 13: NAVSIM v1 ablation results. Configurations are listed in Table 12 .
Setting
Random seed
Mean ± SD
1234
20260901
20260910
Exp0
89.1
89.2
89.0
89.1±0.1
Exp2
91.3
91.4
91.3
91.3±0.1
Exp5
91.0
90.9
91.0
91.0±0.1
Exp7
91.1
91.2
91.2
91.2±0.1
Appendix
Table 14: NAVSIM v1 PDMS across random seeds. The final column reports the mean and sample standard deviation over three runs.
Method
NAVSIM v1 PDMS
NAVSIM v2 EPDMS
SimWAM ( Zhao et al., 2026 )
91.5
90.2
Latent-WAM ( Wang et al., 2026b )
–
89.3
WA-JEPA ( Wang et al., 2026c )
91.8
91.7
SimVTA
91.3
88.2
Appendix
Table 15: Recent WAM results reported by their respective papers. Training data, initialization, and evaluators differ across methods.
Figure 7: Reported planning results across benchmarks. (a) NAVSIM v1 PDMS and v2 EPDMS, with NAVSIM v2 component-score differences ( SimVTA minus the NAVSIM-adapted Epona initialization). (b) HUGSIM RC and HD-Score using benchmark aggregation. (c) RC and (d) HD-Score for DrivoR and SimVTA by scenario difficulty. Values correspond to Tables 1 , 2 , and 3 .
In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring video dynamics priors to action prediction, but many still couple planning with future-video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future-frame generation at inference. This design supports multiple pretrained video backbones and independent action-expert scaling within a shared attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves 91.9 PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
Zongchuang Zhao, Xin Zhou, Tianyang Xu +6
Huazhong University of Science & Technology · Dongfeng Research & Development Institute
Generalization is a central challenge in autonomous driving, as real-world deployment requires robust performance under unseen scenarios, sensor domains, and environmental conditions. Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction, yet their generalization across datasets and sensor configurations remains limited. In addition, their loosely coupled planning paradigm often leads to poor video-trajectory consistency during visual imagination. To overcome these limitations, we propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process. DriveVA inherits rich priors on motion dynamics and physical plausibility from well-pretrained large-scale video generation models to capture continuous spatiotemporal evolution and causal interaction patterns. To this end, DriveVA employs a DiT-based decoder to jointly predict future action sequences (trajectories) and videos, enabling tighter alignment between planning and scene evolution. We also introduce a video continuation strategy to strengthen long-duration rollout consistency. DriveVA achieves an impressive PDM-based planning performance of 90.9 PDM score on the NAVSIM benchmark. Extensive experiments also demonstrate the zero-shot capability and cross-domain generalization of DriveVA, which reduces average L2 error and collision rate by 78.9% and 83.3% on nuScenes and 52.5% and 52.4% on the Bench2Drive built on CARLA v2 compared with the state-of-the-art world-model-based planner.
Mengmeng Liu, Diankun Zhang, Jiuming Liu +7
University of Twente, The Netherlands · Xiaomi EV, China · University of Cambridge, United Kingdom +1
Driving world models learn rich predictive representations of the surrounding environment from visual observations, yet accurate visual prediction does not necessarily translate into effective trajectory planning. We argue that a key bottleneck lies in the mismatch between visual world states and raw geometric trajectories, which may limit the planner's ability to exploit action-relevant semantics encoded by the world model. To address this issue, we propose World-Model Alignment for Latent Trajectories (WALT), which learns a compact generative trajectory latent space by transferring information from a frozen pretrained driving world model without modifying the world model itself. Rather than directly generating raw waypoints, WALT maps them into compact representations through a dual-branch trajectory autoencoder and transfers semantic knowledge from the frozen visual world model into this trajectory space, encouraging the learned action representation to capture scene-level cues relevant to future motion and planning. Beyond our proposed formulation, we systematically study latent learning based on Joint-Embedding Predictive Architectures (JEPA) and feature alignment following Representation Alignment (REPA) to investigate how trajectory-only representation learning affects downstream planning. We evaluate WALT on the NAVSIM benchmarks. Relative to the raw-waypoint baseline, WALT improves PDMS from 89.4 to 89.8 on NAVSIMv1 and EPDMS from 87.3 to 87.9 on NAVSIMv2 while reducing trajectory planner FLOPs by 30.5%. These results suggest that preserving world representations while extracting action-relevant information provides an effective interface for world-model-based trajectory planning.
Mingkai Jia, Jiaxin Guo, Zhijian Shu +5
The Hong Kong University of Science and Technology. · Horizon Robotics. · The Chinese University of Hong Kong. +2