World Action Models (WAMs) connect future scene prediction with action planning for autonomous driving. Each driving record pairs an executed ego trajectory with its observed future, providing a factual outcome for learning this connection. We present TrajWAM, a trajectory-conditioned WAM that improves vision-language driving through auxiliary bird's-eye-view (BEV) future learning. An ego-trajectory query conditions DINOv3-aligned BEV predictions from shared planner representations. Training combines factual reconstruction under the executed trajectory with one-sided feature separation under an unlabeled perturbed trajectory. Together, these objectives train the shared planner to generate driving waypoints. Native inference directly generates waypoints and bypasses the auxiliary future-prediction branch. On all 220 Bench2Drive routes, TrajWAM achieves 91.53 Driving Score and 75.45% Success Rate, exceeding a schedule-matched planning-only fine-tuning baseline by 5.31 Driving Score points and eight successful routes.
Figures & tables
Figure 1: Motivation and overview of TrajWAM. (a) Trajectory-agnostic future supervision predicts future features and driving waypoints from shared scene representations. Introducing trajectory queries creates a supervision challenge: the executed query has an observed future, while alternative queries remain unlabeled. (b) Factual reconstruction and query separation jointly train the driving policy, which generates waypoints directly at inference. Upper right: reported Driving Score and ability comparisons.
Figure 2: Overview of TrajWAM. (a) Shared planning and trajectory-conditioned auxiliary learning. (b) Logged and perturbed trajectory queries. (c) FiLM conditioning. Sections 3.4 – 3.5 detail the readout and objectives. Solid arrows denote training and inference; dashed arrows denote training-only computation.
Method
Venue
Paradigm
Expert
Closed-loop
DS ↑
SR(%) ↑
Efficiency ↑
Smoothness ↑
TCP* [ 29 ]
NeurIPS ’22
E2E
Think2Drive
40.70
15.00
54.26
47.80
ThinkTwice* [ 6 ]
CVPR ’23
E2E
Think2Drive
62.44
31.23
69.33
16.22
DriveAdapter* [ 5 ]
ICCV ’23
E2E
Think2Drive
64.22
33.08
70.22
16.01
VAD [ 9 ]
ICCV ’23
E2E
Think2Drive
42.35
15.00
157.94
46.01
GenAD [ 39 ]
ECCV ’24
E2E
Think2Drive
44.81
15.90
-
-
Table 1: Bench2Drive Base Set closed-loop results. Venue reports conference/preprint and year; ∗ denotes expert-feature distillation. Bold/underlined values mark the best/second-best listed results.
Method
Venue
Paradigm
Ability (%) ↑
Merging
Overtaking
Emergency Brake
Give Way
Traffic Sign
Mean
TCP* [ 29 ]
NeurIPS ’22
E2E
16.18
20.00
20.00
10.00
6.99
14.63
ThinkTwice* [ 6 ]
CVPR ’23
E2E
27.38
18.42
35.82
50.00
54.23
37.17
DriveAdapter* [ 5 ]
ICCV ’23
E2E
28.82
26.38
48.76
50.00
56.43
42.08
AD-MLP [ 35 ]
arXiv ’23
E2E
0.00
0.00
0.00
0.00
4.35
0.87
UniAD-Base [ 3 ]
CVPR ’23
E2E
14.10
17.78
21.67
10.00
14.21
15.55
Table 2: Bench2Drive multi-ability results (%). Venue and expert-feature distillation ( ∗ ) follow Table 1 . Bold/underlined values mark the best/second-best listed results.
Model
Factual DINO
Traj.-FiLM
Sep.
DS ↑
SR ↑
Base Planner (planning-only FT)
–
–
–
86.22
71.36%
Base Planner (unconditioned future FT)
✓
–
–
85.34
73.18%
TrajWAM w/o separation
✓
✓
–
89.34
74.55%
TrajWAM
✓
✓
✓
91.53
75.45%
Table 3: Phase-II component ablation on 220 routes under the setup in Section 4.1 . Factual DINO denotes target reconstruction, unconditioned in Base Planner (unconditioned future FT); Traj.-FiLM comprises the ActionEncoder and latent FiLM; Sep. denotes Equation 10 . The Base Planner (planning-only FT) control is shared with Table 11 .
Training variant
Seed 42 DS ↑
Seed 43 DS ↑
Mean DS ± SD
Planning-only FT
82.64
83.16
82.90±0.37
Unconditioned future FT
82.11
82.49
82.30±0.27
TrajWAM w/o separation
86.28
86.52
86.40±0.17
TrajWAM (additive)
86.07
85.83
85.95±0.17
TrajWAM (correct pairing)
88.69
88.21
88.45±0.34
Shuffled pairing
84.02
84.38
84.20±0.25
Table 4: Matched two-seed DS comparisons on the fixed 120-route subset. Summary values are mean ± sample standard deviation across seeds 42 and 43. All rows share the Phase-I initialization and 24-epoch training budget.
Figure 3: Executed-query benefits under different training variants, averaged over seeds 42 and 43 and 15 routes. Positive Δ MSE favors executed over mismatched queries on 371 fixed windows. Bars show conditional route-bootstrap 95% intervals; the bottom row directly compares correct and shuffled training.
Figure 4: Selected closed-loop comparisons with DeepSight under matching route, scenario, and weather configurations. Each case shows TrajWAM above DeepSight. Recorded DeepSight outcomes are (a) vehicle collision, (b) collision and route deviation, and (c) blocked-agent termination; TrajWAM completes all three routes successfully. Arrows indicate chronological order within each row, with actual simulation times and speeds. Columns use nearby ego positions in (a), equal elapsed times in (b), and independent time sequences in (c) to show progression versus persistent standstill.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Future-feature predictions under longitudinal deceleration perturbations. Top: rainy example from Accident_Town04_Route205_Weather23 , frame 51, perturbation seed 2042. Bottom: snowy example from Accident_Town04_Route160_Weather3 , frame 51, seed 2029. Columns span t through t+2 s in 0.5 s increments. Rows show the factual DINOv3 target, predictions under the logged and perturbed queries, factual cosine error, and label-free query sensitivity. Appendix B describes the shared-PCA visualization and native-space metric computation.
Figure 6: Future-feature predictions under lateral coordinate perturbations. Top: nighttime example from Accident_Town03_Route102_Weather20 , frame 99, perturbation seed 2033. Bottom: rainy example from Accident_Town04_Route205_Weather23 , frame 163, seed 2040. The columns, rows, and visualization procedure follow Figure 5 .
Outcome
Routes
Fraction
Strict success
166
75.45%
Completed with disqualifying infraction
24
10.91%
Did not complete
30
13.64%
Route deviation
12
5.45%
Route timeout
12
5.45%
Vehicle blocked
5
2.27%
Appendix
Table 5: Mutually exclusive outcomes on all 220 routes. The four termination reasons partition the 30 incomplete routes.
Quantity
Base Planner (Phase I)
TrajWAM
Parameters
4.063B
4.086B ( +0.554% )
Payload
8.755GB
8.800GB ( +45.04 MB)
Native path
Planner + PID
Planner + PID
Appendix
Table 6: Stored checkpoint footprint and native execution path for Base Planner (Phase I) and TrajWAM, using decimal units. Parameter counts and payloads include auxiliary tensors that are bypassed during native inference.
Model or reference
Future MSE ↓
Cosine ↑
Public DeepSight
0.02329
0.82575
TrajWAM
0.03029
0.76771
External additive reference (B)
0.02340
0.82475
Public DeepSight: repeat predicted t
0.02463
0.81594
TrajWAM: repeat predicted t
0.03110
0.76099
External additive reference: repeat predicted t
0.02471
0.81506
Appendix
Table 7: Single-checkpoint external-reference comparison on all 374 evaluation windows from 15 newly collected episodes. Metrics average the four future offsets and then routes equally. The public and additive models are prediction references with different training histories. The last row uses privileged actual current-BEV input.
Model
Metric
t
t+0.5
t+1.0
t+1.5
t+2.0
Public DeepSight
MSE ↓
0.02378
0.02330
0.02317
0.02335
0.02333
Cosine ↑
0.82259
0.82600
0.82661
0.82531
0.82505
TrajWAM
MSE ↓
0.03042
0.03025
0.03005
0.03024
0.03062
Cosine ↑
0.76726
0.76843
0.76949
0.76784
0.76509
External additive reference
MSE ↓
0.02382
0.02337
0.02327
0.02348
0.02346
Cosine ↑
0.82208
0.82529
0.82565
0.82413
0.82393
Appendix
Table 8: Single-checkpoint per-offset prediction for the external-reference comparison on the same 374 windows. Each entry averages windows within a route and then the 15 routes equally; Table 7 averages the four future columns.
Figure 7: Single-checkpoint trajectory-pairing effects for the TrajWAM checkpoint in Table 7 , across all 371 matched evaluation windows. Each window receives equal weight in the empirical cumulative distribution. We define ΔMSE=MSEmismatched−MSEexecuted , averaged over the four future offsets from 0.5 to 2.0s . The dashed line marks zero; positive values indicate lower error with the executed query.
Future
t=0
Model
Arithmetic
Energy
Variance
Energy
Variance
F
Native BF16
7.2099
4.6981
6.7815
4.4884
F
FP32 replay
6.6529
4.1715
6.2456
3.9822
B
Native BF16
0.9000
0.3505
0.9000
0.3517
B
FP32 replay
0.5221
0
0.5221
0
Appendix
Table 9: Single-checkpoint responses to the same trajectory pair across 252 windows, using the external-reference models from Appendix E.1 . All entries are in units of 10−6 . F is TrajWAM; B is the external additive reference. The FP32 rows use the analytic readout difference.
Measurement
DeepSight
TrajWAM
Generation (s)
3.4926
3.5086
File to waypoints (s)
3.7539
3.7676
PID (ms)
0.1940
0.1982
Peak allocated memory (GiB)
8.1813
8.2217
Mean output tokens
96.25
93.25
Appendix
Table 10: Native planner microbenchmark. Times are medians over 12 measurements; memory is the peak allocated GPU memory. The DeepSight reference is its public checkpoint.
Model
DS ↑
SR (%) ↑
Base Planner (Phase I)
85.94
70.91%
Base Planner (planning-only FT)
86.22
71.36%
TrajWAM
91.53
75.45%
Appendix
Table 11: Schedule-matched Phase-II comparison on all 220 Bench2Drive routes. Models and objectives follow Section 4.1 .
Variant
Training seed
DS ↑
SR (%) ↑
Planning-only FT
42
82.64
64.17
Planning-only FT
43
83.16
65.00
Unconditioned future FT
42
82.11
65.83
Unconditioned future FT
43
82.49
66.67
TrajWAM w/o separation
42
86.28
66.67
TrajWAM w/o separation
43
86.52
67.50
Appendix
Table 12: Individual training-seed results on the same 120 routes. SR values and their paired differences are rounded after aggregation.
Comparison
Seeds
Mean Δ DS
95% CI
Mean Δ SR
95% CI
Full − planning-only FT
42, 43
+5.55
[+3.62,+7.50]
+4.17
[−0.83,+9.17]
Full − no separation
42, 43
+2.05
[+0.54,+3.54]
+1.67
[−2.92,+6.25]
No separation − unconditioned
42, 43
+4.10
[+2.46,+5.81]
+0.83
[−2.92,+5.00]
Full − additive
42, 43
+2.50
[+1.03,+3.99]
+2.50
[−1.67,+7.08]
Appendix
Table 13: Paired gains on the 120-route subset. SR differences are percentage points. Intervals are conditional, scenario-stratified paired route-bootstrap intervals; each comparison uses only its listed common training seeds.
Comparison
Seeds
Route-equal Δ DS
Scenario-equal Δ DS
95% CI
Signs ( +/0/− )
Full − planning-only FT
42, 43
+5.55
+5.55
[+3.42,+7.68]
24 / 16 / 4
Full − no separation
42, 43
+2.05
+2.10
[+0.46,+3.75]
22 / 19 / 3
No separation − unconditioned
42
+4.17
+4.07
[+2.14,+6.08]
21 / 20 / 3
Full − additive
42
+2.62
+2.76
[+1.28,+4.28]
20 / 20 / 4
Appendix
Table 14: Scenario-balanced analysis on the fixed subset. The first two rows average seeds 42 and 43; the last two use seed 42. Sign counts list positive, tied, and negative scenario-level mean DS differences. Intervals use the same conditional stratified route resampling with equal scenario weights.
Comparison
Seeds
Full-subset Δ DS
Route-equal range
Scenario-equal range
Full − planning-only FT
42, 43
+5.55
[+5.05,+5.93]
[+5.09,+6.02]
Full − no separation
42, 43
+2.05
[+1.65,+2.60]
[+1.69,+2.60]
No separation − unconditioned
42
+4.17
[+3.77,+4.56]
[+3.53,+4.52]
Full − additive
42
+2.62
[+2.15,+3.06]
[+2.34,+3.16]
Appendix
Table 15: Leave-one-scenario-type-out analysis. Each range spans the 44 recomputed mean DS differences after removing one scenario type and renormalizing the remaining weights. These are sensitivity ranges. The first two rows average seeds 42 and 43; the last two use seed 42.
Training / comparison
Seed 42: DS / SR
Seed 43: DS / SR
Mean DS ± SD
Mean SR ± SD
Shuffled pairing
84.02 / 66.67
84.38 / 67.50
84.20±0.25
67.08±0.59
TrajWAM (correct pairing)
88.69 / 69.17
88.21 / 68.33
88.45±0.34
68.75±0.59
Correct − shuffled
+4.67 / +2.50
+3.83 / +0.83
+4.25[2.64,5.89]
+1.67[−2.50,5.84]
Appendix
Table 16: Correct versus within-group shuffled training pairing on the fixed 120 routes. Both variants use seeds 42 and 43. SR and its differences are percentages and percentage points, respectively. The last row reports paired differences and conditional scenario-stratified route-bootstrap 95% intervals.
Model
Mean DS ↑
Future MSE ↓
Persistence MSE ↓
Query gain ↑
95% CI
Unconditioned future FT
82.30
0.03110
0.03170
—
—
TrajWAM w/o separation
86.40
0.02985
0.03055
+1.60
[−0.60,+4.00]
Matched additive
85.95
0.03055
0.03115
+0.70
[−1.00,+2.80]
TrajWAM
88.45
0.03020
0.03101
+6.00
[+0.80,+12.50]
Appendix
Table 17: Matched two-seed planning–prediction diagnosis. DS is the mean on 120 closed-loop routes; future and persistence MSE use 374 windows. Query gains and their conditional 95% intervals use 371 windows and are in units of 10−5 . Unconditioned supervision has no trajectory-query comparison.
Training / comparison
Mean query gain
95% CI
Shuffled pairing
+0.30
[−1.20,+1.80]
Correct pairing
+6.00
[+0.80,+12.50]
Correct − shuffled
+5.70
[+0.50,+11.80]
Appendix
Table 18: Executed-query benefits after correct or shuffled training pairing. All gains and conditional paired route-bootstrap 95% intervals are in units of 10−5 MSE, averaged over seeds 42 and 43.
World models for autonomous driving should anticipate the consequences that matter for driving decisions rather than reconstruct every detail of future scenes. However, predictive representations alone are not guaranteed to distinguish candidate actions that lead to different planning outcomes.We propose DA-WAM, a decision-aligned world action model that couples a Joint-Embedding Predictive Architecture (JEPA) with candidate-level trajectory scoring. During joint training, the JEPA objective grounds future representations in observed scene dynamics, while the planning objectives encourage these representations to capture distinctions relevant to safety and utility. An action-conditioned predictor generates a distinct future latent for each candidate trajectory, and a multi-head trajectory scorer uses the corresponding latent to estimate planning factors and overall utility. In addition, safety-critical post-training with hard negatives further improves discrimination between geometrically similar trajectories with different safety outcomes. DA-WAM achieves state-of-the-art performance among learned camera-based planners on NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Ablation studies demonstrate the effectiveness of each component in driving decision-making.
Ruiguo Zhong, Xiaolong Chen, Benshan Ma +5
The Hong Kong University of Science and Technology (Guangzhou) · Leapmotor · The Hong Kong University of Science and Technology
Vision-Language Models (VLMs) and world models have demonstrated strong capabilities in semantic understanding and future scene evolution for autonomous driving. However, effectively integrating both capabilities for trajectory generation remains challenging. In this work, we propose DriveWLA, a unified World-Language-Action model based on Mixture-of-Transformers (MoT) that jointly models semantic understanding, future scene dynamics, and trajectory generation. DriveWLA consists of three experts: a World Expert for future prediction, a Language Expert for semantic understanding, and an Action Expert for trajectory generation. We further introduce Cross-Expert Attention, which enables controlled information flow across experts while aligning semantic and dynamic representations. To improve trajectory generation beyond imitation learning, we apply Flow-GRPO reinforcement learning post-training to refine the Action Expert. Extensive experiments show that DriveWLA achieves superior performance in both open-loop and closed-loop evaluation. DriveWLA achieves 92.3 PDMS on NAVSIM v1 and 90.1 EPDMS on NAVSIM v2, demonstrating strong performance among VLM-based and world-model-based planners. Without specific fine-tuning, DriveWLA further demonstrates strong zero-shot generalization to HUGSIM, nuScenes, and Bench2Drive. These results validate the effectiveness of our unified World-Language-Action modeling, establishing DriveWLA as a generalizable framework for autonomous driving.
Shuzheng Wang, Haoxuan Li, Shengzhen Li +5
Huazhong University of Science and Technology · Yinwang Intelligent Technology Co. Ltd.
In autonomous driving, World-Action Models (WAMs) have improved end-to-end planning by transferring video dynamics priors to action prediction, but many still couple planning with future-video generation at inference, incurring substantial computational overhead. We present SimWAM, a simple yet effective WAM that leverages future-video prediction solely as a training-time supervision signal. It co-trains a pretrained video expert and a lightweight action expert with joint flow matching. An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without future-frame generation at inference. This design supports multiple pretrained video backbones and independent action-expert scaling within a shared attention interface, while preserving the joint learning objective. Moreover, we apply reinforcement learning to optimize a compositional driving reward beyond trajectory imitation. Experiments show that SimWAM achieves 91.9 PDMS on NAVSIM with a favorable trade-off between accuracy and latency among world-model-based planners, while transferring zero-shot to nuScenes. It also achieves competitive planning accuracy on WOD-E2E and PhysicalAI-Autonomous-Vehicles. These results position SimWAM as a plain yet solid baseline for efficient autonomous driving. The code and model weights are available at https://github.com/H-EmbodVis/SimWAM/.
Zongchuang Zhao, Xin Zhou, Tianyang Xu +6
Huazhong University of Science & Technology · Dongfeng Research & Development Institute