Driving scenes keep changing during model computation, making both observation freshness and action-delivery time central to world-action planning. To retain rich generative world priors in autonomous driving without stalling real-time control, we present SWIFT (Slow World, Instant Freshness, and Timely action), a deployment-oriented asynchronous world-action system that moves world updates off the observation-to-action critical path. On a background GPU, a slow Video DiT streams surround-view history into read-only predictive world context, trained with a predictive span that covers both the update interval and background world computation. On a foreground GPU, a lightweight freshness carrier encodes current surround-view images into a gated key/value bank at every step, letting a fast Action DiT combine live visual feedback with the latest completed world context without waiting. On open-loop NAVSIM, SWIFT reaches 94.2 PDMS at 198.4 ms per decision with a trajectory scorer; on closed-loop Bench2Drive, it achieves an 80.30 Driving Score at 109.9 ms foreground latency. SWIFT further transfers to the Waymo E2E hidden test (8.0050 Rater Feedback Score) and nuScenes (0.54 m mean endpoint error, 0.03% mean collision rate). Across all four benchmarks, SWIFT achieves the best results among compared world-action methods, establishing a favorable accuracy-latency trade-off.
Figures & tables
Figure 1: Pretraining gains and decision latency. (a) NAVSIM PDMS of SWIFT without the scorer, by pretraining stage. (b) PDMS versus decision latency (log scale) for world-action methods in Tab. 2 , as reported on each method’s hardware; ours is the foreground decision on one H20 while the world model updates on a second. Stars mark SWIFT without and with the scorer.
Figure 2: Current visual evidence at every planning step. A chunk-based synchronous WAM admits new visual evidence when a chunk is processed and runs world and action inference in series. Our Action DiT combines the latest completed world K/V with current surround-view evidence at every planning step while the next world update is still in flight. Timing is schematic.
Figure 3: SWIFT architecture. The slow Video DiT converts clean camera history into cached world K/V Cs and residuals Xs . The freshness carrier supplies current-view K/V Ft∣s at every planning step while world updates proceed in the background. Conditioned on ego state and navigation, the fast Action DiT reads both banks to generate trajectories. The NAVSIM configuration is shown, with seven views and candidate scoring. Noisy future video provides training targets only; no video is decoded at inference, and the dashed decoder, video panel, and candidate labels are illustrative.
Figure 4: Asynchronous context reuse. The upper timeline shows background world updates and their predictive spans. The lower timeline shows foreground decisions reading the latest completed cache together with current inputs. Filled trajectory markers are executed; hollow markers are replanned. Each predictive span covers the update interval plus the world update latency. Spacing is schematic.
Quantity
Symbol
NAVSIM
Bench2Drive
Decision interval
Δ
0.5 s
0.1 s
World update interval
Tu
1.0 s
1.0 s
World update latency
τw
760.6 ms
941.5 ms
First-read delay
τd
1.0 s
1.0 s
Context age
at
1.0, 1.5 s
1.0–1.9 s
Predictive span
Tp
2.0 s
2.0 s
Table 1: Timing of the asynchronous schedule. The chunk step first reads a new world context ( at=τd ); freshness steps follow.
Figure 5: Attention visibility. Rows A0–A3 are the decisions in one training sequence. Each decision’s action queries read its world-history window (R) and its own freshness (F) and action (A) tokens under one softmax over the available K/V. Noised future-video tokens (V, training targets) and other decisions’ freshness and action tokens are blocked. Colored cells show visible attention and blank cells show blocked attention.
Figure 6: Trajectory scorer. Candidate trajectories and the current ego condition form queries. Four decoder layers combine candidate self-attention with cross-attention to detached world and freshness features. Six heads predict driving sub-scores; the PDMS aggregation of five of them selects the final trajectory, and the sixth is auxiliary.
Method
Year
In.
Ann.
NC ↑
DAC ↑
EP ↑
TTC ↑
Comfort ↑
PDMS ↑
Latency (ms) ↓
World-action methods
DriveLaW [ 59 ]
2026
1V
×
99.0
97.1
81.3
96.7
100.0
89.1
710.0
DriveVLA-W0 [ 31 ]
2026
1V
×
98.7
99.1
83.3
95.3
99.3
90.2
–
SV-WAM [ 55 ]
2026
6V
✓
98.6
98.6
83.9
95.1
100.0
90.2
341.6
PerceptDrive [ 39 ]
2026
1V
✓
99.2
99.0
82.9
98.0
100.0
90.4
68.0
4D-WAM [ 12 ]
2026
1V
×
99.1
97.9
84.6
96.6
100.0
90.9
790.0
Table 2: NAVSIM v1 comparison on navtest . In.: camera views (V) and LiDAR (L). Ann.: perception or HD-map label supervision, including derived simulator sub-scores. Latencies follow each paper’s hardware ( † : our H20 rerun); ours is the foreground decision on one H20 while the world model updates on a second. SWIFT uses a 7 s history and the chunk-step age ( Sec. 4.1 ). Bold: best accuracy per column; dashes: unavailable.
Figure 7: Qualitative results. The top two rows show NAVSIM plans; brighter green indicates higher scorer output and red denotes ground truth. The bottom two rows show closed-loop Bench2Drive examples of rainy-night parking and a pedestrian crossing. Boxes and traffic-light insets are simulator ground-truth overlays.
Method
DS ↑
SR (%) ↑
Emerg. Brake ↑
Give Way ↑
Latency (ms) ↓
MindDriver [ 69 ]
65.48
39.55
–
–
1000.0
Raw2Drive [ 62 ]
71.36
50.24
60.00
50.00
604.0
World4Scorer [ 42 ]
73.27
46.36
–
–
–
MomWorld [ 50 ]
74.07
50.00
61.71
50.00
–
GraphWorld [ 49 ]
76.71
44.22
49.09
50.00
–
PerlAD [ 14 ]
78.70
57.27
68.33
50.00
–
Table 3: Bench2Drive comparison. Camera-only world-action methods on 220 routes under the standard protocol. Emergency Brake and Give Way (%) assess responses to current events. SWIFT: mean ± sample std over three seeds of one model; bold: best mean; dashes: unreported. Injected delay: Tab. 21 .
Method
Public val.
Hidden test
Qwen2.5-VL 7B [ 44 ]
5.40
–
Qwen2.5-VL 3B [ 44 ]
5.59
–
Poutine-Base (CoVLA) [ 44 ]
7.74
–
SWIFT (zero-shot)
7.7841
–
Other end-to-end planners
RAP-DINO [ 11 , 19 ]
7.91
–
Table 4: Waymo E2E comparison. Representative methods on public validation and hidden test (RFS ↑ ). Top: zero-shot transfer. Bold: SWIFT; RL: reinforcement learning.
Model
RFS ↑
ADE 3
ADE 5
FDE 5
SWIFT (zero-shot)
7.7841
1.3353
2.8773
7.1533
SWIFT (fine-tuned)
8.2723
1.0826
2.3365
6.0575
Table 5: Waymo public-validation transfer results. Both systems use eight one-step candidates ( K=8 , NODE=1 ) at at=0 across all 478 scenes (479 decisions). Errors are in meters.
Method
RFS ↑
ADE 3 ↓
ADE 5 ↓
Vision-language-action planners
AutoVLA [ 73 ]
7.5566
1.3507
2.9580
IRL-VLA [ 22 , 27 ]
7.8900
1.2184
2.8235
Poutine-Base [ 44 ]
7.91
1.27
2.94
Poutine [ 44 , 73 ]
7.9860
1.2055
2.7419
World-action models
Table 6: Waymo hidden-test comparison. Representative methods with published precision retained. ADE is in meters. Bold: SWIFT.
Metric / category
Value
Overall RFS
8.005038
Construction
8.446138
Intersection
8.112481
Pedestrian
8.085993
Cyclist
7.9328837
Multi Lane Maneuver
7.914063
Table 7: Official Waymo hidden-test results of SWIFT (fine-tuned). RFS is the equal-weight mean over all 11 scenario categories. Displacement errors are in meters.
Method
Fine-tuned
Input
Aux.
L2 (m) ↓
Collision rate (%) ↓
1 s
2 s
3 s
Avg.
1 s
2 s
3 s
Avg.
ST-P3
Yes
Camera
Map&Box&Depth
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
UniAD
Yes
Camera
Map&Box&Motion
0.48
0.96
1.65
1.03
0.05
0.17
0.71
0.31
VAD-Base
Yes
Camera
Map&Box&Motion
0.54
1.15
1.98
1.22
0.04
0.39
1.17
0.53
GenAD
Yes
Camera
Map&Box&Motion
0.36
0.83
1.55
0.91
0.06
0.23
1.00
0.43
Epona
Yes
Camera ∗
None
0.61
1.17
1.98
1.25
0.01
0.22
0.85
0.36
Table 8: nuScenes open-loop planning comparison. Published rows follow the SimWAM compilation [ 71 , 46 ] . Camera ∗ uses the front view; Aux. denotes dense perception prediction tasks. SWIFT rows use 6,019 windows and average the 1, 2, and 3 s endpoints. Bold marks the best values at the displayed precision.
Figure 8: Freshness under delayed world context. SWIFT (fine-tuned) RFS and ADE 5 with freshness enabled or masked (top); paired RFS gain and pointwise 95% scene-bootstrap intervals (bottom). Both conditions coincide at at=0 .
at (s)
RFS on/off
ADE 5 on/off
RFS gain
95% interval
SWIFT (fine-tuned)
0.0
8.1418/8.1418
2.3582/2.3582
0.0000
[0.0000, 0.0000]
0.1
8.0972/8.0273
2.3740/2.3859
0.0699
[-0.0324, 0.1780]
0.2
8.0509/7.9004
2.4117/2.4595
0.1505
[0.0512, 0.2593]
0.3
8.0502/7.8700
2.4148/2.4990
0.1802
[0.0471, 0.3139]
0.4
8.0606/7.8376
2.4244/2.5601
0.2230
[0.0814, 0.3798]
Table 9: Paired Waymo freshness results. Both models use K=1 on all 478 scenes (479 decisions per age). Paired columns show on/off values; ADE 5 is in meters.
Category
n
RFS on
RFS off
Gain
Construction
15
8.3648
8.3668
-0.0020
Cut ins
20
8.6655
8.2505
0.4150
Cyclist
71
8.3676
8.1399
0.2277
Foreign Object Debris
78
8.0461
7.8824
0.1638
Intersections
116
7.8786
7.8275
0.0510
Multi-Lane Maneuvers
42
7.5356
7.5341
0.0014
Table 10: SWIFT (fine-tuned) category breakdown. Mean over seven nonzero ages; n denotes decisions per age, totaling 479.
Context age at (s)
Freshness on
Masked
Reliance
0.0
90.9
90.8
0.1
0.5
90.8
90.0
0.8
1.0 (chunk)
90.7
89.0
1.7
1.5 (freshness)
90.9
87.8
3.1
2.0
89.4
83.8
5.6
Table 11: NAVSIM context age with the TAEW2.1 generator. Reliance is the paired freshness-on minus masked PDMS.
Model
Pass
Fcar/Fw
ms
PDMS ↑
Masked
Reliance
DINOv3
no
0.34%
16.97
90.6
88.9
1.7
TAEW2.1
no
0.30%
17.84
90.7
89.0
1.7
TransFuser
no
0.13%
19.23
90.3
88.5
1.8
Native Wan-VAE
shared
18.00%
1418.51
90.6
89.0
1.6
Table 12: Freshness carriers at the chunk-step age. Seven-view NAVSIM, at=1.0 s, K=1 , NODE=1 , no scorer; carrier cost on H20. Masked: PDMS with Ft∣s removed at inference; Reliance: their difference.
Fresh training input views × H × W
at (s)
RFS ↑
ADE 5 (m) ↓
6 × 480 × 832
0
8.0897
2.4150
0.1–0.7
8.0444
2.4213
6 × 240 × 416
0
8.0176
2.3579
0.1–0.7
7.9613
2.4603
8 × 480 × 832
0
8.1348
2.3799
0.1–0.7
8.0795
2.4231
Table 13: Fresh-input configurations after three epochs of fine-tuning. At at=0 the carrier is bypassed; nonzero-age rows average freshness-enabled results over at=0.1,…,0.7 s. All rows cover 478 scenes and 479 decisions per age.
Span Tp ( Hb )
at=1.0 s
at=1.5 s
at=2.0 s
1.0 s (1)
89.9
89.2
88.6
2.0 s (2), SWIFT
90.7
90.9
89.4
4.0 s (4)
90.2
89.7
89.6
Table 14: Predictive span and context age on NAVSIM. PDMS without the scorer, with the TAEW2.1 carrier. Ages 1.0 and 1.5 s follow the deployment schedule; 2.0 s is outside it.
Variant
PDMS ↑
Without scorer, K=1 and NODE=1
Reference
90.7
Trained without world context Cs
87.9
Freshness carrier masked (eval. only)
89.0
Fixed value gates gℓ=1
90.1
Default 4-frame history, padded (eval. only)
89.3
Table 15: Design ablations on NAVSIM. Each row changes one factor relative to its group’s reference.
Factor
Variant
Ref.
PDMS ↑
Without scorer ( K=1 , one ODE step)
Training ages
at∈{0.5,…,2.0} s
90.7
90.6
ODE steps
NODE=2 / NODE=5
90.7
90.6 / 90.4
With scorer ( K=53 )
Candidates
vocabulary-only pool
94.2
93.5
Gaussian-only pool
94.2
93.7
Table 16: Additional NAVSIM component ablations. Each row changes one factor, except candidate pools, whose composition and count both vary.
System
Mean L2 (m)
Mean C (%)
C 3 (%)
Single-frame context
0.5974
0.5760
0.9802
SWIFT: clean history
0.5323
0.5760
0.7642
Imagine then execute
1.0717
0.7753
1.1962
Table 17: Information-flow ablation on nuScenes. Endpoint error and collision (C) are averaged over 1, 2, and 3 s; C 3 is collision at 3 s. Lower is better.
Variant
Metric
1 s
2 s
3 s
Single-frame context
L2 (m)
0.2076
0.5252
1.0594
C (%)
0.2492
0.4984
0.9802
SWIFT: clean history
L2 (m)
0.1869
0.4713
0.9388
C (%)
0.4486
0.5150
0.7642
Imagine then execute
L2 (m)
0.3933
0.9968
1.8249
C (%)
0.4486
0.6812
1.1962
Table 18: Information-flow endpoint metrics. L2 and collision C use all 6,019 validation windows.
Variant
L2 (m)
C (%)
C 3 (%)
Single-frame context
0.6829
0.6577
1.1526
SWIFT: clean history
0.6091
0.6381
0.8986
Imagine then execute
1.2317
0.8726
1.4065
Table 19: Complete-future sensitivity on 5,119 windows. L2 and C average endpoints; C 3 is collision at 3 s.
Execution
GPUs
Latency (ms) ↓
PDMS ↑
Synchronous, at=0
1
927.6
93.9
Asynchronous, at=1.0 s (chunk)
2
198.4
94.2
Asynchronous, at=1.5 s (freshness)
2
198.4
94.0
Table 20: Synchronous and asynchronous NAVSIM execution. PDMS with the scorer and H20 decision latency. Both asynchronous rows evaluate the same model.
Delay (ms)
DS ↑
SR (%) ↑
0
80.30±0.32
56.36±0.79
100
75.62±0.58
52.12±1.14
500
50.24±1.66
16.67±1.84
1000
11.32±0.50
0.91±0.45
Table 21: Bench2Drive under injected delay. Commands take effect after each delay as the scene advances; 0 ms is the standard protocol. Mean ± sample std over three seeds.
Delay (ms)
Seed
DS
SR (%)
Successes
0
1
80.58
56.82
125
0
2
79.95
55.45
122
0
3
80.38
56.82
125
100
1
74.98
50.91
112
100
2
76.13
53.18
117
100
3
75.74
52.27
115
Table 22: Bench2Drive results for each evaluation seed. Rows vary injected delay and seed. Columns report DS, SR, and successful routes out of 220.
Method
ms
GPU
Operating point
NAVSIM comparators
DriveLaW [ 59 ]
710.0
H20
5-step planning
SV-WAM [ 55 ]
341.6
H20
2-step online total
PerceptDrive [ 39 ]
68.0
MI308X
25-step end-to-end
4D-WAM [ 12 ]
790.0
4090
10-step default plan
UNIVERSE [ 36 ]
376.0
n/s
trajectory inference
Table 23: Comparator timing sources. A dagger denotes our official-code H20 rerun; n/s denotes an unspecified device.
Figure 9: Autoregressive imagination and planning. From 29 observed frames ending at t=0 , the pretrained model generates about 8 s of front-view futures without future pose or action conditioning. Rows show U-turn, following, pull-over, and right-turn maneuvers. The right panels show ego plans obtained from the generated images over two successive 4 s windows.
Method
Domain (views)
Reused slow context
Cache between updates
Current observations
AHA-WAM [ 7 ]
Manip. (1–3)
Planner K/V
Adapted per chunk
Routing queries from current view
GlanceWAM [ 56 ]
Manip. (2–3)
Generated lookahead latent
Held
Video DiT on current frame
DualWAM [ 72 ]
Manip. (global, wrist)
Partially denoised plan
Refined per window
Wrist-view WAM
ZimaBlue [ 58 ]
Manip. (3)
K/V from video-action rollout
Reused unchanged
Observation latents in fast DiT
SWIFT
Driving (6–7)
K/V of clean history
Read-only
Lightweight encoder, own K/V bank
Table 24: Asynchronous world-action designs. Entries follow each paper’s description. Views counts camera inputs; the last column gives the path by which current observations reach action steps between slow updates.
Direct-policy World Action Models (WAM) have demonstrated the benefits of decoupling action generation from explicit future visual prediction for efficient inference in autonomous driving and embodied control. However, they still require tens of denoising steps to achieve competitive action quality, incurring substantial latency and hindering real-time deployment. Simply reducing the number of denoising steps will collapse the nonlinear multi-step denoising trajectory into coarse updates, substantially degrading action quality. To reduce the denoising steps while enabling real-time action generation, we present an effective world action model with one-step action generation, namely OSWAM, for autonomous driving and embodied control. We argue that it is the absence of structured action priors in random noise initialization that forces direct-policy WAMs to recover coherent action sequences through iterative denoising. Hence, we construct imagination tokens to initialize the diffusion process, providing physical priors and action-conditioned representations. We further introduce privileged information distillation with future conditioning to enhance future-aware action generation. As a result, OSWAM can efficiently and effectively generate high quality actions in just one diffusion step. Our experiments demonstrate that OSWAM achieves comparable or even better results in both autonomous driving and embodied control than direct-policy WAM and Joint-WAM that require tens of steps. It obtains approximately 3× speedup over direct-policy WAM baselines and supports real-time compiled inference with latency as low as 40ms on an NVIDIA RTX 5090 GPU. The project page is available at https://gwenzhang.github.io/OSWAM/
World action models~(WAMs) have shown great promise for autonomous driving and urban navigation. Built upon Vision-Language-Action models or video generation models, existing approaches suffer key limitations: (1) High inference latency due to future observation prediction at test time, and (2) tightly coupled video and action modeling leading to representational mismatch and degraded generalization. To address both issues, we propose Metis, an end-to-end WAM framework that decouples video generation and action prediction. Specifically, Metis employs a Mixture-of-Transformers architecture with dedicated experts for video generation and action prediction, preserving the intrinsic distributional properties of each task. To enhance efficiency, we introduce an asymmetric attention mask that enables joint training of both experts while allowing the action model to bypass explicit video generation during inference. This design ensures training-inference consistency and significantly reduces computational costs without compromising planning performance. Extensive experiments demonstrate state-of-the-art performance on the NAVSIM navhard and navtest benchmarks and the CityWalker navigation benchmark, validating both the generalizability and efficiency across diverse tasks. Real-robot deployments further confirm the practical feasibility of our approach.
Jingyu Li, Zhe Liu, Dongnan Hu +10
Fudan University · Shanghai Innovation Institute · The University of Hong Kong +5
A plausible scene evolution depends on the maneuver being considered, while a good maneuver depends on how the scene may evolve. Existing World Action Models (WAMs) largely miss this reciprocity, treating world prediction and action generation as either isolated parallel branches or rigid predict-then-plan pipelines. We formalize this perspective as World-Action Interactive Models (WAIMs), and instantiate it in autonomous driving with DAWN (Denoising Actions and World iNteractive model), a simple yet strong latent generative baseline. DAWN operates in a compact semantic latent space and couples a World Predictor with a World-Conditioned Action Denoiser: the predicted world hypothesis conditions action denoising, while the denoised action hypothesis is fed back to update the world prediction, so that both are recursively refined during inference. Rather than eliminating test-time world evolution altogether or rolling out the full future in pixel space, DAWN performs a short explicit latent rollout that is sufficient to support long-horizon trajectory generation in complex interactive scenes. Experiments show that DAWN achieves strong planning performance and favorable safety-related results across multiple autonomous driving benchmarks. More broadly, our results suggest that interactive world-action generation is a principled path toward truly actionable world models.
Hongbo Lu, Liang Yao, Chenghao He +6
1COWARobot Co. Ltd · 2Shanghai Jiao Tong University · 3Hohai University