World models are increasingly used in autonomous driving to predict how a scene evolves under the ego vehicle's actions, supporting counterfactual reasoning and model-based planning. However, existing evaluations focus on prediction fidelity and planning performance, and rarely verify that the predictions depend on the conditioning action. In this paper, we study action dependence in action-conditioned semantic forecasting, which predicts future semantic segmentation from past segmentations and the ego action. We compare a forecaster with feature-wise and input-level action conditioning against an identical model without action input, and measure action dependence with test-time action interventions scored by standard segmentation metrics. The evaluation is validated in a synthetic environment with known action-dependent dynamics and applied to Cityscapes and BDD100K, both within each dataset and across the two vehicle platforms. Experiments show that the executed action improves mid-term mIoU by at most 2.5 points on Cityscapes and 0.6 points on BDD100K, whereas a wrong action reduces it by up to 23.2 and 7.0 points, below the model without action input. On real data, feature-wise conditioning depends 1.6 times more on the action than input-level conditioning at similar accuracy, whereas both depend on it equally in the synthetic environment, where the history does not determine the action. These results show that an accuracy gain indicates that a driving world model uses its action, but not how strongly its forecasts depend on it. Code is available at https://github.com/binayksingh/does-action-matter.
Figures & tables
Figure 1: Overview of action-conditioned semantic forecasting. A U-Net maps the four context segmentations to logits, which are added to the scaled last context map and normalised by a softmax; the prediction is fed back for the next step. The ego action enters through FiLM, input concatenation, or not at all (bottom left; the no-action model is the U-Net alone). At test time, the future action is replaced to measure action dependence (bottom right). The thumbnails show a Cityscapes validation scene in which the vehicle turns left ( vˉ=4.6 m/s, ωˉ=0.36 rad/s) and the mid-term predictions of a trained FiLM model; the shuffled action ( 6.5 m/s, 0.00 rad/s) and the training-mean action ( 6.3 m/s, 0.00 rad/s) both correspond to straight driving.
Cityscapes (gtFine)
BDD100K (pseudo-labels)
Short term
Mid term
Short term
Mid term
Method
mIoU ↑
MO ↑
mIoU ↑
MO ↑
mIoU ↑
MO ↑
mIoU ↑
MO ↑
Copy-last
55.77
52.51
40.82
32.62
49.06
38.33
38.20
26.37
Flow warping
66.17
64.82
46.05
39.79
52.86
42.43
39.80
28.50
S2S-dil [ 12 ] †
59.4
55.3
47.8
40.8
–
–
–
–
Bayes-WD-SL [ 1 ] †
64.8
–
45.9
–
–
–
–
–
Table 1: Semantic forecasting results with the executed action (mIoU and MO-mIoU in %; mean ± std over 3 seeds). Models are trained and evaluated on the same dataset. † Published results use weaker single-frame segmenters than ours and are given for reference only. Best value per column among methods run in our pipeline in bold.
Figure 2: Per-class IoU change of FiLM over the no-action model on Cityscapes at mid term (executed action, mean over three seeds).
Figure 3: Change in mid-term mIoU relative to the no-action model when the future action is replaced at test time. Models are trained and evaluated on the same dataset; mean ± std over three seeds.
True
Change under intervention
Train / eval
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Cityscapes
No action
57.02 ± 0.15
0.00
0.00
0.00
0.00
Concat
59.14 ± 0.18
− 14.78
− 7.09
− 11.92
− 9.94
FiLM
59.47 ± 0.14
− 23.17
− 8.46
− 17.03
− 17.08
BDD100K
No action
45.04 ± 0.07
0.00
0.00
0.00
0.00
Concat
45.15 ± 0.04
− 4.50
− 0.19
− 2.64
− 4.49
Table 2: Mid-term mIoU (%) with the executed action and its change when the future action is replaced (shuffled: another sequence’s action; zero yaw; stop; training-set mean). Changes are averaged over seeds; their standard deviation is at most 2.47.
True
Change under intervention
Train / eval
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Cityscapes
No action
70.16 ± 0.08
0.00
0.00
0.00
0.00
Concat
71.02 ± 0.06
− 6.99
− 3.53
− 5.22
− 3.97
FiLM
71.27 ± 0.06
− 13.20
− 5.28
− 9.19
− 8.21
BDD100K
No action
54.92 ± 0.08
0.00
0.00
0.00
0.00
Concat
54.97 ± 0.07
− 1.27
− 0.08
− 1.05
− 0.88
Table 3: Short-term mIoU (%) with the executed action and its change under the interventions of Tab. 2 , averaged over seeds.
Eval: Cityscapes
Eval: BDD100K
Train
Model
True
Shuffled
True
Shuffled
Cityscapes
No action
57.02 ± 0.15
0.00
40.59 ± 0.18
0.00
Concat
59.14 ± 0.18
− 14.78
42.11 ± 0.18
− 7.94
FiLM
59.47 ± 0.14
− 23.17
41.36 ± 0.09
− 13.42
BDD100K
No action
50.36 ± 0.17
0.00
45.04 ± 0.07
0.00
Concat
50.40 ± 0.38
− 4.61
45.15 ± 0.04
− 4.50
Table 4: Transfer between platforms: mid-term mIoU (%) with the executed action and its change under the shuffled action, for every training and evaluation dataset.
True
Change under intervention
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Copy-last
55.63
–
–
–
–
Flow warping
59.40
–
–
–
–
No action
59.99
0.00
0.00
0.00
0.00
Concat
86.44
− 30.02
− 19.42
− 30.62
− 24.75
FiLM
87.95
− 31.57
− 20.67
− 32.02
− 26.14
Table 5: Synthetic validation environment: mid-term mIoU (%) with the executed action and its change under each intervention (one run per model). Baselines do not take an action.
Method
Params
Δ Params
GFLOPs
Latency (ms) ↓
mIoU ↑
Δshuf
(M) ↓
(M)
↓
1 / 3 steps
mid
mid
No action
12.55
–
200.2
19.8 / 58.6
57.02
0.00
Concat
12.56
+0.01
201.7
20.0 / 59.1
59.14
− 14.78
FiLM
13.07
+0.51
200.2
20.5 / 60.7
59.47
− 23.17
Table 6: Model size and inference cost at 256×512 (fp32, batch 1, one NVIDIA V100-PCIE-32GB). Δ Params is relative to the no-action model; GFLOPs count convolutions and matrix products of one forecasting step. mIoU and Δshuf (change under the shuffled action) are mid-term values on Cityscapes validation.
A plausible scene evolution depends on the maneuver being considered, while a good maneuver depends on how the scene may evolve. Existing World Action Models (WAMs) largely miss this reciprocity, treating world prediction and action generation as either isolated parallel branches or rigid predict-then-plan pipelines. We formalize this perspective as World-Action Interactive Models (WAIMs), and instantiate it in autonomous driving with DAWN (Denoising Actions and World iNteractive model), a simple yet strong latent generative baseline. DAWN operates in a compact semantic latent space and couples a World Predictor with a World-Conditioned Action Denoiser: the predicted world hypothesis conditions action denoising, while the denoised action hypothesis is fed back to update the world prediction, so that both are recursively refined during inference. Rather than eliminating test-time world evolution altogether or rolling out the full future in pixel space, DAWN performs a short explicit latent rollout that is sufficient to support long-horizon trajectory generation in complex interactive scenes. Experiments show that DAWN achieves strong planning performance and favorable safety-related results across multiple autonomous driving benchmarks. More broadly, our results suggest that interactive world-action generation is a principled path toward truly actionable world models.
Hongbo Lu, Liang Yao, Chenghao He +6
1COWARobot Co. Ltd · 2Shanghai Jiao Tong University · 3Hohai University
VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action--semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry--language--action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, with ablation studies examining the contributions of semantic and action conditioning. Compared with baseline, our method reduces future-depth RMSE by 13.7% and improves δ1 by 4.11 percentage points on NAVSIM motion subset. Controlled action-input interventions further show that the model uses the supplied action information. These results support semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.
Zhaoyang Liu, Zhigui Lin, Kun Jiang +4
Tsinghua University · Nanyang Technological University
Driving world models are often interpreted as counterfactual simulators for observed driving episodes: given a factual driving log, they are asked what would have happened under an alternative ego action. In this paper, we identify a fundamental mismatch between this goal and direct action-conditioned prediction. The direct prediction uses the shared history and the alternative action but not the factual continuation observed after that history. It can therefore generate a plausible future without preserving what actually happened in this episode. We formalize this gap using the causal recipe of abduction, action, and prediction and study it in a setting with a short time horizon, where the alternative ego action does not alter how surrounding agents evolve. To make the gap measurable, we construct a controlled simulation benchmark with factual outcomes and matched counterfactual outcomes. Across two representative world models, direct predictions fail to match the counterfactual ground truth, supporting our analysis. As a constructive check of this analysis, we introduce a deliberately simple, training-free pipeline that moves observed evidence into the counterfactual view and lets the frozen model complete what remains unknown. Even this simple construction raises the overall recovered fraction substantially and reduces perceptual distance to the matched counterfactual on both models. We hope this work draws attention to this gap and motivates better counterfactual prediction methods for driving world models.
Jiaru Zhang, Can Cui, Yi Xu +3
Purdue University · Bosch Center for Artificial Intelligence