World models are increasingly used in autonomous driving to predict how a scene evolves under the ego vehicle's actions, supporting counterfactual reasoning and model-based planning. However, existing evaluations focus on prediction fidelity and planning performance, and rarely verify that the predictions depend on the conditioning action. In this paper, we study action dependence in action-conditioned semantic forecasting, which predicts future semantic segmentation from past segmentations and the ego action. We compare a forecaster with feature-wise and input-level action conditioning against an identical model without action input, and measure action dependence with test-time action interventions scored by standard segmentation metrics. The evaluation is validated in a synthetic environment with known action-dependent dynamics and applied to Cityscapes and BDD100K, both within each dataset and across the two vehicle platforms. Experiments show that the executed action improves mid-term mIoU by at most 2.5 points on Cityscapes and 0.6 points on BDD100K, whereas a wrong action reduces it by up to 23.2 and 7.0 points, below the model without action input. On real data, feature-wise conditioning depends 1.6 times more on the action than input-level conditioning at similar accuracy, whereas both depend on it equally in the synthetic environment, where the history does not determine the action. These results show that an accuracy gain indicates that a driving world model uses its action, but not how strongly its forecasts depend on it. Code is available at https://github.com/binayksingh/does-action-matter.
Figures & tables
Figure 1: Overview of action-conditioned semantic forecasting. A U-Net maps the four context segmentations to logits, which are added to the scaled last context map and normalised by a softmax; the prediction is fed back for the next step. The ego action enters through FiLM, input concatenation, or not at all (bottom left; the no-action model is the U-Net alone). At test time, the future action is replaced to measure action dependence (bottom right). The thumbnails show a Cityscapes validation scene in which the vehicle turns left ( vˉ=4.6 m/s, ωˉ=0.36 rad/s) and the mid-term predictions of a trained FiLM model; the shuffled action ( 6.5 m/s, 0.00 rad/s) and the training-mean action ( 6.3 m/s, 0.00 rad/s) both correspond to straight driving.
Cityscapes (gtFine)
BDD100K (pseudo-labels)
Short term
Mid term
Short term
Mid term
Method
mIoU ↑
MO ↑
mIoU ↑
MO ↑
mIoU ↑
MO ↑
mIoU ↑
MO ↑
Copy-last
55.77
52.51
40.82
32.62
49.06
38.33
38.20
26.37
Flow warping
66.17
64.82
46.05
39.79
52.86
42.43
39.80
28.50
S2S-dil [ 12 ] †
59.4
55.3
47.8
40.8
–
–
–
–
Bayes-WD-SL [ 1 ] †
64.8
–
45.9
–
–
–
–
–
Table 1: Semantic forecasting results with the executed action (mIoU and MO-mIoU in %; mean ± std over 3 seeds). Models are trained and evaluated on the same dataset. † Published results use weaker single-frame segmenters than ours and are given for reference only. Best value per column among methods run in our pipeline in bold.
Figure 2: Per-class IoU change of FiLM over the no-action model on Cityscapes at mid term (executed action, mean over three seeds).
Figure 3: Change in mid-term mIoU relative to the no-action model when the future action is replaced at test time. Models are trained and evaluated on the same dataset; mean ± std over three seeds.
True
Change under intervention
Train / eval
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Cityscapes
No action
57.02 ± 0.15
0.00
0.00
0.00
0.00
Concat
59.14 ± 0.18
− 14.78
− 7.09
− 11.92
− 9.94
FiLM
59.47 ± 0.14
− 23.17
− 8.46
− 17.03
− 17.08
BDD100K
No action
45.04 ± 0.07
0.00
0.00
0.00
0.00
Concat
45.15 ± 0.04
− 4.50
− 0.19
− 2.64
− 4.49
Table 2: Mid-term mIoU (%) with the executed action and its change when the future action is replaced (shuffled: another sequence’s action; zero yaw; stop; training-set mean). Changes are averaged over seeds; their standard deviation is at most 2.47.
True
Change under intervention
Train / eval
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Cityscapes
No action
70.16 ± 0.08
0.00
0.00
0.00
0.00
Concat
71.02 ± 0.06
− 6.99
− 3.53
− 5.22
− 3.97
FiLM
71.27 ± 0.06
− 13.20
− 5.28
− 9.19
− 8.21
BDD100K
No action
54.92 ± 0.08
0.00
0.00
0.00
0.00
Concat
54.97 ± 0.07
− 1.27
− 0.08
− 1.05
− 0.88
Table 3: Short-term mIoU (%) with the executed action and its change under the interventions of Tab. 2 , averaged over seeds.
Eval: Cityscapes
Eval: BDD100K
Train
Model
True
Shuffled
True
Shuffled
Cityscapes
No action
57.02 ± 0.15
0.00
40.59 ± 0.18
0.00
Concat
59.14 ± 0.18
− 14.78
42.11 ± 0.18
− 7.94
FiLM
59.47 ± 0.14
− 23.17
41.36 ± 0.09
− 13.42
BDD100K
No action
50.36 ± 0.17
0.00
45.04 ± 0.07
0.00
Concat
50.40 ± 0.38
− 4.61
45.15 ± 0.04
− 4.50
Table 4: Transfer between platforms: mid-term mIoU (%) with the executed action and its change under the shuffled action, for every training and evaluation dataset.
True
Change under intervention
Model
mIoU ↑
Shuffled
Zero yaw
Stop
Mean
Copy-last
55.63
–
–
–
–
Flow warping
59.40
–
–
–
–
No action
59.99
0.00
0.00
0.00
0.00
Concat
86.44
− 30.02
− 19.42
− 30.62
− 24.75
FiLM
87.95
− 31.57
− 20.67
− 32.02
− 26.14
Table 5: Synthetic validation environment: mid-term mIoU (%) with the executed action and its change under each intervention (one run per model). Baselines do not take an action.
Method
Params
Δ Params
GFLOPs
Latency (ms) ↓
mIoU ↑
Δshuf
(M) ↓
(M)
↓
1 / 3 steps
mid
mid
No action
12.55
–
200.2
19.8 / 58.6
57.02
0.00
Concat
12.56
+0.01
201.7
20.0 / 59.1
59.14
− 14.78
FiLM
13.07
+0.51
200.2
20.5 / 60.7
59.47
− 23.17
Table 6: Model size and inference cost at 256×512 (fp32, batch 1, one NVIDIA V100-PCIE-32GB). Δ Params is relative to the no-action model; GFLOPs count convolutions and matrix products of one forecasting step. mIoU and Δshuf (change under the shuffled action) are mid-term values on Cityscapes validation.