Traffic elements such as traffic lights and road signs play a fundamental role in human driving decisions and should naturally influence end-to-end driving performance. However, existing end-to-end driving research predominantly focuses on dynamic road participants (e.g., vehicles and pedestrians), while the role of traffic elements remains largely unexplored. The community still lacks a systematic study quantifying their impact, largely because public datasets rarely provide structured traffic-element annotations and modern driving systems vary widely in architecture and training paradigm. In this work, we present the first systematic investigation of traffic element awareness for end-to-end autonomous driving. We construct a unified research infrastructure by augmenting multiple public driving datasets with comprehensive traffic-element annotations. To support diverse model families, we adopt a minimal and universal integration design that incorporates traffic-element signals into existing pipelines in a plug-and-play manner with negligible architectural modification. We evaluate this design across modern paradigms, including perception-prediction-planning pipelines, vision-language-action models (VLA), regression-based planners, diffusion-based policies, and trajectory-scoring frameworks, on nuScenes, NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Across all paradigms and datasets, this simple integration consistently improves driving performance, demonstrating that traffic element awareness provides a robust and generalizable signal for end-to-end driving systems. Notably, on the challenging NAVSIM-v2 benchmark, our approach significantly improves state-of-the-art architectures and data pipelines, establishing a new state of the art.
Figures & tables
Figure 2: Definitions of driving topologies. (a) LCTE topology mapping traffic elements ( light / sign ) to the ego-lane in FV. (b) BEV representation of centerlines , ego-related centerlines , and LCLC topology .
Figure 3: (a) Pipeline for 3D traffic element extraction. We first perform 2D traffic-element detection and monocular depth estimation on the FV image, then combine them and LiDAR geometry to localize each traffic element as a 3D center point ( ∙ ). (b) Overview of our method. Multi-view images are encoded into vision queries and enhanced by an auxiliary 3D traffic element detection task. In parallel, predicted topological adjacency matrices are formatted into structured language and processed by a frozen language encoder. The resulting topology queries are concatenated with the BEV queries to guide the planning decoder in generating the ego-vehicle trajectory.
Method
Auxiliary Task
L2 (m) ↓
Collision Rate (%) ↓
FPS ↑
1s
2s
3s
Avg.
1s
2s
3s
Avg.
NMP ⋄ [ 95 ]
Det & Motion
0.53
1.25
2.67
1.48
0.04
0.12
0.87
0.34
-
FF ⋄ [ 30 ]
FreeSpace
0.55
1.20
2.54
1.43
0.06
0.17
1.07
0.43
-
EO ⋄ [ 41 ]
FreeSpace
0.67
1.36
2.78
1.60
0.04
0.09
0.88
0.33
-
ST-P3 [ 31 ]
Det & Map & Depth
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
1.6
UniAD [ 32 ]
Det&Track&Map&Motion&Occ
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
1.8
Table 1: Comparison of methods on the nuScenes dataset . ⋄ : Lidar-based methods. †: The ego status is utilized. FPS is measured on an NVIDIA A100 GPU.
Method
Backbone
Venue
NC ↑
DAC ↑
TTC ↑
Comf. ↑
EP ↑
PDMS ↑
PDM - Closed [ 14 ]
–
PMLR’23
94.6
99.8
89.9
86.9
99.9
89.1
Human driver [ 15 ]
–
NeurIPS’24
100
100
100
99.9
87.5
94.8
Ego - stat. MLP [ 15 ]
–
NeurIPS’24
93.0
77.3
83.6
100
62.8
65.6
UniAD [ 32 ]
ResNet34
CVPR’23
97.8
91.9
92.9
100
78.8
83.4
VAD-v2 [ 9 ]
ResNet34
ICLR’26
98.1
94.8
94.3
100
80.6
86.2
ReCogDrive [ 49 ]
InternViT
ICLR’26
97.9
97.3
94.9
100
87.3
90.8
Table 2: Camera-only methods on the NAVSIM-v1 benchmark ( navtest ) .
Stage 1
Stage 2
Method
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS ↑
ZTRS† [ 51 ]
98.9
97.6
100
100
66.7
98.9
96.2
96.7
44.0
91.1
90.4
95.8
99.0
63.6
89.8
60.4
97.6
66.1
45.5
DiffVLA† [ 38 ]
95.7
99.2
100
100
85.9
96.4
97.1
95
84.2
81.2
88.8
94.6
99.0
86.0
76.4
59.8
98.6
80.4
45.0
GTRS-Dense† [ 52 ]
98.9
94.9
99.1
100
76.1
98.4
93.8
94.9
37.8
89.9
90.5
94.1
99.3
77.6
88.5
56.0
92.0
30.2
41.9
GuideFlow† [ 57 ]
97.8
97.1
100
100
81.4
98.5
91.4
92.8
34.2
87.3
92.3
98.0
96.9
75.8
85.5
59.3
95.4
53.5
46.7
Regression-based Planner
Table 3: Comparison to existing methods on the NAVSIM-v2 navhard-two-stage benchmark using EPDMS. †: Metrics reported prior to the official benchmark bug fix.
Method
Open-loop Metric
Closed-loop Metric
Latency
Avg. L2 ↓
Driving Score ↑
Success Rate(%) ↑
Efficiency ↑
Comfortness ↑
TCP ∗ [ 91 ]
1.70
40.70
15.00
54.26
47.80
86ms
TCP-ctrl ∗
-
30.47
7.27
55.97
51.51
86ms
TCP-traj ∗
1.70
59.90
30.00
76.54
18.08
86ms
TCP-traj w/o distillation
1.96
49.30
20.45
78.78
22.96
86ms
ThinkTwice ∗ [ 35 ]
0.95
62.44
31.23
69.33
16.22
698ms
Table 4: Closed-Loop Planning Performance in Bench2Drive . Avg. L2 is averaged over the predictions in 2 seconds. ∗ : denotes expert feature distillation. All latency is measured by the averaged inference step-time on CARLA [ 19 ] evaluation in A6000.
Figure 4: Qualitative comparison on NAVSIM-v2 . Left: front-view image with detected traffic elements. Right: predicted trajectories of LTF, LTF+SimScale, and Ours.
Stage 1
Stage 2
Method
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
NC
DAC
DDC
TLC
EP
TTC
LK
HC
EC
EPDMS ↑
LTF [ 13 ]
96.2
79.5
99.1
99.5
84.1
95.1
94.2
97.5
79.1
77.7
70.2
84.2
98.0
85.1
75.6
45.4
95.7
75.9
25.1
LTF+TE(2D)
96.4
82.0
99.2
99.6
84.0
95.8
93.1
97.8
79.1
81.6
68.4
85.5
98.6
85.4
78.4
46.9
96.2
76.5
28.1 +3.0
LTF+depth(FV)
96.9
81.8
99.0
99.6
84.2
95.8
92.7
97.8
78.2
81.3
70.1
84.3
98.6
85.7
78.4
45.9
96.3
76.5
27.7 +2.6
LTF+TE(Lidar)
97.2
80.9
99.0
99.5
83.9
96.0
92.9
97.8
79.5
81.1
70.2
85.4
98.6
85.0
79.1
45.9
96.0
77.2
27.9 +2.8
LTF+TE [ 55 ]
96.7
81.8
99.1
99.3
84.1
95.6
92.4
97.6
78.2
81.2
69.4
84.9
98.6
85.6
78.3
46.6
96.2
76.0
27.8 +2.7
Table 5: Ablation study of traffic element (TE) representations on the NAVSIM-v2 using LTF (Latent Transfuser). TE(2D) denotes 2D traffic elements in the front-view ( FV ) image; depth(FV) uses the full FV depth; LiDAR uses the NAVSIM-v2 point-cloud input; TL denotes traffic lights (green/yellow/red light) only.
ID
TE Head
Loss
Pooling
Interaction
EPDMS ↑
w/ BEV
Indep.
Cross-Entropy
Focal
Average
Max
C.A.
Concat
0
25.1
1
✓
✓
✓
24.3 -0.8
2
✓
✓
✓
26.1 +1.0
3
✓
✓
✓
25.4 +0.3
4
✓
✓
✓
✓
24.9 -0.2
Table 6: Ablation study of traffic element integration on the NAVSIM-v2 using LTF (Latent Transfuser). We systematically investigate the design choices for the prediction head, loss function, spatial pooling strategy, and planning interaction method.
Figure 5: Robustness analysis on NAVSIM-v2 with LTF. We corrupt the predicted TE representation at inference time by adding TE depth noise, randomly dropping TE detections, or injecting false-positive TE predictions. The performance decreases smoothly under stronger corruption while remaining above the LTF baseline, indicating robustness to moderate upstream TE perception noise.
ID
Topology
Encoder
Source
Scope
L2 (m) ↓
Collision Rate (%) ↓
LCTE
LCLC
BERT
GCN
GT [ 81 ]
Pred [ 88 ]
Global
Ego
1s
2s
3s
Avg.
1s
2s
3s
Avg.
0
0.41
0.70
1.05
0.72
0.07
0.17
0.41
0.22
1
✓
✓
✓
✓
0.34
0.60
0.95
0.63
0.06
0.14
0.27
0.16
2
✓
✓
✓
✓
0.34
0.59
0.93
0.62
0.06
0.16
0.33
0.18
3
✓
✓
✓
✓
✓
0.33
0.58
0.94
0.62
0.04
0.16
0.28
0.16
4
✓
✓
✓
✓
✓
0.34
0.60
0.93
0.62
0.07
0.18
0.43
0.23
Table 7: Ablation study of topology information integration on the nuScenes dataset. We ablate the topology types, encoding methods, topology source and interaction scope upon the VAD baseline.
Figure 6: Stepwise performance improvements of our traffic element detector on the OpenLane-V2 validation set, evaluated by the DETt metric.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Frames
TE Inst.
mAP 50
mAP 50:95
Recall
Precision
NAVSIM manual subset
100
1,126
0.908
0.725
0.915
0.918
Appendix
Table A.1 : Cross-dataset validation of the OpenLane-V2-trained traffic-element detector on a manually annotated NAVSIM subset. The subset contains 100 frames from 20 scenes and 1,126 traffic-element instances.
Figure A.1 : Failure case of depth estimation. The left image shows the front-view camera image, where the green traffic light is highlighted by the red bounding box. The right image shows the corresponding predicted depth map.
Figure A.2 : Illustration of topology extraction. Blue lines denote centerlines, while green lines indicate the topology relations between green traffic lights and the corresponding centerlines.
Subset
Depth
Planning
AbsRel ↓
RMSE log↓
δ1↑
L2@1s ↓
L2@2s ↓
L2@3s ↓
Col@1s ↓
Col@2s ↓
Col@3s ↓
All
0.095
0.201
0.920
0.36
0.61
0.92
0.09
0.14
0.28
Night
0.160
0.279
0.793
0.49
0.85
1.33
0.00
0.15
0.55
Day
0.088
0.193
0.934
0.34
0.58
0.88
0.10
0.13
0.25
Appendix
Table A.2 : Day/night breakdown on nuScenes with the VAD-based traffic-element model. Nighttime scenes exhibit worse depth quality, which correlates with degraded planning accuracy.
Figure A.5 : Overview of the VAD-based architecture with the proposed traffic element and topology branches.
Figure A.6 : Overview of the LTF-based architecture with the proposed traffic element prediction branch.
Figure A.7 : Additional qualitative comparison between VAD and Ours on five traffic-element-rich scenes (a)-(e) from the nuScenes dataset. For each case, the left shows the multi-view camera images, and the right shows the corresponding BEV planning visualizations of VAD and Ours. From top to bottom, the overlaid numbers denote the per-frame L2 error at 1s/2s/3s, collision rate at 1s/2s/3s, and the driving command , respectively.
Figure A.8 : Temporal qualitative comparison on nuScenes . (a) Multi-view images at three consecutive time steps t0 , t1 , t2 , where the forward traffic light changes from red to green . (b) BEV planning trajectories predicted by VAD over the same three frames. (c) Corresponding BEV planning trajectories predicted by Ours.
Figure A.9 : Temporal qualitative comparison on NAVSIM . The left shows front-view images at three consecutive time steps, where the green boxes indicate the detected traffic elements. The right compares the corresponding planning trajectories of LTF, LTF+SimScale, and Ours.
Figure A.10 : Additional qualitative comparison on synthetic NAVSIM-v2 scenes . The left shows front-view images with detected traffic elements (green boxes), and the right compares the corresponding planning trajectories. Cases (a)–(c) compare DiffusionDrive, DiffusionDrive+SimScale, and Ours, while (d) compares DrivoR, DrivoR+SimScale, and Ours.
Method
L2@1s ↓
L2@2s ↓
L2@3s ↓
Col@1s ↓
Col@2s ↓
Col@3s ↓
VAD
0.49
0.85
1.25
0.00
0.00
0.00
Ours
0.37
0.63
0.98
0.00
0.00
0.00
Appendix
Table A.3 : Yellow-light subset on nuScenes. We report open-loop planning metrics on samples containing yellow-light traffic elements.
Method
L2@1s ↓
L2@2s ↓
L2@3s ↓
Col@1s ↓
Col@2s ↓
Col@3s ↓
VAD
0.41
0.70
1.05
0.07
0.17
0.41
Rule w/o Topo
0.57
0.97
1.43
0.02
0.38
0.76
Rule w/ Topo
0.42
0.71
1.07
0.02
0.24
0.35
Ours
0.34
0.56
0.92
0.04
0.21
0.26
Appendix
Table A.4 : Rule-based traffic-element baselines on nuScenes with VAD. “Rule w/o Topo” stops for any red light within 30 meters, while “Rule w/ Topo” stops only for ego-lane-associated red lights.
Encoder
L2@1s ↓
L2@2s ↓
L2@3s ↓
Col@1s ↓
Col@2s ↓
Col@3s ↓
FPS ↑
BERT
0.34
0.59
0.92
0.04
0.21
0.26
5.4
DistilBERT
0.35
0.61
0.94
0.05
0.15
0.26
5.6
Graph Transformer
0.39
0.65
0.99
0.11
0.19
0.34
5.5
Appendix
Table A.5 : Topology encoder variants on nuScenes with the VAD backbone. DistilBERT achieves performance comparable to BERT at slightly lower cost, while Graph Transformer is less effective.
Figure A.11 : Illustration of a limitation of the standard nuScenes metrics . Using the ground-truth HD map , we additionally count the number of collisions between the predicted ego-vehicle boxes and the lane/map boundaries (shown in the upper-left corner of each panel). (a) VAD, (b) Orion, and (c) Ours. Even in these simple scenes, Ours yields zero boundary collisions, while the other methods still produce boundary violations, showing that L2 and collision rate alone may not fully capture map-consistent planning quality.
Method
Boundary Collision Rate (%) ↓
1s
2s
3s
Avg.
VAD
1.09
1.81
2.85
1.92
Ours
1.01
1.39
1.93
1.44
Appendix
Table A.6: Boundary collision rate on nuScenes. Lower is better.
End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .
Christian Löwens, Thorben Funke, Alexandru Paul Condurache
Bosch Research · University of Lübeck · Automated Driving, Bosch
This letter presents PrismAD, a decoupled end-to-end autonomous driving framework based on a Semantic Mixture-of-Planners. Existing planners usually aggregate heterogeneous scene tokens into a coupled representation space, forcing a single planning branch to jointly model agent interaction, road geometry, and driving intention. Such coupling may weaken factor-specific reasoning and obscure the contribution of different planning cues. To address this limitation, PrismAD partitions scene tokens into interaction, geometry, and intent groups, and assigns them to independent planning experts with the same architecture but separate parameters. Each expert learns a specialized motion-planning representation, while a semantics-aware router adaptively aggregates expert predictions with separate routing weights for motion prediction and ego planning. Sparse top-K activation with noisy gating is further introduced to improve routing robustness and reduce unnecessary expert computation. Extensive experiments on the nuScenes open-loop dataset and NeuroNCAP closed-loop benchmark demonstrate that PrismAD exhibits competitive performance. Our code will be released soon.
Kang Ding, Zhigui Lin, Hongsong Wang +5
School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · School of Cyberspace Security, Southeast University, Nanjing 210096, China. +3
End-to-end autonomous driving, which bypasses traditional modular pipelines by directly predicting future trajectories from sensor inputs, has recently achieved substantial progress. However, existing methods often overlook the causal inter-dependencies in ego-vehicle planning, ignoring the reciprocal relations between the ego vehicle and surrounding agents. This causal oversight leads to inconsistent and unreliable trajectory predictions, especially in interaction-critical scenarios where ego decisions and neighboring agent behaviors must be reasoned about jointly. To address this limitation, we propose CaAD, a Causality-aware end-to-end Autonomous Driving framework that captures these dependencies within a shared latent scene representation. First, we propose an ego-centric joint-causal modeling module that builds on the marginal prediction branch, and learns causal dependencies between the ego vehicle and interaction-relevant agents. Second, we employ a causality-aware policy alignment stage implemented with joint-mode embeddings to align the stochastic ego policy with planning-oriented closed-loop feedback computed from surrounding traffic and map context. On the Bench2Drive and NAVSIM benchmarks, CaAD demonstrates strong closed-loop planning performance, achieving a Driving Score of 87.53 and Success Rate of 71.81 on Bench2Drive, and a PDMS of 91.1 on NAVSIM. The project page is available at https://moonseokha.github.io/CaAD/.