Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving

Zongzheng Zhang; Jijun Wang; Saining Zhang; Shuo Wang; Yiru Wang; Hai Yang; Yang Chen; Yuwen Heng

cs.CVAug 18, 2026

Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving

Authors: Zongzheng Zhang, Jijun Wang, Saining Zhang, Shuo Wang, Yiru Wang, Hai Yang, Yang Chen, Yuwen Heng,

Organizations: Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China · Bosch Corporate Research, Shanghai, China

Abstract

Traffic elements such as traffic lights and road signs play a fundamental role in human driving decisions and should naturally influence end-to-end driving performance. However, existing end-to-end driving research predominantly focuses on dynamic road participants (e.g., vehicles and pedestrians), while the role of traffic elements remains largely unexplored. The community still lacks a systematic study quantifying their impact, largely because public datasets rarely provide structured traffic-element annotations and modern driving systems vary widely in architecture and training paradigm. In this work, we present the first systematic investigation of traffic element awareness for end-to-end autonomous driving. We construct a unified research infrastructure by augmenting multiple public driving datasets with comprehensive traffic-element annotations. To support diverse model families, we adopt a minimal and universal integration design that incorporates traffic-element signals into existing pipelines in a plug-and-play manner with negligible architectural modification. We evaluate this design across modern paradigms, including perception-prediction-planning pipelines, vision-language-action models (VLA), regression-based planners, diffusion-based policies, and trajectory-scoring frameworks, on nuScenes, NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Across all paradigms and datasets, this simple integration consistently improves driving performance, demonstrating that traffic element awareness provides a robust and generalizable signal for end-to-end driving systems. Notably, on the challenging NAVSIM-v2 benchmark, our approach significantly improves state-of-the-art architectures and data pipelines, establishing a new state of the art.

Figures & tables

Figure 2: Definitions of driving topologies. (a) LCTE topology mapping traffic elements ( light / sign ) to the ego-lane in FV. (b) BEV representation of centerlines , ego-related centerlines , and LCLC topology .

Figure 3: (a) Pipeline for 3D traffic element extraction. We first perform 2D traffic-element detection and monocular depth estimation on the FV image, then combine them and LiDAR geometry to localize each traffic element as a 3D center point ( $\bullet$ ). (b) Overview of our method. Multi-view images are encoded into vision queries and enhanced by an auxiliary 3D traffic element detection task. In parallel, predicted topological adjacency matrices are formatted into structured language and processed by a frozen language encoder. The resulting topology queries are concatenated with the BEV queries to guide the planning decoder in generating the ego-vehicle trajectory.

Method	Auxiliary Task	L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$				FPS $\uparrow$
Method	Auxiliary Task	1s	2s	3s	Avg.	1s	2s	3s	Avg.	FPS $\uparrow$
NMP $\diamond$ [ 95 ]	Det & Motion	0.53	1.25	2.67	1.48	0.04	0.12	0.87	0.34	-
FF $\diamond$ [ 30 ]	FreeSpace	0.55	1.20	2.54	1.43	0.06	0.17	1.07	0.43	-
EO $\diamond$ [ 41 ]	FreeSpace	0.67	1.36	2.78	1.60	0.04	0.09	0.88	0.33	-
ST-P3 [ 31 ]	Det & Map & Depth	1.33	2.11	2.90	2.11	0.23	0.62	1.27	0.71	1.6
UniAD [ 32 ]	Det&Track&Map&Motion&Occ	0.44	0.67	0.96	0.69	0.04	0.08	0.23	0.12	1.8

Table 1: Comparison of methods on the nuScenes dataset .

\diamond

: Lidar-based methods. †: The ego status is utilized. FPS is measured on an NVIDIA A100 GPU.

Method	Backbone	Venue	NC $\uparrow$	DAC $\uparrow$	TTC $\uparrow$	Comf. $\uparrow$	EP $\uparrow$	PDMS $\uparrow$
PDM - Closed [ 14 ]	–	PMLR’23	94.6	99.8	89.9	86.9	99.9	89.1
Human driver [ 15 ]	–	NeurIPS’24	100	100	100	99.9	87.5	94.8
Ego - stat. MLP [ 15 ]	–	NeurIPS’24	93.0	77.3	83.6	100	62.8	65.6
UniAD [ 32 ]	ResNet34	CVPR’23	97.8	91.9	92.9	100	78.8	83.4
VAD-v2 [ 9 ]	ResNet34	ICLR’26	98.1	94.8	94.3	100	80.6	86.2
ReCogDrive [ 49 ]	InternViT	ICLR’26	97.9	97.3	94.9	100	87.3	90.8

Table 2: Camera-only methods on the NAVSIM-v1 benchmark ( navtest ) .

	Stage 1									Stage 2
Method	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	EPDMS $\uparrow$
ZTRS† [ 51 ]	98.9	97.6	100	100	66.7	98.9	96.2	96.7	44.0	91.1	90.4	95.8	99.0	63.6	89.8	60.4	97.6	66.1	45.5
DiffVLA† [ 38 ]	95.7	99.2	100	100	85.9	96.4	97.1	95	84.2	81.2	88.8	94.6	99.0	86.0	76.4	59.8	98.6	80.4	45.0
GTRS-Dense† [ 52 ]	98.9	94.9	99.1	100	76.1	98.4	93.8	94.9	37.8	89.9	90.5	94.1	99.3	77.6	88.5	56.0	92.0	30.2	41.9
GuideFlow† [ 57 ]	97.8	97.1	100	100	81.4	98.5	91.4	92.8	34.2	87.3	92.3	98.0	96.9	75.8	85.5	59.3	95.4	53.5	46.7
Regression-based Planner

Table 3: Comparison to existing methods on the NAVSIM-v2 navhard-two-stage benchmark using EPDMS. †: Metrics reported prior to the official benchmark bug fix.

Method	Open-loop Metric	Closed-loop Metric				Latency
Method	Avg. L2 $\downarrow$	Driving Score $\uparrow$	Success Rate(%) $\uparrow$	Efficiency $\uparrow$	Comfortness $\uparrow$	Latency
TCP $\ast$ [ 91 ]	1.70	40.70	15.00	54.26	47.80	86ms
TCP-ctrl $\ast$	-	30.47	7.27	55.97	51.51	86ms
TCP-traj $\ast$	1.70	59.90	30.00	76.54	18.08	86ms
TCP-traj w/o distillation	1.96	49.30	20.45	78.78	22.96	86ms
ThinkTwice $\ast$ [ 35 ]	0.95	62.44	31.23	69.33	16.22	698ms

Table 4: Closed-Loop Planning Performance in Bench2Drive . Avg. L2 is averaged over the predictions in 2 seconds.

\ast

: denotes expert feature distillation. All latency is measured by the averaged inference step-time on CARLA [ 19 ] evaluation in A6000.

Figure 4: Qualitative comparison on NAVSIM-v2 . Left: front-view image with detected traffic elements. Right: predicted trajectories of LTF, LTF+SimScale, and Ours.

	Stage 1									Stage 2
Method	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	NC	DAC	DDC	TLC	EP	TTC	LK	HC	EC	EPDMS $\uparrow$
LTF [ 13 ]	96.2	79.5	99.1	99.5	84.1	95.1	94.2	97.5	79.1	77.7	70.2	84.2	98.0	85.1	75.6	45.4	95.7	75.9	25.1
LTF+TE(2D)	96.4	82.0	99.2	99.6	84.0	95.8	93.1	97.8	79.1	81.6	68.4	85.5	98.6	85.4	78.4	46.9	96.2	76.5	28.1 +3.0
LTF+depth(FV)	96.9	81.8	99.0	99.6	84.2	95.8	92.7	97.8	78.2	81.3	70.1	84.3	98.6	85.7	78.4	45.9	96.3	76.5	27.7 +2.6
LTF+TE(Lidar)	97.2	80.9	99.0	99.5	83.9	96.0	92.9	97.8	79.5	81.1	70.2	85.4	98.6	85.0	79.1	45.9	96.0	77.2	27.9 +2.8
LTF+TE [ 55 ]	96.7	81.8	99.1	99.3	84.1	95.6	92.4	97.6	78.2	81.2	69.4	84.9	98.6	85.6	78.3	46.6	96.2	76.0	27.8 +2.7

Table 5: Ablation study of traffic element (TE) representations on the NAVSIM-v2 using LTF (Latent Transfuser). TE(2D) denotes 2D traffic elements in the front-view ( FV ) image; depth(FV) uses the full FV depth; LiDAR uses the NAVSIM-v2 point-cloud input; TL denotes traffic lights (green/yellow/red light) only.

ID	TE Head		Loss		Pooling		Interaction		EPDMS $\uparrow$
ID	w/ BEV	Indep.	Cross-Entropy	Focal	Average	Max	C.A.	Concat	EPDMS $\uparrow$
0									25.1
1	✓		✓		✓				24.3 -0.8
2		✓		✓	✓				26.1 +1.0
3		✓	✓		✓				25.4 +0.3
4	✓		✓		✓		✓		24.9 -0.2

Table 6: Ablation study of traffic element integration on the NAVSIM-v2 using LTF (Latent Transfuser). We systematically investigate the design choices for the prediction head, loss function, spatial pooling strategy, and planning interaction method.

Figure 5: Robustness analysis on NAVSIM-v2 with LTF. We corrupt the predicted TE representation at inference time by adding TE depth noise, randomly dropping TE detections, or injecting false-positive TE predictions. The performance decreases smoothly under stronger corruption while remaining above the LTF baseline, indicating robustness to moderate upstream TE perception noise.

ID	Topology		Encoder		Source		Scope		L2 (m) $\downarrow$				Collision Rate (%) $\downarrow$
ID	LCTE	LCLC	BERT	GCN	GT [ 81 ]	Pred [ 88 ]	Global	Ego	1s	2s	3s	Avg.	1s	2s	3s	Avg.
0									0.41	0.70	1.05	0.72	0.07	0.17	0.41	0.22
1		✓	✓		✓			✓	0.34	0.60	0.95	0.63	0.06	0.14	0.27	0.16
2	✓		✓		✓			✓	0.34	0.59	0.93	0.62	0.06	0.16	0.33	0.18
3	✓	✓	✓		✓			✓	0.33	0.58	0.94	0.62	0.04	0.16	0.28	0.16
4	✓	✓		✓	✓			✓	0.34	0.60	0.93	0.62	0.07	0.18	0.43	0.23

Table 7: Ablation study of topology information integration on the nuScenes dataset. We ablate the topology types, encoding methods, topology source and interaction scope upon the VAD baseline.

Figure 6: Stepwise performance improvements of our traffic element detector on the OpenLane-V2 validation set, evaluated by the

\text{DET}_{t}

metric.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Frames	TE Inst.	mAP 50	mAP 50:95	Recall	Precision
NAVSIM manual subset	100	1,126	0.908	0.725	0.915	0.918

Appendix

Table A.1 : Cross-dataset validation of the OpenLane-V2-trained traffic-element detector on a manually annotated NAVSIM subset. The subset contains 100 frames from 20 scenes and 1,126 traffic-element instances.

Figure A.1 : Failure case of depth estimation. The left image shows the front-view camera image, where the green traffic light is highlighted by the red bounding box. The right image shows the corresponding predicted depth map.

Figure A.2 : Illustration of topology extraction. Blue lines denote centerlines, while green lines indicate the topology relations between green traffic lights and the corresponding centerlines.

Subset	Depth			Planning
Subset	AbsRel $\downarrow$	RMSE ${}_{\log}\downarrow$	$\delta_{1}\uparrow$	L2@1s $\downarrow$	L2@2s $\downarrow$	L2@3s $\downarrow$	Col@1s $\downarrow$	Col@2s $\downarrow$	Col@3s $\downarrow$
All	0.095	0.201	0.920	0.36	0.61	0.92	0.09	0.14	0.28
Night	0.160	0.279	0.793	0.49	0.85	1.33	0.00	0.15	0.55
Day	0.088	0.193	0.934	0.34	0.58	0.88	0.10	0.13	0.25

Appendix

Table A.2 : Day/night breakdown on nuScenes with the VAD-based traffic-element model. Nighttime scenes exhibit worse depth quality, which correlates with degraded planning accuracy.

Figure A.5 : Overview of the VAD-based architecture with the proposed traffic element and topology branches.

Figure A.6 : Overview of the LTF-based architecture with the proposed traffic element prediction branch.

Figure A.7 : Additional qualitative comparison between VAD and Ours on five traffic-element-rich scenes (a)-(e) from the nuScenes dataset. For each case, the left shows the multi-view camera images, and the right shows the corresponding BEV planning visualizations of VAD and Ours. From top to bottom, the overlaid numbers denote the per-frame L2 error at 1s/2s/3s, collision rate at 1s/2s/3s, and the driving command , respectively.

Figure A.8 : Temporal qualitative comparison on nuScenes . (a) Multi-view images at three consecutive time steps $t_{0}$ , $t_{1}$ , $t_{2}$ , where the forward traffic light changes from red to green . (b) BEV planning trajectories predicted by VAD over the same three frames. (c) Corresponding BEV planning trajectories predicted by Ours.

Figure A.9 : Temporal qualitative comparison on NAVSIM . The left shows front-view images at three consecutive time steps, where the green boxes indicate the detected traffic elements. The right compares the corresponding planning trajectories of LTF, LTF+SimScale, and Ours.

Figure A.10 : Additional qualitative comparison on synthetic NAVSIM-v2 scenes . The left shows front-view images with detected traffic elements (green boxes), and the right compares the corresponding planning trajectories. Cases (a)–(c) compare DiffusionDrive, DiffusionDrive+SimScale, and Ours, while (d) compares DrivoR, DrivoR+SimScale, and Ours.

Method	L2@1s $\downarrow$	L2@2s $\downarrow$	L2@3s $\downarrow$	Col@1s $\downarrow$	Col@2s $\downarrow$	Col@3s $\downarrow$
VAD	0.49	0.85	1.25	0.00	0.00	0.00
Ours	0.37	0.63	0.98	0.00	0.00	0.00

Appendix

Table A.3 : Yellow-light subset on nuScenes. We report open-loop planning metrics on samples containing yellow-light traffic elements.

Method	L2@1s $\downarrow$	L2@2s $\downarrow$	L2@3s $\downarrow$	Col@1s $\downarrow$	Col@2s $\downarrow$	Col@3s $\downarrow$
VAD	0.41	0.70	1.05	0.07	0.17	0.41
Rule w/o Topo	0.57	0.97	1.43	0.02	0.38	0.76
Rule w/ Topo	0.42	0.71	1.07	0.02	0.24	0.35
Ours	0.34	0.56	0.92	0.04	0.21	0.26

Appendix

Table A.4 : Rule-based traffic-element baselines on nuScenes with VAD. “Rule w/o Topo” stops for any red light within 30 meters, while “Rule w/ Topo” stops only for ego-lane-associated red lights.

Encoder	L2@1s $\downarrow$	L2@2s $\downarrow$	L2@3s $\downarrow$	Col@1s $\downarrow$	Col@2s $\downarrow$	Col@3s $\downarrow$	FPS $\uparrow$
BERT	0.34	0.59	0.92	0.04	0.21	0.26	5.4
DistilBERT	0.35	0.61	0.94	0.05	0.15	0.26	5.6
Graph Transformer	0.39	0.65	0.99	0.11	0.19	0.34	5.5

Appendix

Table A.5 : Topology encoder variants on nuScenes with the VAD backbone. DistilBERT achieves performance comparable to BERT at slightly lower cost, while Graph Transformer is less effective.

Figure A.11 : Illustration of a limitation of the standard nuScenes metrics . Using the ground-truth HD map , we additionally count the number of collisions between the predicted ego-vehicle boxes and the lane/map boundaries (shown in the upper-left corner of each panel). (a) VAD, (b) Orion, and (c) Ours. Even in these simple scenes, Ours yields zero boundary collisions, while the other methods still produce boundary violations, showing that L2 and collision rate alone may not fully capture map-consistent planning quality.

Method	Boundary Collision Rate (%) $\downarrow$
	1s	2s	3s	Avg.
VAD	1.09	1.81	2.85	1.92
Ours	1.01	1.39	1.93	1.44

Appendix

Table A.6: Boundary collision rate on nuScenes. Lower is better.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 227 papers

Driving on Memory
Aug 31, 2026Christian Löwens, Thorben Funke, Alexandru Paul ConduracheAutonomous Driving Benchmarks End-to-End Autonomous Driving
Bosch Research · University of Lübeck · Automated Driving, Bosch
PrismAD: Decoupled Planning via Semantic Mixture-of-Planners for End-to-End Autonomous Driving
Jul 11, 2026Kang Ding, Zhigui Lin, Hongsong Wang +5End-to-End Autonomous Driving Mixture of Experts
School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing 100084, China. · School of Cyberspace Security, Southeast University, Nanjing 210096, China. +3
Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling
May 13, 2026Seokha Moon, Minseung Lee, Joon Seo +2Causal Representation Learning End-to-End Autonomous Driving
Korea University · Kakao Mobility

Aug 31, 2026J/K move · Enter open · S save

Driving on Memory

Christian Löwens, Thorben Funke, Alexandru Paul Condurache

Bosch Research · University of Lübeck · Automated Driving, Bosch

Autonomous Driving Benchmarks End-to-End Autonomous Driving Autonomous Driving Safety Evaluation Autonomous Driving

Open paper AskPDF arXiv

End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .

Abstract

End-to-End Autonomous Driving

Explore similar work

Driving on Memory

PrismAD: Decoupled Planning via Semantic Mixture-of-Planners for End-to-End Autonomous Driving

Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling