Multi-Modal Traffic Sign Detection with Semantic Attributes for Autonomous Driving

Meda Lazar; Sourab Sridhar; Shashwata Gupta; Alexandra Tripcea; Varun Ravi; Senthil Yogamani

cs.CVAug 21, 2026

Multi-Modal Traffic Sign Detection with Semantic Attributes for Autonomous Driving

Authors: Meda Lazar, Sourab Sridhar, Shashwata Gupta, Alexandra Tripcea, Varun Ravi, Senthil Yogamani

Organizations: Arriver System Software S.r.l., Romania · Qualcomm Auto Ltd Sweden Filial · Qualcomm Auto Ltd., UK · Automated Driving, Qualcomm Technologies, Inc

Abstract

Reliable traffic sign detection is a prerequisite for the global deployment of autonomous driving systems, where regulatory compliance and road safety depend on perceiving signs correctly across regions, ranges, and weather conditions. Despite recent progress, vision-based methods continue to face three fundamental limitations: poor cross-regional generalization due to high diversity across countries, degraded performance on small-object detection at long ranges (traffic signs occupy as little as $10{\times}10$ pixels at 200m), and fragile temporal tracking under the strongly non-linear perspective distortion that occurs as a vehicle approaches a sign. In this paper, we address the problem of robust, long-range, region-agnostic traffic sign perception by combining camera and Light Detection and Ranging (LiDAR) sensing. We present a multi-modal detection framework whose Intensity-Aware Deformable Fusion module aligns retro-reflective LiDAR cues with camera features, anchoring detection on geometric invariants rather than region-specific visual appearance. We further introduce a dual motion-model tracker that explicitly accounts for non-linear perspective transformations during vehicle approach, substantially improving temporal consistency over linear motion assumptions. Additionally, we develop a semantic attribute classification pipeline that estimates occlusion level, readability, sign embeddedness, and road relevance, providing actionable context to downstream planning. Extensive evaluation on our dataset, spanning 60+ countries and 2,500+ hours of driving data, shows that the proposed pipeline achieves an Object Miss Ratio (OMR) of 0.49% across 221,068 evaluation sequences, demonstrating globally generalizable traffic sign perception in commercial-grade autonomous driving systems.

Figures & tables

Fig. 1: Traffic sign samples from the Qualcomm internal dataset. The samples highlight the geographic diversity and the wide range of regulatory standards captured by our dataset.

Fig. 2: Global geographic coverage of the Qualcomm dataset used in our development.

Dataset	Annotated Objects	Annotation Type	Region	Modality
Qualcomm	142,000,000	2D/3D Bounding Box	60+ Countries	Image + LiDAR
Zenseact (ZOD) [ 22 ]	446,000	2D/3D Bounding Box	14 (Europe)	Image + LiDAR
Mapillary (MTSD) [ 21 ]	300,000	2D Bounding Box	60 Countries	Image Only
TT100K [ 18 ]	26,000	2D Bounding Box	China	Image Only
GTSDB [ 17 ]	852	2D Bounding Box	Germany	Image Only

TABLE I: Overview of traffic sign datasets and comparative attributes. ZOD: Zenseact Open Dataset; MTSD: Mapillary Traffic Sign Dataset; TT100K: Tsinghua-Tencent 100K; GTSDB: German Traffic-Sign Detection Benchmark.

Fig. 3: Proposed architecture. The pipeline has three stages. (i) A multi-modal detector fuses image features with depth and intensity maps, generated from a temporally accumulated LiDAR point cloud, through an Intensity-Aware Deformable Fusion module, and produces detections via a multi-scale detection head. (ii) Detections are linked into tracked bounding boxes by a dual motion-model tracker. (iii) Tracked boxes are post-processed and passed through an attribute classifier to produce semantic attributes. Architectural details are provided in Section IV .

Panel of Figure 5 — Fig. 4: Examples of detections generated by our detection framework on different scenarios including roundabouts, construction sites, and cities (best viewed by zooming in on the PDF image).

Panel of Figure 6 — Fig. 5: Examples of our proposed tracking framework on edge cases such as adverse weather conditions, roundabout driving, and occluded objects (best viewed by zooming in on the PDF image).

Method	AP	AP 50	AP 75	AP s	AP m	AP l
Zenseact dataset
Model trained on ZOD	0.55	0.78	0.60	0.44	0.80	0.86
Qualcomm dataset
Model trained on ZOD	0.38	0.63	0.39	0.27	0.54	0.71
Model trained on Qualcomm	0.65	0.86	0.73	0.51	0.80	0.90

TABLE II: Multi-modal 2D object detection results. Best values per dataset are highlighted in bold. AP follows the COCO protocol; AP 50 and AP 75 are evaluated at IoU thresholds 0.50 and 0.75, respectively; AP s , AP m , AP l correspond to small (

<32^{2}

px), medium (

32^{2}

–

96^{2}

px), and large (

>96^{2}

px) objects.

Algorithm	Recall	Precision
Detector + OC-SORT [ 32 ]	0.68	0.70
Detector + BoostTrack++ (Default Parameters) [ 36 ]	0.69	0.72
Detector + Baseline Tracker (Tuned Parameters)	0.71	0.71
+ Interpolation	0.72	0.70
+ Second order kinematic model	0.73	0.71
+ Third order kinematic model	0.74	0.71

TABLE III: Object tracker results and ablation study. The baseline tracker is BoostTrack++ [ 36 ] for which we have configured tracking parameters empirically; “+Interpolation” adds linear interpolation across tracking outputs; “+Second-order kinematic model” adds a constant-acceleration Kalman filter; “+Third-order kinematic model” additionally enables the constant-jerk filter. Best Recall and Precision are highlighted in bold; the last row is our final configuration.

Image Backbone	Training Strategy	Class
		0		1
		P	R	P	R
ViT-Base [ 49 ]	From Scratch	0.61	0.74	0.93	0.87
ViT-Base [ 46 ]	LoRA [ 47 ]	0.64	0.71	0.92	0.89
ViT-DINOv2 [ 48 ]	Frozen	0.47	0.71	0.78	0.91
ViT-DINOv2 [ 48 ]	LoRA [ 47 ]	0.72	0.74	0.93	0.92

TABLE IV: Binary Occlusion Classification attribute estimation results: 0 = not occluded, 1 = occluded. LoRA [ 47 ] is applied to ViT-Base [ 46 ] and DINOv2 [ 48 ] ; ViT-Base [ 46 ] is also trained from scratch. P represents Precision, while R represents Recall. Best results are in bold.

Image Backbone	Training Strategy	Class (%)
		0		25		50		75		100
		P	R	P	R	P	R	P	R	P	R
ViT-Base [ 49 ]	From Scratch	0.61	0.74	0.54	0.52	0.52	0.52	0.56	0.28	0.54	0.68
ViT-Base [ 46 ]	LoRA [ 47 ]	0.64	0.71	0.55	0.61	0.50	0.54	0.45	0.37	0.58	0.51
ViT-DINOv2 [ 48 ]	Frozen	0.47	0.71	0.38	0.45	0.39	0.31	0.35	0.13	0.49	0.52
ViT-DINOv2 [ 48 ]	LoRA [ 47 ]	0.72	0.74	0.61	0.63	0.52	0.57	0.44	0.36	0.58	0.59

TABLE V: Occlusion classification attribute estimation results across backbones (LoRA [ 47 ] enabled only for DINOv2 [ 48 ] ). Class column reports the occlusion percentage; ViT-Base [ 46 ] is also trained from scratch. P represents Precision, while R represents Recall. Best results are in bold.

Modality	Class
	Readable		NotReadable		BackOfSign
	P	R	P	R	P	R
Camera Only	0.89	0.92	0.96	0.89	0.81	0.97
Camera + LiDAR	0.89	0.93	0.96	0.90	0.83	0.97

TABLE VI: Readability attribute estimation results with camera-only vs. camera+LiDAR input (DINOv2 backbone [ 48 ] + LoRA [ 47 ] ). P represents Precision, while R represents Recall. Bold marks the better value per (class, metric) pair across the two input configurations.

Panel of Figure 12 — Fig. 6: Examples of occlusion classification. Bounding-box color encodes the predicted occlusion level: blue = 0%, turquoise = 25%, yellow = 50%, green = 75%, red = 100%.

Panel of Figure 13 — Fig. 7: Examples of readability classification. The color of the bounding box represents the predicted class: blue = NotReadable, red = BackOfSign, green = Readable.

Attribute	Recall	Precision
Embedded	0.99	0.99

TABLE VII: Embedded attribute classifier results using precision and recall metrics on the full evaluation set, including both embedded and non-embedded traffic signs.

Panel of Figure 15 — Fig. 8: Examples of edge cases that are correctly classified by the Embedded classifier as NotEmbedded.

Method	Class	Recall	Precision
Lane-distance
Lane-distance	Relevant	0.89	0.54
Lane-distance	NotRelevant	0.62	0.92
Lane-distance + point-inside
Lane-distance + point-inside	Relevant	0.89	0.64
Lane-distance + point-inside	NotRelevant	0.67	0.90

TABLE VIII: Relevance attribute classifier results using class-wise precision and recall metrics. Lane-distance : a sign is Relevant if its distance to the nearest lane is below 5 m. Point-inside + lane-distance : same distance criterion plus the additional constraint that the sign must lie inside a lane-derived polygon. Bold marks the better value per (class, metric) cell across the two methods.

Method	Traffic Sign Type	Modality	AP
Zenseact baseline model
Zenseact	Front	LiDAR	0.42
Zenseact	Back	LiDAR	0.15
Zenseact	All	LiDAR	0.28
Proposed method
Ours	All	LiDAR	0.60

TABLE IX: 3D detection results comparison on the Zenseact public dataset . Zenseact baseline model results for all traffic sign types is obtained by averaging. Best AP is highlighted in bold.

Panel of Figure 18 — Fig. 9: Examples of embedded classifications. Green bounding boxes indicate signs classified as embedded; pink bounding boxes indicate the traffic sign that contains them.

Panel of Figure 19 — Fig. 10: Examples of relevance classification. Bounding-box color encodes the predicted class: blue = NotRelevant, red = Relevant.

Method	AP	AP 0-50	AP 50-100	AP 100-150	AP 150-200
Qualcomm dataset
Single Sweep	0.55	0.65	0.52	0.34	0.14
Multi Sweeps	0.60	0.68	0.58	0.43	0.20

TABLE X: 3D detection results on the Qualcomm dataset , comparing single-sweep and multi-sweep LiDAR accumulation. Best AP per range bin is highlighted in bold; the row in blue is our final configuration.

Method	AP	AP 50	AP 75	AP s	AP m	AP l
Qualcomm dataset
Camera-only baseline	0.62	0.83	0.69	0.46	0.77	0.89
Camera and LiDAR Fusion	0.65	0.86	0.73	0.51	0.80	0.90

TABLE XI: Ablation of the LiDAR contribution in the 2D detector on the Qualcomm dataset. The Camera-only baseline is the same Qualcomm-trained backbone with no LiDAR input; “+ LiDAR” adds the depth and intensity maps fused via the Intensity-Aware Deformable Fusion module described in Section IV . Best value per metric is in bold.

Panel of Figure 22 — Fig. 11: Distribution of dataset metadata tags: (left) weather, (center) road type, (right) time of day.

Category	Sequences	Object Miss Ratio
Time of Day
Day	145,412	0.67%
Night	63,641	0.17%
Dusk/Dawn	12,395	0.36%
Road Type
Urban	95,806	0.32%

TABLE XII: Object Miss Ratio (OMR) breakdown by time of day, road type, and weather condition on the Qualcomm dataset (lower is better). Rare weather categories (Thunderstorm, ViolentRain, Hail) contain very few sequences and should be treated as indicative rather than statistically conclusive.

Explore similar work

Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment
Mar 24, 2026Guoyang Zhao, Weiqing Qi, Kai Zhang +7Autonomous Driving Datasets Autonomous Driving Perception
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Lingnan University, Hong Kong SAR, China · Shenzhen Unity Drive Innovation Technology Co., Ltd., Shenzhen 518063, China +1
Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree
Oct 12, 2026Hongsen Cao, Mona Jaber, Shanxin Yuan +1Camera-LiDAR Fusion 3D Object Detection
School of Electronic Engineering and Computer Science, Queen Mary University of London, London, United Kingdom.
VGGT-MPR: VGGT-Enhanced Multimodal Place Recognition in Autonomous Driving Environments
Feb 23, 2026Jingyi Xu, Zhangshuo Qi, Zhongmiao Yan +5Camera-LiDAR Fusion Autonomous Driving
Shanghai Jiao Tong University. · Beijing Institute of Technology. · National University of Defense Technology.

Mar 24, 2026J/K move · Enter open · S save

Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment

Guoyang Zhao, Weiqing Qi, Kai Zhang, Chenguang Zhang+6

The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Lingnan University, Hong Kong SAR, China · Shenzhen Unity Drive Innovation Technology Co., Ltd., Shenzhen 518063, China+1

Autonomous Driving Datasets Autonomous Driving Perception Autonomous Driving Benchmarks VLMs for Autonomous Driving

Open paper AskPDF arXiv

Traffic Sign Recognition (TSR) is a core perception capability for autonomous driving, where robustness to cross-region variation, long-tailed categories, and semantic ambiguity is essential for reliable real-world deployment. Despite steady progress in recognition accuracy, existing traffic sign datasets and benchmarks offer limited diagnostic insight into how different modeling paradigms behave under these practical challenges. We present TS-1M, a large-scale and globally diverse traffic sign dataset comprising over one million real-world images across 454 standardized categories, together with a diagnostic benchmark designed to analyze model capability boundaries. Beyond standard train-test evaluation, we provide a suite of challenge-oriented settings, including cross-region recognition, rare-class identification, low-clarity robustness, and semantic text understanding, enabling systematic and fine-grained assessment of modern TSR models. Using TS-1M, we conduct a unified benchmark across three representative learning paradigms: classical supervised models, self-supervised pretrained models, and multimodal vision-language models (VLMs). Our analysis reveals consistent paradigm-dependent behaviors, showing that semantic alignment is a key factor for cross-region generalization and rare-category recognition, while purely visual models remain sensitive to appearance shift and data imbalance. Finally, we validate the practical relevance of TS-1M through real-scene autonomous driving experiments, where traffic sign recognition is integrated with semantic reasoning and spatial localization to support map-level decision constraints. Overall, TS-1M establishes a reference-level diagnostic benchmark for TSR and provides principled insights into robust and semantic-aware traffic sign perception. Project page: https://guoyangzhao.github.io/projects/ts1m.

Abstract

Explore similar work

Traffic Sign Recognition in Autonomous Driving: Dataset, Benchmark, and Field Experiment

Mining Cross-Modal Perception Gaps in Autonomous Driving: When Camera and LiDAR Disagree

VGGT-MPR: VGGT-Enhanced Multimodal Place Recognition in Autonomous Driving Environments