Real-World Perception for Autonomous Driving in Adverse Weather: Enhancing Standard Detectors via Foundation-Guided Auto-Annotation

Sepideh Gohari; Goodarz Mehr; Azim Eskandarian

cs.CVSep 22, 2026

Real-World Perception for Autonomous Driving in Adverse Weather: Enhancing Standard Detectors via Foundation-Guided Auto-Annotation

Authors: Sepideh Gohari, Goodarz Mehr, Azim Eskandarian

Organizations: Autonomous Robots and Vehicles Laboratory (ARVL), College of Engineering, Virginia Commonwealth University (VCU), Richmond, Virginia, United States

Abstract

Standard deployment-ready object detectors for autonomous vehicles degrade in adverse weather and lighting conditions without being trained on extensive domain-specific data. While large-scale vision foundation models offer robust zero-shot generalization, their high computational cost makes them impractical for real-time deployment. To bridge this gap, we propose a foundation-guided auto-annotation pipeline that enhances standard detectors without architectural changes. We first benchmark three distinct models, YOLOv8, Co-DETR, and SAM3, on our custom real-world driving dataset spanning 25 unique operational scenarios across various route, weather, and lighting conditions. Based on our analysis, SAM3 demonstrates superior accuracy and resilience across all scenarios. Thus, we deploy it as an offline auto-annotator to generate pseudo-labels on the unannotated subset of our dataset. Fine-tuning the baseline YOLOv8 on these annotations yields a 16.04% higher overall mean Average Precision (mAP) and improves cross-environmental stability compared to the baseline model, highlighted by a 32.73% and 28.65% mAP increase in Residential Direct Sunlight and Highway Fog, respectively. These results demonstrate that standard detectors can achieve environmental resilience without the need for extensive manual annotation or architectural modifications.

Figures & tables

Fig. 1: End-to-end framework of the proposed foundation-guided perception pipeline.

Panel of Figure 2 — Fig. 2: Class distribution of the manually annotated ground-truth subset ( $N=3,169$ total instances). (a) Instance count for each object class broken down by route type (Campus, Highway, Residential, Rural) and total count. (b) Overall percentage distribution of object classes across the ground-truth.

Environment	Condition	YOLOv8	CoDETR	SAM3
Campus	Direct Sunlight	29.71	48.44	53.47
	Fog	36.68	68.66	68.93
	Low Light	32.61	54.10	46.06
	No Light	37.19	38.63	51.72
	Rain	38.77	57.55	55.27
	Snow	30.88	52.10	55.58

TABLE I: Performance comparison (

\text{mAP}_{50:95}

(%)) under varying weather and lighting conditions across four route types.

Model Architecture	Mean mAP ( $\mu$ )	STD ( $\sigma$ )
Baseline YOLO (Pre-trained)	34.19	10.68
Fine-Tuned YOLO (SAM3-Guided)	50.23	9.96
Overall Improvement ( $\Delta$ )	+16.04	-0.72

TABLE II: Overall Performance Summary: Baseline vs. Fine-Tuned YOLOv8.

Fig. 3: Performance improvements (mAP 50-95 ) of the SAM3-guided fine-tuned YOLOv8 model versus the pre-trained baseline, sorted by the magnitude of improvement. Grey markers indicate the baseline accuracy, while colored markers denote the fine-tuned model across all 25 scenarios.

	(a) Ground Truth	(b) Baseline YOLOv8	(c) Co-DETR	(d) SAM3	(e) Fine-Tuned YOLOv8
Res. Direct Light
High. No Light
Campus Fog
Rural Rain

Fig. 4: Qualitative detection comparison across four operational scenarios: Residential Direct Sunlight, Highway No Light, Campus Fog, and Rural Rain. Columns display (a) Ground Truth manual annotations alongside outputs from (b) Baseline YOLOv8, (c) Co-DETR, (d) SAM3, and (e) Fine-tuned YOLOv8. Note that while baseline YOLOv8 misses heavily obscured targets, the foundation-guided fine-tuned model successfully recovers those bounding boxes (Best viewed in color and zoomed in).

Sepideh Gohari received the B.S. degree in electrical engineering and the M.S. degree in artificial intelligence and robotics from Ferdowsi University of Mashhad, Mashhad, Iran. She is currently pursuing the Ph.D. degree in electrical and computer engineering at the Autonomous Robots and Vehicles Lab (ARVL), at Virginia Commonwealth University (VCU), Richmond, VA, USA. Her research interests include computer vision, autonomous driving, robotics, and artificial intelligence.

Table 7

Goodarz Mehr received the B.Sc. degree in mechanical engineering from Sharif University of Technology, Tehran, Iran, in 2016 and the M.Sc. and Ph.D. degrees in mechanical engineering from Virginia Tech, Blacksburg, VA, USA, in 2023 and 2024, respectively. He is currently a postdoctoral research associate at Autonomous Robots and Vehicles Lab (ARVL) at Virginia Commonwealth University (VCU), Richmond, VA, USA. His research interests include multi-agent robotics, stochastic planning models, and cooperative perception.

Table 8

Azim Eskandarian (IEEE Fellow) received the B.S. and D.Sc. degrees in mechanical engineering from George Washington University (GWU), Washington, D.C., USA, and the M.S. degree in mechanical engineering from Virginia Tech, Blacksburg, VA, USA. He has been the Alice T. and William H. Goodwin Jr. Dean of the College of Engineering at Virginia Commonwealth University (VCU), Richmond, USA, since 2023, where he also established the Autonomous Robots and Vehicle Laboratory (ARVL). Before that, he was a Professor and the Head of the Department of Mechanical Engineering at Virginia Tech since 2015, where he became the Nicholas and Rebecca Des Champs Chair Professor in 2018, and where he established the Autonomous Systems and the Intelligent Machines Laboratory, to conduct research in intelligent and autonomous vehicles and mobile robotics. As a Professor at George Washington University, he was the Co-Founder of the National Crash Analysis Center in 1992, the founding director of the Center for Intelligent Systems Research from 1995 to 2015, and the director of the University’s area of excellence in Transportation Safety and Security from 2003 to 2015. He was an Assistant Professor at The Pennsylvania State University in York, PA, USA, from 1989 to 1992 and an Engineer/Project Manager in the industry from 1983 to 1989. Dr. Eskandarian is a Fellow of IEEE, ASME, and SAE. He was elected to the Virginia Academy of Science, Engineering, and Medicine (VASEM) in 2026. He received the IEEE Intelligent Transportation Society Outstanding Researcher Award in 2017 and the GWU School of Engineering Outstanding Researcher Award in 2013. He served as Editor-in-Chief of the IEEE Transactions on Intelligent Transportation Systems from 2019 to 2023.

Table 9

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 373 papers

CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving
May 19, 2026Sherif Khairy, Catherine M. EliasImage Enhancement Autonomous Driving Perception
Computer Science & Engineering Department, German University in Cairo (GUC), Egypt · C-DRiVeS Lab: Cognitive Driving Research in Vehicular Systems, Cairo, Egypt
HG-Lane: High-Fidelity Generation of Lane Scenes under Adverse Weather and Lighting Conditions without Re-annotation
Mar 10, 2026Daichao Zhao, Qiupu Chen, Feng He +2Lane Detection Autonomous Driving Datasets
Shanghai Jiao Tong University · School of Artificial Intelligence, Henan University · University of Science and Technology of China +2
XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions
May 12, 2026Chih-Hsin Chen, Yu-Tung Liu, Amar Fadillah +2Autonomous Driving Datasets Autonomous Driving Benchmarks
Department of Electronic Engineering, National Taipei University of Technology · Adobe Inc.

May 19, 2026J/K move · Enter open · S save

CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving

Sherif Khairy, Catherine M. Elias

Computer Science & Engineering Department, German University in Cairo (GUC), Egypt · C-DRiVeS Lab: Cognitive Driving Research in Vehicular Systems, Cairo, Egypt

Image Enhancement Autonomous Driving Perception Object Detection Real-Time Object Detection

Open paper AskPDF arXiv

Adverse weather (rain, fog, sand, and snow) degrades camera-based object detection in autonomous vehicles. Existing enhancement-then-detect approaches stall the safety-critical perception loop, violating hard real-time requirements. Progress on this problem is also constrained by an under-recognized evaluation ceiling: ground truth annotated on degraded images cannot credit a detector that recovers objects the annotators themselves could not see, so a genuinely useful enhancement can register as a near-flat F1 gain. This paper presents CADENet (Condition-Adaptive Asynchronous Dual-stream Enhancement Network), a training-free three-thread system: Thread S (YOLOv11n) delivers detections at full frame rate with zero added latency; Thread Q applies condition-adaptive enhancement (CAPE) and fuses results via entropy-guided NMS (EG-NMS) without blocking Thread S; Thread E provides CLIP zero-shot weather classification, so new weather categories require only a new text prompt, with no labeled data and no retraining. Evaluated on 1327 DAWN images (YOLOv11m, IoU = 0.5, confidence = 0.25), CADENet achieves Recall = 0.0103 (micro), F1 = 0.0230 on snow, and F1 = 0.0038 on rain. We formalize the annotation completeness bias on DAWN-class data, so the reported F1 values are lower bounds on the true gain; recall is the annotation-gap-immune headline metric. Thread S sustains approximately 44 FPS regardless of enhancement load. No model retraining or additional sensor hardware is required.

Abstract

Autonomous Driving Perception

Explore similar work

CADENet: Condition-Adaptive Asynchronous Dual-Stream Enhancement Network for Adverse Weather Perception in Autonomous Driving

HG-Lane: High-Fidelity Generation of Lane Scenes under Adverse Weather and Lighting Conditions without Re-annotation

XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions