Analysis of SWIR Imaging Detection Performance Under Adverse Environmental Conditions for Autonomous Driving Systems

Rohan Mehra; Alexandre Riffard; Yannis Loumouamou; Mathieu Labussière

cs.CVOct 5, 2026

Analysis of SWIR Imaging Detection Performance Under Adverse Environmental Conditions for Autonomous Driving Systems

Authors: Rohan Mehra, Alexandre Riffard, Yannis Loumouamou, Mathieu Labussière

Organizations: Université Clermont Auvergne, Clermont Auvergne INP, CNRS, Institut Pascal, F-63000 Clermont-Ferrand, France

Abstract

Short-wave infrared (SWIR) imaging has emerged as a promising modality for autonomous driving, yet its practical benefits over RGB remain poorly characterized across diverse conditions. This paper presents a systematic comparative study of paired RGB and SWIR object detection on the RASMD dataset, covering four weather conditions and two real-time detection architectures, with various fine-tunings evaluated against a unified ground truth. Overall, RGB demonstrates comparable or superior performance in most scenarios, while RF-DETR exhibits greater robustness across varying conditions. Beyond aggregate metrics, we propose a sensor-dominance mining framework that combines multi-model agreement with targeted manual inspection to identify scenarios where one sensing modality provides more reliable detections using largely unannotated paired data. This analysis reveals that SWIR offers clear advantages in four safety-critical situations, including windshield glare, water droplets on the windshield, low-contrast object visibility, and long-range vehicle detection. The findings suggest that SWIR should be viewed as a complementary modality that enhances perception in rare but challenging conditions. The datasets will be available upon request, and all code and trained model weights are publicly released at https://github.com/comsee-research/swir-adverse-env-analysis.

Figures & tables

Panel of Figure 1 — Fig. 1: Paired RGB-SWIR images from the RASMD dataset [ 1 ] across different weather conditions: (a) Sunny, (b) Cloudy, (c) Rainy, and (d) Snowy. Each image presents a diagonal split showing RGB (bottom-left) and SWIR (top-right).

Fig. 2: High-level overview of the experimental pipeline. The block diagram illustrates the progression of tasks undertaken in this study, which are discussed in detail in the subsequent sections.

Class	Train (1713 images)	Test (800 images)	Total (2513 images)
Car	7,531	5,092	12,623
Truck	1,845	897	2,742
Person	480	828	1,308
Bus	421	566	987
Motorcycle	87	228	315
Bicycle	60	217	277

TABLE I: Class-wise distributions in the RASMD-IP dataset. Training set cover all weather and scenes uniformly, whereas the test set prioritizes per-class support for significant evaluations.

Camera	Model	Fine-Tuning	F1 w	Rec w	Prec w	AP person	AP car	AP truck	AP bus	AP bicy.	AP moto.	mAP
RGB	YOLOv8	PT	0.6994	0.5841	0.8813	0.3711	0.4793	0.3415	0.2764	0.2383	0.3014	0.3347 $\pm$ 0.0084
		RGB	0.7956	0.7357	0.8701	0.2945	0.5280	0.5329	0.4570	0.2401	0.3247	0.3962 $\pm$ 0.0072
		Mix	0.7892	0.7214	0.8752	0.2505	0.5286	0.5131	0.4194	0.2149	0.2825	0.3681 $\pm$ 0.0072
	RF-DETR	PT	0.7770	0.6905	0.8994	0.3973	0.5556	0.4521	0.3600	0.2711	0.4340	0.4116 $\pm$ 0.0084
		RGB	0.8176	0.7634	0.8842	0.3518	0.5694	0.5680	0.4952	0.3442	0.2960	0.4374 $\pm$ 0.0075
		Mix	0.8136	0.7494	0.8951	0.3113	0.5425	0.5547	0.4600	0.2822	0.3122	0.4105 $\pm$ 0.0074

TABLE II: Quantitative evaluation across combined weather conditions for YOLOv8 and RF-DETR on paired RGB and SWIR data. Modality-specific fine-tuning improves performance over pre-trained models, while mixed fine-tuning gives competitive cross-modality performance. Bold indicates the best, and underline the the second best per camera.

Panel of Figure 5 — Fig. 3: Performance comparison across weather conditions (Sunny, Cloudy, Rainy, Snowy) and sensor modalities (RGB, SWIR) of YOLO and RF-DETR: (a) Weighted Precision, (b) Weighted Recall, (c) Weighted F1-score, and (d) bootstrapped mAP mean. RF-DETR generally outperforms YOLO under the same fine-tuning. Pretrained models show the lower recall than fine-tuned but competitive precision. RGB performs better overall, while RGB and SWIR are comparable under snowy conditions. Better visualized in color.

Method	RGB +	SWIR +	Equal	Discarded
M1: Any Vote	14,596 (14.7%)	10,344 (10.4%)	13,358 (13.4%)	61,183 (61.5%)
M2: Majority Vote	8,514 (8.6%)	4,945 (5.0%)	24,839 (25.0%)	61,183 (61.5%)
M3: Unanimous Vote	3,649 (3.7%)	1,668 (1.7%)	32,981 (33.2%)	61,183 (61.5%)

TABLE III: Distribution of number of frame pairs across the three segregation methods. RGB + and SWIR + columns indicate frames where that respective sensor strictly outperformed the other. Equal denotes same performance, while Discarded represents conflicted cases. Percentages represent the proportion of the total dataset.

Condition	Modality	Weighted Metrics			Binary Class Metrics			Class-wise AP
Condition	Modality	Prec.	Recall	F-1	Prec.	Recall	AP	Car	Truck	Person
Hidden	RGB	0.9358	0.7631	0.8333	0.9454	0.7685	0.5420	0.6095	0.5739	0.1380
Hidden	SWIR	0.9370	0.8062	0.8638	0.9451	0.8116	0.6212	0.6723	0.6545	0.2978
Very-Far	RGB	0.9186	0.7288	0.8108	0.9322	0.7408	0.5029	0.5017	0.5088	-
Very-Far	SWIR	0.9296	0.7887	0.8520	0.9536	0.8092	0.5723	0.5753	0.5541	-
Glare	RGB	0.8719	0.3318	0.4746	0.8537	0.3318	0.1950	0.1845	-	-

TABLE IV: Validation of four SWIR-better filtered scenarios done by RF-DETR (modality-specific models). Best values between modalities are in bold , and only statistically significant classes are considered for class-wise AP. Results indicate a clear SWIR advantage, primarily driven by improved recall.

Panel of Figure 8 — Fig. 4: Visual comparison of detections in four SWIR-advantageous mined scenarios produced by RF-DETR-Mix model: (a) glare on the windshield, where glare saturates RGB while SWIR enables vehicle detection; (b) water on the windshield, preventing detections in RGB due to scattering while vehicles are detected in SWIR; (c) motorcyclist hidden due to low contrast in RGB but detected in SWIR, and (d) distant vehicles, where reduced atmospheric scattering in SWIR enables detection of a car and a truck. Each pair shows RGB (left) and SWIR (right). Images are cropped and zoomed for visualization clarity.

Panel of Figure 9 — Fig. 5: Temporal analysis of vehicle detection during tunnel traversal, highlighting SWIR robustness to glare. Graphs above shows detection counts from RGB and SWIR for two vehicles using RF-DETR and YOLOv8x under Mixed and modality-specific fine-tuning (curves smoothed with $\sigma=2$ for visualization). (a) Tunnel entry: due to the rapid transition from bright to dark regions, RGB detects two vehicles while SWIR detects only one; (b) Glare region: RGB pixels saturates while SWIR detects both vehicles; (c) Tunnel exit: under normal illumination, both sensors detect one vehicle.

Explore similar work

XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions
May 12, 2026Chih-Hsin Chen, Yu-Tung Liu, Amar Fadillah +2Object Detection Cross-Dataset Benchmark
Department of Electronic Engineering, National Taipei University of Technology · Adobe Inc.
Descriptor: LYNRED Mobility Dataset Multimodal Detection Subset (LYNRED-MDS)
Jul 2, 2026Loïc Arbez, Jessy Matias, Xavier Brenière +2Pedestrian Detection Multimodal Dataset
Thoth · INRIA center at University Grenoble Alpes, 38330 Montbonnot-Saint-Martin, France · Lynred, 38113, Veurey-Voroize, France +2
InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection
Jul 4, 2026Zichao Feng, Haodong Zhu, Jingying Yang +8Infrared Infrared Small Target Detection
Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · Communication University of China, Beijing, China +1

May 12, 2026, cs.CVJ/K move · Enter open · S save

XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions

Chih-Hsin Chen, Yu-Tung Liu, Amar Fadillah, Kuan-Ting Lai+1

Department of Electronic Engineering, National Taipei University of Technology · Adobe Inc.

Object Detection Cross-Dataset Benchmark Robotwin

Open paper AskPDF arXiv

Autonomous driving and intelligent transportation systems remain vulnerable under extreme weather. The U.S. Federal Highway Administration reports that roughly 745,000 crashes and 3,800 fatalities per year are weather-related, and recent regulatory investigations have examined failures of Level-2/3 driving systems under reduced-visibility conditions. However, datasets commonly used to evaluate weather robustness remain limited in scale, diversity, and realism. In this paper, we introduce XWOD (Extreme Weather Object Detection), a large-scale real-world traffic-object detection benchmark containing 10,010 images and 42,924 bounding boxes across seven extreme weather conditions: rain, snow, fog, haze/sand/dust, flooding, tornado, and wildfire. The dataset covers six traffic-object categories, including car, person, truck, motorcycle, bicycle, and bus. XWOD extends the weather taxonomy from one to seven conditions, and is the first to cover the emerging class of climate-amplified hazards, such as flooding, tornado, and wildfire. To evaluate the quality of our data, we train standard YOLO-family detectors on XWOD and test them zero-shot on external weather benchmarks, achieving mAP

_{50}

scores of 63.00% on RTTS, 59.94% on DAWN, and 61.12% on WEDGE, compared with the corresponding published YOLO-based baselines of 40.37%, 32.75%, and 45.41%, respectively, representing relative improvements of 56%, 83%, and 35%. These cross-dataset results show that XWOD provides a strong source domain for learning weather-robust traffic perception. We release the dataset, splits, baseline weights, and reproducible evaluation code under a research-use license.

Abstract

Explore similar work

XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions

Descriptor: LYNRED Mobility Dataset Multimodal Detection Subset (LYNRED-MDS)

InfraNet: Quality-Aware RGB Guidance for Efficient Infrared Object Detection