Heuristic Style Transfer for Real-Time, Efficient Weather Attribute Detection
Authors: Hamed Ouattara, Pierre Duthon, Pascal Houssam Salmane, Frédéric Bernardin, Omar Ait Aider
Organizations: Intelligent Transportation Systems Research Team(I.T.S), Cerema, 8, rue Bernard Palissy, Clermont-Ferrand, 63017, France. · Intelligent Transportation Systems Research Team(I.T.S), Cerema, 1 avenue du Colonel Roche, Toulouse, 31400, France. · Clermont Auvergne INP, Université Clermont Auvergne, Pascal Institute, CNRS, 4 Av. Blaise Pascal, Clermont-Ferrand, 63178, France.
We present lightweight and efficient architectures to detect weather conditions from RGB images, predicting the weather type (sunny, rain, snow, fog) and 11 complementary attributes such as intensity, visibility, and ground condition, for a total of 53 classes across the tasks. This work examines to what extent weather conditions manifest as variations in visual style. We investigate style-inspired techniques, including Gram matrices, a truncated ResNet-50 targeting lower and intermediate layers, and PatchGAN-style architectures, within a multi-task framework with attention mechanisms. Two families are introduced: RTM (ResNet50-Truncated-MultiTasks) and PMG (PatchGAN-MultiTasks-Gram), together with their variants. Our contributions include automation of Gram-matrix computation, integration of PatchGAN into supervised multi-task learning, and local style capture through local Gram for improved spatial coherence. We also release a dataset of 503,875 images annotated with 12 weather attributes under a Creative Commons Attribution (CC-BY) license. The models achieve F1 scores above 96 percent on our internal test set and above 78 percent in zero-shot evaluation on several external datasets, confirming their generalization ability. The PMG architecture, with fewer than 5 million parameters, runs in real time with a small memory footprint, making it suitable for embedded systems. The modular design of the models also allows style-related or weather-related tasks to be added or removed as needed.
Figures & tables
Figure 1: Joint detection of 12 weather-related attributes with ResNet50-Truncated-MultiTasks (RTM). This example illustrates the simultaneous prediction of Weather Type, Visibility, etc. The aim is to describe the environment from the standpoint of weather and its effects, using 12 key characteristics (e.g., Ground Condition, useful for adapting speed on wet pavement) with a single real-time model. The architecture is modular: task heads can be disabled at inference to reduce computational cost, without impacting other tasks thanks to independent classifiers and attention mechanisms.
Figure 2: Simulation of weather conditions via style transfer with CycleGAN [ 4 ] . Left columns: real (source) images. Right columns: stylized outputs reproducing rain, snow, or fog at varying intensities. This example shows that an unpaired style-transfer model can simulate visual signatures of weather (rain streaks, luminous haze, color drift), illustrating that weather-related changes can be expressed as changes in visual style and motivating our investigation of this link.
Figure 3: Snow simulation via style transfer with CycleGAN [ 4 ] . Left: real image; right: “snow” stylized image reproducing coherent cues (overcast sky, snow-covered ground, attenuated vegetation, tracks). These results suggest that unpaired style transfer, guided by a PatchGAN discriminator, can capture stylistic signatures related to multiple meteorological attributes—which motivates our multi-task approach, where the local sensitivity of PatchGAN is treated as a potential “sensor” for such cues.
Figure 4: Neural style transfer [ 45 ] : a synthetic image is optimized to preserve the structure of the content images while reproducing the style statistics (Gram matrices) of the style images.
Figure 5: Receptive field in a convolutional network: deeper layers aggregate information over larger spatial extents, and descriptors increasingly reflect global properties.
Figure 6: Annotation example. Each image is annotated on 13 criteria(each one has a number of classes, 53 of which are related to the weather). 12 related to weather conditions and 1 related to viewpoint.
Figure 7: Distribution of the 13(53 classes related to the weather) annotated attributes. The split is near-balanced for Weather Type, but several tasks are strongly imbalanced, notably Road Spray, Water on Windshield, and Snow on Windshield. To limit learning bias, we apply per-task, per-class weighting during training.
Figure 8: Example images showing rain, snow, fog, and clear weather, by day and by night.
Table 1: Summary of models and hyperparameters optimized by evolutionary search.
Task
ResNet50
RTM (no att.)
RTM (with att.)
RTMG
PM (no att.)
PM (with att.)
PMG
Weather condition classification
Weather Type
0.8629
0.9949
0.9944
0.9605
0.9598
0.9792
0.9877
Weather Intensity
0.8486
0.9883
0.9908
0.9605
0.9434
0.9761
0.9845
Visibility and sky condition
Visibility
0.8542
0.9871
0.9865
0.9564
0.9382
0.9689
0.9778
Sky Condition
0.8686
0.9870
0.9873
0.9494
0.9491
0.9652
0.9821
Table 2: F1 scores per task on the test set (25,000 images). Comparison of truncated ResNet models (RTM, RTMG - 25 epochs) and PatchGAN (PM, PMG - 100 epochs). Best values per row are in bold. RTM achieves the best overall mean, with performance close to PMG. External datasets will refine the comparison in terms of generalization and robustness.
Figure 13: Grad - CAM for the Sky Condition task. For RTM without attention, maps are extracted at the output of the truncated ResNet-50; for PM without attention, at the trunk output before the classifier; for RTM and PM with attention as well as RTMG and PMG, at the output of the attention block dedicated to the task. Overall, attention better guides focus toward relevant regions (sky, clouds).
Figure 14: t - SNE of embeddings for Ground Condition (50,388 images). Legend (colors as shown in the figure): Dry , Wet , Partly Wet , Partly Snow , Wet and Snowy , Unknown . Subfigures: (a) RTM without attention, (b) RTM with attention, (c) RTMG. Attention strengthens inter-class separation (RTM with attention, RTMG), with coherent semantic proximities ( Wet and Snowy between Wet and Snowy ).
Figure 15: t - SNE of embeddings for Weather Intensity (50,388 images). Legend (colors as shown in the figure): Low , Average , High , None . Subfigures: (a) PM without attention, (b) PM with attention, (c) PMG. Attention variants improve separation; PMG produces more compact clusters and sharper boundaries.
Model
Image2Weather (F1)
MWI (F1)
Kaggle (F1)
RTM (truncated, no att.)
0.8014
0.8922
0.9660
RTM (truncated, with att.)
0.8109
0.9033
0.9800
RTMG
0.8153
0.8936
0.9730
PM (no att.)
0.7721
0.8387
0.9208
PM (with att.)
0.7567
0.8690
0.9389
PMG
0.7917
0.8774
0.9527
Table 3: Zero-shot generalization on external datasets not seen during training (F1). Evaluation limited to the Weather Type task. External labels are harmonized to {sun, fog, rain, snow}: sun ← {Clear, Sun and Clear}, fog ← {Fog, Fog and Rain, Fog and Snow}, rain ← {Rain}, snow ← {Snow}.
Figure 16: Example images from external datasets (unseen during training).
Figure 17: Examples of errors due to annotation mismatches. For instance, the Fog class in MWI covers Fog , Fog + Rain , and Fog + Snow in our scheme, whereas our taxonomy explicitly distinguishes precipitations.
Figure 18: Example images from the external WEAPD dataset.
MWI (20k images) [ 57 ] — metric: Accuracy
RTM
Acc = 0.9731
classifier only
PMG
Acc = 0.9544
classifier only
SLM3 (2024) [ 58 ]
Acc = 0.9001
published value
WEAPD (6,877 images) [ 59 ] — metric: F1
RTM
F1 = 0.9002
classifier only
PMG
F1 = 0.7874
classifier only
Table 4: Fine-tuning per benchmark (encoder and attention frozen, classifier retrained, 30 epochs). Training on 80% of the data, validation 10%, test 10%. Metrics follow each reference (Accuracy or F1). Best score per block in bold.
In this paper, we present three neural network architectures designed for real-time classification of weather conditions (sunny, rain, snow, fog) from images. These models, inspired by recent advances in style transfer, aim to capture the stylistic elements present in images. One model, called "Multi-PatchGAN", is based on PatchGANs used in well-known architectures such as Pix2Pix and CycleGAN, but here adapted with multiple patch sizes for detection tasks. The second model, "Truncated ResNet50", is a simplified version of ResNet50 retaining only its first nine layers. This truncation, determined by an evolutionary algorithm, facilitates the extraction of high-frequency features essential for capturing subtle stylistic details. Finally, we propose "Truncated ResNet50 with Gram Matrix and Attention", which computes Gram matrices for each layer during training and automatically weights them via an attention mechanism, thus optimizing the extraction of the most relevant stylistic expressions for classification. These last two models outperform the state of the art and demonstrate remarkable generalization capability on several public databases. Although developed for weather detection, these architectures are also suitable for other appearance-based classification tasks, such as animal species recognition, texture classification, disease detection in medical imaging, or industrial defect identification.
Hamed Ouattara, Pascal Houssam Salmane, Pierre Duthon +2
Cerema, Intelligent Transportation Systems Research Team, France · Institut Pascal, France
Photographs taken in adverse weather conditions often suffer from blurriness, occlusion, and low brightness due to interference from rain, snow, and fog. These issues can significantly hinder the performance of subsequent computer vision tasks, making the removal of weather effects a crucial step in image enhancement. Existing methods primarily target specific weather conditions, with only a few capable of handling multiple weather scenarios. However, mainstream approaches often overlook performance considerations, resulting in large parameter sizes, long inference times, and high memory costs. In this study, we introduce the WeatherRemover model, designed to enhance the restoration of images affected by various weather conditions while balancing performance. Our model adopts a UNet-like structure with a gating mechanism and a multi-scale pyramid vision Transformer. It employs channel-wise attention derived from convolutional neural networks to optimize feature extraction, while linear spatial reduction helps curtail the computational demands of attention. The gating mechanisms, strategically placed within the feed-forward and downsampling phases, refine the processing of information by selectively addressing redundancy and mitigating its influence on learning. This approach facilitates the adaptive selection of essential data, ensuring superior restoration and maximizing efficiency. Additionally, our lightweight model achieves an optimal balance between restoration quality, parameter efficiency, computational overhead, and memory usage, distinguishing it from other multi-weather models, thereby meeting practical application demands effectively. The source code is available at https://github.com/RICKand-MORTY/WeatherRemover.
Weikai Qu, Sijun Liang, Cheng Pan +6
Guangdong University of Technology · Sanda University · University of Queensland +5
Existing vision-language models (VLMs) have demonstrated impressive performance in reasoning-based segmentation. However, current benchmarks are primarily constructed from high-quality images captured under idealized conditions. This raises a critical question: when visual cues are severely degraded by adverse weather conditions such as rain, snow, or fog, can VLMs sustain reliable reasoning segmentation capabilities? In response to this challenge, we introduce WeatherReasonSeg, a benchmark designed to evaluate VLM performance in reasoning-based segmentation under adverse weather conditions. It consists of two complementary components. First, we construct a controllable reasoning dataset by applying synthetic weather with varying severity levels to existing segmentation datasets, enabling fine-grained robustness analysis. Second, to capture real-world complexity, we curate a real-world adverse-weather reasoning segmentation dataset with semantically consistent queries generated via mask-guided LLM prompting. We further broaden the evaluation scope across five reasoning dimensions, including functionality, application scenarios, structural attributes, interactions, and requirement matching. Extensive experiments across diverse VLMs reveal two key findings: (1) VLM performance degrades monotonically with increasing weather severity, and (2) different weather types induce distinct vulnerability patterns. We hope WeatherReasonSeg will serve as a foundation for advancing robust, weather-aware reasoning.
Wanjun Du, Zifeng Yuan, Tingting Chen +3
Beijing Jiaotong University · National University of Singapore · Monash University