Heuristic Style Transfer for Real-Time, Efficient Weather Attribute Detection

Hamed Ouattara; Pierre Duthon; Pascal Houssam Salmane; Frédéric Bernardin; Omar Ait Aider

cs.CVApr 15, 2026

Heuristic Style Transfer for Real-Time, Efficient Weather Attribute Detection

Authors: Hamed Ouattara, Pierre Duthon, Pascal Houssam Salmane, Frédéric Bernardin, Omar Ait Aider

Organizations: Intelligent Transportation Systems Research Team(I.T.S), Cerema, 8, rue Bernard Palissy, Clermont-Ferrand, 63017, France. · Intelligent Transportation Systems Research Team(I.T.S), Cerema, 1 avenue du Colonel Roche, Toulouse, 31400, France. · Clermont Auvergne INP, Université Clermont Auvergne, Pascal Institute, CNRS, 4 Av. Blaise Pascal, Clermont-Ferrand, 63178, France.

Abstract

We present lightweight and efficient architectures to detect weather conditions from RGB images, predicting the weather type (sunny, rain, snow, fog) and 11 complementary attributes such as intensity, visibility, and ground condition, for a total of 53 classes across the tasks. This work examines to what extent weather conditions manifest as variations in visual style. We investigate style-inspired techniques, including Gram matrices, a truncated ResNet-50 targeting lower and intermediate layers, and PatchGAN-style architectures, within a multi-task framework with attention mechanisms. Two families are introduced: RTM (ResNet50-Truncated-MultiTasks) and PMG (PatchGAN-MultiTasks-Gram), together with their variants. Our contributions include automation of Gram-matrix computation, integration of PatchGAN into supervised multi-task learning, and local style capture through local Gram for improved spatial coherence. We also release a dataset of 503,875 images annotated with 12 weather attributes under a Creative Commons Attribution (CC-BY) license. The models achieve F1 scores above 96 percent on our internal test set and above 78 percent in zero-shot evaluation on several external datasets, confirming their generalization ability. The PMG architecture, with fewer than 5 million parameters, runs in real time with a small memory footprint, making it suitable for embedded systems. The modular design of the models also allows style-related or weather-related tasks to be added or removed as needed.

Figures & tables

Figure 1: Joint detection of 12 weather-related attributes with ResNet50-Truncated-MultiTasks (RTM). This example illustrates the simultaneous prediction of Weather Type, Visibility, etc. The aim is to describe the environment from the standpoint of weather and its effects, using 12 key characteristics (e.g., Ground Condition, useful for adapting speed on wet pavement) with a single real-time model. The architecture is modular: task heads can be disabled at inference to reduce computational cost, without impacting other tasks thanks to independent classifiers and attention mechanisms.

Figure 2: Simulation of weather conditions via style transfer with CycleGAN [ 4 ] . Left columns: real (source) images. Right columns: stylized outputs reproducing rain, snow, or fog at varying intensities. This example shows that an unpaired style-transfer model can simulate visual signatures of weather (rain streaks, luminous haze, color drift), illustrating that weather-related changes can be expressed as changes in visual style and motivating our investigation of this link.

Figure 3: Snow simulation via style transfer with CycleGAN [ 4 ] . Left: real image; right: “snow” stylized image reproducing coherent cues (overcast sky, snow-covered ground, attenuated vegetation, tracks). These results suggest that unpaired style transfer, guided by a PatchGAN discriminator, can capture stylistic signatures related to multiple meteorological attributes—which motivates our multi-task approach, where the local sensitivity of PatchGAN is treated as a potential “sensor” for such cues.

Figure 4: Neural style transfer [ 45 ] : a synthetic image is optimized to preserve the structure of the content images while reproducing the style statistics (Gram matrices) of the style images.

Figure 5: Receptive field in a convolutional network: deeper layers aggregate information over larger spatial extents, and descriptors increasingly reflect global properties.

Figure 6: Annotation example. Each image is annotated on 13 criteria(each one has a number of classes, 53 of which are related to the weather). 12 related to weather conditions and 1 related to viewpoint.

Figure 7: Distribution of the 13(53 classes related to the weather) annotated attributes. The split is near-balanced for Weather Type, but several tasks are strongly imbalanced, notably Road Spray, Water on Windshield, and Snow on Windshield. To limit learning bias, we apply per-task, per-class weighting during training.

Figure 8: Example images showing rain, snow, fog, and clear weather, by day and by night.

Figure 9: RTM: truncated ResNet $\rightarrow$ per-task attention $\rightarrow$ multi-task classifiers.

Figure 10: RTMG: truncated ResNet $\rightarrow$ Gram (style) $\rightarrow$ (attention) $\rightarrow$ multi-task heads.

Figure 11: PMG: local Gram (style tokens) + attention mechanisms + task-conditioned pooling.

Figure 12: PM: PatchGAN trunk $\rightarrow$ per-task spatial attention $\rightarrow$ aggregation $\rightarrow$ per-task outputs.

Model	Summary (idea and training)	Optimized hyperparameters (DEAP)
RTM	Truncated ResNet-50 (no avgpool/fc ) with lightweight per-task attention (GAP+MLP ablation). Optimized under SGD; $K$ -fold F1 as the objective.	truncate_layer , batch_size , lr , head depth/width ( hidden_dims , num_layers ); use_attention (if mode=evolve ); encoder freezing (training option).
RTMG	RTM augmented with global Gram over intermediate channels (style correlations); same training protocol (SGD, $K$ -fold).	All RTM HPOs plus gram_matrix_size ; use_attention (if attention_mode=evolve ).
PM	Multi-task PatchGAN (implicit non-overlapping patches) with TaskHeadImproved (optional SE, spatial softmax with temperature $\tau$ ); $K$ -fold F1 objective.	batch_size , lr , patch_size . Attention parameters not evolved but configurable via CLI : attn_tau , attn_use_se , attn_softmax_spatial , attn_tv_lambda .
PMG	PatchGAN + local Grams per implicit patch $\rightarrow$ token projection $\rightarrow$ (optional) lightweight attention mechanisms + task-conditioned pooling. Optimized under AdamW + Cosine; AMP/accumulation available.	batch_size , lr , weight_decay , patch_size , patch_div , ndf , gram_channels , d_model , attention mechanisms_layers , attention mechanisms_heads , use_token_attention , use_channel_attention , use_focal , focal_gamma .

Table 1: Summary of models and hyperparameters optimized by evolutionary search.

Task	ResNet50	RTM (no att.)	RTM (with att.)	RTMG	PM (no att.)	PM (with att.)	PMG
Weather condition classification
Weather Type	0.8629	0.9949	0.9944	0.9605	0.9598	0.9792	0.9877
Weather Intensity	0.8486	0.9883	0.9908	0.9605	0.9434	0.9761	0.9845
Visibility and sky condition
Visibility	0.8542	0.9871	0.9865	0.9564	0.9382	0.9689	0.9778
Sky Condition	0.8686	0.9870	0.9873	0.9494	0.9491	0.9652	0.9821

Table 2: F1 scores per task on the test set (25,000 images). Comparison of truncated ResNet models (RTM, RTMG - 25 epochs) and PatchGAN (PM, PMG - 100 epochs). Best values per row are in bold. RTM achieves the best overall mean, with performance close to PMG. External datasets will refine the comparison in terms of generalization and robustness.

Figure 13: Grad - CAM for the Sky Condition task. For RTM without attention, maps are extracted at the output of the truncated ResNet-50; for PM without attention, at the trunk output before the classifier; for RTM and PM with attention as well as RTMG and PMG, at the output of the attention block dedicated to the task. Overall, attention better guides focus toward relevant regions (sky, clouds).

Figure 14: t - SNE of embeddings for Ground Condition (50,388 images). Legend (colors as shown in the figure): Dry , Wet , Partly Wet , Partly Snow , Wet and Snowy , Unknown . Subfigures: (a) RTM without attention, (b) RTM with attention, (c) RTMG. Attention strengthens inter-class separation (RTM with attention, RTMG), with coherent semantic proximities ( Wet and Snowy between Wet and Snowy ).

Figure 15: t - SNE of embeddings for Weather Intensity (50,388 images). Legend (colors as shown in the figure): Low , Average , High , None . Subfigures: (a) PM without attention, (b) PM with attention, (c) PMG. Attention variants improve separation; PMG produces more compact clusters and sharper boundaries.

Model	Image2Weather (F1)	MWI (F1)	Kaggle (F1)
RTM (truncated, no att.)	0.8014	0.8922	0.9660
RTM (truncated, with att.)	0.8109	0.9033	0.9800
RTMG	0.8153	0.8936	0.9730
PM (no att.)	0.7721	0.8387	0.9208
PM (with att.)	0.7567	0.8690	0.9389
PMG	0.7917	0.8774	0.9527

Table 3: Zero-shot generalization on external datasets not seen during training (F1). Evaluation limited to the Weather Type task. External labels are harmonized to {sun, fog, rain, snow}: sun

\leftarrow

{Clear, Sun and Clear}, fog

\leftarrow

{Fog, Fog and Rain, Fog and Snow}, rain

\leftarrow

{Rain}, snow

\leftarrow

{Snow}.

Figure 16: Example images from external datasets (unseen during training).

Figure 18: Example images from the external WEAPD dataset.

MWI (20k images) [ 57 ] — metric: Accuracy
RTM	Acc = 0.9731	classifier only
PMG	Acc = 0.9544	classifier only
SLM3 (2024) [ 58 ]	Acc = 0.9001	published value
WEAPD (6,877 images) [ 59 ] — metric: F1
RTM	F1 = 0.9002	classifier only
PMG	F1 = 0.7874	classifier only

Table 4: Fine-tuning per benchmark (encoder and attention frozen, classifier retrained, 30 epochs). Training on 80% of the data, validation 10%, test 10%. Metrics follow each reference (Accuracy or F1). Best score per block in bold.

Image Classification

Image classification assigns semantic labels to images, enabling recognition of objects, scenes, conditions, and fine-grained categories.

See all 390 papers

Style-Based Neural Architectures for Real-Time Weather Classification
Apr 20, 2026Hamed Ouattara, Pascal Houssam Salmane, Pierre Duthon +2Convolutional Neural Networks Image Classification
Cerema, Intelligent Transportation Systems Research Team, France · Institut Pascal, France
WeatherRemover: All-in-one Adverse Weather Removal with Multi-scale Feature Map Compression
Apr 8, 2026Weikai Qu, Sijun Liang, Cheng Pan +6All-in-One Image Restoration
Guangdong University of Technology · Sanda University · University of Queensland +5
WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models
Mar 18, 2026Wanjun Du, Zifeng Yuan, Tingting Chen +3VLM Robustness VLM Reasoning
Beijing Jiaotong University · National University of Singapore · Monash University

Apr 20, 2026J/K move · Enter open · S save

Style-Based Neural Architectures for Real-Time Weather Classification

Hamed Ouattara, Pascal Houssam Salmane, Pierre Duthon, Frédéric Bernardin+1

Cerema, Intelligent Transportation Systems Research Team, France · Institut Pascal, France

Convolutional Neural Networks Image Classification

Open paper AskPDF arXiv

In this paper, we present three neural network architectures designed for real-time classification of weather conditions (sunny, rain, snow, fog) from images. These models, inspired by recent advances in style transfer, aim to capture the stylistic elements present in images. One model, called "Multi-PatchGAN", is based on PatchGANs used in well-known architectures such as Pix2Pix and CycleGAN, but here adapted with multiple patch sizes for detection tasks. The second model, "Truncated ResNet50", is a simplified version of ResNet50 retaining only its first nine layers. This truncation, determined by an evolutionary algorithm, facilitates the extraction of high-frequency features essential for capturing subtle stylistic details. Finally, we propose "Truncated ResNet50 with Gram Matrix and Attention", which computes Gram matrices for each layer during training and automatically weights them via an attention mechanism, thus optimizing the extraction of the most relevant stylistic expressions for classification. These last two models outperform the state of the art and demonstrate remarkable generalization capability on several public databases. Although developed for weather detection, these architectures are also suitable for other appearance-based classification tasks, such as animal species recognition, texture classification, disease detection in medical imaging, or industrial defect identification.

Abstract

Image Classification

Explore similar work

Style-Based Neural Architectures for Real-Time Weather Classification

WeatherRemover: All-in-one Adverse Weather Removal with Multi-scale Feature Map Compression

WeatherReasonSeg: A Benchmark for Weather-Aware Reasoning Segmentation in Visual Language Models