WeatherRemover: All-in-one Adverse Weather Removal with Multi-scale Feature Map Compression
Authors: Weikai Qu, Sijun Liang, Cheng Pan, Zikuan Yang, Guanchi Zhou, Xianjun Fu, Bo Liu, Changmiao Wang, Ahmed Elazab
Organizations: Guangdong University of Technology · Sanda University · University of Queensland · Shenzhen MSU-BIT University · Zhejiang college of Security Technology · Northwest China Research Institute of Electronic Equipment · Shenzhen Research Institute of Big Data · Shenzhen University and Misr Higher Institute for Commerce and Computers
Photographs taken in adverse weather conditions often suffer from blurriness, occlusion, and low brightness due to interference from rain, snow, and fog. These issues can significantly hinder the performance of subsequent computer vision tasks, making the removal of weather effects a crucial step in image enhancement. Existing methods primarily target specific weather conditions, with only a few capable of handling multiple weather scenarios. However, mainstream approaches often overlook performance considerations, resulting in large parameter sizes, long inference times, and high memory costs. In this study, we introduce the WeatherRemover model, designed to enhance the restoration of images affected by various weather conditions while balancing performance. Our model adopts a UNet-like structure with a gating mechanism and a multi-scale pyramid vision Transformer. It employs channel-wise attention derived from convolutional neural networks to optimize feature extraction, while linear spatial reduction helps curtail the computational demands of attention. The gating mechanisms, strategically placed within the feed-forward and downsampling phases, refine the processing of information by selectively addressing redundancy and mitigating its influence on learning. This approach facilitates the adaptive selection of essential data, ensuring superior restoration and maximizing efficiency. Additionally, our lightweight model achieves an optimal balance between restoration quality, parameter efficiency, computational overhead, and memory usage, distinguishing it from other multi-weather models, thereby meeting practical application demands effectively. The source code is available at https://github.com/RICKand-MORTY/WeatherRemover.
Figures & tables
Fig. 1: Overview of WeatherRemover structure. The WeatherRemover model is structured around a UNet architecture. The left side of the model is dedicated to downsample, incorporating a gating mechanism to manage the flow of information, delineated as three D-Stages. The gating mechanism in D-Stage employs a dual-branch structure, where one branch is activated via the GELU function and both branches are fused through element-wise multiplication. The right side performs upsampling, merges features from matching downsampling layers, and includes three U-Stages. Each U-Stage fuses outputs from the previous layer and matching D-Stage along channels, then applies a 1 × 1 convolution to halve the channels. Each stage utilizes MS-PVT, halving spatial size and doubling channels during downsampling, and reversing this during upsampling.
Fig. 2: The MSA architecture. Within the MSA, the input matrix passes through a 1×1 convolution layer and a depthwise separable convolution to obtain the query. Next, adjust the dimensions of the query to adapt the computation of multi-head attention (”HEADS” represents the number of attention heads). For the generation of the key and value, the input matrix is first compressed by linear SRA, reducing the dimensions from B×D×H×W to B×D×P×P . It then passes through a 1×1 convolution layer and a depthwise separable convolution and is finally split along the channel dimension to obtain the key and value, which are adjusted to have the same dimensions as the query. The query, key, and value are then used to compute attention scores, which are scaled by a learnable parameter.
Fig. 3: The linear SRA flowchart for compressing feature maps. The primary component is an adaptive mean pooling layer. It compresses the tensor from dimensions H×W to P×P , where P represents the size of the feature map output from the adaptive average pooling layer. The input tensor, with an initial dimension of B×D×H×W , is transformed to B×D×P×P after passing through the adaptive mean pooling layer. Subsequently, it is processed by a 1×1 convolutional layer and normalized. Finally, the tensor is activated by an activation function.
Fig. 4: Comparison between PVTv2 and GFN structures. (a) depicts the structure of PVTv2, and (b) illustrates our proposed structure GFN. PVTv2’s serial FN structure was modified with a gating mechanism: the original single-path 1x1 and depthwise separable convolution were split into dual paths, one activated, then fused via element-wise multiplication.
Fig. 5: Overall structure of MS-PVT. MS-PVT is composed of MSA and GFN in series. The outputs of both MSA and GFN are normalized and added to the input. On the right side of both GFN and MSA, there are overall descriptions of the internal data flow. For an input with dimensions B×D×H×W , the output also has dimensions B×D×H×W .
Fig. 6: Desnowing visual comparisons. From left to right, the sequence of images includes the input images, Restormer [ 6 ] , DRSformer [ 11 ] , our model, and the ground truth images. Each image is accompanied by magnified details beneath it for closer examination. Images are selected from the Snow100K-L [ 27 ] test set, focusing on scenes where the background color closely resembles the snowflake color.
Fig. 7: Real-scene desnowing visual comparisons. The figures from left to right are: Input images, Restormer [ 6 ] , DRSformer [ 11 ] , ConvIR-B [ 16 ] , and our model. The zoomed-in details are shown below each image. Images were selected from the DAWN [ 30 ] dataset. The first two rows demonstrate the models’ ability to restore fine details in real-world scenarios, while the third row highlights the performance of the models in removing heavy snow in low-resolution settings, aiming to reveal our limitations under such conditions.
Task
Model
PSNR ↑
SSIM ↑
Time/s
Param
Snow Removal
SnowDiff 64 [ 18 ]
30.43
0.9145
>10
82.9M
SnowDiff 128 [ 18 ]
30.28
0.9000
>10
85.6M
Restormer [ 6 ]
31.48
0.9286
0.15
26.5M
DRSformer [ 11 ]
31.06
0.9100
0.21
33.6M
LMQFormer [ 17 ]
31.24
0.9100
0.02
2.18M
ConvIR-S [ 16 ]
30.49
0.9290
0.02
5.53M
TABLE I: Desnowing results on Snow100K-L.
Fig. 8: Raindrop removal visual comparisons. From left to right, the images are: Input images, Restormer [ 6 ] , DRSformer [ 11 ] , I 2 R 2 Net [ 35 ] , our model, and ground truth images. Each image includes magnified details below. The images are from the RainDrop [ 28 ] dataset’s test set. The first row showcases restoration on backgrounds resembling rain droplet colors, the second highlights dense droplet removal, and the third contrasts models’ impacts on texture and detail preservation.
Task
Model
PSNR ↑
SSIM ↑
Time/s
Param
Raindrop Removal
AttentiveGAN [ 28 ]
31.59
0.9170
3.21
6.2M
DuRN [ 14 ]
31.24
0.9259
0.01
10.2M
RaindropAttn [ 36 ]
31.44
0.9263
0.19
7.3M
IDT [ 37 ]
31.87
0.9313
1.05
16.4M
RainDropDiff 64 [ 18 ]
32.29
0.9422
>10
82.9M
RainDropDiff 128 [ 18 ]
32.43
0.9334
>10
85.6M
TABLE II: Raindrop removal results on RainDrop-A.
Fig. 9: Single-weather deraining and dehazing visual comparisons. From left to right, the images displayed are: The original input images, Restormer [ 6 ] , DRSformer [ 11 ] , I 2 R 2 Net [ 35 ] , our proposed model, and the ground truth images. Below each image, magnified details are provided to emphasize the distinctions in performance. The images are from the Outdoor-Rain [ 29 ] dataset. The first and third rows demonstrate the removal of rain haze from the sky, while the second row compares the impact of different models on preserving the original image details.
Fig. 10: Real-scene rain and dense fog removal visual comparisons. From left to right: Input image, Restormer [ 6 ] , DRSformer [ 11 ] , I 2 R 2 Net [ 35 ] , and our model. To show the fineness of the restoration, there are enlarged details below each image. Images with extreme dense fog from the DAWN [ 30 ] dataset were selected to demonstrate the effectiveness of different models in fog removal and their capability to restore image details.
Task
Model
PSNR ↑
SSIM ↑
Time/s
Param
Deraining & Dehazing
HRGAN [ 29 ]
21.56
0.8550
0.28
40.6M
PCNet [ 38 ]
26.19
0.9015
0.02
0.6M
MPRNet [ 39 ]
28.03
0.9192
0.02
3.6M
RainHazeDiff 64 [ 18 ]
28.38
0.9320
>10
82.9M
RainHazeDiff 128 [ 18 ]
26.84
0.9152
>10
85.6M
Restormer [ 6 ]
31.28
0.9252
0.16
26.5M
TABLE III: Results of deraining and dehazing on the Outdoor-Rain dataset.
Gating
Dataset
GFN
D-Stage
PSNR ↑
SSIM ↑
Time/s
Param
31.1566
0.9164
0.0805
20.10M
✓
32.0181
0.9250
0.0979
23.76M
✓
31.4092
0.9184
0.0944
20.56M
Snow100K
✓
✓
32.2699
0.9268
0.1161
24.32M
32.2264
0.9390
0.0967
20.10M
TABLE IV: Ablation experiments of the gating mechanisms within GFN and D-Stage. Ablated by removing the element-wise multiplications and branches.
Dataset
Method
PSNR ↑
SSIM ↑
Time/s
Param
1 × 1Conv
30.3039
0.9075
0.1085
24.10M
DWConv
31.8930
0.9230
0.1069
18.87M
DWConv-1 × 1Conv
31.8597
0.9223
0.1151
27.94M
Snow100K
1 × 1Conv-DWConv
32.2699
0.9268
0.1161
24.32M
1 × 1Conv
32.5468
0.9418
0.1271
24.10M
DWConv
32.6839
0.9431
0.1272
18.87M
TABLE V: Ablation study results on different datasets using 1×1 convolution and depthwise separable convolution for generating query, key, and value in the MS-PVT.
Dataset
SRA
GFN
PSNR ↑
SSIM ↑
Time/s
Param
31.7547
0.9232
0.0808
18.72M
✓
31.8923
0.9234
0.0962
20.56M
✓
32.1415
0.9267
0.1022
22.49M
Snow100K
✓
✓
32.2699
0.9268
0.1161
24.32M
32.4960
0.9421
0.1027
18.72M
✓
32.6308
0.9409
0.1210
20.56M
TABLE VI: Ablation Study of Linear SRA and GFN on Different Datasets.
MACs/G
Model
RainDrop-A
Snow100K-L
Restormer [ 6 ]
743.5
627.8
DRSformer [ 11 ]
805.3
981.4
TransWeather [ 1 ]
32.6
28.7
LMQFormer [ 17 ]
120.8
91.1
WeatherRemover (Ours)
377.2
301.8
TABLE VII: Comparison of Transformer-based model inference computation on each dataset. Using MACs as a metric.
Model
Memory (MiB)
Param
DRSformer [ 11 ] (w/o MEFC)
140.9
32.6M
DRSformer [ 11 ]
147.8
33.6M
Restormer [ 6 ]
114.2
26.5M
TransWeather [ 1 ]
161.6
38.1M
LMQFormer [ 17 ]
27.1
2.2M
WeatherRemover (Ours)
109.6
24.3M
TABLE VIII: Comparison of Transformer-based model video memory consumption, measured in MiB.
Snow100K
RainDrop
Outdoor-Rain
Model
T4
4090
T4
4090
T4
4090
Restormer [ 6 ]
0.66
0.07
0.99
0.11
0.78
0.08
TransWeather [ 1 ]
0.38
0.04
0.05
0.01
0.05
0.00
DRSformer [ 11 ]
1.06
0.11
2.73
0.30
0.77
0.02
WeatherRemover (Ours)
0.39
0.05
0.66
0.07
0.54
0.05
TABLE IX: Inference Time Comparison of Transformer-Based Multi-Weather Models.
Fig. 11: Model performance vs. resolution. Select raindrops as an example to measure memory usage during inference while disabling gradients.
Model
MACs/M
Memory/MiB
Param
Restormer [ 6 ]
58.06
3.59
189
DRSformer [ 11 ]
318.27
3.59
1036
TransWeather-E [ 1 ]
27.65
11.64
105
TransWeather-D [ 1 ]
55.3
11.64
207
WeatherRemover (Ours)
49.77
3.52
189
TABLE X: Performance comparison of different feed-forward networks.
Adverse weather removal (AWR) in real-world images remains challenging due to heterogeneous and unseen degradations, while distortion-driven training often yields overly smooth results. We propose PVRF, a unified framework that integrates zero-shot soft weather perceptions with velocity-constrained rectified-flow refinement. PVRF introduces an AWR-specific question answering module (AWR-QA) that uses frozen vision--language models (VLMs) to estimate soft probabilities of weather types and low-level attribute scores. These perceptions condition restoration networks via attribute-modulated normalization (AMN) and weather-weighted adapters (WWA), producing an anchor estimate for refinement. We then learn a terminal-consistent residual rectified flow with perception-adaptive source perturbation and a terminal-consistent velocity parameterization to stabilize learning near the terminal regime. Extensive experiments show that PVRF improves both fidelity and perceptual quality over state-of-the-art baselines, with strong cross-dataset generalization on single and combined degradations. Code will be released at https://github.com/dongw22/PVRF.
Wei Dong, Han Zhou, Terry Ji +6
McMaster University · Shanghai Jiao Tong University
In this work, we present our winning solution for the 8th UG2+ Challenge (CVPR 2026) Track 1: Image Restoration under All-weather Conditions. Our method is built upon the X-Restormer baseline, which captures both channel-wise global dependencies and spatially-local structural information through its dual-attention design (Multi-DConv Head Transposed Attention and Overlapping Cross-Attention), augmented with the spatially-adaptive input scaling mechanism from Restormer-Plus. We adopt a two-stage training strategy with dual-model ensemble inference. In the first stage, Model B is trained from scratch on a large-scale diverse dataset randomly sampled from the FoundIR training set (approximately 800 GB out of 4.84 TB), covering five degradation types: blur, haze, rain, snow, and composite conditions such as co-occurring rain and haze. In the second stage, Model A is fine-tuned on the WeatherStream dataset (rain and snow splits) using Model B's final checkpoint as pretrained initialization, enabling efficient domain adaptation with a substantially smaller dataset. To better preserve structural details during training, we propose a novel Gradient-Guided Edge-Aware (GGEA) Loss, which applies Sobel operators to the ground-truth image to construct a spatially adaptive weight map that assigns higher supervision to edge and high-frequency regions. This is incorporated alongside L1 and Multi-Scale SSIM losses in a unified training objective. At inference time, predictions from the two models are fused via a weighted average, out = 0.4 x outA + 0.6 x outB, where the higher weight assigned to Model B reflects its stronger generalization ability from large-scale pretraining. With these strategies, our proposed method successfully ranks 1st in the challenge.
Adverse weather causes diverse and complex image degradations, severely compromising the reliability of computer vision systems. Existing all-in-one restoration models attempt to address multiple degradation types within a unified framework, but often lack explicit spatial and semantic modeling of degradation characteristics, limiting their adaptability to diverse weather conditions. To address this limitation, we propose a Degradation-Aware Cross-Modal Prompt Compensation Network (DCMPC-Net) that leverages cross-modal degradation cues from a pretrained vision-language model to condition restoration features within a unified backbone. Specifically, our DCMPC-Net mainly consists of the Cross-Modal Prompt Generator (CMPG), Prompt-Guided Attention Alignment Module (PGAAM), and Dual Feature Compensation Module (DFCM). The CMPG integrates textual embeddings with visual features to produce degradation-aware prompts that encode degradation-related semantic and contextual cues. These prompts are injected into the decoder via a PGAAM, which adaptively aligns semantic information with degraded regions to facilitate context-aware restoration. To further enhance structural fidelity, DFCM is introduced that disentangles degradation artifacts from scene structures, thereby improving the reconstruction of fine textures and detailed content. By integrating cross-modal semantic guidance with spatial alignment and structural enhancement, DCMPC-Net achieves robust and perceptually consistent restoration across diverse weather conditions. Extensive experiments show that DCMPC-Net outperforms state-of-the-art methods in both task-specific and unified settings, achieving superior accuracy and visual fidelity.
Wanshu Fan, Yunzhe Zhang, Yue Shen +5
School of Software Engineering, Dalian University, Dalian, China · School of Mathematical Sciences, Dalian University of Technology, Dalian, China · The Hong Kong Polytechnic University, Hong Kong, China +2