Organizations: Huanjiang Laboratory, Zhuji, P.R. China and School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, P.R.China · School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, P.R.China · School of Mechanical Engineering, Suzhou University of Science and Technology, Suzhou, P.R.China
Raindrops can cause occlusion and distortion in the background scenes due to their adherence to windows or camera lenses. Existing raindrop removal methods concentrate on designing sophisticated CNN or Transformer architectures to recover distorted and missing texture. In this paper, we try to integrate location information and physical model into off-the-shelf CNN or Transformer architectures to help improve their performance. Specifically, we notice that existing methods deploy a preprocessing sub-network to generate a binary or soft mask to indicate the raindrop location, which will increase the network parameters and computational complexity. In contrast, a location-aware learning branch is embedded to teach the encoder in the training phase with the capability of perceiving the position of the raindrops. Note that this location-aware learning branch can be removed during the inference process (achieving performance improvements at no cost). Furthermore, instead of directly reconstructing the raindrop-free image (i.e., background scene), we devise a physics-based reconstruction scheme to first learn the transparency matrix and the raindrop layer. The latent background layer is then reversely derived based on the physical model. By combining the above-mentioned components, we propose our location-aware learning and physics-based reconstruction (LLPR) framework for this challenging ill-posed problem. We also collect a real-world raindrop-degraded image dataset, which is challenging for single-image raindrop removal (SIRR) methods. Extensive experimental results demonstrate the effectiveness and generality of our LLPR framework, achieving superior performance against state-of-the-art SIRR methods. The code will be made available upon acceptance.
Figures & tables
Fig. 1: Top: Schematic diagram of different settings. E, D, and L denote encoder, decoder, and location, respectively. Bottom: Comparison of raindrop removal performance. This is a photograph of a billboard taken through an automobile’s windshield by an iPhone in a real-world rainy day (wild scene).
Fig. 2: PSNR vs. number of parameters for various algorithms. We compare our LLPR framework with some state-of-the-art raindrop removal methods. The results are measured on Test-a dataset.
Fig. 3: The diagram of our location-aware learning and physics-based reconstruction (LLPR) framework. It mainly consists of a four-level U-net-like backbone. The rectangular boxes with background colors represent the core modules.
Fig. 4: Image samples of Test-wild dataset which are captured under actual rainy weather conditions with diverse scenarios.
Setting
DRSformer
Restormer
PSNR
SSIM
PSNR
SSIM
(1)
Baseline
32.42
0.9401
32.28
0.9388
(2)
(1)+PR
32.55
0.9410
32.45
0.9399
(3)
(2)+LL
32.72
0.9416
32.55
0.9403
(4)
(3)+ Lcycle
32.80
0.9420
32.60
0.9410
TABLE I: Ablation study of the key components (physics-based reconstruction scheme, location-aware learning branch, and cycle loss function Lcycle ) of our LLPR framework.
Fig. 5: SSIM vs. number of iterations during the training phase. We record the SSIM values at certain iteration interval on Test-a.
Baseline
+LL
+LL-E1
+LL-E1234
PSNR
32.42
32.56
32.50
32.22
SSIM
0.9401
0.9408
0.9403
0.9378
TABLE II: Ablation study of connecting the location-aware learning branch to different positions. Ei denotes level i of the encoder. The quantitative results are measured upon Test-a.
Baseline
+LL(Diff)
+LL(Binary)
PSNR
32.42
32.56
32.13
SSIM
0.9401
0.9408
0.9401
# Param. (Tr.)
2,069,778
2,145,234
2,144,082
# Param. (Te.)
2,069,778
2,069,778
2,069,778
TABLE III: Ablation study of different supervisions on the learned location map M . The # Param. (Tr./Te.) is obtained by statistical analysis of the network structure during the training/testing phase.
Fig. 6: Visualization of learned masks and results of +LL(diff) and +LL(binary). Since the learned mask of +LL(diff) contains negative values, we adopt absolute values for better visualization.
Fig. 7: The user study result on our Test-wild dataset. We compare our LLPR(DRSformer) with 4 competitors.
Baseline
+LL
+AGAN style
PSNR
32.42
32.56
32.50
SSIM
0.9401
0.9408
0.9401
# Param. (Tr.)
2,069,778
2,145,234
2,237,731
# Param. (Te.)
2,069,778
2,069,778
2,237,731
Memory (GB)
∼ 8
∼ 8
∼ 12
TABLE IV: Ablation study of different location learning styles. The # Param. (Tr./Te.) is obtained by statistical analysis of the network structure during the training/testing phase.
Type
Method
Venue
Test-a
Test-b
Overhead
PSNR
SSIM
PSNR
SSIM
# Param. (M)
# FLOPs (G)
Restoration
DuRN [ 11 ]
CVPR’19
31.24
0.9259
25.32
0.8173
10.18
46.46
MAXIM [ 12 ]
CVPR’22
31.87
0.9352
25.74
0.8270
14.10
216.00
Histoformer [ 13 ]
ECCV’24
33.06
0.9441
-
-
-
-
Multi-task
All-in-One [ 18 ]
CVPR’20
31.12
0.9268
-
-
44.00
-
TransWeather [ 19 ]
CVPR’22
30.17
0.9157
-
-
31.00
-
TABLE V: Benchmark results of various raindrop removal methods on Test-a and Test-b datasets. We report PSNR, SSIM, number of parameters (# Param.), and number of floating-point operations (# FLOPs) to perform comprehensive comparisons. Note that # FLOPs is measured on color images with 256×256 resolution. Bold represents the state-of-the-art performance.
Fig. 8: Visual comparisons of various methods on Test-a dataset. Please zoom in on screen for a better view.
Fig. 9: Visual comparisons of various methods on Test-b dataset. Please zoom in on screen for a better view.
NIQE ↓
MUSIQ ↑
PaQ-2-PiQ ↑
HyperIQA ↑
Input
4.758
59.91
66.66
0.5659
AGAN [ 7 ]
5.116
63.28
70.13
0.6226
DuRN [ 11 ]
4.630
63.80
70.67
0.6186
IDT [ 23 ]
3.334
64.87
70.95
0.6344
Ours
3.283
65.24
71.09
0.6514
TABLE VI: Quantitative results of competing methods on Test-wild in terms of no-reference metrics. ↑ denotes that the higher value indicates the better performance. ↓ denotes that the lower value indicates the better performance. Bold represents the state-of-the-art performance.
Fig. 10: Visual comparisons of various methods on our collected Test-wild dataset. Please zoom in on screen for a better view.
Raindrops adhered to camera lens or windshield are inevitable in rainy scenes and can become an issue for many computer vision systems such as autonomous driving. Because raindrop appearance is affected by too many parameters, therefore it is unlikely to find an effective model based solution. Learning based methods are also problematic, because traditional learning method cannot properly model the complex appearance. Whereas deep learning method lacks sufficiently large and realistic training data. To solve it, in our work, we propose the first photo-realistic dataset of synthetic adherent raindrops for training. The rendering is physics based with consideration of the water dynamic, geometric and photometry. The dataset contains various types of rainy scenes and particularly the rainy driving scenes. Based on the modeling of raindrop imagery, we introduce a detection network which has the awareness of the raindrop refraction as well as its blurring. Based on that, we propose the removal network that can well recover the image structure. Rigorous experiments demonstrate the state-of-the-art performance of our proposed framework.
Zhixiang Hao, Shaodi You, Yu Li +2
State Key Laboratory of VR Technology and Systems, Beihang University, Beijing, China · 61-CSIRO · Tencent +2
Adherent raindrops on camera lenses are a pervasive source of real-world image degradation, undermining outdoor visual perception for downstream applications such as robotics and 3D scene reconstruction. However, most existing raindrop datasets and methods target only background-focused 2D images, overlooking the equally common raindrop-focused case and the impact of raindrops on 3D scene reconstruction. Moreover, they are almost exclusively daytime, lacking the complex artificial lighting and reflections of nighttime raindrop scenes. To address these issues, we introduce OmniRaindrop, a real-world dataset and benchmark that jointly evaluates 2D raindrop removal and 3D scene reconstruction across raindrop-focused and background-focused captures, under both daytime and nighttime conditions. OmniRaindrop provides spatially aligned raindrop-focused, background-focused, and rain-free ground-truth image sets: a 2D subset of 15,186 aligned pairs/triplets and a 3D subset of 40 real-world scenes evenly split between day and night, totaling 3,822 multi-view images. On top of this data, the benchmark traces how raindrop degradation propagates from 2D restoration to 3D reconstruction by testing representative restoration methods and their effects on pose estimation, point cloud initialization, and 3DGS optimization. Comprehensive experiments reveal that raindrop-focused scenes are substantially harder than background-focused ones, and that errors in 2D restoration and pose/point cloud initialization cascade into degraded 3D reconstruction, pointing to clear directions for more robust 2D restoration and 3DGS design. Building on the dataset, we have organized two consecutive NTIRE challenges (2025 and 2026) on Day and Night Raindrop Removal for Dual-Focused Images.
Zhiqiang Teng, Tingting Chen, Beibei Lin +4
Beijing Jiaotong University · National University of Singapore
When capturing images through glass surfaces or windshields on rainy days, raindrops and reflections frequently co-occur to significantly reduce the visibility of captured images. This practical problem lacks attention and needs to be resolved urgently. Prior de-raindrop, de-reflection, and all-in-one models have failed to address this composite degradation. To this end, we first formally define the unified removal of raindrops and reflections (UR3) task for the first time and construct a real-shot dataset, namely RainDrop and ReFlection (RDRF), which provides a new benchmark with substantial, high-quality, diverse image pairs. Then, we propose a novel diffusion-based framework (i.e., DiffUR3) with several target designs to address this challenging task. By leveraging the powerful generative prior, DiffUR3 successfully removes both types of degradations. Extensive experiments demonstrate that our method achieves state-of-the-art performance on our benchmark and on challenging in-the-wild images.
Xingyu Liu, Zewei He, Yu Chen +4
Huanjiang Laboratory, Zhuji, China · School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, China · Hangzhou Institute of Technology, Xidian University, Hangzhou, China +1