Organizations: Huanjiang Laboratory, Zhuji, P.R. China and School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, P.R.China · School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, P.R.China · School of Mechanical Engineering, Suzhou University of Science and Technology, Suzhou, P.R.China
Raindrops can cause occlusion and distortion in the background scenes due to their adherence to windows or camera lenses. Existing raindrop removal methods concentrate on designing sophisticated CNN or Transformer architectures to recover distorted and missing texture. In this paper, we try to integrate location information and physical model into off-the-shelf CNN or Transformer architectures to help improve their performance. Specifically, we notice that existing methods deploy a preprocessing sub-network to generate a binary or soft mask to indicate the raindrop location, which will increase the network parameters and computational complexity. In contrast, a location-aware learning branch is embedded to teach the encoder in the training phase with the capability of perceiving the position of the raindrops. Note that this location-aware learning branch can be removed during the inference process (achieving performance improvements at no cost). Furthermore, instead of directly reconstructing the raindrop-free image (i.e., background scene), we devise a physics-based reconstruction scheme to first learn the transparency matrix and the raindrop layer. The latent background layer is then reversely derived based on the physical model. By combining the above-mentioned components, we propose our location-aware learning and physics-based reconstruction (LLPR) framework for this challenging ill-posed problem. We also collect a real-world raindrop-degraded image dataset, which is challenging for single-image raindrop removal (SIRR) methods. Extensive experimental results demonstrate the effectiveness and generality of our LLPR framework, achieving superior performance against state-of-the-art SIRR methods. The code will be made available upon acceptance.
Figures & tables
Fig. 1: Top: Schematic diagram of different settings. E, D, and L denote encoder, decoder, and location, respectively. Bottom: Comparison of raindrop removal performance. This is a photograph of a billboard taken through an automobile’s windshield by an iPhone in a real-world rainy day (wild scene).
Fig. 2: PSNR vs. number of parameters for various algorithms. We compare our LLPR framework with some state-of-the-art raindrop removal methods. The results are measured on Test-a dataset.
Fig. 3: The diagram of our location-aware learning and physics-based reconstruction (LLPR) framework. It mainly consists of a four-level U-net-like backbone. The rectangular boxes with background colors represent the core modules.
Fig. 4: Image samples of Test-wild dataset which are captured under actual rainy weather conditions with diverse scenarios.
Setting
DRSformer
Restormer
PSNR
SSIM
PSNR
SSIM
(1)
Baseline
32.42
0.9401
32.28
0.9388
(2)
(1)+PR
32.55
0.9410
32.45
0.9399
(3)
(2)+LL
32.72
0.9416
32.55
0.9403
(4)
(3)+ Lcycle
32.80
0.9420
32.60
0.9410
TABLE I: Ablation study of the key components (physics-based reconstruction scheme, location-aware learning branch, and cycle loss function Lcycle ) of our LLPR framework.
Fig. 5: SSIM vs. number of iterations during the training phase. We record the SSIM values at certain iteration interval on Test-a.
Baseline
+LL
+LL-E1
+LL-E1234
PSNR
32.42
32.56
32.50
32.22
SSIM
0.9401
0.9408
0.9403
0.9378
TABLE II: Ablation study of connecting the location-aware learning branch to different positions. Ei denotes level i of the encoder. The quantitative results are measured upon Test-a.
Baseline
+LL(Diff)
+LL(Binary)
PSNR
32.42
32.56
32.13
SSIM
0.9401
0.9408
0.9401
# Param. (Tr.)
2,069,778
2,145,234
2,144,082
# Param. (Te.)
2,069,778
2,069,778
2,069,778
TABLE III: Ablation study of different supervisions on the learned location map M . The # Param. (Tr./Te.) is obtained by statistical analysis of the network structure during the training/testing phase.
Fig. 6: Visualization of learned masks and results of +LL(diff) and +LL(binary). Since the learned mask of +LL(diff) contains negative values, we adopt absolute values for better visualization.
Fig. 7: The user study result on our Test-wild dataset. We compare our LLPR(DRSformer) with 4 competitors.
Baseline
+LL
+AGAN style
PSNR
32.42
32.56
32.50
SSIM
0.9401
0.9408
0.9401
# Param. (Tr.)
2,069,778
2,145,234
2,237,731
# Param. (Te.)
2,069,778
2,069,778
2,237,731
Memory (GB)
∼ 8
∼ 8
∼ 12
TABLE IV: Ablation study of different location learning styles. The # Param. (Tr./Te.) is obtained by statistical analysis of the network structure during the training/testing phase.
Type
Method
Venue
Test-a
Test-b
Overhead
PSNR
SSIM
PSNR
SSIM
# Param. (M)
# FLOPs (G)
Restoration
DuRN [ 11 ]
CVPR’19
31.24
0.9259
25.32
0.8173
10.18
46.46
MAXIM [ 12 ]
CVPR’22
31.87
0.9352
25.74
0.8270
14.10
216.00
Histoformer [ 13 ]
ECCV’24
33.06
0.9441
-
-
-
-
Multi-task
All-in-One [ 18 ]
CVPR’20
31.12
0.9268
-
-
44.00
-
TransWeather [ 19 ]
CVPR’22
30.17
0.9157
-
-
31.00
-
TABLE V: Benchmark results of various raindrop removal methods on Test-a and Test-b datasets. We report PSNR, SSIM, number of parameters (# Param.), and number of floating-point operations (# FLOPs) to perform comprehensive comparisons. Note that # FLOPs is measured on color images with 256×256 resolution. Bold represents the state-of-the-art performance.
Fig. 8: Visual comparisons of various methods on Test-a dataset. Please zoom in on screen for a better view.
Fig. 9: Visual comparisons of various methods on Test-b dataset. Please zoom in on screen for a better view.
NIQE ↓
MUSIQ ↑
PaQ-2-PiQ ↑
HyperIQA ↑
Input
4.758
59.91
66.66
0.5659
AGAN [ 7 ]
5.116
63.28
70.13
0.6226
DuRN [ 11 ]
4.630
63.80
70.67
0.6186
IDT [ 23 ]
3.334
64.87
70.95
0.6344
Ours
3.283
65.24
71.09
0.6514
TABLE VI: Quantitative results of competing methods on Test-wild in terms of no-reference metrics. ↑ denotes that the higher value indicates the better performance. ↓ denotes that the lower value indicates the better performance. Bold represents the state-of-the-art performance.
Fig. 10: Visual comparisons of various methods on our collected Test-wild dataset. Please zoom in on screen for a better view.
Huanjiang Laboratory, Zhuji, China · School of Aeronautics and Astronautics, Zhejiang University, Hangzhou, China · Hangzhou Institute of Technology, Xidian University, Hangzhou, China +1