Selective Generative Restoration for Video Deraining via Uncertainty-Aware Diffusion
Authors: Yi'ang Chen, Han Li, Laibin Chang, Hui Liu, Kui Jiang, Junjun Jiang
Organizations: School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Computer and Artificial Intelligence, Zhengzhou University, Zhengzhou 450001, China · Meituan Group, Shanghai 200000, China
Video deraining is challenging due to rain streaks, occlusions, and atmospheric effects. Existing methods mainly follow a deterministic restoration paradigm, assuming that clean scenes can be uniquely recovered from rainy observations. However, rain degradation is spatially heterogeneous. Reliable regions require faithful reconstruction, whereas severely occluded regions contain ambiguous content that require stronger prior-driven inference. Therefore, conventional deterministic restoration often produces over-smoothed results, while global generative restoration may introduce unnecessary hallucinations. We introduce a Selective Generative Restoration paradigm for video deraining, where generative priors are adaptively activated according to restoration uncertainty. Based on this principle, we propose Selective Generative Video Deraining (SGVD), an uncertainty-aware diffusion framework that enables adaptive fidelity preservation and generative recovery. Specifically, SGVD exploits stochastic diversity between dual one-step diffusion predictions to estimate region-level restoration uncertainty, which determines where observation fidelity should be preserved and where generative inference should be activated. Furthermore, an anchor-guided trajectory consistency mechanism is proposed to regularize stochastic diffusion evolution across frames, reducing temporal generation drift and improving video coherence. To achieve practical efficiency, SGVD adopts one-step diffusion inference while maintaining uncertainty-aware generative capability. Extensive experiments demonstrate that SGVD achieves superior restoration performance, perceptual quality, and temporal consistency compared with state-of-the-art video deraining methods.
Figures & tables
Fig. 1: Motivation of Selective Generative Restoration. Deterministic restoration tends to produce over-smoothed details in severely degraded regions, while global generative restoration may introduce content hallucination in reliable regions. In contrast, the proposed paradigm performs region-adaptive restoration by preserving fidelity in low-uncertainty regions and selectively activating generative recovery in high-uncertainty regions.
Fig. 2: Overall architecture of the proposed SGVD for video deraining. The rainy video is first encoded into the latent space by a frozen VAE encoder. The Uncertainty-Guided Selective Generation (USG) module estimates a spatial uncertainty map from the discrepancy between dual-noise clean latent predictions, and adaptively combines the generative latent with the fidelity-preserving LQ latent. The resulting conditional latent is injected into the frozen DiT backbone through a lightweight trainable adapter. Meanwhile, Anchor-Guided Trajectory Consistency (ATC) regularizes each frame's distance to a shared five-frame anchor across trajectory states, promoting temporal coherence without evaluating this training loss at inference.
Dataset
Metric
DiffIR2VR [ 41 ]
Diff-Plugin [ 42 ]
T3-Diff [ 43 ]
WeatherDiff [ 31 ]
DiffBIR [ 44 ]
ZSVD [ 20 ]
DOVE [ 45 ]
RainMamba [ 11 ]
DeLiVR [ 10 ]
SGVD
NTURain-Syn
PSNR ↑
23.7304
24.1138
25.3339
25.4756
25.9601
25.9677
26.2488
36.3430
36.7115
26.5174
SSIM ↑
0.6850
0.7161
0.8633
0.8110
0.7802
0.8027
0.8534
0.9705
0.9684
0.8810
LPIPS ↓
0.2755
0.1137
0.1350
0.1471
0.1221
0.1937
0.1333
0.0257
0.0281
0.1310
NIQE ↓
4.2206
3.8413
3.4850
3.4752
3.6846
3.7548
4.8571
3.6070
3.5866
3.4224
CLIP-IQA ↑
0.3601
0.3273
0.2877
0.2918
0.2583
0.2452
0.2537
0.2766
0.2666
0.3565
MUSIQ ↑
57.3385
59.4466
57.1062
58.2231
53.3840
53.1633
52.7594
57.2592
56.4605
58.3749
TABLE I: Quantitative comparisons with representative video restoration and task-specific video deraining methods on synthetic and real-world video deraining datasets. The best and second-best results are highlighted in bold and underline , respectively.
Fig. 3: Visual comparisons on synthetic video deraining datasets, including NTURain [ 6 ] and RainSynLight25 [ 18 ] .
Fig. 4: Visual comparisons on real-world rainy videos from the NTURain-Real dataset [ 6 ] .
Method
Object Detection
Object Tracking
mAP 1↑
mAP 2↑
IDF1 ↑
MOTA ↑
MOTP ↑
Rainy Input
39.73
44.69
64.5
28.0
19.9
ZSVD
38.86
43.32
64.8
26.4
20.3
DOVE
41.35
45.87
65.7
30.1
20.6
WeatherDiff
37.82
42.12
63.5
29.5
18.8
DiffBIR
38.06
44.56
65.3
27.9
20.4
TABLE II: Object detection and tracking results on the RVDT benchmark. mAP 1 and mAP 2 denote the detection results obtained by YOLO-v3 and MEGA, respectively. The tracking results are obtained using GTR.
Fig. 5: Visualization of diffusion posterior uncertainty in SGVD. The uncertainty map is estimated from the discrepancy between two stochastic one-step diffusion predictions. Severely degraded regions exhibit stronger uncertainty responses, while reliable observations remain relatively low-uncertainty, indicating that the estimated uncertainty effectively reflects spatial variations in restoration ambiguity.
Metric
NTURain-Syn
RainSynLight25
RainSynComplex25
Spearman Correlation ( ρs ) ↑
0.912
0.910
0.933
TABLE III: Spearman correlation between the estimated uncertainty map and restoration error on synthetic video deraining datasets. Higher values indicate that regions with larger estimated uncertainty tend to exhibit larger restoration errors.
Model
USG
ATC
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
CLIP-IQA ↑
MUSIQ ↑
WE ↓
D1
23.0013
0.7819
0.1899
3.5131
0.3221
57.8211
0.0287
D2
✓
26.2416
0.8732
0.1362
3.5114
0.3487
57.9213
0.0309
D3
✓
25.9347
0.8638
0.1449
3.7052
0.3316
57.2845
0.0225
SGVD
✓
✓
26.5174
0.8810
0.1310
3.4224
0.3565
58.3749
0.0207
TABLE IV: Ablation study of the core components of SGVD on the NTURain-Syn test set. USG mainly improves fidelity and perceptual recovery, whereas ATC primarily improves temporal stability.
Fig. 6: Temporal slice comparison of rainy input, SGVD without ATC, full SGVD, and ground truth (top to bottom). ATC improves structural consistency across frames and reduces temporal artifacts, yielding a lower warping error (WE) .
Metric
Neighboring-frame
Shared-anchor ATC
PSNR ↑
26.3309
26.5174
SSIM ↑
0.8763
0.8810
LPIPS ↓
0.1257
0.1310
NIQE ↓
3.5280
3.4224
CLIP-IQA ↑
0.3671
0.3565
MUSIQ ↑
57.9113
58.3749
TABLE V: Temporal regularization comparison on NTURain-Syn. Best results are bold.
Method
Steps
Time (ms/frame)
FPS
FLOPs (G/frame)
Total (M)
Trainable (M)
SGVD
1
296
3.37
21548
7298.47
71.67
SGVD
30
2078
0.48
226611
7298.47
71.67
SGVD w/o probes
1
133
7.52
11372
7298.47
71.67
TABLE VI: End-to-end SGVD efficiency with and without uncertainty probing on 49-frame inputs at 640×480 resolution.
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
CLIP-IQA ↑
MUSIQ ↑
WE ↓
Start Only
19.8903
0.5467
0.2796
3.3830
0.3528
57.8211
0.0256
Condition Only
17.0639
0.6489
0.3435
3.7909
0.1627
42.4441
0.0221
Both (SGVD)
26.5174
0.8810
0.1310
3.4224
0.3565
58.3749
0.0207
TABLE VII: Effect of LQ latent utilization strategies on the NTURain [ 6 ] synthetic dataset.
Number of Probes
PSNR ↑
LPIPS ↓
NIQE ↓
MUSIQ ↑
WE ↓
w/o Uncertainty
25.6821
0.1478
3.8126
56.9821
0.0318
Single Probe
26.0712
0.1404
3.6468
57.5316
0.0284
Two Probes
26.5174
0.1310
3.4224
58.3749
0.0207
Four Probes
26.4938
0.1332
3.4517
58.1263
0.0205
TABLE VIII: Effect of the number of stochastic posterior probes used for uncertainty estimation. The two-probe configuration achieves the best overall accuracy–quality–efficiency trade-off.
Variant
λFM
λCM
λATC
PSNR ↑
SSIM ↑
LPIPS ↓
MUSIQ ↑
WE ↓
Group1
1.0
1.0
0.05
26.2847
0.8746
0.1328
57.9482
0.0231
Group2
1.0
1.0
0.10
26.5174
0.8810
0.1310
58.3749
0.0207
Group3
1.0
1.5
0.15
26.4263
0.8789
0.1358
58.1637
0.0220
Group4
1.0
0.5
0.50
26.3518
0.8767
0.1341
58.0126
0.0214
TABLE IX: Sensitivity analysis of the weighting coefficients for flow matching, one-step consistency, and anchor-guided trajectory consistency.
Prompt
PSNR ↑
SSIM ↑
LPIPS ↓
NIQE ↓
CLIP-IQA ↑
MUSIQ ↑
WE ↓
Original
26.5174
0.8810
0.1310
3.4224
0.3565
58.3749
0.0207
Short
26.3683
0.8740
0.1359
3.4231
0.3433
57.9668
0.0211
Empty
26.3680
0.8743
0.1368
3.4251
0.3467
58.2029
0.0211
TABLE X: Text-conditioning comparison on NTURain-Syn. Best results are bold.
Metric
25 frames
49 frames
81 frames
PSNR ↑
26.2957
26.5174
26.6539
SSIM ↑
0.8784
0.8810
0.8897
LPIPS ↓
0.1339
0.1310
0.1316
NIQE ↓
3.4480
3.4224
3.3489
CLIP-IQA ↑
0.3497
0.3565
0.3609
MUSIQ ↑
58.1333
58.3749
58.6926
TABLE XI: Input-length evaluation using a model trained on 49-frame clips. Best results are bold.
Fig. 7: Restoration examples in challenging scenes. From left to right: rain streak map, uncertainty overlay, and reconstruction error.
School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China