Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Computer Vision and Pattern Recognition Laboratory, School of Engineering Science, Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland
Although diffusion-based zero-shot image restoration and enhancement methods have achieved great success, applying them to video restoration or enhancement will lead to severe temporal flickering. In this paper, we propose to systematically utilize both homologous and heterogeneous video diffusion models to assist the image-based method in maintaining more temporal consistency for zero-shot video restoration and enhancement. We propose homologous latents fusion, heterogenous latents fusion, and a CoQ fusion ratio strategy to utilize both homologous and heterogenous text-to-video diffusion models to complement the image method. Moreover, we propose temporal-strengthening post-processing to utilize the image-to-video diffusion model to further improve temporal consistency. Our method is training-free and can be applied to any diffusion-based image restoration and enhancement methods. Experimental results demonstrate the superiority of the proposed method.
Figures & tables
Fig. 1: Overview of our framework.
Fig. 2: Our CoQ Fusion Ratio Strategy. We utilize video quality metrics as test-time verifiers within searching paths.
Methods
Backbone
PSNR ↑
SSIM ↑
CLIP-IQA ↑
LPIPS ↓
WE ↓
FVD ↓
DOVER ↑
t-LPIPS ↓
VMAF ↑
TDAN (sup.)
-
25.47
0.7359
0.3088
0.1869
0.5096
155.3
7.119
3.65
79.99
BasicVSR++ (sup.)
-
28.12
0.8035
0.4823
0.1387
0.4683
122.5
7.849
1.84
82.75
FMA-Net (sup.)
-
28.29
0.8314
0.5121
0.1357
0.4923
117.6
8.207
1.06
84.56
VRT (sup.)
-
29.58
0.8464
0.5224
0.1221
0.3264
93.1
8.928
0.66
84.08
MIA-SR (sup.)
-
29.63
0.8466
0.5320
0.1219
0.3315
94.6
8.644
0.82
85.21
IART (sup.)
-
29.69
0.8472
0.5329
0.1252
0.3212
92.2
9.105
0.59
85.90
TABLE I: Quantitative comparison with state-of-the-art methods for zero-shot 4 × video super-resolution. The best results are highlighted in bold and the second best results are underlined. The WE and t-LPIPS values have been multiplied by 100.
Fig. 3: Visual quality comparison for zero-shot 4 × video super-resolution. The first and second rows list the results of frames t and t+1 , the third and fourth rows list the results of the corresponding patches. Zoom in for better observation.
Methods
Backbone
PSNR ↑
SSIM ↑
CLIP-IQA ↑
LPIPS ↓
WE ↓
FVD ↓
DOVER ↑
t-LPIPS ↓
VMAF ↑
VRT (sup.)
-
18.98
0.4505
0.2093
0.7155
0.1816
1694.8
7.581
0.86
81.94
PSLD
SDXL
19.69
0.4996
0.1905
0.4546
1.5298
662.0
5.573
3.12
76.02
SVI
-
18.25
0.4839
0.2010
0.5871
1.3463
1218.3
7.202
2.35
79.47
VISION-XL
SDXL
19.82
0.5068
0.2147
0.4232
1.2623
457.7
7.298
1.84
80.66
PSLD+ZVRV
SDXL
20.47
0.5342
0.4813
0.3675
0.1632
326.4
8.456
0.67
84.53
TABLE II: Quantitative comparison with state-of-the-art methods for zero-shot video deblurring. The best results are highlighted in bold and the second best results are underlined. The WE and t-LPIPS values have been multiplied by 100.
Methods
Backbone
PSNR ↑
SSIM ↑
CLIP-IQA ↑
LPIPS ↓
WE ↓
FVD ↓
DOVER ↑
t-LPIPS ↓
VMAF ↑
FastLLVE (sup.)
-
10.62
0.6911
0.2524
0.2674
0.1299
1465.2
2.093
15.76
49.33
UDU (unsup.)
-
6.24
0.4346
0.3560
0.3662
0.1048
1618.7
3.682
10.39
60.01
SGZ
-
15.03
0.6987
0.3835
0.1392
0.2454
723.6
5.715
1.10
81.81
PSLD
SDv1.5
20.37
0.8281
0.3086
0.1186
0.6254
547.3
5.102
9.34
65.63
PSLD+ZVRV
SDv1.5
20.69
0.8453
0.3955
0.0930
0.1017
251.9
8.965
0.78
83.22
TABLE III: Quantitative comparison with state-of-the-art methods for zero-shot low-light video enhancement. The best results are highlighted in bold and the second best results are underlined. The WE and t-LPIPS values have been multiplied by 100.
Fig. 4: Visual quality comparison for zero-shot low-light video enhancement. The first and second rows list the results of frames t and t+1 , the third and fourth rows list the results of the corresponding patches. Zoom in for better observation.
Methods
Backbone
PSNR ↑
SSIM ↑
CLIP-IQA ↑
LPIPS ↓
WE ↓
FVD ↓
DOVER ↑
t-LPIPS ↓
VMAF ↑
Upscale-A-Video
SDv2.0
23.76
0.5676
0.7028
0.3114
0.5958
585.3
5.962
1.06
79.49
SeedVR
SDv3.0
22.27
0.5162
0.8268
0.3070
0.5634
717.2
8.739
1.23
83.61
DiT4SR
SDv3.0
23.41
0.6235
0.9283
0.2944
1.8274
917.6
4.305
11.29
73.24
DiT4SR+ZVRV
SDv3.0
23.66
0.6387
0.9363
0.2909
0.5526
839.7
7.821
1.20
82.80
TSD-SR
SDv3.0
25.15
0.6809
0.8702
0.1985
1.0295
357.8
6.464
2.12
77.93
TSD-SR+ZVRV
SDv3.0
25.22
0.6866
0.8761
0.1811
0.4731
329.2
8.983
0.64
84.36
TABLE IV: Quantitative comparison with state-of-the-art methods for 4 × blind video super-resolution on the DAVIS dataset. The best results are highlighted in bold and the second best results are underlined. The WE and t-LPIPS values have been multiplied by 100.
Fig. 5: Visual quality comparison for 4 × blind video super-resolution. The first and second rows list the results of frames t and t+1 , the third and fourth rows list the results of the corresponding patches. Zoom in for better observation.
Methods
Upscale-A-Video
SeedVR
DiffBIR+ZVRV
DiT4SR+ZVRV
TSD-SR+ZVRV
Time (s) ↓
4.12
10.55
61.29
21.83
2.96
TABLE V: Inference Time on GPU 96G H20.
Fusion Ratio Strategy
PSNR ↑
SSIM ↑
CLIP-IQA ↑
LPIPS ↓
WE ↓
FVD ↓
DOVER ↑
t-LPIPS ↓
VMAF ↑
CoQ Fusion Ratio Strategy
M =2, r =0.45
26.74
0.6953
0.8472
0.1649
0.4186
258.5
7.251
0.92
83.42
M =3, r =0.45
27.15
0.7196
0.8599
0.1573
0.3952
240.8
8.854
0.63
85.01
M =4, r =0.45
27.42
0.7388
0.8691
0.1395
0.3755
231.7
9.076
0.41
86.37
M =4, r =0.50
27.20
0.7316
0.8610
0.1432
0.3764
239.5
8.983
0.51
86.04
M =4, r =0.40
27.09
0.7272
0.8623
0.1408
0.3783
234.1
8.779
0.50
85.79
Fixed Fusion Ratio Strategy
26.59
0.6915
0.8347
0.1724
0.4298
265.2
6.451
0.98
82.26
TABLE VI: Ablation study on the CoQ fusion ratio strategy (with different hyperparameters M and r ) and the fixed fusion ratio strategy. The best results are highlighted in bold.
HMLF
×
✓
✓
✓
✓
HTLF
×
×
✓
✓
✓
CFRS
×
×
×
✓
✓
TSPP
×
×
×
×
✓
PSNR ↑
26.50
26.69
27.12
27.35
27.42
SSIM ↑
0.6869
0.6981
0.7201
0.7256
0.7388
CLIP-IQA ↑
0.8340
0.7339
0.7026
0.8446
0.8691
TABLE VII: Ablation study for Homologous Latents Fusion (HMLF), Heterogenous Latents Fusion (HTLF), CoQ Fusion Ratio Strategy (CFRS) and Temporal-Strengthening Post-Processing (TSPP) on 4 × blind video super-resolution task. The WE and t-LPIPS values have been multiplied by 100.
School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Computer Vision and Pattern Recognition Laboratory, School of Engineering Science, Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China · Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden +1