Organizations: MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China · Department of Networked Intelligence, Pengcheng Laboratory, Shenzhen, China · Department of Computer Science and Shenzhen Research Institute, City University of Hong Kong, Kowloon, Hong Kong
Contemporary no-reference image quality assessment (NR-IQA) models can effectively quantify perceived image quality, often achieving strong correlations with human perceptual scores on standard IQA benchmarks. Yet, limited efforts have been devoted to treating NR-IQA models as natural image priors for real-world image enhancement, and consequently comparing them from a perceptual optimization standpoint. In this work, we show -- for the first time -- that NR-IQA models can be plugged into the maximum a posteriori (MAP) estimation framework for image enhancement. This is achieved by performing gradient ascent in the diffusion latent space rather than in the raw pixel domain, leveraging a pretrained differentiable and bijective diffusion process. Likely, different NR-IQA models lead to different enhanced outputs, which in turn provides a new computational means of comparing them. Unlike conventional correlation-based measures, our comparison method offers complementary insights into the respective strengths and weaknesses of the competing NR-IQA models in perceptual optimization scenarios. Additionally, we aim to improve the best-performing NR-IQA model in diffusion latent MAP estimation by incorporating the advantages of other top-performing methods. The resulting model delivers noticeably better results in enhancing real-world images afflicted by unknown and complex distortions, all preserving a high degree of image fidelity.
Figures & tables
Fig. 1: (a) Distorted image used as the initial estimate. (b) Enhancement output obtained by directly maximizing a state-of-the-art NR-IQA model, LIQE [ 11 ] (see Eq. ( 1 )). (c) Enhancement output obtained by MAP estimation (see Eq. ( 2 )), where a reduced-resolution Euclidean distance serves as the fidelity term and the LIQE score [ 11 ] defines the energy-based prior. (d) Enhancement output obtained by diffusion latent MAP estimation (see Eq. ( 17 )). Each image is accompanied by its LIQE-predicted quality score, which ranges from zero to a maximum of five, with higher values indicating better predicted quality.
Fig. 2: Optimized images via MAP estimation (see Eq. ( 2 )) for a range of λ values.
Fig. 3: (a) System diagram of diffusion latent MAP estimation. (b) NR-IQA model comparison and improvement enabled by the proposed diffusion latent MAP estimation framework.
Fig. 4: Representative photographic images used as inputs to the proposed enhancement framework.
NR-IQA Model
Training Set
# Parameters (in Millions)
SRCC Rank (SRCC)
MAP Rank
Δ Rank
NIQE [ 24 ]
–
0.001
8 (0.706)
8
0
CLIPIQA+ [ 40 ]
KonIQ-10k
101.349
2 (0.855)
6
-4
PaQ-2-PiQ [ 33 ]
FLIVE
11.704
5 (0.827)
7
-2
HyperIQA [ 89 ]
KonIQ-10k
27.375
7 (0.776)
4
3
MUSIQ [ 18 ]
KonIQ-10k
27.125
3 (0.853)
2
1
DBCNN [ 27 ]
KonIQ-10k
15.311
6 (0.789)
5
1
TABLE I: Global ranking of eight NR-IQA models, where lower ranks indicate better performance. “Multiple” denotes training on the combined LIVE, CSIQ, BID, CLIVE, KonIQ-10k, and KADID-10k datasets. We define Δ Rank = SRCC Rank − MAP Rank; thus, a positive value indicates a better rank under diffusion latent MAP estimation.
Fig. 5: Global ranking scores of eight NR-IQA models under diffusion latent MAP estimation, where higher scores indicate better performance. Models enclosed by the same colored box are statistically indistinguishable according to a two-tailed z -test at the 5% significance level.
Fig. 6: Visual comparison of MAP-refined images associated with different pairs of NR-IQA models.
Fig. 7: Visual comparison of MAP-refined images using vanilla and ControlNet-augmented Stable Diffusion 2.0.
Fig. 8: (a) Initial image with mixed, realistic degradations. (b) Enhancement output by direct diffusion latent optimization of LIQE [ 11 ] (equivalent to diffusion latent MAP estimation with λ=∞ ), which results in an overall color appearance that undesirably deviates from (a). (c) Enhancement output by the proposed diffusion latent MAP estimation using the same LIQE [ 11 ] .
Fig. 9: Visual comparison of MAP-refined images guided by MS-LIQE and its three predecessor models: LIQE, MUSIQ, and UNIQUE.
Fig. 10: Visual examples of the proposed post-enhancement method applied to task-specific enhancement outputs. From left to right, the columns show two super-resolution (SR) examples, one low-light image enhancement (LLIE) example, and one face restoration (FR) example.
Method
Super Resolution on Real60
Super Resolution on DPED
Face Restoration
MANIQA
Q-Align
VQ-R1
MANIQA
Q-Align
VQ-R1
MANIQA
Q-Align
VQ-R1
SeeSR
0.655 / 0.681
4.46 / 4.69
4.65 / 4.74
0.586 / 0.635
3.00 / 3.62
3.29 / 3.79
0.633 / 0.697
3.96 / 4.50
4.42 / 4.68
SUPIR
0.655 / 0.671
4.46 / 4.77
4.69 / 4.77
0.574 / 0.621
2.70 / 3.40
3.06 / 3.44
0.610 / 0.686
3.68 / 4.46
4.26 / 4.63
StableSR
0.641 / 0.668
3.89 / 4.53
4.51 / 4.73
0.536 / 0.604
2.43 / 3.22
2.72 / 3.56
0.608 / 0.671
3.60 / 4.38
4.11 / 4.60
PiSA-SR
0.688 / 0.703
4.40 / 4.71
4.70 / 4.81
0.617 / 0.658
2.85 / 3.44
3.22 / 3.73
0.663 / 0.701
4.11 / 4.65
4.51 / 4.76
TABLE II: Quantitative comparison of the baseline and post-enhanced outputs on super-resolution and face restoration. Each entry reports baseline / post-enhancement, and higher values indicate better performance for all metrics
Method
MANIQA
Q-Align
VQ-R1
QuadPrior
0.634 / 0.683
3.71 / 4.25
4.12 / 4.44
HVI-CIDNet
0.642 / 0.664
3.64 / 4.18
4.01 / 4.37
RetinexFormer
0.570 / 0.628
3.25 / 3.95
3.63 / 4.03
ISP-UNet
0.643 / 0.678
3.81 / 4.29
4.18 / 4.43
TABLE III: Quantitative comparison of the baseline and post-enhanced outputs on low-light image enhancement
Task
Baseline
Preference (%)
Super-Resolution on Real60
PiSA-SR
85.6
Super-Resolution on DPED
SeeSR
79.6
Face Restoration
PiSA-SR
88.0
Low-light Enhancement
ISP-UNet
76.8
TABLE IV: Percentage of trials in which the post-enhanced output is preferred to the strongest competing baseline for three tasks