Enhancing image quality through super-resolution is crucial in image and video processing. While contemporary deep learning models, such as CNNs and Transformers, excel in pixel-level enhancements, they often overlook the rich semantic context within images. To address this, we introduce a novel approach that synergizes pixel intensity with semantic information for superior detail recovery. Our method involves extracting semantic data, including textual descriptors and spatial positioning, and encoding this information. We also present a Cross-Modal Feature Fusion module for generating semantic feature maps via multimodal training, and a Prompt Traction Head to augment the backbone network with semantic insights, thereby refining image details. Our empirical results across various datasets validate the superiority of our approach.
Figures & tables
Figure 1: Comparison of different image super-resolution model architectures.
Figure 2: The comprehensive framework of the proposed method. The snowflakes in the figure denote that the module is in a frozen state, while the flames indicate that the module is undergoing training.
Figure 3: Semantic Information Processing for Different Types of Images.
Figure 4: Structure of Position Mask Attention.
Figure 5: The structural diagram of the Prompt Traction Head, where δ denotes the tanh function and Conv symbolizes the point convolution operation.
Method
Scale
Set5
Set14
BSD100
Urban100
Manga109
ESRGAN
× 2
37.96 / 0.9593
33.87 / 0.9135
31.62 / 0.8902
32.27 / 0.9307
39.56 / 0.9648
PAN
× 2
38.34 / 0.9625
34.36 / 0.9221
32.00 / 0.9001
33.42 / 0.9402
39.84 / 0.9726
RealSR
× 2
38.19 / 0.9622
34.38 / 0.9223
31.98 / 0.8967
33.89 / 0.9395
39.79 / 0.9721
SwinIR
× 2
38.37 / 0.9617
34.42 / 0.9228
32.02 / 0.9000
33.26 / 0.9387
39.76 / 0.9710
HAT
× 2
38.59 / 0.9629
34.47 / 0.9223
32.12 / 0.9009
33.44 / 0.9418
40.02 / 0.9729
Ours
× 2
38.62 / 0.9636
34.55 / 0.9249
32.10 / 0.9024
33.60 / 0.9425
40.14 / 0.9730
Table 1: Comparative Results on Natural Image Datasets. Our method surpasses other techniques in numerous benchmarks, particularly leveraging ESRGAN as the backbone while achieving significant improvements over it.
Figure 6: Visualization of enhanced image details for select test images, accompanied by captions generated by the semantic extraction module. The displayed PSNR values quantify the image quality.
Method
Scale
PSNR (dB)
SSIM
ESRGAN
× 2
25.53
0.9441
PAN
× 2
25.97
0.9489
RealSR
× 2
26.64
0.9563
SwinIR
× 2
29.13
0.9691
HAT
× 2
29.89
0.9743
Ours
× 2
31.05
0.9773
Table 2: Comparative Performance on Document Image Super-Resolution
Figure 7: Visualization and analysis of the Locally Adaptive Modulation (LAM) effect on text detail enhancement in the BDDI dataset. Our approach significantly improves pixel activation in text-heavy areas.
Method
Scale
BSD100
BDDI
Baseline
× 2
32.10 / 0.9024
31.05 / 0.9773
⊘PE
× 2
31.91 / 0.8982
28.71 / 0.9682
⊘CMA
× 2
31.69 / 0.8936
28.03 / 0.9679
⊘MT
× 2
31.62 / 0.8902
25.53 / 0.9441
Baseline
× 3
29.71 / 0.8615
28.79 / 0.9687
⊘PE
× 3
29.58 / 0.8610
27.16 / 0.9604
Table 3: Ablation study results. ⊘PE denotes the exclusion of the position encoding module, ⊘CMA represents the substitution of the content mask attention, and ⊘MT signifies the absence of multimodal training.
Magnitude
Upscaling Factor
BSD100
BDDI
0.0
× 2
31.94 / 0.9003
30.62 / 0.9711
0.3
× 2
32.10 / 0.9024
31.05 / 0.9773
0.5
× 2
31.54 / 0.8895
30.22 / 0.9696
0.7
× 2
31.88 / 0.8936
29.32 / 0.9603
1.0
× 2
31.47 / 0.8872
26.33 / 0.9481
Table 4: Comparative analysis of model performance with varying content area mask magnitudes. The data indicates that a moderate increase in magnitude can enhance model efficacy.
Weight
Scale
BSD100
BDDI
0.0
× 2
31.25 / 0.8879
25.73 / 0.9467
1.0
× 2
26.13 / 0.8122
21.39 / 0.8951
0.1
× 2
29.37 / 0.8890
24.98 / 0.9381
0.01
× 2
31.68 / 0.8883
29.11 / 0.9695
0.001
× 2
32.10 / 0.9024
31.05 / 0.9773
0.0001
× 2
31.89 / 0.8991
30.04 / 0.9711
Table 5: Comparative experiment outcomes based on varying loss weights. "Weight" refers to the semantic similarity loss weight, with pixel loss and visual feature similarity loss weights fixed at 1.0 and 0.0001, respectively.