Organizations: Department of Electrical and Computer Engineering, Tamkang University, No. 151, Yingzhuan Rd., Tamsui District, New Taipei City 251301, Taiwan
Sparse attention mechanisms, which score all token pairs but propagate only the strongest, now underpin the most efficient Transformers for lightweight image super-resolution. This paper observes that sparsification changes what it means to improve such a network. A dense attention layer has one place where representation quality matters: the aggregation of attended features. A sparse layer has two, because the top-k operator first decides which tokens survive and only then decides what to do with them, and a token discarded at the selection stage cannot be recovered downstream. Selection quality and aggregation quality are therefore separable targets, addressed by modules placed before and after the attention respectively. We test this by pairing a dual-branch spatial enhancement on the input of a progressive focused attention with a wavelet-domain modulation on its output, forming SFMformer. Measuring each module alone and jointly over all fifteen benchmark-scale pairs, we find their gains are not additive: the joint gain exceeds the sum of the individual gains on nine pairs, and the sign of the discrepancy is predicted by how much the weaker module contributes on its own (r = -0.72), so the two compound when they relieve different constraints and overlap when they relieve the same one. Enabling spectral modulation once per block rather than once per layer retains the effect at roughly one-sixth of its cost, keeping the model below one million parameters at every scale. SFMformer ranks first on 28 of 30 PSNR/SSIM entries across five benchmarks and three upscaling factors. We report the cases where the pairing does not help, and deploy the model on a Raspberry Pi 5 to confirm the design is practical under tight resource budgets.
Figures & tables
Figure 1: Overall architecture of SFMformer. A 3×3 convolution extracts shallow features, M cascaded SFM Blocks produce deep features, and a long residual connection followed by pixel-shuffle upsampling reconstructs the HR output.
Figure 2: Structure of the Spatial–Frequency Modulation Block (SFMB). Only the final layer (SFML ∗ ) enables the WMA module.
Figure 3: Internal structure of the Spatial–Frequency Modulation Layer (SFML). The dashed region (LayerNorm + WMA) is enabled only in the last layer of each SFMB.
Figure 4: The Dual Feature Extraction (DFE) module. The hourglass branch captures local spatial neighbourhood information; the 1×1 branch preserves channel semantics.
Figure 5: The Progressive Focused Attention (PFA) mechanism. The attention map of the previous layer is inherited, multiplied element-wise with the current similarity map, and sparsified by a top- Kl operation.
Figure 6: The Wavelet-domain Modulation self-Attention (WMA) module. Features are reduced in channel dimension, decomposed by the DWT into four sub-bands, modulated by channel-wise self-attention and dynamic convolution, and returned to the spatial domain by the IDWT.
Method
Scale
Params
FLOPs
Set5
Set14
BSD100
Urban100
Manga109
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Bicubic
×2
—
—
33.66
0.9299
30.24
0.8688
29.56
0.8431
26.88
0.8403
30.80
0.9339
SwinIR-light [ 5 ]
×2
910K
244G
38.14
0.9611
33.86
0.9206
32.31
0.9012
32.76
0.9340
39.12
0.9783
ELAN-light [ 9 ]
×2
582K
203G
38.17
0.9611
33.94
0.9207
32.30
0.9012
32.76
0.9340
39.11
0.9782
SwinIR-NG [ 40 ]
×2
1181K
274.1G
38.17
0.9612
33.94
0.9205
32.31
0.9013
32.78
0.9340
39.20
0.9781
OmniSR [ 20 ]
×2
772K
194.5G
38.22
0.9613
33.98
0.9210
32.36
0.9020
33.05
0.9363
39.28
0.9784
Table 1: Quantitative comparison of SFMformer with state-of-the-art lightweight methods at three upscaling factors. Bold : best; underline : second best; bicubic is listed for reference and excluded from the ranking. † denotes retraining under our configuration. LKMN [ 42 ] and PromptSR [ 43 ] report ×2 results only, and FLOPs are not available for every quoted method.
Figure 7: PSNR on Urban100 versus parameter count at ×4 . The dashed line marks the 1M-parameter boundary conventionally used to delimit lightweight SR models.
Method
Scale
Params
Set5
Set14
BSD100
Urban100
Manga109
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PFT-light [ 14 ]
×2
776K
38.36
0.9620
34.19
0.9232
32.43
0.9030
33.67
0.9411
39.55
0.9792
PFT-light †
×2
776K
38.35
0.9620
34.19
0.9227
32.44
0.9030
33.61
0.9406
39.64
0.9793
SFMformer (ours)
×2
970K
38.43
0.9622
34.28
0.9235
32.48
0.9036
33.73
0.9414
39.82
0.9796
PFT-light [ 14 ]
×3
783K
34.82
0.9305
30.75
0.8493
29.33
0.8116
29.46
0.8763
34.61
0.9509
PFT-light †
×3
783K
34.82
0.9306
30.75
0.8490
29.34
0.8118
29.37
0.8752
34.68
0.9511
Table 2: Results at a batch size of 32. PFT-light † was retrained by us under settings identical to SFMformer; the PFT-light rows without † quote the original paper [ 14 ] . Bold : best; underline : second best.
Method
DFE
WMA
Scale
Params
Set5
Set14
BSD100
Urban100
Manga109
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PFT-light † (baseline)
×2
776K
38.33
0.9618
34.06
0.9218
32.41
0.9026
33.36
0.9385
39.50
0.9790
+ DFE
✓
×2
890K
38.35
0.9620
34.15
0.9221
32.43
0.9029
33.47
0.9398
39.60
0.9791
+ WMA
✓
×2
856K
38.36
0.9619
34.05
0.9220
32.42
0.9028
33.37
0.9382
39.65
0.9793
+ DFE + WMA (SFMformer)
✓
✓
×2
970K
38.40
0.9620
34.20
0.9227
32.45
0.9032
33.53
0.9397
39.73
0.9794
PFT-light † (baseline)
×3
783K
34.78
0.9303
30.70
0.8481
29.32
0.8122
29.22
0.8728
34.57
0.9505
Table 3: Ablation study of the DFE and WMA modules at all three upscaling factors.
Dataset
Scale
ΔDFE
ΔWMA
Sum
Δboth
ε
Set5
×2
+0.02
+0.03
+0.05
+0.07
+0.02
Set14
×2
+0.09
−0.01
+0.08
+0.14
+0.06
BSD100
×2
+0.02
+0.01
+0.03
+0.04
+0.01
Urban100
×2
+0.11
+0.01
+0.12
+0.17
+0.05
Manga109
×2
+0.10
+0.15
+0.25
+0.23
−0.02
Set5
×3
+0.03
+0.05
+0.08
+0.10
+0.02
Table 4: Module interaction across all benchmark–scale pairs. Δ values are PSNR gains in dB over the PFT-light † baseline; ε is the interaction term of Eq. ( 19 ). Positive ε means the modules compound; negative means their gains overlap.
Figure 8: Visual comparison on image 148026 from BSD100 at ×4 .
Figure 9: Visual comparison on the 12th image of Urban100 at ×4 .
Figure 10: Visual comparison on the 2nd image of Urban100 at ×4 . At the intersections of the X-shaped steel framing, PFT-light and HiT-SRF fade near the horizontal members where SFMformer renders the framing distinctly.
Figure 11: Visual comparison on the 14th image of Urban100 at ×4 . SFMformer recovers the structural shape of the railing almost completely, where the other models produce an incomplete structure or outright blur.
Figure 12: Visual comparison on the 54th image of Urban100 at ×4 . No model recovers the window grid correctly; HiT-SRF cannot separate the panes, PFT-light separates them but skews some, and SFMformer keeps almost all of them square.
Figure 13: The interactive inference interface on the Raspberry Pi 5. (a) Side-by-side view. (b) Magnifier view. (c) Region selection. (d) The corresponding region result.
Dataset
Average time per image (s)
Set5 [ 34 ]
12.85
Set14 [ 35 ]
24.29
BSD100 [ 36 ]
17.36
Urban100 [ 37 ]
97.42
Manga109 [ 38 ]
103.88
Table 5: Average per-image inference time at ×4 on the Raspberry Pi 5 (CPU only).
Single image super-resolution aims to reconstruct high-resolution images from low-resolution inputs. This paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform. The transform learns task-adaptive basis functions in a data-driven manner, enabling adaptive feature modulation with minimal parameter overhead. To further enhance high-frequency detail recovery, we introduce a local feature modulation branch that complements transform-domain processing with depthwise convolution. In addition, a soft complexity adaptive module dynamically fuses the outputs of local convolution and window self-attention branches through a lightweight gating network, adaptively adjusting the fusion ratio based on regional texture characteristics. An adaptive intensity modulation strategy is also incorporated to adjust transform-domain response strength at the sample level, enabling the network to dynamically adjust processing intensity according to input features. Extensive experiments on five benchmark datasets demonstrate that FreeTransformSR achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs. Specifically, FreeTransformSR achieves 32.41 dB on BSD100 x2 and 27.00 dB on Urban100 x4 with only 595K parameters, while delivering faster inference speed than competing methods, making it well-suited for deployment in resource-constrained scenarios. Source code is available at: https://github.com/HJiLi/FreeTransformSR.
Hongji Li, Yunhui Li
Chinese Academy of Sciences Changchun Institute of Optics Fine Mechanics and Physics, Changchun, Jilin 130033, China · University of the Chinese Academy of Sciences, Beijing 100049, China
Transformer-based approaches have revolutionized image super-resolution by modeling long-range dependencies. However, the quadratic computational complexity of vanilla self-attention mechanisms poses significant challenges, often leading to compromises between efficiency and global context exploitation. Recent window-based attention methods mitigate this by localizing computations, but they often yield restricted receptive fields. To mitigate these limitations, we propose Selective Aggregation Transformer (SAT). This novel transformer efficiently captures long-range dependencies, leading to an enlarged model receptive field by selectively aggregating key-value matrices (reducing the number of tokens by 97%) via our Density-driven Token Aggregation algorithm while maintaining the full resolution of the query matrix. This design significantly reduces computational costs, resulting in lower complexity and enabling scalable global interactions without compromising reconstruction fidelity. SAT identifies and represents each cluster with a single aggregation token, utilizing density and isolation metrics to ensure that critical high-frequency details are preserved. Experimental results demonstrate that SAT outperforms the state-of-the-art method PFT by up to 0.22dB, while the total number of FLOPs can be reduced by up to 27%.
This paper introduces a lightweight image super-resolution (SR) network, termed the Multi-scale Spatial Adaptive Attention Network (MSAAN), to address the common dilemma between high reconstruction fidelity and low model complexity in existing SR methods. The core of our approach is a novel Multi-scale Spatial Adaptive Attention Module (MSAA), designed to jointly model fine-grained local details and long-range contextual dependencies. The MSAA comprises two synergistic components: a Global Feature Modulation Module (GFM) that learns coherent texture structures through differential feature extraction, and a Multi-scale Feature Aggregation Module (MFA) that adaptively fuses features from local to global scales using pyramidal processing. To further enhance the network's capability, we propose a Local Enhancement Block (LEB) to strengthen local geometric perception and a Feature Interactive Gated Feed-Forward Module (FIGFF) to improve nonlinear representation while reducing channel redundancy. Extensive experiments on standard benchmarks (Set5, Set14, B100, Urban100, Manga109) across ×2, ×3, and ×4 scaling factors demonstrate that both our lightweight (MSAAN-light) and standard (MSAAN) versions achieve superior or competitive performance in terms of PSNR and SSIM, while maintaining significantly lower parameters and computational costs than state-of-the-art methods. Ablation studies validate the contribution of each component, and visual results show that MSAAN reconstructs sharper edges and more realistic textures.