SFMformer: A Spatial-Frequency Modulation Transformer for Lightweight Image Super-Resolution

Chih-Hsiang Yang; Chia-Min Lin; Ching-Yu Tsai; Yung-Che Wang; Jen-Shiun Chiang

cs.CVAug 18, 2026

SFMformer: A Spatial-Frequency Modulation Transformer for Lightweight Image Super-Resolution

Authors: Chih-Hsiang Yang, Chia-Min Lin, Ching-Yu Tsai, Yung-Che Wang, Jen-Shiun Chiang

Organizations: Department of Electrical and Computer Engineering, Tamkang University, No. 151, Yingzhuan Rd., Tamsui District, New Taipei City 251301, Taiwan

Abstract

Sparse attention mechanisms, which score all token pairs but propagate only the strongest, now underpin the most efficient Transformers for lightweight image super-resolution. This paper observes that sparsification changes what it means to improve such a network. A dense attention layer has one place where representation quality matters: the aggregation of attended features. A sparse layer has two, because the top-k operator first decides which tokens survive and only then decides what to do with them, and a token discarded at the selection stage cannot be recovered downstream. Selection quality and aggregation quality are therefore separable targets, addressed by modules placed before and after the attention respectively. We test this by pairing a dual-branch spatial enhancement on the input of a progressive focused attention with a wavelet-domain modulation on its output, forming SFMformer. Measuring each module alone and jointly over all fifteen benchmark-scale pairs, we find their gains are not additive: the joint gain exceeds the sum of the individual gains on nine pairs, and the sign of the discrepancy is predicted by how much the weaker module contributes on its own (r = -0.72), so the two compound when they relieve different constraints and overlap when they relieve the same one. Enabling spectral modulation once per block rather than once per layer retains the effect at roughly one-sixth of its cost, keeping the model below one million parameters at every scale. SFMformer ranks first on 28 of 30 PSNR/SSIM entries across five benchmarks and three upscaling factors. We report the cases where the pairing does not help, and deploy the model on a Raspberry Pi 5 to confirm the design is practical under tight resource budgets.

Figures & tables

Figure 1: Overall architecture of SFMformer. A $3\times 3$ convolution extracts shallow features, $M$ cascaded SFM Blocks produce deep features, and a long residual connection followed by pixel-shuffle upsampling reconstructs the HR output.

Figure 2: Structure of the Spatial–Frequency Modulation Block (SFMB). Only the final layer (SFML ∗ ) enables the WMA module.

Figure 3: Internal structure of the Spatial–Frequency Modulation Layer (SFML). The dashed region (LayerNorm + WMA) is enabled only in the last layer of each SFMB.

Figure 4: The Dual Feature Extraction (DFE) module. The hourglass branch captures local spatial neighbourhood information; the

1\times 1

branch preserves channel semantics.

Figure 5: The Progressive Focused Attention (PFA) mechanism. The attention map of the previous layer is inherited, multiplied element-wise with the current similarity map, and sparsified by a top- $K^{l}$ operation.

Figure 6: The Wavelet-domain Modulation self-Attention (WMA) module. Features are reduced in channel dimension, decomposed by the DWT into four sub-bands, modulated by channel-wise self-attention and dynamic convolution, and returned to the spatial domain by the IDWT.

Method	Scale	Params	FLOPs	Set5		Set14		BSD100		Urban100		Manga109
Method	Scale	Params	FLOPs	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic	$\times 2$	—	—	33.66	0.9299	30.24	0.8688	29.56	0.8431	26.88	0.8403	30.80	0.9339
SwinIR-light [ 5 ]	$\times 2$	910K	244G	38.14	0.9611	33.86	0.9206	32.31	0.9012	32.76	0.9340	39.12	0.9783
ELAN-light [ 9 ]	$\times 2$	582K	203G	38.17	0.9611	33.94	0.9207	32.30	0.9012	32.76	0.9340	39.11	0.9782
SwinIR-NG [ 40 ]	$\times 2$	1181K	274.1G	38.17	0.9612	33.94	0.9205	32.31	0.9013	32.78	0.9340	39.20	0.9781
OmniSR [ 20 ]	$\times 2$	772K	194.5G	38.22	0.9613	33.98	0.9210	32.36	0.9020	33.05	0.9363	39.28	0.9784

Table 1: Quantitative comparison of SFMformer with state-of-the-art lightweight methods at three upscaling factors. Bold : best; underline : second best; bicubic is listed for reference and excluded from the ranking.

\dagger

denotes retraining under our configuration. LKMN [ 42 ] and PromptSR [ 43 ] report

\times 2

results only, and FLOPs are not available for every quoted method.

Figure 7: PSNR on Urban100 versus parameter count at

\times 4

. The dashed line marks the 1M-parameter boundary conventionally used to delimit lightweight SR models.

Method	Scale	Params	Set5		Set14		BSD100		Urban100		Manga109
Method	Scale	Params	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
PFT-light [ 14 ]	$\times 2$	776K	38.36	0.9620	34.19	0.9232	32.43	0.9030	33.67	0.9411	39.55	0.9792
PFT-light †	$\times 2$	776K	38.35	0.9620	34.19	0.9227	32.44	0.9030	33.61	0.9406	39.64	0.9793
SFMformer (ours)	$\times 2$	970K	38.43	0.9622	34.28	0.9235	32.48	0.9036	33.73	0.9414	39.82	0.9796
PFT-light [ 14 ]	$\times 3$	783K	34.82	0.9305	30.75	0.8493	29.33	0.8116	29.46	0.8763	34.61	0.9509
PFT-light †	$\times 3$	783K	34.82	0.9306	30.75	0.8490	29.34	0.8118	29.37	0.8752	34.68	0.9511

Table 2: Results at a batch size of 32. PFT-light † was retrained by us under settings identical to SFMformer; the PFT-light rows without

\dagger

quote the original paper [ 14 ] . Bold : best; underline : second best.

Method	DFE	WMA	Scale	Params	Set5		Set14		BSD100		Urban100		Manga109
Method	DFE	WMA	Scale	Params	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
PFT-light † (baseline)			$\times 2$	776K	38.33	0.9618	34.06	0.9218	32.41	0.9026	33.36	0.9385	39.50	0.9790
+ DFE	✓		$\times 2$	890K	38.35	0.9620	34.15	0.9221	32.43	0.9029	33.47	0.9398	39.60	0.9791
+ WMA		✓	$\times 2$	856K	38.36	0.9619	34.05	0.9220	32.42	0.9028	33.37	0.9382	39.65	0.9793
+ DFE + WMA (SFMformer)	✓	✓	$\times 2$	970K	38.40	0.9620	34.20	0.9227	32.45	0.9032	33.53	0.9397	39.73	0.9794
PFT-light † (baseline)			$\times 3$	783K	34.78	0.9303	30.70	0.8481	29.32	0.8122	29.22	0.8728	34.57	0.9505

Table 3: Ablation study of the DFE and WMA modules at all three upscaling factors.

Dataset	Scale	$\Delta_{\mathrm{DFE}}$	$\Delta_{\mathrm{WMA}}$	Sum	$\Delta_{\mathrm{both}}$	$\varepsilon$
Set5	$\times 2$	$+0.02$	$+0.03$	$+0.05$	$+0.07$	$+0.02$
Set14	$\times 2$	$+0.09$	$-0.01$	$+0.08$	$+0.14$	+0.06
BSD100	$\times 2$	$+0.02$	$+0.01$	$+0.03$	$+0.04$	$+0.01$
Urban100	$\times 2$	$+0.11$	$+0.01$	$+0.12$	$+0.17$	$+0.05$
Manga109	$\times 2$	$+0.10$	$+0.15$	$+0.25$	$+0.23$	$-0.02$
Set5	$\times 3$	$+0.03$	$+0.05$	$+0.08$	$+0.10$	$+0.02$

Table 4: Module interaction across all benchmark–scale pairs.

\Delta

values are PSNR gains in dB over the PFT-light † baseline;

\varepsilon

is the interaction term of Eq. ( 19 ). Positive

\varepsilon

means the modules compound; negative means their gains overlap.

Figure 8: Visual comparison on image 148026 from BSD100 at $\times 4$ .

Figure 9: Visual comparison on the 12th image of Urban100 at $\times 4$ .

Figure 10: Visual comparison on the 2nd image of Urban100 at $\times 4$ . At the intersections of the X-shaped steel framing, PFT-light and HiT-SRF fade near the horizontal members where SFMformer renders the framing distinctly.

Figure 11: Visual comparison on the 14th image of Urban100 at $\times 4$ . SFMformer recovers the structural shape of the railing almost completely, where the other models produce an incomplete structure or outright blur.

Figure 12: Visual comparison on the 54th image of Urban100 at $\times 4$ . No model recovers the window grid correctly; HiT-SRF cannot separate the panes, PFT-light separates them but skews some, and SFMformer keeps almost all of them square.

Figure 13: The interactive inference interface on the Raspberry Pi 5. (a) Side-by-side view. (b) Magnifier view. (c) Region selection. (d) The corresponding region result.

Dataset	Average time per image (s)
Set5 [ 34 ]	12.85
Set14 [ 35 ]	24.29
BSD100 [ 36 ]	17.36
Urban100 [ 37 ]	97.42
Manga109 [ 38 ]	103.88

Table 5: Average per-image inference time at

\times 4

on the Raspberry Pi 5 (CPU only).

Image Super-Resolution

Image super-resolution reconstructs high-resolution images from lower-resolution observations, balancing detail, fidelity, and computational cost.

See all 248 papers

FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform
Sep 5, 2026Hongji Li, Yunhui LiHybrid CNN-Transformer Architectures Image Super-Resolution
Chinese Academy of Sciences Changchun Institute of Optics Fine Mechanics and Physics, Changchun, Jilin 130033, China · University of the Chinese Academy of Sciences, Beijing 100049, China
SAT: Selective Aggregation Transformer for Image Super-Resolution
Apr 9, 2026Dinh Phu Tran, Thao Do, Saad Wazir +3Image Super-Resolution Efficient ViTs
School of Computing, KAIST, Republic of Korea
Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networks
Feb 22, 2026Sushi Rao, Jingwei LiImage Super-Resolution Multi-Scale Attention
Zhejiang Gongshang University

Sep 5, 2026J/K move · Enter open · S save

FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform

Hongji Li, Yunhui Li

Chinese Academy of Sciences Changchun Institute of Optics Fine Mechanics and Physics, Changchun, Jilin 130033, China · University of the Chinese Academy of Sciences, Beijing 100049, China

Hybrid CNN-Transformer Architectures Image Super-Resolution

Open paper AskPDF arXiv

Single image super-resolution aims to reconstruct high-resolution images from low-resolution inputs. This paper proposes FreeTransformSR, a novel lightweight super-resolution network based on a channel-wise free low-rank learnable transform. The transform learns task-adaptive basis functions in a data-driven manner, enabling adaptive feature modulation with minimal parameter overhead. To further enhance high-frequency detail recovery, we introduce a local feature modulation branch that complements transform-domain processing with depthwise convolution. In addition, a soft complexity adaptive module dynamically fuses the outputs of local convolution and window self-attention branches through a lightweight gating network, adaptively adjusting the fusion ratio based on regional texture characteristics. An adaptive intensity modulation strategy is also incorporated to adjust transform-domain response strength at the sample level, enabling the network to dynamically adjust processing intensity according to input features. Extensive experiments on five benchmark datasets demonstrate that FreeTransformSR achieves competitive PSNR/SSIM performance with significantly fewer parameters and FLOPs. Specifically, FreeTransformSR achieves 32.41 dB on BSD100 x2 and 27.00 dB on Urban100 x4 with only 595K parameters, while delivering faster inference speed than competing methods, making it well-suited for deployment in resource-constrained scenarios. Source code is available at: https://github.com/HJiLi/FreeTransformSR.

Abstract

Image Super-Resolution

Explore similar work

FreeTransformSR: Efficient Lightweight Image Super-Resolution via Free Low-Rank Learnable Transform

SAT: Selective Aggregation Transformer for Image Super-Resolution

Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networks