SAT: Selective Aggregation Transformer for Image Super-Resolution

Dinh Phu Tran; Thao Do; Saad Wazir; Seongah Kim; Seon Kwon Kim; Daeyoung Kim

cs.CVApr 9, 2026

SAT: Selective Aggregation Transformer for Image Super-Resolution

Authors: Dinh Phu Tran, Thao Do, Saad Wazir, Seongah Kim, Seon Kwon Kim, Daeyoung Kim

Organizations: School of Computing, KAIST, Republic of Korea

Abstract

Transformer-based approaches have revolutionized image super-resolution by modeling long-range dependencies. However, the quadratic computational complexity of vanilla self-attention mechanisms poses significant challenges, often leading to compromises between efficiency and global context exploitation. Recent window-based attention methods mitigate this by localizing computations, but they often yield restricted receptive fields. To mitigate these limitations, we propose Selective Aggregation Transformer (SAT). This novel transformer efficiently captures long-range dependencies, leading to an enlarged model receptive field by selectively aggregating key-value matrices (reducing the number of tokens by 97%) via our Density-driven Token Aggregation algorithm while maintaining the full resolution of the query matrix. This design significantly reduces computational costs, resulting in lower complexity and enabling scalable global interactions without compromising reconstruction fidelity. SAT identifies and represents each cluster with a single aggregation token, utilizing density and isolation metrics to ensure that critical high-frequency details are preserved. Experimental results demonstrate that SAT outperforms the state-of-the-art method PFT by up to 0.22dB, while the total number of FLOPs can be reduced by up to 27%.

Figures & tables

Figure 1: The pixel-wise absolute error between HR and SR images from RCAN [ 52 ] and PFT [ 28 ] . These concentrated error areas at high-frequency regions motivate our SAT’s design: we preserve full-resolution Query while compressing Key–Value tokens in homogeneous areas for achieving efficient global attention.

Figure 2: The architecture of the proposed SAT. The Local Transformer Block (LTB) and the Selective Aggregation Transformer Block (SATB) are arranged alternately to construct global-local structure, better capturing deep features for pixel reconstruction.

Figure 3: The illustration of the Selective Aggregation Attention (SAA). SAA aggregates

N

input tokens into

K=k\%\times N

tokens (with

k=3

) to compact the Key-Value matrices, preserving the full-resolution Query matrix to form an efficiently global cross-attention.

Method	Scale	Params	FLOPs	Set5		Set14		B100		Urban100		Manga109
Method	Scale	Params	FLOPs	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
EDSR [ 27 ]	$\times$ 2	42.6M	22.14T	38.11	0.9692	33.92	0.9195	32.32	0.9013	32.93	0.9351	39.10	0.9773
RCAN [ 52 ]		15.4M	7.02T	38.27	0.9614	34.12	0.9216	32.41	0.9027	33.34	0.9384	39.44	0.9786
IPT [ 11 ]		115M	7.38T	38.37	-	34.43	-	32.48	-	33.76	-	-	-
SwinIR [ 26 ]		11.8M	3.04T	38.42	0.9623	34.46	0.9250	32.53	0.9041	33.81	0.9433	39.92	0.9797
CAT-A [ 15 ]		16.5	5.08	38.51	0.9626	34.78	0.9265	32.59	0.9047	34.26	0.9440	40.10	0.9805

Table 1: Comparison between SAT and other SOTA methods at

\times

\times

\times

4 scales for image ISR. The top-2 results are in red and blue .

Method	Scale	Params	FLOPs	Set5		Set14		B100		Urban100		Manga109
Method	Scale	Params	FLOPs	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
CARN [ 2 ]	$\times$ 2	1,592K	222.8G	37.76	0.9590	33.52	0.9166	32.09	0.8978	31.92	0.9256	38.36	0.9765
IMDN [ 23 ]		694K	158.8G	38.00	0.9605	33.63	0.9177	32.19	0.8996	32.17	0.9283	38.88	0.9774
LatticeNet [ 29 ]		756K	169.5G	38.15	0.9610	33.78	0.9193	32.25	0.9005	32.43	0.9302	-	-
SwinIR-light [ 26 ]		910K	244G	38.14	0.9611	33.86	0.9206	32.31	0.9012	32.76	0.9340	39.12	0.9783
ELAN [ 51 ]		582K	203G	38.17	0.9611	33.94	0.9207	32.30	0.9012	32.76	0.9340	39.11	0.9782

Table 2: Comparison between SAT-light and other methods at

\times

\times

4 scales on lightweight benchmark. Top-2 results are in red and blue .

Figure 4: PSNR performance across different keeping ratios on Urban100, and Manga109 datasets.

Panel of Figure 5 — Figure 5: Qualitative comparison of visual results between our SAT and other state-of-the-art SR methods. Best results are marked in bold .

Method	Params	FLOPs	VRAM	Set5	Urban100	Manga109
VSA [ 42 ]	808K	69.4G	60.4GB	32.48	26.74	31.12
SRA [ 44 ]	787K	37.5G	4.7GB	32.40	26.47	30.88
WSA [ 26 ]	809K	43.9G	4.1GB	32.44	26.52	30.92
SAA (Ours)	763K	36.4G	5.3GB	32.48	26.61	31.09

Table 3: Effects of the proposed selective aggregation attention

Method	Complexity	Runtime	Set5	Urban100	Manga109
K-means (20 iters) [ 31 ]	$\mathcal{O}(20NKC)$	113ms	32.39	26.49	30.91
DPC-KNN [ 34 ]	$\mathcal{O}(N^{2}C)$	6534ms	32.50	26.66	31.14
DTA (Ours)	$\mathcal{O}(NKC)$	11ms	32.48	26.61	31.09

Table 4: Effects of Density-driven Token Aggregation algorithm

LTB	SATB	Params	FLOPs	Set5	Urban100	Manga 109
w/o	w/	716K	28.6G	32.45	26.58	31.07
w/	w/o	810K	44.2G	32.41	26.48	30.97
w/	w/	763K	36.4G	32.48	26.61	31.09

Table 5: Effects of SATB and LTB blocks.

Scale	Method	Params	FLOPs	PSNR (Manga109)	Runtime
$\times$ 4	HAT [ 12 ]	20.8M	1.45T	32.48	192ms
$\times$ 4	ATD [ 50 ]	20.3M	1.52T	32.62	228ms
$\times$ 4	IPG [ 35 ]	18.3M	1.30T	32.53	288ms
$\times$ 4	PFT [ 28 ]	19.8M	1.26T	32.63	230ms
$\times$ 4	SAT (Ours)	19.5M	0.94T	32.85	207ms

Table 6: Comparison on model complexity and running time

Explore similar work

CUST: Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution
Jul 13, 2026Jeongsoo KimVision Transformer Efficient ViTs
Independent Researcher
From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution
Jan 13, 2026Chunyu Meng, Wei Long, Shuhang GuEfficient ViTs Sparse Attention
University of Electronic Science and Technology of China
ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration
Mar 3, 2026Leheng Zhang, Wei Long, Yawei Li +3Image Restoration Efficient ViTs
School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China (UESTC), China · Computer Vision Lab, ETH Zürich, Zürich, Switzerland and with the Integrated Systems Laboratory, ETH Zürich, Zürich, Switzerland

Jul 13, 2026J/K move · Enter open · S save

CUST: Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution

Jeongsoo Kim

Independent Researcher

Vision Transformer Efficient ViTs Attention Mechanisms Image Super-Resolution

Open paper AskPDF arXiv

Recently, Vision Transformer (ViT)-based models have exhibited remarkable performance in image super-resolution. However, the quadratic computational complexity of ViTs with respect to spatial resolution severely constrains their efficiency, leading to high latency and massive memory consumption. To alleviate this, various window-based attention mechanisms have been proposed; yet, they inherently compromise the long-range dependency modeling that is the primary advantage of ViTs. To overcome these limitations, we propose the Clustered Unit-level Similarity Transformer (CUST), a novel architecture that efficiently integrates global and local information. Specifically, CUST enables each patch to aggregate and attend to similar patches within a broadened regional scope outside its local window, thereby capturing extensive contextual understanding. Furthermore, it employs overlapping attention windows to capture local dependencies, while explicitly extracting high-frequency details by computing the residual difference between the original features and their downsampled-upsampled counterparts. Comprehensive experiments demonstrate that our proposed model achieves a practical balance between computational efficiency and restoration performance. It achieves a lower memory footprint and faster inference speed compared to recent global context or lightweight models under realistic constraints. Code is available at [https://github.com/jwgdmkj/CUST].

Abstract

Explore similar work

CUST: Clustered Unit-level Similarity Transformer for Lightweight Image Super-Resolution

From Local Windows to Adaptive Candidates via Individualized Exploratory: Rethinking Attention for Image Super-Resolution

ATD: Improved Transformer with Adaptive Token Dictionary for Image Restoration