Disentangled Textual Priors for Diffusion-based Image Super-Resolution

Lei Jiang; Xin Liu; Xinze Tong; Zhiliang Li; Jie Liu; Jie Tang; Gangshan Wu

cs.CVMar 8, 2026

Disentangled Textual Priors for Diffusion-based Image Super-Resolution

Authors: Lei Jiang, Xin Liu, Xinze Tong, Zhiliang Li, Jie Liu, Jie Tang, Gangshan Wu

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China

Abstract

Image Super-Resolution (SR) aims to reconstruct high-resolution images from degraded low-resolution inputs. While diffusion-based SR methods offer powerful generative capabilities, their performance heavily depends on how semantic priors are structured and integrated into the generation process. Existing approaches often rely on entangled or coarse-grained priors that mix global layout with local details, or conflate structural and textural cues, thereby limiting semantic controllability and interpretability. In this work, we propose DTPSR, a novel diffusion-based SR framework that introduces disentangled textual priors along two complementary dimensions: spatial hierarchy (global vs. local) and frequency semantics (low- vs. high-frequency). By explicitly separating these priors, DTPSR enables the model to simultaneously capture scene-level structure and object-specific details with frequency-aware semantic guidance. The corresponding embeddings are injected via specialized cross-attention modules, forming a progressive generation pipeline that reflects the semantic granularity of visual content, from global layout to fine-grained textures. To support this paradigm, we construct DisText-SR, a large-scale dataset containing approximately 95,000 image-text pairs with carefully disentangled global, low-frequency, and high-frequency descriptions. To further enhance controllability and consistency, we adopt a multi-branch classifier-free guidance strategy with frequency-aware negative prompts to suppress hallucinations and semantic drift. Extensive experiments on synthetic and real-world benchmarks show that DTPSR achieves high perceptual quality, competitive fidelity, and strong generalization across diverse degradation scenarios.

Figures & tables

Figure 1: Comparison under severe degradation. Without textual priors, the diffusion model suffers from hallucinations, generating human-like artifacts or misinterpreting walls as ocean textures. Incorporating our disentangled textual priors suppresses such errors and enhances semantic consistency.

Figure 2: The overall architecture of DTPSR. Given an LR image, a global prior and local object priors are extracted, where local priors are disentangled into Low-Frequency (LF) descriptions (shape, layout, color) and High-Frequency (HF) descriptions (texture, edges, details). They are encoded into $\mathbf{e}_{g}$ , $\mathbf{E}_{lf}$ , and $\mathbf{E}_{hf}$ via CLIP, while the LR image is encoded into $\mathbf{f}_{lr}$ by an image encoder. The diffusion process sequentially updates the latent $\mathbf{z}_{t}$ through GTCA, LFCA, and HFCA—yielding intermediate representations $\mathbf{z}_{t}^{g}$ , $\mathbf{z}_{t}^{lf}$ , and $\mathbf{z}_{t}^{hf}$ —and then fuses $\mathbf{f}_{lr}$ via LRCA to produce $\mathbf{z}_{t-1}$ , progressively restoring structures and details.

Figure 3: DisText-SR dataset construction. Given an LR image, we use a segmentation model to extract regions $S_{1},S_{2},\dots,S_{n}$ , and query a frozen MLLM to generate a global description $c_{g}$ and per-segment low-frequency ( $c_{lf}^{(i)}$ ) and high-frequency ( $c_{hf}^{(i)}$ ) texts. They are encoded via CLIP into Global, Low-Frequency (LF) and High-Frequency (HF) embeddings for semantic-guided SR.

Dataset	Metrics	Real-ESRGAN [ 36 ]	BSRGAN [ 44 ]	StableSR [ 34 ]	DiffBIR [ 21 ]	PASD [ 42 ]	SeeSR [ 40 ]	SUPIR [ 43 ]	FaithDiff [ 8 ]	Ours
DIV2K-Val [ 1 ]	PSNR↑	24.12	24.62	22.90	22.87	24.41	23.67	23.16	23.49	22.03
	SSIM↑	0.6303	0.6311	0.5609	0.5548	0.6252	0.6043	0.5441	0.5817	0.5471
	LPIPS↓	0.3168	0.3390	0.3158	0.3645	0.3794	0.3194	0.3625	0.3118	0.3806
	MUSIQ↑	61.29	60.52	61.06	69.29	61.27	68.67	62.59	69.18	71.24
	MANIQA↑	0.3803	0.3502	0.3586	0.5699	0.4042	0.5036	0.5224	0.4309	0.5866
	CLIPIQA↑	0.5212	0.5311	0.6174	0.7156	0.5568	0.6935	0.7040	0.6463	0.7549

Table 1: Quantitative comparison with state-of-the-art methods on DIV2K-Val, RealSR, and DRealSR datasets. The best and second-best results for each metric are highlighted in bold and underlined , respectively.

Figure 4: Qualitative comparison with representative SR methods. Our DTPSR reconstructs sharper textures and more semantically aligned details, especially under complex degradations, compared to both GAN-based ( e.g . , BSRGAN, Real-ESRGAN) and diffusion-based ( e.g . , FaithDiff, SUPIR) approaches. Zoom in for better visual comparison. More qualitative comparisons can be found in Sec. 4 of the supplementary material.

Method	Params	Steps	Inference Time
SUPIR [ 43 ]	17.8B	100	17.25s
FaithDiff [ 8 ]	15.6B	20	13.20s
DTPSR (Ours)	10.5B	50	14.94s

Table 2: Comparison of model complexity and runtime efficiency on the DRealSR dataset. Inference time is reported as the average per-image runtime.

Exp	Global	Local	MANIQA $\uparrow$	CLIP-IQA $\uparrow$	MUSIQ $\uparrow$
3-1	$\times$	$\times$	0.5271	0.7064	67.48
3-2	$\times$	$\checkmark$	0.5851	0.7471	68.86
3-3	$\checkmark$	$\times$	0.5394	0.7211	67.80
3-4	$\checkmark$	$\checkmark$	0.6011	0.7640	69.24

Table 3: Ablation study on the effect of global and local textual priors on the DRealSR dataset.

Exp	HF	LF	MANIQA $\uparrow$	CLIP-IQA $\uparrow$	MUSIQ $\uparrow$
4-1	$\times$	$\checkmark$	0.5850	0.7544	68.96
4-2	$\checkmark$	$\times$	0.5495	0.7301	68.08
4-3	$\checkmark$	$\checkmark$	0.6011	0.7640	69.24

Table 4: Ablation study on the effect of frequency-aware textual priors on the DRealSR dataset.

Exp	Type	MANIQA $\uparrow$	CLIP-IQA $\uparrow$	MUSIQ $\uparrow$
5-1	Mixed	0.5947	0.7527	69.05
5-2	Disentangled	0.6011	0.7640	69.24

Table 5: Comparison of frequency-mixed and frequency-disentangled learning on the DRealSR dataset.

Exp	CFG Strategy	MANIQA $\uparrow$	CLIP-IQA $\uparrow$	MUSIQ $\uparrow$
6-1	None	0.5995	0.6992	66.73
6-2	Single	0.5776	0.7499	67.87
6-3	Multi (Ours)	0.6011	0.7640	69.24

Table 6: Ablation study on multi-branch classifier-free guidance (CFG) on the DRealSR dataset.

Method	MUSIQ↑	MANIQA↑	CLIP-IQA↑
DiffBIR [ 21 ]	68.86	0.5589	0.7030
PASD [ 42 ]	59.99	0.3939	0.4855
SeeSR [ 40 ]	69.82	0.5437	0.6701
SUPIR [ 43 ]	58.51	0.4429	0.6357
FaithDiff [ 8 ]	68.99	0.4644	0.6126
DTPSR-C	71.64	0.5855	0.7061

Table 7: Robustness comparison under textual corruption on the DRealSR dataset. DTPSR-C denotes the variant with randomly corrupted textual descriptions (“None” substitution). Despite artificial textual errors, DTPSR remains superior across perceptual metrics.

Explore similar work

GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution
Apr 28, 2026Fabio D'Oronzio, Federico Putamorsi, Leonardo Zini +2Diffusion Models for Image Restoration Diffusion Model Fine-Tuning
University of Modena and Reggio Emilia, Italy
TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution
Oct 9, 2026Zhiming Hu, Ran Zhang, Angela Ye +7Image Super-Resolution Diffusion Models for Image Restoration
AIC-Toronto, Samsung Electronics · AMD · University of Toronto +2
DreamSR: Towards Ultra-High-Resolution Image Super-Resolution via a Receptive-Field Enhanced Diffusion Transformer
May 15, 2026Qingji Dong, Hang Dong, Mingqin Chen +2Diffusion Models for Image Restoration Diffusion Model Guidance
ByteDance Inc.

Apr 28, 2026J/K move · Enter open · S save

GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution

Fabio D'Oronzio, Federico Putamorsi, Leonardo Zini, Marcella Cornia+1

University of Modena and Reggio Emilia, Italy

Diffusion Models for Image Restoration Diffusion Model Fine-Tuning Conditional Diffusion Models Image Super-Resolution

Open paper AskPDF arXiv

Despite recent advances, single-image super-resolution (SR) remains challenging, especially in real-world scenarios with complex degradations. Diffusion-based SR methods, particularly those built on Stable Diffusion, leverage strong generative priors but commonly rely on text conditioning derived from semantic captioning. Such textual descriptions provide only high-level semantics and lack the spatially aligned visual information required for faithful restoration, leading to a representation gap between abstract semantics and spatially aligned visual details. To address this limitation, we propose GramSR, a one-step diffusion-based SR framework that replaces text conditioning with dense visual features extracted from the low-resolution input using a pre-trained DINOv3 encoder. GramSR adopts a three-stage LoRA architecture, where pixel-level, semantic-level, and texture-level LoRA modules are trained sequentially. The pixel-level module focuses on degradation removal using

\ell_2

loss, the semantic-level module enhances perceptual details via LPIPS and CSD losses, and the texture-level module enforces feature correlation consistency through a Gram matrix loss computed from DINOv3 features. At inference, independent guidance scales enable flexible control over degradation removal, semantic enhancement, and texture preservation. Extensive experiments on standard SR benchmarks demonstrate that GramSR consistently outperforms existing one-step diffusion-based methods, achieving superior structural fidelity and texture realism. The code for this work is available at: https://github.com/aimagelab/GramSR.

Abstract

Explore similar work

GramSR: Visual Feature Conditioning for Diffusion-Based Super-Resolution

TGSR: Text-Grounded Diffusion for Faithful Image Super-Resolution

DreamSR: Towards Ultra-High-Resolution Image Super-Resolution via a Receptive-Field Enhanced Diffusion Transformer