Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

Wanzhou Lei; Cuifeng Shen; Yanjin He; Maohua Li; Hua Yuan; Per-Olof Persson; Tao Lan; Kan Liu

cs.CVOct 3, 2026

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

Authors: Wanzhou Lei, Cuifeng Shen, Yanjin He, Maohua Li, Hua Yuan, Per-Olof Persson, Tao Lan, Kan Liu,

Organizations: University of California, Berkeley · Alibaba Group · University of Michigan, Ann Arbor · Nanjing University ·

Abstract

In an image latent space, the embeddings of high-resolution, natural, and sharp images form a manifold. Degradation of high-resolution images pushes their embeddings off this manifold. Real-world super-resolution (SR) then becomes the task of mapping the degraded embedding back onto this manifold --- not anywhere on the manifold, but to the point that preserves what the input still carries, both its semantics and pixel details. Every published method implements this mapping in a reconstruction-oriented latent space or pixel space. We claim these spaces are the wrong substrates for SR. Low-resolution and degraded images are embedded far from the manifold, making the mapping difficult and expensive. The lack of semantic information in these substrates also makes it difficult to navigate to the faithful point on the manifold, causing severe hallucination when degradation is heavy. Thus, restoring in a suitable latent space is crucial to the SR task. We show that the latent space of 23 fused layers of a frozen DINOv3-L is one such space that makes the SR task easier. Degraded images are embedded near the manifold. Moreover, this substrate contains a hierarchy of information, from pixel record to degradation robust semantics, guiding the model to find the faithful point on the manifold. On this substrate, a 415M decoder is trained under reconstruction and adversarial objectives to map the degraded embeddings back and decode to pixel space in one pass. The resulting model, RAESR, attains the best fidelity--perception trade-off among state-of-the-art adversarial and diffusion-based restorers on RealSR, DRealSR, LSDIR and DIV2K-Val, at 37 ms per 512 by 512 image on a single H20 GPU. Swapping the substrate for a VAE latent under an identical recipe loses on every metric.

Figures & tables

Figure 1: The choice of latent space decides the task. (a) A reconstruction-oriented latent. (b) The fused DINOv3-L latent (ours). Strips: decoded images along the straight latent line $z_{\alpha}=(1-\alpha)\,z_{\mathrm{LR}}+\alpha\,z_{\mathrm{HR}}$ in each space.

Figure 2: Perceptual-Fidelity and Perceptual-Efficiency tradeoffs. Circle area scales with inference parameter count. The vertical axis fuses the six perceptual metrics, each min–max normalized and re-oriented, then averaged with equal weight. (a) against LPIPS; (b) against measured latency (one H20, batch

1

). Construction and robustness checks of the fused score: App. 8.4 .

Figure 3: (a) Per layer, clean inputs: PSNR of a small decoder overfit to a few images (blue) and held-out ImageNet linear-probe accuracy (red). (b, c)

R^{2}

of a linear map from the fused latent

z

to each layer, on clean and training-degraded inputs.

Figure 4: The critic. The restored image and its ground truth pass through the same frozen DINOv3-B, and each of its

12

layers is read by its own trainable head that emits one logit per token.

Method	NFE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$	Params (M)	Lat. (ms)	Mem (GiB)
RAESR	1	24.08	0.6269	0.3007	0.1970	4.319	70.51	0.6713	0.6815	0.7049	719	37.2	1.57
RAESR D	1	23.39	0.6245	0.3127	0.2097	4.485	70.44	0.6652	0.6755	0.7033	502	30.0	1.18
Adversarial restorers
Real-ESRGAN	1	24.46	0.6648	0.3205	0.2208	5.282	60.50	0.5388	0.5270	0.5068	17	20.0	0.17
Real-HAT-GAN	1	25.56	0.6829	0.3110	0.2270	5.837	55.56	0.5310	0.4433	0.4087	21	156.7	0.73
Diffusion trained from scratch

Table 1:

\times 4

real-world SR, macro-averaged over the four benchmarks . NFE: network evaluations per image. RAESR D : distilled. Bold best, underline second; per-benchmark tables in App. 8 .

Figure 5: Qualitative $\times 4$ comparison, all sixteen panels. Full frames with the region boxed; right: that region across every method at $4\times$ nearest-neighbour magnification, ours in red . Blocks from top: LSDIR, RealSR, DIV2K-Val. Best viewed zoomed in. App. 16 shows other examples.

Comparison	RAESR votes	Baseline votes	Ties	RAESR preference
RAESR vs. Real-HAT-GAN	$163$	$26$	$26$	$86.2\%$
RAESR vs. DiffBIR	$133$	$55$	$30$	$70.7\%$
RAESR vs. SUPIR	$109$	$80$	$22$	$57.7\%$
RAESR vs. TSD-SR	$101$	$91$	$27$	$52.6\%$

Table 2: Human preference, two-alternative forced choice with input and ground truth shown.

Figure 6: Raw metric curves along the latent line

z_{\alpha}=(1-\alpha)\,z_{\mathrm{LR}}+\alpha\,z_{\mathrm{HR}}

, decoded at eleven points. Curves are means over

2{,}000

images.

Figure 7: Left: relative change in each metric when one eight-layer band of the average is removed; negative means the metric improves. Right: the same interventions on one LSDIR image.

Figure 8: Swapping the substrate, keeping everything else. Bold marks the better score of the two. Two further examples in Fig. 11 .

Method	Reconstruction	Ref. perceptual	Adversarial	NR-IQA
Real-ESRGAN	L1	VGG feature	GAN	none
StableSR	latent $\epsilon$ -MSE; CFW: L1	CFW: LPIPS	CFW: GAN	none
ResShift (4-step)	latent MSE	LPIPS	—	none
OSEDiff	pixel MSE $+$ VSD	LPIPS	—	none
DiffBIR	MSE / latent $\epsilon$ -MSE	—	—	none
SUPIR	(training code unreleased)	—	—	none stated

Table 3: Training signals of the baselines (coarse; official paper/code). Ref. perc. = reference-based perceptual loss; NR-IQA = a no-reference quality model as a training signal.

Adversarial restorers
Method	NFE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
RAESR	1	19.77	0.4354	0.3485	0.1923	3.312	73.58	0.6987	0.7192	0.7108
RAESR D	1	19.92	0.4409	0.3629	0.2098	3.416	73.13	0.6859	0.7062	0.7039
Real-ESRGAN	1	19.95	0.4694	0.4135	0.2526	3.942	66.22	0.5663	0.5997	0.5899
Real-HAT-GAN	1	20.55	0.4791	0.4289	0.2769	4.345	59.37	0.5433	0.4852	0.4570
Diffusion trained from scratch

Table 4: Per-benchmark results: LSDIR (250 images). Marks and protocol as in Table 1 .

Adversarial restorers
Method	NFE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
RAESR	1	25.00	0.7195	0.2703	0.2012	4.587	70.49	0.6757	0.6705	0.6841
RAESR D	1	23.45	0.7045	0.2864	0.2141	4.856	70.77	0.6748	0.6718	0.6852
Real-ESRGAN	1	25.00	0.7473	0.2753	0.2076	5.807	60.43	0.5500	0.5163	0.4581
Real-HAT-GAN	1	26.55	0.7655	0.2480	0.1969	6.348	57.11	0.5513	0.4458	0.3737
Diffusion trained from scratch

Table 5: Per-benchmark results: RealSR (100 images). Marks and protocol as in Table 1 .

Adversarial restorers
Method	NFE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
RAESR	1	27.85	0.7613	0.3030	0.2120	5.348	67.41	0.6449	0.6517	0.7091
RAESR D	1	26.52	0.7567	0.3102	0.2207	5.531	67.40	0.6405	0.6477	0.7132
Real-ESRGAN	1	28.61	0.8051	0.2819	0.2089	6.698	54.27	0.4903	0.4624	0.4516
Real-HAT-GAN	1	30.02	0.8324	0.2605	0.2063	7.294	50.56	0.4915	0.4053	0.3916
Diffusion trained from scratch

Table 6: Per-benchmark results: DRealSR (93 images). Marks and protocol as in Table 1 .

Adversarial restorers
Method	NFE	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
RAESR	1	23.71	0.5915	0.2810	0.1827	4.027	70.55	0.6658	0.6845	0.7158
RAESR D	1	23.66	0.5958	0.2913	0.1943	4.136	70.46	0.6595	0.6764	0.7108
Real-ESRGAN	1	24.29	0.6372	0.3112	0.2141	4.679	61.06	0.5485	0.5297	0.5277
Real-HAT-GAN	1	25.10	0.6547	0.3066	0.2279	5.361	55.20	0.5378	0.4370	0.4124
Diffusion trained from scratch

Table 7: Per-benchmark results: DIV2K-Val (3000 images). Marks and protocol as in Table 1 .

Figure 9: Layer-pair geometry on training-degraded inputs: chance-corrected, variance-matched containment $\Omega$ (left) and linear CKA against layer distance (right).

Threshold	RAE	SD3-VAE	SD2.1-VAE
NIQE $\leq 5$	$0.47$	$0.65$	$0.68$
NIQE $\leq 6$	$0.39$	$0.54$	$0.54$
NIQE $\leq 7$	$0.33$	$0.43$	$0.43$
LPIPS $\leq 0.25$	$0.59$	$0.71$	$0.73$
LPIPS $\leq 0.35$	$0.50$	$0.63$	$0.63$
LPIPS $\leq 0.50$	$0.39$	$0.48$	$0.46$

Table 8: Path fraction

\alpha

at which the mean raw curve first crosses each threshold (smaller: the decoded line turns natural, or reaches the ground truth, earlier). The RAE column is shared by both studies, which use the same RAE rows.

Figure 10: The latent line with the SD2.1 VAE , on the same images and with the same protocol and thresholds as Fig. 6 . The number above each arrow is the gap between the two threshold crossings.

Condition	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
A unmodified	23.72	0.6265	0.3012	0.1971	4.320	70.45	0.6717	0.6820	0.7059
B no-shallow (layers $2$ – $9$ )	16.43	0.5179	0.5194	0.3807	5.021	70.66	0.6585	0.6454	0.5994
C no-deep (layers $17$ – $24$ )	24.60	0.6565	0.3079	0.2088	4.687	66.75	0.6134	0.6177	0.6622
D random band (3-seed mean)	20.48	0.6069	0.3728	0.2733	4.884	69.59	0.6402	0.6497	0.6482
seed 0	22.67	0.6356	0.3270	0.2308	4.814	68.25	0.6230	0.6432	0.6541
seed 1	19.90	0.6030	0.3777	0.2820	4.931	70.19	0.6500	0.6531	0.6572

Table 9: Decoder channel ablation , macro-averaged over the four benchmarks. B–D replace eight of the

23

averaged layers with training-set mean tokens at inference; E additionally replaces the broadcast term of layer

24

by its training-set mean. D is the mean of three random bands. Bold marks the cells the dissociation turns on: fidelity collapsing without the shallow band and improving without the deep one, further without the broadcast.

Released recipe (no quality metric in the loss)
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	NIQE $\downarrow$	MUSIQ $\uparrow$	MANIQA $\uparrow$	TOPIQ $\uparrow$	CLIP $\uparrow$
RAESR (released)	23.72	0.6265	0.3012	0.1971	4.320	70.45	0.6717	0.6820	0.7059
SD3-VAE substrate	23.61	0.6033	0.3734	0.2495	4.344	69.76	0.6300	0.6632	0.7011
on LSDIR	19.73	0.4242	0.4291	0.2481	3.274	73.33	0.6492	0.7023	0.7210
on RealSR	24.25	0.6831	0.3390	0.2521	4.694	69.26	0.6313	0.6481	0.6801
on DRealSR	27.15	0.7320	0.3711	0.2618	5.528	65.82	0.6052	0.6286	0.6776

Table 10: Encoder-substrate ablation under the released recipe; the DINOv3 substrate leads on all nine metrics.

Figure 11: Swapping the substrate: two further examples. Format as in Fig. 8 . The tractor becomes a pond and the cathedral’s tracery is redrawn as generic stonework.

Figure 12: LSDIR. The first sample is the example of Fig. 5 . Best viewed zoomed in.

Figure 13: RealSR. The first sample is the example of Fig. 5 . Best viewed zoomed in.

Figure 14: DIV2K-Val. The first sample is the example of Fig. 5 . Best viewed zoomed in.

Figure 15: DRealSR. Best viewed zoomed in.

Image Super-Resolution

Image super-resolution reconstructs high-resolution images from lower-resolution observations, balancing detail, fidelity, and computational cost.

See all 248 papers

Degradation-Aware and Structure-Preserving Diffusion for Real-World Image Super-Resolution
Apr 13, 2026Yang Ji, Zonghao Chen, Zhihao Xue +1Degradation-Aware Image Restoration Image Super-Resolution
KUNBYTE Hangzhou, Zhejiang Province, China · GOLDMYE Xuhui District, Shanghai, China
When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution
Aug 10, 2026Yu Shi, Yuyao Zhang, Yu-wing TaiDiffusion Model Guidance Diffusion Transformer
Dartmouth College
DACESR: Degradation-Aware Conditional Embedding for Real-World Image Super-Resolution
Feb 27, 2026Xiaoyan Lei, Wenlong Zhang, Biao Luo +3Image Super-Resolution
School of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450002, Henan, China. · Shanghai Artificial Intelligence Laboratory, Shanghai 200233, China. · School of Automation, Central South University, Changsha 410083, China. +1

Apr 13, 2026J/K move · Enter open · S save

Degradation-Aware and Structure-Preserving Diffusion for Real-World Image Super-Resolution

Yang Ji, Zonghao Chen, Zhihao Xue, Junqin Hu

KUNBYTE Hangzhou, Zhejiang Province, China · GOLDMYE Xuhui District, Shanghai, China

Degradation-Aware Image Restoration Image Super-Resolution Diffusion Models for Image Restoration Image Restoration Diffusion Models

Open paper AskPDF arXiv

Real-world image super-resolution is particularly challenging for diffusion models because real degradations are complex, heterogeneous, and rarely modeled explicitly. We propose a degradation-aware and structure-preserving diffusion framework for real-world SR. Specifically, we introduce Degradation-aware Token Injection, which encodes lightweight degradation statistics from low-resolution inputs and fuses them with semantic conditioning features, enabling explicit degradation-aware restoration. We further propose Spatially Asymmetric Noise Injection, which modulates diffusion noise with local edge strength to better preserve structural regions during training. Both modules are lightweight add-ons to the adopted diffusion SR framework, requiring only minor modifications to the conditioning pipeline. Experiments on DIV2K and RealSR show that our method delivers competitive no-reference perceptual quality and visually more realistic restoration results than recent baselines, while maintaining a favorable perception--distortion trade-off. Ablations confirm the effectiveness of each module and their complementary gains when combined. The code and model are publicly available at https://github.com/jiyang0315/DASP-SR.git.

Abstract

Image Super-Resolution

Explore similar work

Degradation-Aware and Structure-Preserving Diffusion for Real-World Image Super-Resolution

When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution

DACESR: Degradation-Aware Conditional Embedding for Real-World Image Super-Resolution