Domain-Specific Latent Representations Improve the Fidelity of Diffusion-Based Medical Image Super-Resolution

Sebastian Cajas; Ashaba Judith; Rahul Gorijavolu; Sahil Kapadia; Hillary Clinton Kasimbazi; Leo Kinyera; Emmanuel Paul Kwesiga; Sri Sri Jaithra Varma Manthena

cs.CVApr 14, 2026

Domain-Specific Latent Representations Improve the Fidelity of Diffusion-Based Medical Image Super-Resolution

Authors: Sebastian Cajas, Ashaba Judith, Rahul Gorijavolu, Sahil Kapadia, Hillary Clinton Kasimbazi, Leo Kinyera, Emmanuel Paul Kwesiga, Sri Sri Jaithra Varma Manthena,

Organizations: MIT Critical Data, Massachusetts Institute of Technology, Cambridge, MA, USA. · Department of Biomedical Engineering, Mbarara University of Science and Technology, Uganda. · School of Medicine, Johns Hopkins University, Baltimore, MD, USA. · Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA. · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. · Department of Radiology and Radiotherapy, Makerere University, Uganda. · Department of Radiotherapy, Uganda Cancer Institute, Uganda. · Department of Radiology and Radiological Science, School of Medicine, Johns Hopkins University, Baltimore, MD, USA. · Department of Public Health Sciences, Cavendish University, Uganda. · Department of Applied Natural Sciences, Technical University of Applied Sciences Lübeck (TH Lübeck), Lübeck, Germany. · Department of Ophthalmology, Federal University of S˜ao Paulo, Brazil. · Division of Pulmonary, Critical Care and Sleep Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA. · Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA, USA.

Abstract

Latent diffusion models for medical image super-resolution universally inherit variational autoencoders designed for natural photographs. We show that this default choice, not the diffusion architecture, is the dominant constraint on reconstruction quality. In a controlled experiment holding all other pipeline components fixed, replacing the generic Stable Diffusion VAE with MedVAE, a domain-specific autoencoder pretrained on more than 1.6 million medical images, yields +2.91 to +3.29 dB PSNR improvement across knee MRI, brain MRI, and chest X-ray (n = 1,820; Cohen's d = 1.37 to 1.86, all p < 10^{-20}, Wilcoxon signed-rank). Wavelet decomposition localises the advantage to the finest spatial frequency bands encoding anatomically relevant fine structure. Ablations across inference schedules, prediction targets, and generative architectures confirm the gap is stable within plus or minus 0.15 dB, while hallucination rates remain comparable between methods (Cohen's h < 0.02 across all datasets), establishing that reconstruction fidelity and generative hallucination are governed by independent pipeline components. These results provide a practical screening criterion: autoencoder reconstruction quality, measurable without diffusion training, predicts downstream SR performance (R^2 = 0.67), suggesting that domain-specific VAE selection should precede diffusion architecture search. Code and trained model weights are publicly available at https://github.com/sebasmos/latent-sr.

Figures & tables

Figure 1: Latent diffusion SR pipeline. Both methods share an identical LDM UNet (purple); only the VAE encoder/decoder differs. Top: SD-VAE SR uses a generic VAE trained on natural images (4-channel, $32\times 32$ latent). Bottom: MedVAE SR (proposed) uses a domain-specific VAE trained on medical images (3-channel, $64\times 64$ latent). The controlled substitution isolates the VAE’s contribution to SR fidelity.

Dataset	Method	PSNR (dB)	MS-SSIM	LPIPS ‡
MRNet †	Bicubic	23.79	0.906	0.541
	ESRGAN	23.28	0.897	0.425
	SwinIR	22.48	0.882	0.446
	SD-VAE SR	22.34	0.870	0.173
	MedVAE SR ∗	25.26	0.936	0.135
	MedVAE-AE §	27.85	0.966	0.109

Table 1: SR reconstruction quality across three datasets. PSNR (dB), MS-SSIM, and LPIPS for all methods. Bold: best SR method per dataset per metric.

\dagger

: MRNet is a refinement task (LR = HR = 256

\times

256); bicubic copies LR unchanged.

\ddagger

: BraTS/CXR bicubic PSNR exceeds diffusion SR due to MSE minimisation vs. perceptual quality tradeoff. Bicubic interpolation achieves competitive PSNR because it avoids hallucination artifacts, but produces blurry outputs with high LPIPS (low perceptual quality).

\dagger\dagger

: BraTS results use the validation split (

n=700

); the flow-matching ablation (Appendix Table S3 ) uses the test split (

n=720

Dataset	$n$	MedVAE PSNR	SD-VAE PSNR	$\Delta$ PSNR	95% CI	Cohen’s $d$	$p$ -value
MRNet	120	$25.26\pm 1.58$	$22.34\pm 1.55$	$+2.91$	$[+2.52,+3.30]$	1.86	$9.9\times 10^{-22}$
BraTS	700	$26.42\pm 2.10$	$23.51\pm 1.99$	$+2.91$	$[+2.69,+3.12]$	1.42	$7.8\times 10^{-120}$
CXR	1,000	$28.87\pm 2.62$	$25.58\pm 2.17$	$+3.29$	$[+3.09,+3.51]$	1.37	$2.3\times 10^{-117}$

Table 2: Statistical significance of MedVAE vs. SD-VAE SR. Per-image Wilcoxon signed-rank test and bootstrap effect sizes (

n_{\text{bootstrap}}=10{,}000

Figure 2: SR reconstruction quality across three datasets. (a) PSNR (dB) and (b) LPIPS for all methods on MRNet (knee MRI), BraTS (brain MRI), and MIMIC-CXR (chest X-ray). Error bars: $\pm 1$ standard deviation across validation images. Improvement brackets (dark blue) show MedVAE SR vs. SD-VAE SR delta: $+X$ dB for PSNR with 95% bootstrap CI, $-\Delta$ for LPIPS (lower = better). MedVAE SR achieves better PSNR than SD-VAE SR and lower LPIPS than bicubic interpolation across all three datasets, occupying the Pareto frontier of the quality-perception tradeoff [ 44 ] . The MedVAE-AE ceiling (light blue, hatched) represents encode-then-decode without diffusion and sets the theoretical maximum for each VAE.

Figure 3: AE reconstruction ceiling predicts SR quality ( $R^{2}=0.67$ ). Scatter plot of VAE autoencoder ceiling PSNR (encode-then-decode without diffusion, x-axis) vs. diffusion SR output PSNR (y-axis) for all method–dataset combinations ( $n=6$ ). Points: MedVAE SR (blue) and SD-VAE SR (red); shapes: MRNet (circle), BraTS (square), CXR (triangle). Line: OLS regression with 95% confidence band (Pearson $r=0.82$ , $p<0.001$ ). The strong linear relationship indicates that AE reconstruction fidelity, computed without training any diffusion model, is a reliable screening criterion for SR quality in any domain-specific LDM pipeline.

Panel of Figure 4 — Figure 4: Multi-resolution frequency analysis. Rows: MRNet, BraTS, MIMIC-CXR. Left: per-subband PSNR (3-level Haar, 10 bands) for MedVAE SR (blue) vs. SD-VAE SR (red). Right: log radial FFT spectrum (HR: dashed). Gains concentrate at HH 1 (+1.18, +1.41, +0.70 dB), while LL 3 differences $\leq$ 0.05 dB.

Panel of Figure 5 — Figure 5: Visual comparison and pixel-level difference maps — BraTS (brain MRI, $4\times$ SR). Top: side-by-side comparison of AE reconstruction, MedVAE SR, SD-VAE SR, and HR ground truth. Bottom: mean absolute pixel difference $|\text{SR}-\text{HR}|$ (hot colormap; brighter $=$ larger error). MedVAE SR produces cooler (darker) error maps, indicating better reconstruction of fine anatomical structure. Scale bar in HR panel: 20 mm (pixel spacing $\approx$ 0.938 mm; 240 mm FOV resized to 256 px). Qualitatively consistent patterns for MRNet (knee MRI) and MIMIC-CXR are shown in Appendix Figs. S5 – S6 .

Figure 6: Multi-resolution latent embedding comparison (all datasets). Rows: MRNet (knee MRI), BraTS (brain MRI), MIMIC-CXR (chest X-ray). (a) PSNR and (b) cosine similarity between SR latents and HR latents at progressively coarser spatial pooling scales ( $64\times 64$ to $1\times 1$ ). MedVAE SR achieves higher SR-to-HR latent fidelity at all scales, confirming that the domain-specific latent space better preserves HR information throughout the spatial hierarchy.

Figure 7: Statistical analysis of SR quality. (a) Cohen’s $d$ forest plot. Each row shows the effect size (Cohen’s $d$ ) for the MedVAE SR vs. SD-VAE SR PSNR gap on one dataset (MRNet, BraTS, CXR), with horizontal error bars indicating 95% bootstrap confidence intervals ( $n_{\text{bootstrap}}=10{,}000$ ). The vertical dashed line marks $d=0.8$ (conventional “large” effect threshold). All three datasets show large-to-very-large effect sizes ( $d>1.3$ ). (b) Per-image PSNR distribution (violin plots). Each violin shows the distribution of per-image PSNR values for MedVAE SR (blue) and SD-VAE SR (red) across the three validation sets; horizontal lines indicate medians. MedVAE SR achieves consistently higher PSNR. (c) Bland-Altman intensity agreement plots. Each panel shows one validation set; $x$ -axis: mean of SR and HR per-image pixel intensities; $y$ -axis: difference SR $-$ HR. Solid lines: mean bias; dashed lines: 95% limits of agreement (LoA). Both methods show near-zero bias; detailed statistics are given in Appendix Table S9 .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

$T$	PSNR (dB)	MS-SSIM	LPIPS	sec/sample	Speedup
50	25.97	0.9317	0.241	0.18	$14.6\times$
100	26.04	0.9355	0.216	0.30	$8.7\times$
250	26.01	0.9410	0.168	0.69	$3.8\times$
500	25.79	0.9405	0.152	1.33	$2.0\times$
1000	25.25	0.9355	0.135	2.62	$1.0\times$

Appendix

Table S1 : Supplementary Table S1. Inference step ablation on MRNet (

T=50

–

1{,}000

, MedVAE SR, S1 latents). PSNR is essentially flat across all step counts; LPIPS improves with more steps. All main results (Table 1 ) use

T=100

unless stated otherwise.

Configuration	PSNR (dB)	$\Delta$ vs baseline
Baseline ( $x_{0}$ + cosine, no extras)	26.05	–
+ EMA (decay $=0.9999$ )	25.98	$-0.07$
+ SNR weighting	25.73	$-0.32$
+ EMA + SNR combined	25.70	$-0.35$

Appendix

Table S2 : Supplementary Table S2. LDM training configuration ablation (MRNet,

T=100

). Baseline

x_{0}

+ cosine + unweighted L1 is already optimal.

Dataset	Method	PSNR	LPIPS	MS-SSIM	Speedup
MRNet	DDPM $T=100$	26.04	0.216	0.9355	$1\times$
MRNet	Flow ( $T=16$ Euler)	24.29	0.138	0.9180	$16\times$
BraTS	DDPM $T=100$	27.13	0.083	0.9400	$1\times$
BraTS	Flow ( $T=16$ Euler)	25.97	0.074	0.9256	$16\times$
CXR	DDPM $T=100$	28.87	0.127	0.9400	$1\times$
CXR	Flow ( $T=16$ Euler)	28.21	0.101	0.9275	$16\times$

Appendix

Table S3 : Supplementary Table S3. Flow matching (rectified flow) vs. DDPM. All datasets,

T=16

Euler steps vs.

T=100

DDPM. Flow matching achieves

16\times

speedup with

0.7

–

1.2

dB PSNR cost but better LPIPS. Note: BraTS results here use the test split (

n=720

); Table 1 of the main text uses the validation split (

n=700

), which accounts for the difference in reported BraTS PSNR (27.13 dB here vs. 26.42 dB in Table 1 ).

Dataset	Scale	Bicubic PSNR (dB)	Bicubic SSIM	MedVAE SR LPIPS
BraTS	$2\times$	$34.91\pm 1.80$	$0.976\pm 0.006$	0.088
	$4\times$	$29.91\pm 1.84$	$0.980\pm 0.005$
	$8\times$	$26.27\pm 1.76$	$0.822\pm 0.035$
CXR	$2\times$	$34.30\pm 2.92$	$0.953\pm 0.015$	0.127
	$4\times$	$30.47\pm 2.86$	$0.977\pm 0.008$
	$8\times$	$26.99\pm 2.78$	$0.774\pm 0.066$

Appendix

Table S4 : Supplementary Table S4. Multi-scale bicubic baseline context. PSNR (dB) and SSIM for bicubic interpolation at 2

\times

, 4

\times

, 8

\times

SR on BraTS and CXR.

Dataset	$n$	$\Delta$ PSNR	$95\%$ CI	Cohen’s $d$	Interpretation
MRNet	120	$+2.91$	$[+2.52,+3.30]$	1.858	Very large
BraTS	700	$+2.91$	$[+2.69,+3.12]$	1.421	Large
CXR	1,000	$+3.29$	$[+3.09,+3.51]$	1.370	Large

Appendix

Table S5 : Supplementary Table S5. Effect sizes (Cohen’s

d

) and statistical significance for MedVAE vs. SD-VAE SR. Bootstrap

95\%

confidence intervals,

n_{\text{bootstrap}}=10{,}000

Subband	Description	MedVAE (MRNet)	SD-VAE (MRNet)	$\bm{\Delta}$ MRNet	$\bm{\Delta}$ BraTS	$\bm{\Delta}$ CXR
		(dB)	(dB)	(dB)	(dB)	(dB)
LL 3	Low-low (coarsest approx.)	$14.51\pm 3.15$	$14.46\pm 2.82$	$+0.05$	$+0.02$	$-0.20$
LH 3	Low-high level 3	$15.78\pm 2.69$	$15.48\pm 2.08$	$+0.30$	$-0.04$	$-0.54$
HL 3	High-low level 3	$13.29\pm 2.68$	$13.13\pm 1.97$	$+0.16$	$-0.13$	$-0.39$
HH 3	High-high level 3	$14.08\pm 2.40$	$13.86\pm 1.96$	$+0.22$	$-0.07$	$-0.75$
LH 2	Low-high level 2	$18.10\pm 1.55$	$17.83\pm 1.59$	$+0.27$	$-0.08$	$-0.81$

Appendix

Table S6 : Supplementary Table S6. Per-frequency-band PSNR from 3-level Haar wavelet decomposition. MedVAE SR vs. SD-VAE SR, mean

\pm

std across validation images. Subbands ordered coarsest (LL 3 ) to finest (HH 1 ). Bold

\Delta

: MedVAE advantage

>0.5

dB.

Dataset	Method	Hallucinated % (above noise floor)	Lost % (below noise floor)
MRNet	MedVAE SR	$24.71\pm 8.69$	$27.07\pm 6.56$
MRNet	SD-VAE SR	$25.18\pm 8.81$	$26.94\pm 6.37$
BraTS	MedVAE SR	$12.91\pm 5.78$	$13.87\pm 5.82$
BraTS	SD-VAE SR	$13.30\pm 5.82$	$13.54\pm 5.78$
CXR	MedVAE SR	$3.30\pm 3.61$	$3.81\pm 4.65$
CXR	SD-VAE SR	$3.40\pm 4.03$	$3.97\pm 5.15$

Appendix

Table S7 : Supplementary Table S7. Pixel-level hallucination rates across datasets and methods.

Scale	MRNet cosine sim	BraTS cosine sim	CXR cosine sim
$64\times 64$	$0.689\pm 0.040$	$0.902\pm 0.014$	$0.373\pm 0.216$
$32\times 32$	$0.778\pm 0.039$	$0.922\pm 0.013$	$0.410\pm 0.237$
$16\times 16$	$0.833\pm 0.040$	$0.936\pm 0.012$	$0.439\pm 0.253$
$8\times 8$	$0.880\pm 0.039$	$0.950\pm 0.010$	$0.471\pm 0.270$
$4\times 4$	$0.926\pm 0.029$	$0.961\pm 0.008$	$0.526\pm 0.288$
$2\times 2$	$0.969\pm 0.011$	$0.971\pm 0.001$	$0.569\pm 0.376$

Appendix

Table S8 : Supplementary Table S8. Multi-resolution latent embedding fidelity (cosine similarity). MRNet:

n=120

pairs. BraTS:

n=700

pairs. CXR:

n=1{,}000

pairs.

Dataset	Method	$n$	Bias	SD( $D$ )	Lower LoA	Upper LoA
MRNet	MedVAE SR	120	$-0.0087$	$0.0292$	$-0.0659$	$+0.0484$
MRNet	SD-VAE SR	120	$-0.0066$	$0.0291$	$-0.0636$	$+0.0503$
BraTS	MedVAE SR	700	$-0.0016$	$0.0304$	$-0.0613$	$+0.0580$
BraTS	SD-VAE SR	700	$+0.0000$	$0.0307$	$-0.0601$	$+0.0602$
CXR	MedVAE SR	1,000	$-0.0011$	$0.0537$	$-0.1063$	$+0.1041$
CXR	SD-VAE SR	1,000	$-0.0030$	$0.0546$	$-0.1100$	$+0.1040$

Appendix

Table S9 : Supplementary Table S9. Bland-Altman intensity agreement statistics. All biases are near-zero (

|

bias

|<0.01

), indicating no systematic intensity offset.

Method	Tumor PSNR (dB)	Background PSNR (dB)	Tumor SSIM	Background SSIM
MedVAE SR	$12.30$	$17.41$	$0.142$	$0.737$
SD-VAE SR	$12.48$	$17.39$	$0.137$	$0.726$
Bicubic	$30.41$	$29.94$	$0.834$	$0.922$

Appendix

Table S10 : Supplementary Table S10. BraTS ROI analysis — tumor vs. background reconstruction quality (

n=473

images with annotated tumor masks).

Dataset	MedVAE AE (PSNR)	SD-VAE AE (PSNR)	$\Delta$
MRNet	27.85	23.92	$+3.93$
BraTS	37.87	31.39	$+6.48$
CXR	36.93	32.02	$+4.91$

Appendix

Table S11 : Supplementary Table S11. VAE autoencoder reconstruction ceiling. PSNR (dB) for encode-then-decode without any diffusion component. The MedVAE ceiling exceeds the SD-VAE ceiling by

+3.93

+6.48

dB.

Dataset	MedVAE AE	MedVAE SR	SD-VAE SR	ESRGAN	SwinIR	Bicubic
MRNet †	46	76	73	158	221	161
BraTS	9	38	31	111	83	102
CXR	15	59	48	101	74	111

Appendix

Table S12: Supplementary Table S12. Fréchet Inception Distance (FID) across all methods. Lower is better. MRNet FID values are unreliable (

n=120<2{,}048

minimum).

Figure S1 : Perception-distortion tradeoff across three medical imaging datasets (MRNet, BraTS, MIMIC-CXR). Each point represents one SR method evaluated on one dataset. $x$ -axis: PSNR (dB; higher = less distortion); $y$ -axis: LPIPS (lower = more perceptually faithful). MedVAE SR moves toward the upper-right Pareto frontier relative to SD-VAE SR, achieving simultaneously higher PSNR and lower LPIPS on all three datasets.

Figure S2 : Pixel-level hallucination and content loss — MRNet (knee MRI). Bar chart showing hallucination rate and loss rate (% of pixels) for MedVAE SR and SD-VAE SR.

Figure S3 : Pixel-level hallucination and content loss — BraTS (brain MRI).

Figure S4 : Pixel-level hallucination and content loss — MIMIC-CXR (chest X-ray, $4\times$ SR).

Dataset	$T=50$	$T=100$	$T=250$	$T=1{,}000$	$\Delta$
MRNet	0.962	0.975	0.968	0.956	0.019
BraTS	0.971	0.984	0.978	0.963	0.021
CXR	0.967	0.981	0.975	0.961	0.020

Appendix

Table S13 : Supplementary Table S13. Cosine similarity between SR latents and HR latents across diffusion timestep budgets at the global mean (

1\times 1

) scale. The narrow range (

\Delta\leq 0.025

) confirms that latent fidelity is determined primarily by the VAE architecture.

Panel of Figure 27 — Figure S5 : Visual comparison and pixel-level difference maps — MRNet (knee MRI). Top: side-by-side comparison of AE reconstruction, MedVAE SR, SD-VAE SR, and HR ground truth. Bottom: mean absolute pixel difference $|\text{SR}-\text{HR}|$ (hot colormap; brighter $=$ larger error).

Panel of Figure 28 — Figure S6 : Visual comparison and pixel-level difference maps — MIMIC-CXR (chest X-ray, $4\times$ SR). Top: side-by-side comparison of AE reconstruction, MedVAE SR, SD-VAE SR, and HR ground truth. Bottom: mean absolute pixel difference $|\text{SR}-\text{HR}|$ (hot colormap).

Medical Image Super-Resolution

Medical image super-resolution reconstructs fine spatial detail from low-resolution scans to support clearer anatomy and more useful clinical measurements.

See all 57 papers

MedSR-Vision: Deep Learning Framework for Multi-Domain Medical Image Super-Resolution
May 5, 2026Subhash Gurappa, Trivikram Satharasi, Yashas Hariprasad +1Medical Image Super-Resolution Medical Image Benchmarks
Knight Foundation School of Computing and Information Sciences Florida International University Miami, FL 33199, USA · University of Florida Gainesville, FL 32611, USA · Department of Computer Science California State University, East Bay Hayward, CA 94542, USA
When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution
Aug 10, 2026Yu Shi, Yuyao Zhang, Yu-wing TaiDiffusion Model Guidance Diffusion Transformer
Dartmouth College
Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate
Oct 3, 2026Wanzhou Lei, Cuifeng Shen, Yanjin He +6Image Super-Resolution
University of California, Berkeley · Alibaba Group · University of Michigan, Ann Arbor +2

May 5, 2026J/K move · Enter open · S save

MedSR-Vision: Deep Learning Framework for Multi-Domain Medical Image Super-Resolution

Subhash Gurappa, Trivikram Satharasi, Yashas Hariprasad, Sundararaj Sitharama Iyengar

Knight Foundation School of Computing and Information Sciences Florida International University Miami, FL 33199, USA · University of Florida Gainesville, FL 32611, USA · Department of Computer Science California State University, East Bay Hayward, CA 94542, USA

Medical Image Super-Resolution Medical Image Benchmarks

Open paper AskPDF arXiv

Medical image super-resolution (MedSR) is essential for improving diagnostic precision across diverse imaging modalities such as MRI, CT, X-ray, Ultrasound, and Fundus imaging. Despite rapid advances in deep learning, challenges remain in preserving anatomical accuracy, maintaining perceptual quality, and generalizing across medical domains. This paper presents MedSR-Vision, a novel unified deep learning framework for evaluating and comparing super-resolution models across five modalities: Brain MRI, Chest X-ray, Renal Ultrasound, Nephrolithiasis CT, and Spine MRI, at magnification scales of

\times2

\times3

, and

\times4

. Three representative models namely SRCNN, SwinIR, and Real-ESRGAN are benchmarked using multiple quantitative metrics encompassing fidelity, perceptual realism, and sharpness. Experimental analysis demonstrates that Real-ESRGAN achieves superior perceptual quality and edge recovery at higher scales, SwinIR excels in preserving structural and diagnostic features, and SRCNN provides efficient and stable performance at lower magnifications. The results establish domain-specific insights and practical guidelines for model selection in clinical imaging workflows, offering a standardized evaluation framework for future medical image super-resolution research and deployment.

Abstract

Medical Image Super-Resolution

Explore similar work

MedSR-Vision: Deep Learning Framework for Multi-Domain Medical Image Super-Resolution

When Latents Forget Pixels: Restoring Fidelity in Diffusion Transformer Super-Resolution

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate