Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

Mang Ning; Mingxiao Li; Le Zhang; Lanmiao Liu; Matthew B. Blaschko; Albert Ali Salah; Itir Onal Ertugrul

cs.CVMar 15, 2026

Spectrum Matching: a Unified Perspective for Superior Diffusability in Latent Diffusion

Authors: Mang Ning, Mingxiao Li, Le Zhang, Lanmiao Liu, Matthew B. Blaschko, Albert Ali Salah, Itir Onal Ertugrul

Organizations: Utrecht University · KU Leuven · Mila · Max Planck Institute for Psycholinguistics

Abstract

In this paper, we study the diffusability (learnability) of variational autoencoders (VAEs) in latent diffusion. We first derive distribution-free bounds on the maximum achievable reduction in pixel-space denoising loss. Under a power-law power spectral density (PSD), these bounds establish an ordering between sufficiently separated frequencies, with larger achievable reductions at lower frequencies. Motivated by this analysis and measurements of a trained pixel diffusion model, we seek to make low-frequency denoising gains useful for generating perceptually meaningful semantic content in latent diffusion. We propose the Spectrum Matching Hypothesis: latents with superior diffusability should (i) follow a flattened power-law PSD (Encoding Spectrum Matching, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (Decoding Spectrum Matching, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching clarifies prior observations of over-noisy or over-smoothed latents and provides a unified spectral interpretation of prior methods, including VA-VAE and EQ-VAE. Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available at https://github.com/forever208/SpectrumMatching.

Figures & tables

Figure 1: Class-averaged predictability $R_{t}(\omega)$ of a pixel diffusion model (JiT-H/16 on ImageNet $256\times 256$ ).

Figure 2: Diagram of ESM and DSM in a typical VAE for latent diffusion.

Figure 3

Prior research	Observations / Method	Spectrum Matching View
Scale Equivariance ( Skorokhodov et al., 2025 )	Observation: standard VAE latents exhibit excessive high-frequency energy relative to RGB images.	ESM regularizes excessive high-frequency energy toward a flattened power-law PSD.
FreqWarm ( Lai et al., 2025 )	Observation: low-frequency image information can be encoded in high-frequency latent components.	DSM encourages correspondence between latent and image frequency bands.
SSVAE ( Liu et al., 2025 )	Observation: non-uniform latent spectra improve diffusability compared with uniform spectra.	ESM ensures that the latent has a power-law spectrum.
REPA-E ( Leng et al., 2025 )	Observation: over-smoothed latent space leads to bad diffusion modeling.	ESM ensures that the latent has a power-law spectrum.
UAE ( Fan et al., 2025 )	Method: aligns low-frequency latent DCT coefficients with those of a frozen semantic teacher (e.g., DINOv2).	The teacher provides a spectral reference for low-frequency latent bands, related to ESM.
VA-VAE ( Yao et al., 2025 )	Method: aligns linearly projected latents with frozen DINOv2 features.	The measured latent PSD approximates ESM’s flattened power-law target.

Table 1: A Unified Spectral Interpretation of Prior Approaches and Empirical Observations.

Figure 3: Directional image (right) maintains the spatial structure of the original image (left).

Models	f8d4					f16d16
Models	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	steps	gFID $\downarrow$	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	steps	gFID $\downarrow$
SD-VAE	1.31	31.23	0.788	125k	6.63	1.72	30.08	0.753	150k	6.88
Scale Equivariance	1.24	30.98	0.781	100k	5.87	1.31	29.70	0.741	100k	5.46
ESM-AE	1.55	31.63	0.793	100k	5.91	1.56	31.30	0.782	100k	5.84
DSM-AE	1.44	30.07	0.760	100k	4.44	1.39	29.32	0.745	100k	4.49
ESM+DSM	1.37	30.04	0.765	100k	4.48	1.31	29.45	0.745	100k	4.38

Table 2: Reconstruction and generation on CelebA

256\times 256

. Steps denote diffusion training iterations; bold marks the best gFID.

Figure 4: PCA visualization (top three principal components) of the latent space of different autoencoders.

Figure 5: PSD distributions of RGB images and autoencoder latents on CelebA

256\times 256

Models	f16d16			gFID (without REPA)				gFID (with REPA)
Models	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	100k	200k	300k	400k	100k	200k	300k	400k
SD-VAE	1.20	25.44	0.659	27.43	15.88	12.29	10.62	22.25	10.52	7.79	6.83
DSM-AE	1.38	24.15	0.620	22.87	13.34	10.64	9.55	18.80	8.98	6.92	6.26

Table 3: Reconstruction and generation results on ImageNet

256\times 256

(training from scratch).

Models	rFID $\downarrow$	PSNR $\uparrow$	gFID $\downarrow$
Models	rFID $\downarrow$	PSNR $\uparrow$	SiT-B (80 epochs)	SiT-XL (100 epochs)	SiT-XL (200 epochs)
SD-VAE ( $f8d4$ )	0.55	26.11	3.68	2.15	1.98
EQ-VAE ( $f8d4$ )	0.61	25.97	3.31	2.02	1.79
VA-VAE ( $f16d32$ )	0.28	27.71	3.39	2.08	1.89
DSM-AE ( $f8d4$ )	0.66	25.85	3.20	1.82	1.66

Table 4: Reconstruction and generation on ImageNet

256\times 256

(fine-tuning pretrained SD-VAE).

Classifier-Free Guidance = 1.8
Models	100k	200k	300k	400k
REPA	23.92	10.67	7.13	5.68
iREPA	17.73	8.51	6.23	5.07
REPA-DoG	18.52	8.60	6.15	4.98

Table 5: gFID on ImageNet

256\times 256

using DINOv2-B as encoder on SiT-B/2.

Figure 6: Impact of processing the feature representation of DINOv3-B ( Siméoni et al., 2025 ) . Comparison of token-wise cosine similarity maps. The red star indicates the reference token, while the heatmap represents its affinity with the spatial query tokens.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Spectrum distributions of the input image and DINOv2 features on ImageNet $256\times 256$ .

Figure 8: Spectrum distributions of the input image and VA-VAE latents on ImageNet $256\times 256$ .

Figure 9: DCT-domain comparison of downsampling and DSM. Left: the rectangular low-frequency region used to approximate

2\times

average-pooling, with frequency-dependent rescaling omitted. Right: an example of DSM’s triangular high-frequency mask. Orange regions are retained; gray regions are zeroed out.

Dataset	CelebA 256 (f8d4, f16d16)				ImageNet 256 (f16d16)
Model	SD-VAE	Scale Equivariance	ESM-AE	DSM-AE	SD-VAE	DSM-AE
network parameters	83.7M (f8d4), 64.4M (f16d16)				64.4M
VAE learning rate	$5\times 10^{-5}$				$5\times 10^{-5}$
GAN learning rate	$5\times 10^{-5}$				$5\times 10^{-5}$
GAN start step	50k				50k
learning rate schedule	constant with warmup				constant with warmup

Appendix

Table 6: Training hyperparameters of autoencoders on CelebA

256\times 256

and ImageNet

256\times 256

Model	REPA	iREPA	REPA-DoG
diffusion network	SiT-B/2
foundation model	DINOv2-B
encoder depth	4
alignment coefficient	0.5
learning rate	$10^{-4}$
batch size	256

Appendix

Table 7: Training hyperparameters of REPA, iREPA, and REPA-DoG on ImageNet

256\times 256

Figure 10: The frequency in the DCT space follows a zigzag order where low frequencies lie in the top-left corner, and high frequencies are in the bottom right. Therefore, we design the frequency mask as a set of triangular shapes to perform different levels of high-frequency filtering. On the right side, we show some concrete samples of the mask $M$ using $n=4$ , $n=8$ and $n=12$ in the $8\times 8$ DCT space.

Models	$\delta$	$\beta$	f16d16
Models	$\delta$	$\beta$	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	steps	gFID $\downarrow$
ESM-AE	0.6	0.01	1.56	30.52	0.767	100k	5.90
ESM-AE	1.0	0.01	1.56	31.30	0.782	100k	5.84
ESM-AE	1.4	0.01	1.61	31.62	0.790	100k	6.57
ESM-AE	1.0	0.1	1.63	30.86	0.777	100k	6.82
ESM-AE	1.0	0.01	1.56	31.30	0.782	100k	5.84

Appendix

Table 8: Ablation study of ESM-AE on CelebA

256\times 256

. Diffusion steps denote the number of training iterations at convergence.

Models	Mask set $\mathcal{M}$	f16d16
Models	Mask set $\mathcal{M}$	rFID $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	steps	gFID $\downarrow$
DSM-AE	$n=\{0,8,12\}$	1.40	29.64	0.749	100k	5.02
DSM-AE	$n=\{0,8,10,12\}$	1.39	29.32	0.745	100k	4.49
DSM-AE	$n=\{0,6,8,10,12\}$	1.38	29.52	0.746	100k	4.61
DSM-AE	$n=\{0,8,10,12,13\}$	1.23	28.71	0.728	100k	5.33

Appendix

Table 9: Ablation study of DSM-AE on CelebA

256\times 256

. Diffusion steps denote the number of training iterations at convergence.

Latent Diffusion Models

Latent diffusion models denoise compact representations to generate or transform images, video, audio, and other complex data efficiently.

See all 207 papers

Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
Dec 5, 2025Shizhan Liu, Xinran Deng, Zhuoyi Yang +3Variational Autoencoders Spectral Regularization
Zhipu AI, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China
Distribution Matching Variational AutoEncoder
Dec 8, 2025Sen Ye, Jianning Pei, Mengde Xu +4Variational Autoencoders Generative Modeling
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · University of Chinese Academy of Sciences · Tencent +1
SpeeDiff: Scalable Pixel-Anchored End-to-End Latent Diffusion Model
Oct 26, 2026Bingliang Zhang, Wenda Chu, Yizhuo Li +5Latent Diffusion Models Diffusion Transformer
ByteDance Seed · California Institute of Technology · Meta

Dec 5, 2025J/K move · Enter open · S save

Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability

Shizhan Liu, Xinran Deng, Zhuoyi Yang, Jiayan Teng+2

Zhipu AI, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China

Variational Autoencoders Spectral Regularization Video Diffusion Models Latent Diffusion Models

Open paper AskPDF arXiv

Latent diffusion models pair VAEs with diffusion backbones, and the structure of VAE latents strongly influences the difficulty of diffusion training. However, existing video VAEs typically focus on reconstruction fidelity, overlooking latent structure. We present a statistical analysis of video VAE latent spaces and identify two spectral properties essential for diffusion training: a spatio-temporal frequency spectrum biased toward low frequencies, and a channel-wise eigenspectrum dominated by a few modes. To induce these properties, we propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction. Experiments show that our Spectral-Structured VAE (SSVAE) achieves a

3\times

speedup in text-to-video generation convergence and a 10% gain in video reward, outperforming strong open-source VAEs. The code is available at https://github.com/zai-org/SSVAE.

Abstract

Latent Diffusion Models

Explore similar work

Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability

Distribution Matching Variational AutoEncoder

SpeeDiff: Scalable Pixel-Anchored End-to-End Latent Diffusion Model