In this paper, we study the diffusability (learnability) of variational autoencoders (VAEs) in latent diffusion. We first derive distribution-free bounds on the maximum achievable reduction in pixel-space denoising loss. Under a power-law power spectral density (PSD), these bounds establish an ordering between sufficiently separated frequencies, with larger achievable reductions at lower frequencies. Motivated by this analysis and measurements of a trained pixel diffusion model, we seek to make low-frequency denoising gains useful for generating perceptually meaningful semantic content in latent diffusion. We propose the Spectrum Matching Hypothesis: latents with superior diffusability should (i) follow a flattened power-law PSD (Encoding Spectrum Matching, ESM) and (ii) preserve frequency-to-frequency semantic correspondence through the decoder (Decoding Spectrum Matching, DSM). In practice, we apply ESM by matching the PSD between images and latents, and DSM via shared spectral masking with frequency-aligned reconstruction. Importantly, Spectrum Matching clarifies prior observations of over-noisy or over-smoothed latents and provides a unified spectral interpretation of prior methods, including VA-VAE and EQ-VAE. Experiments suggest that Spectrum Matching yields superior diffusion generation on CelebA and ImageNet datasets, and outperforms prior approaches. Finally, we extend the spectral view to representation alignment (REPA): we show that the directional spectral energy of the target representation is crucial for REPA, and propose a DoG-based method to further improve the performance of REPA. Our code is available at https://github.com/forever208/SpectrumMatching.
Figures & tables
Figure 1: Class-averaged predictability Rt(ω) of a pixel diffusion model (JiT-H/16 on ImageNet 256×256 ).
Figure 2: Diagram of ESM and DSM in a typical VAE for latent diffusion.
Figure 3
Prior research
Observations / Method
Spectrum Matching View
Scale Equivariance ( Skorokhodov et al., 2025 )
Observation: standard VAE latents exhibit excessive high-frequency energy relative to RGB images.
ESM regularizes excessive high-frequency energy toward a flattened power-law PSD.
FreqWarm ( Lai et al., 2025 )
Observation: low-frequency image information can be encoded in high-frequency latent components.
DSM encourages correspondence between latent and image frequency bands.
SSVAE ( Liu et al., 2025 )
Observation: non-uniform latent spectra improve diffusability compared with uniform spectra.
ESM ensures that the latent has a power-law spectrum.
REPA-E ( Leng et al., 2025 )
Observation: over-smoothed latent space leads to bad diffusion modeling.
ESM ensures that the latent has a power-law spectrum.
UAE ( Fan et al., 2025 )
Method: aligns low-frequency latent DCT coefficients with those of a frozen semantic teacher (e.g., DINOv2).
The teacher provides a spectral reference for low-frequency latent bands, related to ESM.
VA-VAE ( Yao et al., 2025 )
Method: aligns linearly projected latents with frozen DINOv2 features.
The measured latent PSD approximates ESM’s flattened power-law target.
Table 1: A Unified Spectral Interpretation of Prior Approaches and Empirical Observations.
Figure 3: Directional image (right) maintains the spatial structure of the original image (left).
Models
f8d4
f16d16
rFID ↓
PSNR ↑
SSIM ↑
steps
gFID ↓
rFID ↓
PSNR ↑
SSIM ↑
steps
gFID ↓
SD-VAE
1.31
31.23
0.788
125k
6.63
1.72
30.08
0.753
150k
6.88
Scale Equivariance
1.24
30.98
0.781
100k
5.87
1.31
29.70
0.741
100k
5.46
ESM-AE
1.55
31.63
0.793
100k
5.91
1.56
31.30
0.782
100k
5.84
DSM-AE
1.44
30.07
0.760
100k
4.44
1.39
29.32
0.745
100k
4.49
ESM+DSM
1.37
30.04
0.765
100k
4.48
1.31
29.45
0.745
100k
4.38
Table 2: Reconstruction and generation on CelebA 256×256 . Steps denote diffusion training iterations; bold marks the best gFID.
Figure 4: PCA visualization (top three principal components) of the latent space of different autoencoders.
Figure 5: PSD distributions of RGB images and autoencoder latents on CelebA 256×256 .
Models
f16d16
gFID (without REPA)
gFID (with REPA)
rFID ↓
PSNR ↑
SSIM ↑
100k
200k
300k
400k
100k
200k
300k
400k
SD-VAE
1.20
25.44
0.659
27.43
15.88
12.29
10.62
22.25
10.52
7.79
6.83
DSM-AE
1.38
24.15
0.620
22.87
13.34
10.64
9.55
18.80
8.98
6.92
6.26
Table 3: Reconstruction and generation results on ImageNet 256×256 (training from scratch).
Models
rFID ↓
PSNR ↑
gFID ↓
SiT-B (80 epochs)
SiT-XL (100 epochs)
SiT-XL (200 epochs)
SD-VAE ( f8d4 )
0.55
26.11
3.68
2.15
1.98
EQ-VAE ( f8d4 )
0.61
25.97
3.31
2.02
1.79
VA-VAE ( f16d32 )
0.28
27.71
3.39
2.08
1.89
DSM-AE ( f8d4 )
0.66
25.85
3.20
1.82
1.66
Table 4: Reconstruction and generation on ImageNet 256×256 (fine-tuning pretrained SD-VAE).
Classifier-Free Guidance = 1.8
Models
100k
200k
300k
400k
REPA
23.92
10.67
7.13
5.68
iREPA
17.73
8.51
6.23
5.07
REPA-DoG
18.52
8.60
6.15
4.98
Table 5: gFID on ImageNet 256×256 using DINOv2-B as encoder on SiT-B/2.
Figure 6: Impact of processing the feature representation of DINOv3-B ( Siméoni et al., 2025 ) . Comparison of token-wise cosine similarity maps. The red star indicates the reference token, while the heatmap represents its affinity with the spatial query tokens.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Spectrum distributions of the input image and DINOv2 features on ImageNet 256×256 .
Figure 8: Spectrum distributions of the input image and VA-VAE latents on ImageNet 256×256 .
Figure 9: DCT-domain comparison of downsampling and DSM. Left: the rectangular low-frequency region used to approximate 2× average-pooling, with frequency-dependent rescaling omitted. Right: an example of DSM’s triangular high-frequency mask. Orange regions are retained; gray regions are zeroed out.
Dataset
CelebA 256 (f8d4, f16d16)
ImageNet 256 (f16d16)
Model
SD-VAE
Scale Equivariance
ESM-AE
DSM-AE
SD-VAE
DSM-AE
network parameters
83.7M (f8d4), 64.4M (f16d16)
64.4M
VAE learning rate
5×10−5
5×10−5
GAN learning rate
5×10−5
5×10−5
GAN start step
50k
50k
learning rate schedule
constant with warmup
constant with warmup
Appendix
Table 6: Training hyperparameters of autoencoders on CelebA 256×256 and ImageNet 256×256 .
Model
REPA
iREPA
REPA-DoG
diffusion network
SiT-B/2
foundation model
DINOv2-B
encoder depth
4
alignment coefficient
0.5
learning rate
10−4
batch size
256
Appendix
Table 7: Training hyperparameters of REPA, iREPA, and REPA-DoG on ImageNet 256×256 .
Figure 10: The frequency in the DCT space follows a zigzag order where low frequencies lie in the top-left corner, and high frequencies are in the bottom right. Therefore, we design the frequency mask as a set of triangular shapes to perform different levels of high-frequency filtering. On the right side, we show some concrete samples of the mask M using n=4 , n=8 and n=12 in the 8×8 DCT space.
Models
δ
β
f16d16
rFID ↓
PSNR ↑
SSIM ↑
steps
gFID ↓
ESM-AE
0.6
0.01
1.56
30.52
0.767
100k
5.90
ESM-AE
1.0
0.01
1.56
31.30
0.782
100k
5.84
ESM-AE
1.4
0.01
1.61
31.62
0.790
100k
6.57
ESM-AE
1.0
0.1
1.63
30.86
0.777
100k
6.82
ESM-AE
1.0
0.01
1.56
31.30
0.782
100k
5.84
Appendix
Table 8: Ablation study of ESM-AE on CelebA 256×256 . Diffusion steps denote the number of training iterations at convergence.
Models
Mask set M
f16d16
rFID ↓
PSNR ↑
SSIM ↑
steps
gFID ↓
DSM-AE
n={0,8,12}
1.40
29.64
0.749
100k
5.02
DSM-AE
n={0,8,10,12}
1.39
29.32
0.745
100k
4.49
DSM-AE
n={0,6,8,10,12}
1.38
29.52
0.746
100k
4.61
DSM-AE
n={0,8,10,12,13}
1.23
28.71
0.728
100k
5.33
Appendix
Table 9: Ablation study of DSM-AE on CelebA 256×256 . Diffusion steps denote the number of training iterations at convergence.
Latent diffusion models pair VAEs with diffusion backbones, and the structure of VAE latents strongly influences the difficulty of diffusion training. However, existing video VAEs typically focus on reconstruction fidelity, overlooking latent structure. We present a statistical analysis of video VAE latent spaces and identify two spectral properties essential for diffusion training: a spatio-temporal frequency spectrum biased toward low frequencies, and a channel-wise eigenspectrum dominated by a few modes. To induce these properties, we propose two lightweight, backbone-agnostic regularizers: Local Correlation Regularization and Latent Masked Reconstruction. Experiments show that our Spectral-Structured VAE (SSVAE) achieves a 3× speedup in text-to-video generation convergence and a 10% gain in video reward, outperforming strong open-source VAEs. The code is available at https://github.com/zai-org/SSVAE.
Shizhan Liu, Xinran Deng, Zhuoyi Yang +3
Zhipu AI, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China
Most visual generative models compress images into a latent space before applying diffusion or autoregressive modelling. Yet, existing approaches such as VAEs and foundation model aligned encoders implicitly constrain the latent space without explicitly shaping its distribution, making it unclear which types of distributions are optimal for modeling. We introduce Distribution-Matching VAE (DMVAE), which explicitly aligns the encoder's latent distribution with an arbitrary reference distribution via a distribution matching constraint. This generalizes beyond the Gaussian prior of conventional VAEs, enabling alignment with distributions derived from self-supervised features, diffusion noise, or other prior distributions. With DMVAE, we can systematically investigate which latent distributions are more conducive to modeling, and we find that SSL-derived distributions provide an excellent balance between reconstruction fidelity and modeling efficiency, reaching gFID equals 3.2 on ImageNet with only 64 training epochs. Our results suggest that choosing a suitable latent distribution structure (achieved via distribution-level alignment), rather than relying on fixed priors, is key to bridging the gap between easy-to-model latents and high-fidelity image synthesis. Code is avaliable at https://github.com/sen-ye/dmvae.
Sen Ye, Jianning Pei, Mengde Xu +4
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · University of Chinese Academy of Sciences · Tencent +1
We present Scalable Pixel-anchored End-to-end Diffusion (SpeeDiff), a latent diffusion method that jointly trains the VAE and the diffusion model from scratch. In principle, joint training allows the diffusion loss gradient to directly guide the VAE encoder, encouraging the formation of a generation-friendly latent space and potentially yielding faster convergence than the conventional two-stage approach with a pretrained frozen VAE. However, a naive end-to-end implementation severely degrades performance, as unrestricted backpropagation of the diffusion loss leads to latent space collapse. Our main technical contribution is a simple yet effective Tweedie Pixel Reconstruction (TPR) loss, which provides additional pixel-level feedback by decoding a predicted clean latent from an intermediate noisy state using Tweedie's formula, thereby alleviating collapse. Furthermore, our method enables jointly scaling a fully transformer-based architecture and enhances representation alignment within the end-to-end framework. Our SpeeDiff-XL model achieves over 140x and 61x faster training compared to Vanilla SiT and REPA, respectively, while attaining an FID of 1.50 without guidance on ImageNet 256x256 generation. With a more efficient 32x compressed VAE, our model further reaches an FID of 1.53 without guidance on ImageNet 512x512 generation.
Bingliang Zhang, Wenda Chu, Yizhuo Li +5
ByteDance Seed · California Institute of Technology · Meta