In latent diffusion models, expanding the latent dimension of variational autoencoders (VAEs) is essential to push the upper bound of reconstruction quality. However, without explicit structural constraints, such unregularized expansion easily yields a rugged and ill-conditioned latent manifold, causing generated latents to drift off the manifold during generation (i.e., off-manifold problem). Recent works smooth this manifold by using external foundation models (e.g., DINOv2) as a topological regularizer. However, we argue that tying the latent space to frozen external priors inherently restricts generalization: bound by their pre-training distributions and semantic biases, they may fail to capture faithful visual details on out-of-distribution domains. Observing that low-dimensional latents naturally form a smoother, generation-friendly manifold, we propose dual-manifold VAE (DM-VAE) to leverage internal representations as a topological regularizer. Specifically, DM-VAE pairs a high-dimensional reconstructive branch with a low-dimensional branch that extracts a clean topological skeleton, anchoring the high-dimensional manifold to this smooth subspace. Building upon this framework, we further introduce topological representation alignment (TRA) to supervise intermediate generator features during training, and internal guidance with topological skeleton (IG_TS) to guide structural components while preserving high-frequency details at inference. On ImageNet-1K (256x256), our foundation-model-free DM-VAE achieves a state-of-the-art rFID of 0.12 and a generation FID50K of 1.28 (w/o CFG) and 1.20 (w/CFG), and further reaches 1.11 when orthogonally combined with DINOv2.
Figures & tables
Figure 1: Qualitative comparison of out-of-distribution image reconstruction. All methods are trained on ImageNet-1K. Without relying on external foundation models , our DM-VAE ( f16d64 ) exhibits superior generalization. Average PSNR [ 20 ] on above three samples: RAE [ 52 ] (16.32 dB), Ours (low branch) (27.19 dB), and Ours (high branch) ( 30.10 dB ).
Figure 2: Comparison of reconstruction paradigms in LDMs. Unlike (a) vanilla VAEs [ 38 ] that suffer from unconstrained, rugged high-dimensional manifolds, or (b) foundation-model-aligned [ 49 ] and (c) foundation-model-based [ 52 ] VAEs that rely on external priors ( e.g . , Dino [ 34 ] ) with limited generalization ability, (d) our foundation-model-free DM-VAE constructs an internal topological regularizer. Specifically, given an encoder E mapping the input to a high-dimensional latent z and a main decoder D for reconstruction, we compress z via Pdown into a low-dimensional branch (comprising zlow and a lightweight decoder Dlow ). By enforcing the alignment between z and the lifted z^ , our method compels the high-dimensional manifold to inherit a smooth, generation-friendly geometry from zlow . (e) Smooth vs . rugged manifolds, where “smooth” denotes a geometry dominated by low-frequency principal structures with bounded high-frequency oscillations along normal directions ( cf . , Appendix B).
Figure 3: rFID vs. gFID 10K comparison on DiT-XL (80 epochs). When scaling from d32 to d64 , DM-VAE with IGTS effectively overcomes the reconstruction–generation trade-off without external foundation models .
Figure 4: Overview of topological representation alignment (TRA) and internal guidance with topological skeleton ( IGTS ).
Figure 5: Qualitative generation results on ImageNet using DiT-XL with our foundation-model-free DM-VAE ( f16d32 ) and IGTS at only 64 training epochs ( without CFG ).
Methods
rFID ↓
PSNR ↑
LPIPS ↓
SSIM ↑
In-domain reconstruction performance
MAE-VAE ( f16d32 ) [ 49 ]
0.28
28.33
0.091
0.80
VA-VAE ( f16d32 ) [ 49 ]
0.28
27.96
0.096
0.79
RAE [ 52 ]
0.57
19.20
0.254
0.44
RAEv2 [ 39 ]
0.31
23.29
0.148
0.61
SD-VAE ( f16d32 ) [ 38 ]
0.26
28.59
0.089
0.80
Table 1: In-domain (ImageNet-1K) and out-of-distribution (OOD) reconstruction performance comparison. Without relying on external foundation models, our DM-VAE achieves state-of-the-art reconstruction fidelity across both benchmarks.
Methods
rFID ↓
FID 10K ↓ -B
FID 10K ↓ -XL
80 epo.
160 epo.
80 epo.
160 epo.
With foundation-model alignment
MAE-VAE ( f16d32 ) [ 49 ]
0.28
25.37
19.89
9.93
8.18
VA-VAE ( f16d32 ) [ 49 ]
0.28
20.34
15.73
8.22
6.75
DM-VAE Dino ( f16d32 )
0.27
14.06
12.11
6.99
5.97
+ IG TS
6.72
6.51
4.86
4.33
Table 2: Reconstruction and generation performance comparison on ImageNet using DiT [ 35 ] without CFG. Subscripts highlight FID10K improvements over the corresponding SD-VAE. Results are reported at 80 and 160 generative training epochs, with VAEs trained within 50 epochs. For IG TS , we consistently apply a guidance scale of 1.7 across the timestep interval of [0,0.6] .
Method
Epochs
#Params
Generation@256 w/o CFG
Generation@256 w/ CFG or Autoguidance
FID ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
FID ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
Latent Diffusion with Self-supervised Representation Models
REPA [ 50 ]
80
675M
7.90
5.06
122.6
0.70
0.65
-
-
-
-
-
800
5.90
5.69
157.8
0.70
0.69
1.42
4.70
305.7
0.80
0.65
REPA-E [ 27 ]
80
675M
3.46
4.17
159.8
0.77
0.63
1.67
4.12
266.3
0.80
0.63
800
1.83
-
217.3
-
-
1.26
4.11
314.9
0.79
0.66
Table 3: Quantitative comparison of state-of-the-art generative models on ImageNet-1K ( 256×256 ). We benchmark our method against vanilla latent diffusion models and foundation-model-aligned LDMs across different training epochs and model capacities. Our method establishes new state-of-the-art results with faster convergence in both foundation-model-free and foundation-model-assisted regimes. Note that our results in the “w/o CFG” column are evaluated with single-pass IGTS .
VAE
Align Target
Guidance
FID 50K ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
SD-VAE [ 38 ]
None
None
25.37
5.47
52.22
0.57
0.64
DM-VAE
None
None
19.03
5.47
69.06
0.61
0.64
HB ( z )
None
19.52
5.28
67.03
0.61
0.63
HB ( z )
IG [ 54 ]
10.98
4.08
91.40
0.70
0.61
LB ( zlow )
None
18.59
5.32
69.77
0.62
0.63
LB ( zlow )
IG TS
9.94
4.33
97.55
0.71
0.60
Table 4: Ablation study on DM-VAE, representation alignment, and internal guidance strategy at 80 epochs. Results are obtained under the f16d32 setting using DiT-B [ 35 ] without CFG. HB/LB denote the high/low-dimensional branch: IG [ 54 ] requires full-channel ( C -dim) HB predictions, whereas LB alignment predicts only the Clow -dim skeleton and thus pairs with IGTS .
Figure 6: Empirical verification of spectral decomposition and temporal SNR decoupling on 5K ImageNet validation images ( f16d64 , Clow=32 ). (a) Radially averaged 2D Fourier PSD in pixel space shows that the skeleton reconstruction D(z^) preserves global low-frequency structures, whereas the detail increment D(z)−D(z^) rises by over 44 dB in relative power ratio ( PSDres/PSDfull ) from k=1 to k=128 . (b) Transition rate curves dWk/dlogSNRk (Eqs. ( 16 )–( 17 )) parameterized by empirical σbase,σdetail on globally unit-variance-normalized latents confirm that while SD-VAE exhibits substantial base–detail overlap ( tdetail∗=0.68 ), DM-VAE pushes the detail transition to tdetail∗=0.91 , isolating structural velocity learning within [0,0.6] .
Figure 7: Qualitative visualization of latent subspace decomposition on ImageNet validation samples using DM-VAE ( f16d64 , Clow=32 ). From top to bottom: input image x , full reconstruction D(z) , decoded skeleton D(z^) , and residual ∣D(z)−D(z^)∣ . While D(z^) preserves global structure and color layout, the residual isolates fine-grained edges and high-frequency textures.
Methods
FID 50K ↓
sFID ↓
IS ↑
Prec. ↑
Rec. ↑
Improved-SiT-B † [ 32 ]
25.1
6.54
58.8
0.57
0.64
Improved-REPA-B † [ 50 ]
19.1
6.88
76.5
0.60
0.66
DDT-B (DM-VAE)
15.2
6.08
84.0
0.63
0.66
DDT-B (DM-VAE Dino )
11.0
5.47
94.8
0.70
0.62
+ IG TS
5.1
5.23
136.9
0.77
0.57
Improved-SiT-XL † [ 32 ]
10.9
5.30
103.4
0.66
0.65
Table 5: Generation performance comparison on ImageNet at 80 generative training epochs using DDT [ 45 ] without CFG. † denotes baselines improved by SwiGLU [ 43 ] , RoPE [ 41 ] , and RMSNorm [ 43 ] . Our DM-VAEs adopt the f16d32 configuration. For IG TS , we consistently apply a guidance scale of 1.7 across the timestep interval of [0,0.6] .
Figure 8: Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE ( f16d64 ) at 400 epochs.
Figure 9: Qualitative generation results using DiT-XL with our foundation-model-free DM-VAE ( f16d64 ) at 400 epochs.
Most visual generative models compress images into a latent space before applying diffusion or autoregressive modelling. Yet, existing approaches such as VAEs and foundation model aligned encoders implicitly constrain the latent space without explicitly shaping its distribution, making it unclear which types of distributions are optimal for modeling. We introduce Distribution-Matching VAE (DMVAE), which explicitly aligns the encoder's latent distribution with an arbitrary reference distribution via a distribution matching constraint. This generalizes beyond the Gaussian prior of conventional VAEs, enabling alignment with distributions derived from self-supervised features, diffusion noise, or other prior distributions. With DMVAE, we can systematically investigate which latent distributions are more conducive to modeling, and we find that SSL-derived distributions provide an excellent balance between reconstruction fidelity and modeling efficiency, reaching gFID equals 3.2 on ImageNet with only 64 training epochs. Our results suggest that choosing a suitable latent distribution structure (achieved via distribution-level alignment), rather than relying on fixed priors, is key to bridging the gap between easy-to-model latents and high-fidelity image synthesis. Code is avaliable at https://github.com/sen-ye/dmvae.
Sen Ye, Jianning Pei, Mengde Xu +4
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · University of Chinese Academy of Sciences · Tencent +1
Latent diffusion models have established a new state-of-the-art in high-resolution visual generation. Integrating Vision Foundation Model priors improves generative efficiency, yet existing latent designs remain largely heuristic. These approaches often struggle to unify semantic discriminability, reconstruction fidelity, and latent compactness. In this paper, we propose Geometric Autoencoder (GAE), a principled framework that systematically addresses these challenges. By analyzing various alignment paradigms, GAE constructs an optimized low-dimensional semantic supervision target from VFMs to provide guidance for the autoencoder. Furthermore, we leverage latent normalization that replaces the restrictive KL-divergence of standard VAEs, enabling a more stable latent manifold specifically optimized for diffusion learning. To ensure robust reconstruction under high-intensity noise, GAE incorporates a dynamic noise sampling mechanism. Empirically, GAE achieves compelling performance on the ImageNet-1K 256×256 benchmark, reaching a gFID of 1.82 at only 80 epochs and 1.31 at 800 epochs without Classifier-Free Guidance, significantly surpassing existing state-of-the-art methods. Beyond generative quality, GAE establishes a superior equilibrium between compression, semantic depth and robust reconstruction stability. These results validate our design considerations, offering a promising paradigm for latent diffusion modeling. Code and models are publicly available at https://github.com/sii-research/GAE.
Hangyu Liu, Jianyong Wang, Yutao Sun
Shanghai Innovation Institute · Tsinghua University
Representation Autoencoders (RAE) replace traditional VAE with pretrained vision encoders. In this paper, we systematically investigate several design choices and find three insights which simplify and improve RAE. First, we study a generalized formulation where the representation is defined as sum of the last k encoder layers rather than solely the final layer. This simple change greatly improves reconstruction without encoder finetuning or specialized data (e.g., text, faces). Second, we study the prevalent assumption that RAE (using pretrained representation as encoder) replaces representation alignment (REPA), which distills the same representation to intermediate layers instead. Through large-scale empirical analysis, we uncover a surprising finding: RAE and REPA exhibit complementary working mechanisms, allowing the same representation to be used as both encoder and target for intermediate diffusion layers. Finally, the original RAE struggles with classifier-free guidance (CFG) and requires training a second, weaker diffusion model for AutoGuidance (AG). We show that REPA itself can be viewed as x-prediction in RAE latent space. By simply re-parameterizing the output of the DiT model, it can provide guidance for "free". Overall, RAEv2 leads to more than 10x faster convergence over the original RAE, achieving a state-of-the-art gFID of 1.06 in just 80 epochs on ImageNet-256. On FDr6, RAEv2 achieves a state-of-the-art 2.17 at just 80 epochs compared to the previous best 3.26 (800 epochs) without any post-training. This motivates EPFID@k (epochs to reach unguided gFID < k) as a measure of training efficiency. RAEv2 attains an EPFID@2 of 35 epochs, versus 177 for the original RAE. We also validate our approach across diverse settings for text-to-image generation and navigation world models, showing consistent improvements. The code is available at https://raev2.github.io.