Dual-Manifold Variational Autoencoders
Organizations: The Hong Kong University of Science and Technology
Abstract
In latent diffusion models, expanding the latent dimension of variational autoencoders (VAEs) is essential to push the upper bound of reconstruction quality. However, without explicit structural constraints, such unregularized expansion easily yields a rugged and ill-conditioned latent manifold, causing generated latents to drift off the manifold during generation (i.e., off-manifold problem). Recent works smooth this manifold by using external foundation models (e.g., DINOv2) as a topological regularizer. However, we argue that tying the latent space to frozen external priors inherently restricts generalization: bound by their pre-training distributions and semantic biases, they may fail to capture faithful visual details on out-of-distribution domains. Observing that low-dimensional latents naturally form a smoother, generation-friendly manifold, we propose dual-manifold VAE (DM-VAE) to leverage internal representations as a topological regularizer. Specifically, DM-VAE pairs a high-dimensional reconstructive branch with a low-dimensional branch that extracts a clean topological skeleton, anchoring the high-dimensional manifold to this smooth subspace. Building upon this framework, we further introduce topological representation alignment (TRA) to supervise intermediate generator features during training, and internal guidance with topological skeleton (IG_TS) to guide structural components while preserving high-frequency details at inference. On ImageNet-1K (256x256), our foundation-model-free DM-VAE achieves a state-of-the-art rFID of 0.12 and a generation FID50K of 1.28 (w/o CFG) and 1.20 (w/CFG), and further reaches 1.11 when orthogonally combined with DINOv2.
Figures & tables
| Methods | rFID | PSNR | LPIPS | SSIM |
|---|---|---|---|---|
| In-domain reconstruction performance | ||||
| MAE-VAE ( f16d32 ) [ 49 ] | 0.28 | 28.33 | 0.091 | 0.80 |
| VA-VAE ( f16d32 ) [ 49 ] | 0.28 | 27.96 | 0.096 | 0.79 |
| RAE [ 52 ] | 0.57 | 19.20 | 0.254 | 0.44 |
| RAEv2 [ 39 ] | 0.31 | 23.29 | 0.148 | 0.61 |
| SD-VAE ( f16d32 ) [ 38 ] | 0.26 | 28.59 | 0.089 | 0.80 |
| Methods | rFID | FID 10K -B | FID 10K -XL | ||
|---|---|---|---|---|---|
| 80 epo. | 160 epo. | 80 epo. | 160 epo. | ||
| With foundation-model alignment | |||||
| MAE-VAE ( f16d32 ) [ 49 ] | 0.28 | 25.37 | 19.89 | 9.93 | 8.18 |
| VA-VAE ( f16d32 ) [ 49 ] | 0.28 | 20.34 | 15.73 | 8.22 | 6.75 |
| DM-VAE ( f16d32 ) | 0.27 | 14.06 | 12.11 | 6.99 | 5.97 |
| + IG TS | 6.72 | 6.51 | 4.86 | 4.33 | |
| Method | Epochs | #Params | Generation@256 w/o CFG | Generation@256 w/ CFG or Autoguidance | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FID | sFID | IS | Prec. | Rec. | FID | sFID | IS | Prec. | Rec. | |||
| Latent Diffusion with Self-supervised Representation Models | ||||||||||||
| REPA [ 50 ] | 80 | 675M | 7.90 | 5.06 | 122.6 | 0.70 | 0.65 | - | - | - | - | - |
| 800 | 5.90 | 5.69 | 157.8 | 0.70 | 0.69 | 1.42 | 4.70 | 305.7 | 0.80 | 0.65 | ||
| REPA-E [ 27 ] | 80 | 675M | 3.46 | 4.17 | 159.8 | 0.77 | 0.63 | 1.67 | 4.12 | 266.3 | 0.80 | 0.63 |
| 800 | 1.83 | - | 217.3 | - | - | 1.26 | 4.11 | 314.9 | 0.79 | 0.66 | ||
| VAE | Align Target | Guidance | FID 50K | sFID | IS | Prec. | Rec. |
|---|---|---|---|---|---|---|---|
| SD-VAE [ 38 ] | None | None | 25.37 | 5.47 | 52.22 | 0.57 | 0.64 |
| DM-VAE | None | None | 19.03 | 5.47 | 69.06 | 0.61 | 0.64 |
| HB ( ) | None | 19.52 | 5.28 | 67.03 | 0.61 | 0.63 | |
| HB ( ) | IG [ 54 ] | 10.98 | 4.08 | 91.40 | 0.70 | 0.61 | |
| LB ( ) | None | 18.59 | 5.32 | 69.77 | 0.62 | 0.63 | |
| LB ( ) | IG TS | 9.94 | 4.33 | 97.55 | 0.71 | 0.60 |
| Methods | FID 50K | sFID | IS | Prec. | Rec. |
|---|---|---|---|---|---|
| Improved-SiT-B † [ 32 ] | 25.1 | 6.54 | 58.8 | 0.57 | 0.64 |
| Improved-REPA-B † [ 50 ] | 19.1 | 6.88 | 76.5 | 0.60 | 0.66 |
| DDT-B (DM-VAE) | 15.2 | 6.08 | 84.0 | 0.63 | 0.66 |
| DDT-B (DM-VAE ) | 11.0 | 5.47 | 94.8 | 0.70 | 0.62 |
| + IG TS | 5.1 | 5.23 | 136.9 | 0.77 | 0.57 |
| Improved-SiT-XL † [ 32 ] | 10.9 | 5.30 | 103.4 | 0.66 | 0.65 |
Variational Autoencoders
Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.
See all 221 papers