LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation

Zhengqiang Zhang; Lingchen Sun; Rongyuan Wu; Qiaosi Yi; Xiangtao Kong; Chaodong Xiao; Lei Zhang

cs.CVSep 29, 2026

LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation

Authors: Zhengqiang Zhang, Lingchen Sun, Rongyuan Wu, Qiaosi Yi, Xiangtao Kong, Chaodong Xiao, Lei Zhang

Organizations: The Hong Kong Polytechnic University · OPPO Research Institute

Abstract

Latent Diffusion Models (LDMs) typically adopt a two-stage pipeline: an auto-encoder (AE) is first pre-trained to define a latent space, then a diffusion model is trained to perform denoising within it. Such a two-stage design introduces a representation mismatch, as the latent space is optimized for reconstruction rather than adapting the denoising dynamics. We reveal that the LDM itself is an AE, and consequently present LDM-is-AE, an end-to-end one-stage LDM training framework that eliminates the need for a separately trained tokenizer. Our key observation is that the LDM backbone actually performs a latent-to-feature-to-latent transformation at each denoising step, which can be interpreted as an internal decoding--encoding process. Leveraging this structure, we split the DiT backbone into two reciprocal components, DiT-E (i.e., DiT Encoding) and DiT-D (i.e., DiT Decoding), and impose image-space supervision on the intermediate features across all timesteps. Our model encourages the internal representation to align with the image domain throughout denoising, thereby establishing an explicit latent-to-image-to-latent path. At the zero-noise timestep, our model further performs an image-to-latent-to-image mapping, corresponding to an auto-encoding process. As a result, LDM-is-AE jointly learns latent representations and denoising dynamics in an end-to-end manner, yielding a diffusion-native latent space tailored to the generation process. Experiments demonstrate that LDM-is-AE exhibits highly competitive generation performance, achieving an FID of 1.80 and 1.90 on 256x256 and 512x512 class-conditional image generation, respectively.

Figures & tables

Figure 1 : Illustration of the auto-encoding nature of LDM. (a) The LDM backbone (i.e., a DiT) naturally performs a latent-to-feature-to-latent transformation. (b) Image-space supervision can align intermediate features with the image domain across timesteps. (c) At the zero-noise timestep ( $t=1$ ), an explicit latent-to-image-to-latent path is established, which in turn enables the reverse image-to-latent-to-image auto-encoding process.

Figure 2 : Architecture and training design of LDM-is-AE. (a) Network architecture and training pipeline. (b) Time-aware auxiliary feature mixing.

	Method	Models	Repr.	Total Params (M)	Epochs	Aux. Data	Training FLOPs	w/ CFG
	Method	Models	Repr.	Total Params (M)	Epochs	Aux. Data	Training FLOPs	FID $\downarrow$	IS $\uparrow$
Two-stage	DiT-XL/2 [ 26 ]	Gen.+AE	Fixed	759	1400	[ 17 ]	45.4	2.27	278
	SiT-XL/2 [ 23 ]	Gen.+AE	Fixed	759	1400	[ 17 ]	45.4	2.06	270
	LightningDiT [ 40 ]	Gen.+AE+VFM	Fixed	745	800	-	19.1	1.35	295
	REPA-SiT [ 42 ]	Gen.+AE+VFM	Fixed	759	800	[ 17 ]	25.9	1.29	306
	DDT-XL/2 [ 37 ]	Gen.+AE+VFM	Fixed	759	400	[ 17 ]	19.2	1.26	311

Table 1 : Class-conditional ImageNet generation at

256\times 256

resolution. For Models , Gen. means generator, AE means autoencoder, Dec. means decoder, and VFM indicates the vision foundation model. For Repr. , Pixel means pixel diffusion, Fixed means fixed latent representation in training, and Dynamic means dynamically evolved representation in training. Aux. Data denotes external training data beyond ImageNet, and Training FLOPs report the generator-only training cost (

\times 10^{19}

Figure 3 : Class-conditional ImageNet samples generated by LDM-is-AE at $256\times 256$ resolution.

	Method	Models	Repr.	Total Params (M)	w/ CFG
	Method	Models	Repr.	Total Params (M)	FID $\downarrow$	IS $\uparrow$
Two- stage	DiT-XL/2 [ 26 ]	Gen.+AE	Fixed	759	3.04	241
	SiT-XL/2 [ 23 ]	Gen.+AE	Fixed	759	2.62	252
	REPA-SiT-XL/2 [ 42 ]	Gen.+AE+VFM	Fixed	759	2.08	275
One- stage	ADM-G [ 8 ]	Gen.	Pixel	559	7.72	173
	RIN [ 14 ]	Gen.	Pixel	320	3.95	216

Table 2 : Class-conditional ImageNet generation at

512\times 512

resolution.

Panel of Figure 5 — Figure 5 : Ablations on the three main design choices in LDM-is-AE.

Method	FID $\downarrow$	IS $\uparrow$	sFID $\downarrow$	Prec. $\uparrow$	Recall $\uparrow$
SiT-B/2	45.19	37.40	26.16	0.471	0.613
+ LDM-is-AE	41.68	40.70	27.34	0.485	0.626
JiT-B/16	30.27	54.80	22.01	0.5249	0.6753
+ LPIPS	27.57	61.09	21.71	0.5623	0.6662
+ LDM-is-AE	25.95	67.45	20.38	0.5545	0.6836

Table 4 : Ablation on architecture backbone and LPIPS supervision. All models are evaluated without classifier-free guidance. Bold marks our method and the best result in each column.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Sin.Obj.	Two.Obj	Counting	Colors	Pos	Color.Attr.	Overall $\uparrow$
PixArt- $\alpha$ [ 5 ]	0.98	0.50	0.44	0.80	0.08	0.07	0.48
SD3 [ 10 ]	0.98	0.84	0.66	0.74	0.40	0.43	0.68
PixNerd [ 36 ]	0.97	0.86	0.44	0.83	0.71	0.53	0.73
DeCo [ 24 ]	1.00	0.92	0.72	0.91	0.80	0.79	0.86
LDM-is-AE	0.99	0.95	0.75	0.93	0.62	0.74	0.83

Appendix

Table 5 : Text-to-image generation on the GenEval benchmark.

Figure 6 : Additional class-conditional ImageNet samples generated by LDM-is-AE. All images are produced at a resolution of $256\times 256$ .

Figure 7 : Reconstruction examples and latent visualizations for the auto-encoding path.

Scales $(w(t),w_{\text{lpips}})$	FID $\downarrow$	IS $\uparrow$
$(1.0,1.0)$	17.69	119
$(1.0,0.3)$	19.67	108
$(1.0,3.0)$	18.28	96
$(0.3,1.0)$	26.59	68
$(3.0,1.0)$	17.94	91

Appendix

Table 6 : Sensitivity to the loss weights

w(t)

and

w_{\text{lpips}}

of Eq. 7 . All models are evaluated without classifier-free guidance.

Explore similar work

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
May 8, 2026Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov +4Diffusion Language Models Latent Diffusion Model
Constructor University, Bremen, Germany · HSE University, Moscow, Russia · Applied AI Institute, Moscow, Russia +2
TextLDM: Language Modeling with Continuous Latent Diffusion
May 8, 2026Jiaxiu Jiang, Jingjing Ren, Wenbo Li +10Identity-Conditioned Latent Diffusion Models Multimodal Generation
Joy Future Academy · HIT · HKUST(GZ)
DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence
Sep 30, 2026Xu Huang, Ye Huang, Zijun Liao +6Generative Image Compression Token Compression
Peking University · Singapore University of Technology and Design

May 8, 2026, cs.CLJ/K move · Enter open · S save

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

Viacheslav Meshchaninov, Alexander Shabalin, Egor Chimbulatov, Nikita Gushchin+3

Constructor University, Bremen, Germany · HSE University, Moscow, Russia · Applied AI Institute, Moscow, Russia+2

Diffusion Language Models Latent Diffusion Model Diffusion Models Latent Space

Open paper AskPDF arXiv

Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent diffusion modeling is constructing a suitable latent space. In this work, we present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained language model with a trainable encoder, yielding latents that are easy to both denoise and decode into tokens. We show that naive joint training produces a low-quality diffusion model, and propose a simple training recipe consisting of an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models while being

2{\text -}13\times

faster, indicating that jointly learning the latent space is a key step toward making latent diffusion competitive for text generation.

Abstract

Explore similar work

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

TextLDM: Language Modeling with Continuous Latent Diffusion

DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence