cs.CVNov 17, 2026

Sample-Aligned Decoders for Latent Generative Models

Authors: Yu Lei, Yi Li, Hongcheng Gao, Yipeng Wei, Liang Li, Li Su, Qingming Huang

Organizations: Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Tsinghua University

Abstract

Latent generative models synthesize images by decoding generated latents, yet their decoder is trained only to reconstruct encoder latents and never sees the latents it decodes at test time, so the residual errors of the generator are rendered directly into the final image. Existing remedies retrain the generator, add cost at every sampling step, or make the decoder robust to perturbed real latents, assuming that generator errors resemble such perturbations. We find that they do not. Generated latents lie on the encoder latent manifold as closely as real ones, yet follow a different distribution. The decoder should therefore be adapted to the distribution of generated latents, which raises two challenges: (1) unpaired supervision, since a generated latent has no ground-truth image, and (2) latent-space drift, since a decoder freed from paired targets forgets the latent space it must decode. We propose Sample-Aligned Decoders (SAD), a post-training framework that fine-tunes only the decoder of a frozen generator at no inference cost. Generated-Sample Distribution Alignment matches decoded generated latents to unpaired real images in a frozen foundation-model feature space, and Encoder Anchor Preservation keeps the decoder close to the original one on encoder latents. Our analysis shows that the alignment is effective only on generated latents, where it corrects each sample in proportion to how much more often the generator produces such latents than the encoder; on encoder latents it can only work against the anchor. SAD improves four state-of-the-art ImageNet generators across three tokenizers, e.g., FID-50K from 1.15 to 0.98 for REPA-E, complements discriminator guidance, and extends to text-to-image generation. Our code and models are available at https://torpedo2003.github.io/SAD/.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Efficient Fréchet Distributional Decoder Alignment for Latent Generative Models

    Oct 12, 2026Xianghong Fang, Jiajun Zhu, Esma Aimeur +2Flow-Based Generative Modeling

  2. A Latent Distribution Perspective on Evaluating and Improving Latent Generative Models

    Sep 21, 2026Xianghong Fang, Wenjie Shu, Tongda Xu +3Distribution ShiftGenerative AI Evaluation

  3. End-to-End Training for Unified Tokenization and Latent Denoising

    Mar 23, 2026Shivam Duggal, Xingjian Bai, Zongze Wu +5Latent Diffusion ModelsImage Generation