Latent Diffusion Models

Follow this topic Ask about this topic

Latent diffusion models first encode data into a compact space, often with a variational autoencoder, then learn to generate it by reversing a noise process. Researchers use U-Nets and diffusion transformers, with text, images, masks, or physical variables as conditions, for tasks ranging from ImageNet image generation to medical imaging, audio, 3D shapes, and scientific reconstruction. Current work explores better latent spaces, faster sampling, and controls for detail, structure, and uncertainty. A central tension is that stronger compression can reduce compute but lose fine detail or physical fidelity, while good reconstruction alone does not guarantee that a latent space is easy to model.

Latest papers

207 papers19 papers in the last four weeks, up 36% on the four weeks before. 0.1% of all new papers.

SpeeDiff: Scalable Pixel-Anchored End-to-End Latent Diffusion Model
Oct 26, 2026Bingliang Zhang, Wenda Chu, Yizhuo Li +5Latent Diffusion Models Diffusion Transformer
ByteDance Seed · California Institute of Technology · Meta
Domain-Adapted Latent Diffusion Models for CT-to-PET Image Translation
Oct 26, 2026Yann Bourdé, Lea Bogensperger, Arnela Hadzic +1Medical Image Synthesis Image-to-Image Translation
Department of Quantitative Biomedicine, University of Zurich · Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz
UltraFlash: Accelerating Megapixel Visual Synthesis
Oct 24, 2026Phuc Lai, Anh Nguyen, Phong Nguyen +1Diffusion Model Inference Acceleration Diffusion-Based Image Generation
Qualcomm AI Research†
RoAR-VAE: Improving Latent Diffusability through Robust Prediction
Oct 19, 2026Yang Yue, Fangyun Wei, Yitong Wang +4Image Tokenization Variational Autoencoders
Tsinghua University · Microsoft Research
Learning the Dose Manifold: A Latent Diffusion Framework for Volumetric Dose Prediction in Automated Radiotherapy Treatment Planning
Oct 19, 2026J. Boyer-Chammard, R. Vauclin, M. C. Biston +7Radiotherapy Dose Prediction 3D Diffusion Models
Multimodal-Conditioned Latent Diffusion Models for Spatiotemporal Brain MRI Forecasting
Oct 18, 2026Antonio Scardace, Silvia Ingala, Sebastiano Battiato +1Brain MRI Alzheimer's Disease Progression Prediction
University of Catania · Copenhagen University Hospital – Rigshospitalet · Copenhagen University Hospital – Herlev and Gentofte +2
GOLD: Geometric Optimized Latent Diffusion for Structure-Aware RNA Inverse Folding
Oct 16, 2026Qi Si, Xuyang Liu, Penglei Wang +6Molecular Generation Inverse Design
Shanghai Academy of Artificial Intelligence for Science · School of Biomedical Engineering, Shanghai Jiao Tong University · Community Science Department, Kuaishou +2
Learning to Be Guided: Shaping Latent Geometry for Classifier-Free Guidance
Oct 16, 2026Haoming Liu, Junhao Zhu, Shaoyi Zheng +4Latent Space Geometry Diffusion-Based Image Generation
New York University · New York University Shanghai · Georgia Institute of Technology
UltraDiff: Transferring High-Fidelity Priors to Compressed Latent Spaces for High-Resolution Image Generation
Oct 15, 2026Jingjing Ren, Haitian Zheng, Haoyu Chen +5Latent Diffusion Models Diffusion Model Distillation
The Hong Kong University of Science and Technology (Guangzhou) · Adobe Research
Image Tokenizers Should Be Trained Between Images
Oct 12, 2026Ole Beißwenger, Jan-Niklas Dihlmann, Hendrik LenschImage Tokenization Latent Diffusion Models
University of Tübingen
OmniRSI: Taming Latent Diffusion for Unified Multimodal Remote Sensing Image Generation
Oct 11, 2026Hebaixu Wang, Jing Zhang, Haonan Guo +3Multimodal Generation Unified Multimodal Models
Wuhan University & Zhongguancun Academy
APEX: Asynchronous Prefix Denoising for Extreme Compression
Oct 9, 2026David Dinkevich, Nisan Chiprut, Yoav HaCohen +1Latent Diffusion Models Diffusion Model Compression
Lightricks · Hebrew University of Jerusalem
CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decoding
Oct 8, 2026Andrea Ceron, Michael Schmidt, Alvaro Marcos-Ramiro +2Pixel-Space Diffusion Models Latent Diffusion Models
TU München · BMW AG
Unified Multi-plane Autoregressive Diffusion for 3D Multi-contrast MRI Synthesis
Oct 7, 2026Yejee Shin, Geonhui Son, Jinglu Wang +3Medical Image Synthesis 3D Diffusion Models
Yonsei University · Microsoft Research Asia · Korea Institute of Science and Technology
Local Content-Style Control for Diffusion-based Image Stylization
Oct 6, 2026Amir SemmoLatent Diffusion Models Image Style Transfer
Digital Masterpieces GmbH Potsdam, Germany
PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation
Oct 6, 2026Yu Zhang, Xudong Xu, Xingang PanVariational Autoencoders Latent Diffusion Models
S-Lab, Nanyang Technological University, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China
EagleDepth: Efficient Fine-Grained Depth Estimation via Pixel Diffusion Decoder
Oct 3, 2026Bowen Chai, Tianbao Zhang, Shuyu Wu +3Depth Estimation Monocular Depth Estimation
Shanghai Jiao Tong University · Beihang University
Diffusable Latents from Structure-Agnostic Distillation
Sep 30, 2026Adrien Ramanana Rahary, Nicolas Dufour, Patrick Pérez +1Latent Diffusion Models Diffusion Model Distillation
LIGM, ENPC, IP Paris, CNRS, UGE · Kyutai
PCaPaint: Prostate Cancer Inpainting by Mitigating Shortcut Learning
Sep 29, 2026Levente Lippenszky, Hongxu Yang, Marcell Dömötör +2Synthetic Data Augmentation Latent Diffusion Models
GE HealthCare
LDM-is-AE: Latent Diffusion Model is an Auto-Encoder for End-to-End Image Generation
Sep 29, 2026Zhengqiang Zhang, Lingchen Sun, Rongyuan Wu +4Diffusion Transformer Latent Diffusion Models
The Hong Kong Polytechnic University · OPPO Research Institute
DirectUV: Image-Conditioned UV Texture Generation with Surface-Aware Positional Encoding
Sep 28, 2026Jiantao Lin, Yingjie Xu, Mingzhi Sheng +3Latent Diffusion Models Positional Encoding
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology
Perturb-and-Solve: Efficient Learned-Operator Conditioning for Latent Diffusion Inverse Problems
Sep 27, 2026Abduragim Shtanchaev, Arip Asadulaev, Luiza Labazanova +3Diffusion-Based Inverse Problems Image Restoration
Mohamed bin Zayed University of Artificial Intelligence
An End-to-End Latent-Rollout Approach for Pushing Few-Step ImageNet- $256$ Generation to FID $1.11$ without Fréchet Losses
Sep 26, 2026Xiaoran Xu, Yujing WangFew-Step Diffusion Sampling Latent Diffusion Models
ByteDance
ANaLOG: Anisotropic Native-Latent Operator Guidance for Solving Inverse Problems
Sep 25, 2026Darshan Thaker, Lachlan Ewen MacDonald, René VidalDiffusion Model Guidance Diffusion-Based Inverse Problems
University of Pennsylvania
On the Diffusibility of High-Dimensional Latents
Sep 23, 2026Chao Feng, Zhiyang Xu, Bowei Chen +7Flow Matching Latent Diffusion Models
Cornell University · Virginia Tech · University of Washington +2
Zero-Shot Object Removal via Attention Masking, Latent Anchoring, and Refinement
Sep 23, 2026Arman Taghizadeh, Ulf Krumnack, Kai-Uwe KühnbergerLatent Diffusion Models Image Inpainting
Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany
Physically Based Rendering in the Latent Space
Sep 17, 2026Vuk Radovanovic, Vishesh Gupta, Adrien Gruson +1Physically Based Rendering Physics-Informed Generative Modeling
Trinity College Dublin
Multimodal Conditioning of Fine-Tuned Stable Diffusion XL for Controllable and Culturally Faithful Ulos Motif Generation
Sep 16, 2026Humasak Simanjuntak, Tamara Yunika Sianipar, Bronson T. M Siallagan +2Latent Diffusion Models Image Generation
Faculty of Informatics and Electrical Engineering, Institut Teknologi Del, Indonesia
SlotDiT: Object-Centric Representations for Diffusion Transformers
Sep 15, 2026Gjergj Plepi, Sven BehnkeDiffusion Transformer Latent Diffusion Models
Autonomous Intelligent Systems, Computer Science Institute VI – Intelligent Systems and Robotics, Center for Robotics and the Lamarr Institute for Machine Learning and Artificial Intelligence, University of Bonn, Germany
MedDiME: Efficient Latent Diffusion with Adaptive Masking for Medical Counterfactual Generation
Sep 14, 2026Yan Zeng, Changlu Guo, Anders Nymark Christensen +2Counterfactual Explanations Latent Diffusion Models
Department of Applied Mathematics and Computer Science, Technical University of Denmark

Oct 26, 2026J/K move · Enter open · S save

SpeeDiff: Scalable Pixel-Anchored End-to-End Latent Diffusion Model

Bingliang Zhang, Wenda Chu, Yizhuo Li, Linjie Yang+4

ByteDance Seed · California Institute of Technology · Meta

Latent Diffusion Models Diffusion Transformer Diffusion Model Acceleration

Open paper AskPDF arXiv

We present Scalable Pixel-anchored End-to-end Diffusion (SpeeDiff), a latent diffusion method that jointly trains the VAE and the diffusion model from scratch. In principle, joint training allows the diffusion loss gradient to directly guide the VAE encoder, encouraging the formation of a generation-friendly latent space and potentially yielding faster convergence than the conventional two-stage approach with a pretrained frozen VAE. However, a naive end-to-end implementation severely degrades performance, as unrestricted backpropagation of the diffusion loss leads to latent space collapse. Our main technical contribution is a simple yet effective Tweedie Pixel Reconstruction (TPR) loss, which provides additional pixel-level feedback by decoding a predicted clean latent from an intermediate noisy state using Tweedie's formula, thereby alleviating collapse. Furthermore, our method enables jointly scaling a fully transformer-based architecture and enhances representation alignment within the end-to-end framework. Our SpeeDiff-XL model achieves over 140x and 61x faster training compared to Vanilla SiT and REPA, respectively, while attaining an FID of 1.50 without guidance on ImageNet 256x256 generation. With a more efficient 32x compressed VAE, our model further reaches an FID of 1.53 without guidance on ImageNet 512x512 generation.