Diffusion Models

Ask about this topic

Overview

Diffusion models learn a data distribution by gradually corrupting examples with noise and training a denoiser to reverse that process. Major approaches include DDPMs and score-based SDEs, latent diffusion and diffusion Transformers, flow and consistency methods, and discrete masked or continuous-time Markov diffusion; conditioning and guidance adapt them to specific tasks. They are used for images, video, audio, language, molecules, and scientific forecasting, with evaluations ranging from ImageNet FID to language perplexity and domain-specific measures of accuracy and uncertainty. A central tension is how to make sampling faster and generation more controllable without sacrificing quality, calibration, or diversity, while theory and ongoing audits address generalization, memorization, and privacy.

Latest papers

CardsList

1,192 papers152 papers in the last four weeks, up 77% on the four weeks before. 0.9% of all new papers.

May 11, 2026

HiDream-O1-Image: A Natively Unified Image Generative Foundation Model with Pixel-level Unified Transformer

The evolution of visual generative models has long been constrained by fragmented architectures relying on disjoint text encoders and external VAEs. In this report, we present HiDream-O1-Image, a natively unified generative foundation model via pixel-space Diffusion Transformer, that pioneers a paradigm shift from modular architectures to an end-to-end in-context visual generation engine. By mapping raw image pixels, text tokens, and task-specific conditions into a single shared token space, HiDream-O1-Image achieves a structural unification of multimodal inputs within an Unified Transformer (UiT) architecture. This native encoding paradigm eliminates the need for separate VAEs or disjoint pre-trained text encoders, allowing the model to treat diverse generation and editing tasks as a consistent in-context reasoning process. Extensive experiments show that HiDream-O1-Image excels across various generation tasks, including text-to-image generation, instruction-based editing, and subject-driven personalization. Notably, with only 8B parameters, HiDream-O1-Image (8B) achieves performance parity with or even surpasses established state-of-the-art models with significantly larger parameters (e.g., 27B Qwen-Image). Crucially, to validate the immense scalability of this paradigm, we successfully scale the architecture up to over 200B parameters. Experimental results demonstrate that this massive-scale version HiDream-O1-Image-Pro (200B+) unlocks unprecedented generative capabilities and superior performance, establishing new state-of-the-art benchmarks. Ultimately, HiDream-O1-Image highlights the immense potential of natively unified architectures and charts a highly scalable path toward next-generation multimodal AI.

Qi Cai, Jingwen Chen, Chengmin Gao +22

Diffusion TransformerText-Guided Image EditingUnified Multimodal ModelsPixel-Space Diffusion ModelsMultimodal Foundation ModelsT2I Diffusion Models+1
AskPDF
May 11, 2026

Elucidating Representation Degradation Problem in Diffusion Model Training

Diffusion models have achieved remarkable success, yet their training remains inefficient due to a severe optimization bottleneck, which we term Representation Degradation. As noise levels increase, the outputs of the trained model exhibit progressive structural distortion, which can destabilize training and impair generation quality. Our analysis suggests that this instability is driven by mismatched target recoverability, which is associated with Neural Tangent Kernel (NTK) spectral weakening and effective low-rank behavior. To address this, we propose Elucidated Representation Diffusion (ERD), a plug-and-play framework that dynamically reallocates optimization effort according to effective recoverability. By stabilizing representation learning without external supervision, ERD accelerates convergence and achieves strong empirical performance across diffusion backbones.

Zhipeng Yao, Dazhou Li, Zitong Zhang +6

University of Louisville · Shenyang University of Chemical Technology · Florida International University +2

Deep Learning OptimizationDiffusion Model AccelerationDiffusion Models
AskPDF
May 11, 2026

Heteroscedastic Diffusion for Multi-Agent Trajectory Modeling

Multi-agent trajectory modeling traditionally focuses on forecasting, often neglecting more general tasks like trajectory completion, which is essential for real-world applications such as correcting tracking data. Existing methods also generally predict agents' states without offering any state-wise measure of heteroscedastic uncertainty. Moreover, popular multi-modal sampling methods lack error probability estimates for each generated scene under the same prior observations, which makes it difficult to rank the predictions at inference time. We introduce U2Diffine, a unified diffusion model built to perform trajectory completion while simultaneously offering state-wise heteroscedastic uncertainty estimates. This is achieved by augmenting the standard denoising loss with the negative log-likelihood of the predicted noise, and then propagating the latent space uncertainty to the real state space using a first-order Taylor approximation. We also propose U2Diff, a faster baseline that avoids gradient computation during sampling. This approach significantly increases inference speed, making it as efficient as a standard generative-only diffusion model. For post-processing, we integrate a Rank Neural Network (RankNN) that enables error probability estimation for each generated mode, demonstrating strong correlation with ground truth errors. Our method outperforms state-of-the-art solutions in both trajectory completion and forecasting across four challenging sports datasets (NBA, Basketball-U, Football-U, Soccer-U), underscoring the effectiveness of our uncertainty and error probability estimation.

Guillem Capellera, Antonio Rubio, Luis Ferraz +1

Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Barcelona, 08028, Spain · Kognia Sports Intelligence, 08960, Sant Just Desvern, Spain

Multi-Agent Trajectory PredictionTrajectory PredictionDiffusion Models
AskPDF
May 11, 2026

diffGHOST: Diffusion based Generative Hedged Oblivious Synthetic Trajectories

Trajectories are nowadays valuable information for a wide range of applications. However they are also inherently sensitive, as they contain highly personal information about individuals. Facing this challenge, synthesizing mobility trajectories has emerged as a promising solution to leverage mobility information while preserving privacy. State-of-the-art models, often rely on the false assumptions of generative models implicit privacy and fails to provide privacy guarantees while preserving trajectories utility. Here, we introduce diffGHOST, a conditional diffusion model based on latent space segmentation, designed to answer this challenge. Thus, this paper propose a methodology that identify and mitigate memorization of critical samples using condition segments of a learn latent space.

Florent Guépin, Cheick Tidiani Cisse, Denis Renaud +2

Orange Research, Belfort, France · Orange Business, Belfort, France

Trajectory GenerationSynthetic Data GenerationMemorization in Generative ModelsGenerative ModelingConditional Generative ModelingDiffusion Models
AskPDF
May 11, 2026

GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Tasks

Data-driven medical AI is traditionally formulated as a discriminative mapping from input XX to output YY via a learned function ff, which does not generalize well across heterogeneous data and modalities encountered in real-world clinical settings. In this work, we propose a fundamentally different, generative paradigm. We model the joint distribution P(X,Y)P(X,Y) using diffusion models and reframe inference as a test-time output optimization problem. By guiding the generative process to match observed inputs, our framework enables flexible, gradient-based conditioning at inference time without architectural changes or retraining, effectively supporting arbitrary and previously unseen combinations of observations. Extensive experiments demonstrate strong performance across standard and cross-modality medical image segmentation, few-shot segmentation with only 2 or 4 training samples, degraded-input segmentation, shape completion from sparse and partial observations, and zero-shot application to demonstrate generality. To support these evaluations, we curated and released a large-scale text-shape dataset derived from MedShapeNet. Our results highlight the versatility of generative joint modeling as a foundation for reusable, task-agnostic medical AI systems.

Hantao Zhang, Weidong Guo, Yuhe Liu +5

CVLab, École Polytechnique Fédérale de Lausanne (EPFL), Switzerland · Fudan University, Shanghai, China · Beihang University, Beijing, China +3

Cross-Modality Medical Image SegmentationDiffusion Model GuidanceHealthcareGenerative ModelingDiffusion ModelsMedical Image Segmentation+2
AskPDF
May 11, 2026

Composing diffusion priors with explicit physical context via generative Gibbs sampling

Pretrained diffusion models provide powerful learned priors, but in scientific sampling the target distribution often depends on physical context that is not fully represented by one generative model. We introduce Generative Gibbs for Physics-Aware Sampling (GG-PA), a training-free framework that formulates the composition of learned partial priors and explicit physical context as inference over a joint target distribution in an augmented state space. We derive a Gibbs sampler for this joint target, show that it is asymptotically exact as the diffusion time approaches zero, and prove that in settings with quadratic interactions it remains exact at finite diffusion times. We further introduce replica exchange over diffusion time to accelerate mixing. Experiments on a double-well system, a φ4φ^4 lattice model, and atomistic peptide systems show that GG-PA recovers context-induced distribution shifts and emergent collective behavior in interacting systems using partial priors without retraining. These results demonstrate GG-PA as a practical approach for combining pretrained generative priors with explicit physical context.

Weizhou Wang, Jonathan Weare, Aaron R. Dinner

Department of Chemistry University of Chicago Chicago, IL 60637 · Courant Institute of Mathematical Sciences New York University New York, NY 10012

Diffusion Posterior SamplingDiffusion Models
AskPDF
May 11, 2026

Product-of-Gaussian-Mixture Diffusion Models for Joint Nonlinear MRI Reconstruction

Recently, diffusion models have attracted considerable attention for magnetic resonance image reconstruction due to their high sample quality. However, most existing methods rely on large networks with opaque time-conditioning mechanisms, and require offline coil sensitivity estimation. This results in limited interpretability of the reconstruction process and reduced flexibility in the acquisition setup. To address these limitations, we jointly reconstruct the image and the coil sensitivities by combining the parameter-efficient product-of-Gaussian-mixture diffusion model as an image prior with a classical smoothness prior on the coil sensitivities. The proposed method is fast and robust to both contrast and anatomical distribution shifts as well as changing k-space trajectories. Finally, we propose a more expressive parameterization of the image prior which improves results in denoising and magnetic resonance image reconstruction.

Laurenz Nagler, Martin Zach, Thomas Pock

Graz University of Technology, Institute of Visual Computing, 8010 Graz, Austria. · Ecole Polytechnique Fédérale de Lausanne, Biomedical Imaging Group and Center for Biomedical Imaging, 1015 Lausanne, Switzerland.

Accelerated MRI ReconstructionMagnetic Resonance ImagingDiffusion-Based Inverse ProblemsDiffusion MRIDiffusion Models
AskPDF
May 11, 2026

Infinite Mask Diffusion for Few-Step Distillation

Masked Diffusion Models (MDMs) have emerged as a promising alternative to autoregressive models in language modeling, offering the advantages of parallel decoding and bidirectional context processing within a simple yet effective framework. Specifically, their explicit distinction between masked tokens and data underlies their simple framework and effective conditional generation. However, MDMs typically require many sampling iterations due to factorization errors stemming from simultaneous token updates. We observe that a theoretical lower bound of the factorization error exists, which standard MDMs cannot reduce due to their use of a deterministic single-state mask. In this paper, we propose the Infinite Mask Diffusion Model (IMDM), which introduces a stochastic infinite-state mask to mitigate the theoretical bound while directly inheriting the benefits of MDMs, including the compatibility with pre-trained weights. We empirically demonstrate that MDM fails to perform few-step generation even in a simple synthetic task due to the factorization error bound, whereas IMDM can find an efficient solution for the same task. Finally, when equipped with appropriate distillation methods, IMDM surpasses existing few-step distillation methods at small step counts on LM1B and OpenWebText. Code is available at https://Ugness.github.io/official_imdm.

Jaehoon Yoo, Wonjung Kim, Chanhyuk Lee +1

Korea Advanced Institute of Science and Technology (KAIST)

Masked Diffusion ModelsDiffusion Model DistillationFew-Step Diffusion SamplingDiffusion Language ModelsDiffusion ModelsDiffusion Language Model Distillation
AskPDF
May 11, 2026

Filtering Memorization from Parameter-Space in Diffusion Models

Low-Rank Adaptation (LoRA) has become a widely used mechanism for customizing diffusion models, enabling users to inject new visual concepts or styles through lightweight parameter updates. However, LoRAs can memorize training images, causing generated outputs to reproduce copyrighted or sensitive content. This risk is particularly concerning in LoRA-sharing ecosystems, where users distribute trained LoRAs without releasing the underlying training data. Existing approaches for mitigating memorization rely on access to the training pipeline, training data, or control over the inference process, making them difficult to apply when only the released LoRA weights are available. We propose Base-Anchored Filtering (BAF), a training-free and data-free framework for post-hoc memorization mitigation in diffusion LoRAs. BAF decomposes LoRA updates into spectral channels and measures their alignment with the principal subspace of the pretrained backbone. Channels strongly aligned with this subspace are retained as generalizable adaptations, while weakly aligned channels are suppressed as potential carriers of memorized content. Experiments on multiple datasets and diffusion backbones demonstrate that BAF consistently reduces memorization while preserving or even improving generation quality. Our code is available in the supplementary material.

Yu Zhe, Yang Jiayan, Wei Junhao +2

RIKEN AIP · Institute of Science Tokyo · University of California, Berkeley +1

Low-Rank AdaptationMemorization in Generative ModelsDiffusion Models
AskPDF
May 11, 2026

A Stability Benchmark of Generative Regularizers for Inverse Problems

Generative (diffusion) priors demonstrate remarkable performance in addressing inverse problems in imaging. Yet, for scientific and medical imaging, it is crucial that reconstruction techniques remain stable and reliable under imperfect settings. Typical definitions of stability encompass the notion of ''convergent regularization'', robustness to out-of-distribution data, and to inaccuracies in the forward operator or noise model. We evaluate these properties numerically. Furthermore, we benchmark generative approaches against modern optimization-based methods inspired by the widely used variational techniques. Our results give insights for which settings and applications generative priors can deliver state-of-the-art reconstructions, and on those in which they fall short or may even be problematic.

Alexander Denker, Johannes Hertrich, Sebastian Neumayer

Helmholtz Imaging, Deutsches Elektronen-Synchrotron DESY, Germany · ENS Paris & Inria, France · Chemnitz University of Technology, Germany

Generative ModelingDiffusion-Based Inverse ProblemsInverse ProblemsDiffusion ModelsImage Inverse Problems
AskPDF
May 11, 2026

TrajDLM: Topology-Aware Block Diffusion Language Model for Trajectory Generation

Generating high-fidelity synthetic GPS trajectories is increasingly important for applications in transportation, urban planning, and what-if scenario simulation, especially as privacy concerns limit access to real-world mobility data. Existing trajectory generation models face a trade-off between efficiency and faithfulness to road network topology: continuous-space methods enable fast generation but ignore the road network, while topology-aware approaches rely on search-based autoregressive decoding that limits generation speed. We propose TrajDLM, a topology-aware trajectory generation framework based on block diffusion language models that bridges this gap. TrajDLM models trajectories as sequences of discrete road segments, combining a block diffusion backbone for efficient denoising, topology-aware embeddings from a road network encoder, and topology-constrained sampling to ensure coherent and realistic trajectories. Across three city-scale datasets, TrajDLM achieves strong performance on fine-grained local similarity metrics while being up to 2.8×2.8\times faster than prior work, and demonstrates strong zero-shot transfer across domains, including unseen transportation modes. These results highlight the effectiveness of block-wise discrete diffusion as a scalable approach to accurate and efficient trajectory generation. Our code is available at https://github.com/cruiseresearchgroup/TrajDLM/

Wilson Wongso, Lihuan Li, Arian Prabowo +4

University of New South Wales · Hong Kong University of Science and Technology (Guangzhou)

Trajectory GenerationSynthetic Data GenerationDiffusion Language ModelsDiffusion Models
AskPDF
May 11, 2026

The two clocks and the innovation window: When and how generative models learn rules

Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales: τruleτ_{\mathrm{rule}}, the step at which generations first become rule-valid, and τmemτ_{\mathrm{mem}}, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks, τruleτ_{\mathrm{rule}} and τmemτ_{\mathrm{mem}}, depend on key aspects of the learning setup. Specifically, we show that τruleτ_{\mathrm{rule}} increases with rule complexity and decreases with model capacity, while τmemτ_{\mathrm{mem}} is approximately invariant to the rule and scales nearly linearly with dataset size NN. We define the innovation window as the interval [τrule,τmem][τ_{\mathrm{rule}}, τ_{\mathrm{mem}}]. This window widens with increasing NN and narrows with rule complexity, and may vanish entirely when τrule≥τmemτ_{\mathrm{rule}} \geq τ_{\mathrm{mem}}. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around τruleτ_{\mathrm{rule}}, while training samples' basins begin to dominate around τmemτ_{\mathrm{mem}}. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.

Binxu Wang, Emma Lucia Byrnes Finn, Bingbin Liu

Kempner Institute at Harvard University

Memorization in Generative ModelsGenerative ModelingAutoregressive GenerationNeural Network Training DynamicsDiffusion Models
AskPDF
May 10, 2026

TD3B: Transition-Directed Discrete Diffusion for Allosteric Binder Generation

Protein function is often controlled by ligands that bias the direction of state transitions, such as agonists and antagonists, rather than stabilizing a single conformation. This is especially important for clinically relevant G protein-coupled receptors (GPCRs), where therapeutic efficacy depends on functional directionality. Structure-based design methods optimize binding to static conformations and cannot represent non-reversible, directional effects or systematically distinguish agonist from antagonist behavior. To address this gap, we introduce Transition-Directed Discrete Diffusion for Allosteric Binder Design (TD3B), a sequence-based generative framework that designs binders with specified agonist or antagonist behavior via a directional transition control objective. TD3B combines a target-aware Direction Oracle, a soft binding-affinity gate, and amortized fine-tuning of a pre-trained discrete diffusion model, enabling targeted agonist and antagonist generation decoupled from binding affinity and unattainable by equilibrium-based or inference-only guidance baselines. The code and checkpoints are available at https://huggingface.co/ChatterjeeLab/TD3B.

Hanqun Cao, Aastha Pal, Sophia Tang +4

Department of Computer Science and Engineering, The Chinese University of Hong Kong · Department of Bioengineering, University of Pennsylvania · Department of Computer and Information Science, University of Pennsylvania +1

Molecular GenerationProtein DesignConditional Diffusion ModelsDrug DiscoveryDiffusion ModelsProtein Binder Design
AskPDF
May 10, 2026

Primal-Dual Guided Decoding for Constrained Discrete Diffusion

Discrete diffusion models generate structured sequences by progressively unmasking tokens, but enforcing global property constraints during generation remains an open challenge. We propose primal-dual guided decoding, an inference-time method that formulates constrained generation as a KL-regularised optimisation problem and solves it online via adaptive Lagrangian multipliers. At each denoising step, the method modifies token logits through an additive, constraint-dependent bias, with multipliers updated by mirror descent based on constraint violation. The bias arises as the optimal KL-regularised projection of the constraint, so the constrained distribution remains as close as possible to the model's unconstrained distribution while still satisfying the constraint. The method requires no retraining and no additional model evaluations beyond standard sampling, supports multiple simultaneous constraints, and provides formal bounds on constraint violation. We evaluate our approach on topical text generation, molecular design, and music playlist generation, showing that a single algorithm instantiated via domain-specific scoring functions improves constraint satisfaction while preserving relevant domain-specific quality metrics.

Federico Tomasi, Dmitrii Moor, Alice Wang +1

Spotify

Primal-Dual OptimizationConstrained Generative ModelingConstrained DecodingDiffusion GuidanceConstrained OptimizationDiffusion Models+1
AskPDF
May 10, 2026

Backbone-Equated Diffusion OOD via Sparse Internal Snapshots

Fair comparison between diffusion-based OOD detectors is challenging, as conclusions can vary with backbone choice, corruption parameterization, and test-time budget. We address this issue through a Mutualized Backbone-Equated (MBE) protocol that aligns canonical corruption levels and logical test-time cost across diffusion backbones. Within this setting, we introduce Canonical Feature Snapshots (CFS), a family of detectors that probes a frozen diffusion backbone using only a tiny number of native internal activations at canonical low-noise levels. On a controlled CIFAR-scale benchmark, the strongest one-forward CFS variant is CFS(1x2), while an even smaller decoder-only variant remains highly competitive. This shows that much of the relative-OOD signal exposed by frozen diffusion backbones is concentrated in a small number of sparse internal states, rather than requiring full denoising trajectories or high-capacity downstream heads. We further provide a local diagnostic theory explaining these observations through conditional encoder-decoder complementarity, diagonal-score separation, and low-noise corruption stability. The official implementation is available at https://github.com/RouzAY/cfs-diffusion-ood/.

Yadang Alexis Rouzoumka, Jean Pinsolle, Eugénie Terreaux +3

DEMR, ONERA & SONDRA · Université Paris-Saclay · SONDRA, CentraleSupélec +1

Diffusion-Based OOD DetectionOOD DetectionDiffusion Models
AskPDF
May 10, 2026

PermuQuant: Lowering Per-Group Quantization Error by Reordering Channels for Diffusion Models

Large-scale visual generative models have achieved remarkable performance. However, their high computational and memory costs make deployment challenging in resource-constrained scenarios, such as interactive applications and personal single-GPU usage. Post-training quantization (PTQ) offers a practical solution by compressing pretrained models without expensive retraining. However, existing PTQ methods still suffer from severe quality degradation under extremely low-bit settings. In this paper, we identify channel ordering as an important but underexplored factor in per-group quantization. In this setting, each contiguous group shares one quantization scale. When channels with very different statistics are placed in the same group, the scale can be dominated by outliers and cause large quantization errors. Based on this observation, we propose PermuQuant, a simple and effective PTQ framework for low-bit diffusion models. PermuQuant sorts channels by a joint second-moment criterion before per-group quantization, placing channels with similar activation and weight statistics into the same group. It further uses a calibration-based acceptance rule to apply reordering only when the selected permutation reduces quantization error on calibration data. The selected permutations are absorbed into adjacent modules or applied to weights offline, avoiding explicit runtime permutation operations. Extensive experiments on multiple large diffusion models show that PermuQuant consistently reduces quantization error and outperforms existing PTQ baselines. On FLUX.1-dev with an RTX 5090, PermuQuant achieves up to a 1.7×\times single step speedup and reduces the DiT memory footprint by 3.5×\times under W4A4 NVFP4 quantization. Code will be available at https://github.com/yscheng04/PermuQuant.

Yongsen Cheng, Kai Liu, Kaiwen Tao +5

Shanghai Jiao Tong University · Huawei Noah’s Ark Lab

Diffusion Model QuantizationPost-Training QuantizationLow-Bit QuantizationNeural Network QuantizationDiffusion Models
AskPDF
May 10, 2026

AtteConDA: Attention-Based Conflict Suppression in Multi-Condition Diffusion Models and Synthetic Data Augmentation

Recent conditional image generation methods can improve controllability by generating images that are faithful to conditions such as sketches, human poses, segmentation maps, and depth. By applying these techniques to image augmentation while preserving annotations, generated images can be used as additional training data and can improve recognition performance. However, for high-level driving tasks such as traffic-rule extraction and driving-behavior understanding, simply using annotations as conditions is insufficient. Instead, images must be augmented while preserving the detailed high-level structure of the original scene. One possible solution is to use multiple conditions so that generated images retain diverse structural cues after generation. However, when multiple conditions are used, conflicts among conditions can prevent reliable structure preservation. In this work, we input semantic segmentation, depth, and edges extracted from the original image into a multi-condition image generation model, thereby providing rich structural information as conditions. We further propose a modeling approach for handling conflicts among multiple conditions and show that it enables image generation with stronger structural preservation. We also build a generation framework and evaluation protocol for driving tasks, establishing a basis for comparison with prior and future models. As a result, this work contributes to image generation research by addressing condition conflicts in multi-condition generation and provides an important step toward mitigating data scarcity in high-level autonomous-driving tasks.

Shogo Noguchi

Gunma University

Synthetic Data AugmentationConditional Image GenerationAutonomous DrivingDiffusion Models
AskPDF
May 10, 2026

PGID: Progressive Guided Inversion and Denoising for Robust Watermark Detection

With the proliferation of AI-generated images, digital watermarking has become an essential safeguard for protecting intellectual property and mitigating malicious exploitation. Recent works on semantic watermarking have enabled efficient copyright protection for diffusion models. However, the dependence of semantic watermarking on diffusion inversion for watermark detection creates a critical vulnerability. Imprint removal and forgery attacks exploit this weakness to produce deceptive results. Our analysis reveals that these attacks succeed by displacing watermarked latents into the unwatermarked region, while guiding unwatermarked latents into the watermarked region. Based on that, we propose Progressive Guided Inversion and Denoising (PGID), the first plug-and-play, training-free noise extraction framework designed to defend against both attack strategies. PGID effectively defends by projecting perturbed latents back to the region where they originally belong. The projection is achieved by eliminating intermediate latent deflections and mitigating adversarial perturbations through progressive inversion-denoising cycles. Comprehensive evaluations across multiple schemes demonstrate that PGID successfully restores detection reliability by recovering removed watermarks and identifying forged instances.

Minh Quoc Duong, Chun Tong Lei, Chun Pong Lau

City University of Hong Kong

Diffusion Model InversionDiffusion Model WatermarkingWatermark RemovalImage Forgery DetectionRobust WatermarkingDiffusion Models
AskPDF
May 10, 2026

Attention Sinks in Diffusion Transformers: A Causal Analysis

Attention sinks -- tokens that receive disproportionate attention mass -- are assumed to be functionally important in autoregressive language models, but their role in diffusion transformers remains unclear. We present a causal analysis in text-to-image diffusion, dynamically identifying dominant attention recipients per timestep and suppressing them via paired, training-free interventions on the score and value paths. Across 553 GenEval prompts on Stable Diffusion~3 (with SDXL corroboration), removing these sinks does not degrade text-image alignment (CLIP-T) or preference proxies (ImageReward, HPS-v2) at k=1k{=}1; only under stronger interventions (k ⁣≥ ⁣10k\!\geq\!10) does HPS-v2 exhibit a metric-dependent boundary, while CLIP-T remains robust throughout. The perceptual shifts induced by suppression are nonetheless sink-specific -- ∼ ⁣6×\sim\!6\times larger than equal-budget random masking -- revealing an empirical dissociation between trajectory-level perturbation and semantic alignment in diffusion transformers. \footnote{Code available at https://github.com/wfz666/ICML26-attention-sink.}

Fangzheng Wu, Brian Summa

Department of Computer Science, Tulane University, New Orleans, LA, USA.

Transformer InterpretabilityDiffusion TransformerSelf-AttentionAttention SinksT2I Diffusion ModelsDiffusion Models
AskPDF
May 10, 2026

Discrete Langevin-Inspired Posterior Sampling

We study posterior sampling for inverse problems in discrete state spaces using discrete diffusion models as generative priors. While continuous diffusion models have become widely used for inverse problems, their discrete counterparts remain comparatively underexplored. Existing discrete posterior samplers often rely on continuous relaxations of discrete variables, Gibbs-style updates, or mechanisms specialized to particular corruption processes, which can limit scalability or generality. We propose ΔΔLPS, a Discrete Langevin-Inspired Posterior Sampler that uses gradient information to identify promising discrete moves without leaving the discrete state space. The resulting approach enables efficient parallel updates across all token dimensions and is agnostic to the training paradigm of the discrete diffusion prior, including masked and uniform-state diffusion. We evaluate our method on image restoration tasks across MNIST, CIFAR, and FFHQ, as well as spatial mapping, covering linear, nonlinear, and blind inverse problems. Across these settings, we improve over recent discrete diffusion posterior samplers and are competitive with strong continuous diffusion-based inverse solvers. Our results suggest that fully discrete, gradient-informed posterior samplers offer a scalable and general path toward solving inverse problems over discrete representations.

Chaitanya Amballa, Sattwik Basu, Jorge Vančo Sampedro +1

University of Illinois Urbana-Champaign

Diffusion Models for Image RestorationBayesian Inverse ProblemsDiffusion-Based Inverse ProblemsDiffusion Posterior SamplingDiffusion Models
AskPDF
May 10, 2026

DiffATS: Diffusion in Aligned Tensor Space

Direct diffusion modeling of high-resolution spatiotemporal fields is computationally challenging. Parameter-efficient primitives address this by representing high-dimensional data with a compact set of parameters. In this paper, we construct data-dependent tensor primitives without pretrained compression autoencoders. Our construction starts from Tucker decomposition, which captures low-rank multilinear structure through a core tensor and mode-wise factors. However, Tucker factors are non-unique: the same tensor can be represented by different rotated factors, which complicates generative modeling. We address this issue with orthogonal Procrustes (OP) alignment. Specifically, we select medoid anchor matrices from the data and align the factor matrices to resolve the gauge ambiguity. This yields matrix Grassmannian primitives and tensor Grassmannian primitives that are compact, data-adaptive, and directly decodable by explicit multilinear reconstruction. Theoretically, we prove that the proposed primitive maps are homeomorphisms between low-rank tensors and their corresponding primitive spaces, certifying that the representations are non-degenerate and topologically faithful. Building on these primitives, we propose Diffusion in Aligned Tensor Space (DiffATS), a generative framework that trains diffusion models directly on aligned tensor primitives. Across images, videos, and PDE solutions, DiffATS achieves strong unconditional and conditional generation performance while compressing original data by 3.9×3.9\times to 210×210\times, without relying on any pretrained deep compression autoencoders.

Jinhua Lyu, Tianmin Yu, Brian Kim +3

Northwestern University

Orthogonal Procrustes AlignmentTensor DecompositionDiffusion Models
AskPDF
May 9, 2026

Dependency-Aware Discrete Diffusion for Scene Graph Generation

Scene graphs (SGs) represent objects and their relationships as structured graphs, enabling applications in image generation, robotics, and 3D understanding. Recent work suggests that conditioning image generation on scene graphs improves compositional fidelity compared to text-only prompting. However, since users typically provide text rather than structured graphs, a key challenge is to generate scene graphs from natural language. Prior work on discrete diffusion has demonstrated success in generating generic graphs such as molecules and circuits, but fails to account for the hierarchical structure and strong dependencies between objects, edges, and relations in scene graphs. We address this limitation by introducing a dependency-aware, hierarchically constrained discrete diffusion model for scene graph generation. Our approach decouples structure and semantics across the forward and reverse processes, enabling the model to capture conditional dependencies. At inference time, we perform training-free conditioning to sample text-aligned scene graphs. We evaluate our method on standard SG benchmarks and demonstrate improvements over both continuous and discrete graph generation baselines across graph and layout metrics. When fed to downstream image generation, our approach yields improved compositional alignment compared to text-to-image models, particularly in multi-object scenarios.

Rajalaxmi Rajagopalan, Romit Roy Choudhury

University of Illinois, Urbana-Champaign

Scene Graph GenerationDiffusion ModelsGraph Diffusion
AskPDF
May 9, 2026

SynerDiff: Synergetic Continuous Batching for Fast and Parallel Diffusion Model Inference

The expansion of Artificial Intelligence-generated content service requires diffusion model serving to simultaneously achieve high throughput and low task end-to-end (E2E) latency. However, existing continuous batching methods suffer from severe resource contention during UNet-VAE concurrency, leading to latency spikes. Furthermore, concurrent multi-task scheduling entails a trade-off between UNet throughput and VAE latency across varying scheduling strategies. To address these, we propose SynerDiff, an efficient continuous batching system built on intra-inter level synergy. At the intra-concurrency level, SynerDiff alleviates resource contention by pruning component-specific resource bottlenecks via VAE Chunking and Adaptive Skip-CFG. At the inter-concurrency level, leveraging components' differential sensitivity to scheduling granularities, a threshold-aware scheduler plans concurrent sequences and tunes intra-concurrency decisions to minimize VAE latency while maintaining UNet within high-throughput threshold. Additionally, a feedback controller dynamically adjusts this threshold based on queue loads to boost system capacity ceiling. Experimental results show that, SynerDiff improves throughput by 1.6×\times and decreases both average E2E and P99 tail latencies by up to 78.7%, compared to benchmarks while guaranteeing high image fidelity.

Ziqi Zhou, Peng Yang, Yuxin Liang +2

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China · The State Grid Hubei Electric Power Research Institute, Wuhan, China · Jincheng Yunxiang Big Data Technology Operation Co., Ltd., Shanxi, China

Diffusion Model ServingDiffusion Model Inference AccelerationDiffusion Models
AskPDF
May 8, 2026

TARO: Temporal Adversarial Rectification Optimization Using Diffusion Models as Purifiers

Adversarial purification with diffusion models seeks to project adversarial examples back toward the data manifold, but balancing semantic preservation and robustness against adaptive attacks remains challenging. Recent work shows that standard diffusion purification can fail under adaptive evaluation, while test-time score-based optimization is more resilient. Existing optimization defenses, however, typically rely on a single diffusion noise regime or treat timesteps uniformly, overlooking the distinct roles of coarse and fine denoising scales. We propose Temporal Adversarial Rectification Optimization (TARO), an inference-time purification method that builds a temporally guided score prior from multiple denoising views along the diffusion trajectory. TARO forms a coarse-to-fine residual target: high-noise experts provide globally smoothed structure with reduced adversarial sensitivity, while low-noise experts restore image-specific, class-relevant details. A guidance strength controls this temporal correction, allowing TARO to balance robust global rectification with semantic preservation. Empirically, TARO improves robust accuracy across datasets and adaptive threat models in a zero-shot setting, while remaining compatible with complementary adversarial-likelihood objectives for further robustness gains.

Daniel Wesego, Pedram Rooshenas

Department of Computer Science · University of Illinois Chicago · Chicago, IL 60607

Diffusion Model GuidanceAdversarial RobustnessAdversarial PurificationDiffusion Models
AskPDF
May 8, 2026

Geometry-Aware Discretization Error of Diffusion Models

Practical diffusion sampling requires simulating a reverse-time ODE or SDE with a limited number of denoising steps, making the choice of sampling parameters crucial for minimizing discretization error. Non-asymptotic convergence bounds characterize sampling complexity, but their worst-case constants can obscure target geometry and thereby limit guidance on parameter optimization. Rather than bounding the error, we derive asymptotically exact small-stepsize expansions of Euler-Maruyama weak and Frechet errors for general smooth reverse diffusions, with explicit formulas for Gaussian data. These formulas provide tractable objectives for optimizing diffusion parameters, including the noise and rescaling schedules and the stochasticity coefficient, according to the target's covariance spectrum. In particular, our theory predicts lower optimal stochasticity at smaller step budgets, shows how to adapt the rescaling coefficient to the data power spectrum, and motivates a new family of effective noise schedules. A perturbative extension to Gaussian scale mixtures (GSMs) quantifies how departures from Gaussianity shift the optimal parameters. Finally, experiments on different real image datasets show that FID-optimal parameters agree with the qualitative theoretical predictions.

Samuel Hurault, Thomas Moreau, Gabriel Peyré

Univ. Gustave Eiffel, CNRS, LIGM · Univ. Paris-Saclay, Inria, CEA · CNRS, ENS, PSL Université

Diffusion SamplingFew-Step Diffusion SamplingDiffusion Models
AskPDF
May 8, 2026

Normalizing Trajectory Models

Diffusion-based models decompose sampling into many small Gaussian denoising steps -- an assumption that breaks down when generation is compressed to a few coarse transitions. Existing few-step methods address this through distillation, consistency training, or adversarial objectives, but sacrifice the likelihood framework in the process. We introduce Normalizing Trajectory Models (NTM), which models each reverse step as an expressive conditional normalizing flow with exact likelihood training. Architecturally, NTM combines shallow invertible blocks within each step with a deep parallel predictor across the trajectory, forming an end-to-end network trainable from scratch or initializable from pretrained flow-matching models. Its exact trajectory likelihood further enables self-distillation: a lightweight denoiser trained on the model's own score produces high-quality samples in four steps. On text-to-image benchmarks, NTM matches or outperforms strong image generation baselines in just four sampling steps while uniquely retaining exact likelihood over the generative trajectory.

Jiatao Gu, Tianrong Chen, Ying Shen +3

Apple · UIUC

Normalizing FlowsDiffusion Model DistillationFew-Step Diffusion SamplingDiffusion-Based Image GenerationFlow-Based Generative ModelingDiffusion Models
AskPDF
May 8, 2026

DVD: Discrete Voxel Diffusion for 3D Generation and Editing

We introduce Discrete Voxel Diffusion (DVD), a discrete diffusion framework to generate, assess, and edit sparse voxels for SLat (Structured LATent) based 3D generative pipelines. Although discrete diffusion has not generally displaced continuous diffusion in image-like generation, we show that it can be an effective first-stage prior for sparse voxel scaffolds. By treating voxel occupancy as a native discrete variable, DVD avoids continuous-to-discrete thresholding and provides a simple framework for voxel generation, uncertainty estimation, and editing. Beyond quality gains, DVD provides more interpretable generation dynamics through explicit categorical modeling. Furthermore, we leverage the predictive entropy as a robust uncertainty metric to identify ambiguous voxel regions and complicated samples, facilitating tasks such as data filtering and quality assessment. Finally, we propose a lightweight fine-tuning strategy using block-structured perturbation patterns. This approach empowers the model to inpaint and edit voxels within a single sampling round, requiring negligible auxiliary computation and no additional model evaluations. Code is available at https://github.com/TeCai/DVD.

Zhengrui Xiang, Jiaqi Wu, Fupeng Sun +2

Imperial College London · Math Magic, Hitem3D · Math Magic

3D Shape Editing3D Shape Generation3D Diffusion ModelsDiffusion Models
AskPDF
May 8, 2026

What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion

Tokenizers are a crucial component of latent diffusion models, as they define the latent space in which diffusion models operate. However, existing tokenizers are primarily designed to improve reconstruction fidelity or inherit pretrained representations, leaving unclear what kind of latent space is truly friendly for generative modeling. In this paper, we study this question from the perspective of latent manifold organization. By constructing controlled tokenizer variants, we identify three key properties of a diffusion-friendly latent manifold: coherent spatial structure, local manifold continuity, and global manifold semantics. We find that these properties are more consistent with downstream generation quality than reconstruction fidelity. Motivated by this finding, we propose the Prior-Aligned AutoEncoder (PAE), which explicitly shapes the latent manifold instead of leaving diffusion-friendly manifold to emerge indirectly from reconstruction or inheritance. Specifically, PAE leverages refined priors derived from VFMs and perturbation-based regularization to turn spatial structure, local continuity, and global semantics into explicit training objectives. On ImageNet 256x256, PAE improves both training efficiency and generation quality over existing tokenizers, reaching performance comparable to RAE with up to 13x faster convergence under the same training setup and achieving a new state-of-the-art gFID of 1.03. These results highlight the importance of organizing the latent manifold for latent diffusion models.

Zhengrong Yue, Taihang Hu, Mengting Chen +8

Shanghai Jiao Tong University · Alibaba Group · Beihang University +4

Latent Diffusion ModelsAutoencodersDiffusion-Based Image GenerationDiffusion Models
AskPDF
May 8, 2026

Inference-Time Attribute Distribution Alignment for Unconditional Diffusion

Inference-time controllable generation is essential for real-world applications of unconditional diffusion models. However, most existing techniques focus on individual samples, struggling in applications that require the sample population to follow specific attribute distributions (e.g., demographic balance or semantic proportions). We formalize this setting as the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models. To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process, viewing the process as the rollout of a dynamical system and augmenting it with additive, time-dependent perturbations as control. We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective while penalizing control effort to preserve data fidelity. Experiment results in image generation demonstrate that our proposed plug-and-play approach can better align attribute distributions to diverse and flexible test-time targets compared to baselines, without retraining or finetuning the pretrained diffusion model.

Hao Luan, See-Kiong Ng, Chun Kai Ling

School of Computing, National University of Singapore · Institute of Data Science, National University of Singapore

Controllable Image GenerationInference-Time GuidanceDiffusion Models
AskPDF
May 8, 2026

Towards Photorealistic and Efficient Bokeh Rendering via Diffusion Framework

Existing mobile devices are constrained by compact optical designs, such as small apertures, which make it difficult to produce natural, optically realistic bokeh effects. Although recent learning-based methods have shown promising results, they still struggle with photos captured under high digital zoom levels, which often suffer from reduced resolution and loss of fine details. A naive solution is to enhance image quality before applying bokeh rendering, yet this two-stage pipeline reduces efficiency and introduces unnecessary error accumulation. To overcome these limitations, we propose MagicBokeh, a unified diffusion-based framework designed for high-quality and efficient bokeh rendering. Through an alternative training strategy and a focus-aware masked attention mechanism, our method jointly optimizes bokeh rendering and super-resolution, substantially improving both controllability and visual fidelity. Furthermore, we introduce degradation-aware depth module to enable more accurate depth estimation from low-quality inputs. Experimental results demonstrate that MagicBokeh efficiently produces photorealistic bokeh effects, particularly on real-world low-resolution images, paving the way for future advancements in bokeh rendering. Our code and models are available at https://github.com/vivoCameraResearch/MagicBokeh.

Linxiao Shi, Siming Zheng, Zerong Wang +5

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd. · Shenzhen University of Advanced Technology

Neural RenderingImage Super-ResolutionDiffusion Models
AskPDF