Diffusion Sampling

Follow this topic Ask about this topic

Diffusion sampling starts with noise and repeatedly denoises it to generate data, using reverse stochastic processes such as DDPM or deterministic flows such as DDIM and probability-flow ODEs. Work improves this process with higher-order or adaptive solvers, learned timestep schedules, and methods that reuse or predict model computations; discrete and masked diffusion models also require choices about which tokens to reveal and when. Guidance, particle methods, and search can steer samples toward rewards, constraints, diversity, or solutions to inverse problems, with applications spanning image and text generation, molecules, and scientific modeling. A central tension is reducing steps and computation without losing fidelity, diversity, or correctness, while theory seeks sharper guarantees for error, stability, and sampling cost.

Latest papers

264 papers49 papers in the last four weeks, up 104% on the four weeks before. 0.3% of all new papers.

Feb 19, 2026

MGD: Moment Guided Diffusion for Maximum Entropy Generation

Generating samples from limited information is a fundamental problem across scientific domains. Classical maximum entropy methods provide principled uncertainty quantification from moment constraints but require sampling via MCMC or Langevin dynamics, which typically exhibit exponential slowdown in high dimensions. In contrast, generative models based on diffusion and flow matching efficiently transport noise to data but offer limited theoretical guarantees and can overfit when data is scarce. We introduce Moment Guided Diffusion (MGD), which combines elements of both approaches. Building on the stochastic interpolant framework, MGD samples maximum entropy distributions by solving a stochastic differential equation that guides moments toward prescribed values in finite time, thereby avoiding slow mixing in equilibrium-based methods. We formally obtain, in the large-volatility limit, convergence of MGD to the maximum entropy distribution and derive a tractable estimator of the resulting entropy computed directly from the dynamics. Applications to financial time series, turbulent flows, and cosmological fields using wavelet scattering moments yield estimates of negentropy for high-dimensional multiscale processes.

Etienne Lempereur, Nathanaël Cuvelle--Magar, Florentin Coeurdoux +2

D´epartement d’informatique, ENS, Universit´e PSL, Paris, France · Capital Fund Management, Paris, France · Coll`ege de France, Paris, France +3

Generative Modeling Constrained Generative Modeling Diffusion Sampling

Feb 19, 2026

GASS: Geometry-Aware Spherical Sampling for Disentangled Diversity Enhancement in Text-to-Image Generation

Despite high semantic alignment, modern text-to-image (T2I) generative models still struggle to synthesize diverse images from a given prompt. In this work, we enhance the T2I diversity through a geometric lens. Unlike most existing methods that rely primarily on entropy-based guidance to increase sample dissimilarity, we introduce Geometry-Aware Spherical Sampling (GASS) to enhance diversity by explicitly controlling both prompt-dependent and prompt-independent sources of variation. Specifically, we decompose the diversity measure in CLIP embeddings using two orthogonal directions: the text embedding, which captures semantic variation related to the prompt, and an identified orthogonal direction that captures prompt-independent variation (e.g., backgrounds). Based on this decomposition, GASS increases the geometric projection spread of generated image embeddings along both axes and guides the T2I sampling process via expanded predictions along the generation trajectory. Our experiments on different frozen T2I backbones (U-Net and DiT, diffusion and flow) and benchmarks demonstrate the effectiveness of disentangled diversity enhancement with minimal impact on image fidelity and semantic alignment.

Ye Zhu, Kaleb S. Newman, Johannes F. Lutzeyer +3

Laboratoire d’Informatique (LIX), CNRS, École Polytechnique, IPP, France · Department of Computer Science, Princeton University, USA · FAIR at Meta - Montreal, Canada +3

T2I Generation T2I Diffusion Models Diffusion Sampling

Feb 18, 2026

Enhanced Diffusion Sampling: Efficient Rare Event Sampling and Free Energy Calculation with Diffusion Models

The rare-event sampling problem has long been the central limiting factor in molecular dynamics (MD), especially in biomolecular simulation. Recently, diffusion models such as BioEmu have emerged as powerful equilibrium samplers that generate independent samples from complex molecular distributions, eliminating the cost of sampling rare transition events. However, a sampling problem remains when computing observables that rely on states which are rare in equilibrium, for example folding free energies. Here, we introduce enhanced diffusion sampling, enabling efficient exploration of rare-event regions while preserving unbiased thermodynamic estimators. The key idea is to perform quantitatively accurate steering protocols to generate biased ensembles and subsequently recover equilibrium statistics via exact reweighting. We instantiate our framework in three algorithms: UmbrellaDiff (umbrella sampling with diffusion models), MetaDiff (a batchwise analogue for metadynamics), and

Δ

G-Diff (free-energy differences via tilted ensembles). Across toy systems, protein folding landscapes and folding free energies, our methods achieve fast, accurate, and scalable estimation of equilibrium properties within GPU-minutes to hours per system-closing the rare-event sampling gap that remained after the advent of diffusion-model equilibrium samplers.

Yu Xie, Ludwig Winkler, Lixin Sun +10

Microsoft Research AI for Science

Free Energy Calculation Diffusion Sampling Molecular Dynamics Simulation Rare-Event Estimation

Feb 18, 2026

Steering diffusion models with quadratic rewards: a fine-grained analysis

Inference-time algorithms are an emerging paradigm in which pre-trained models are used as subroutines to solve downstream tasks. Such algorithms have been proposed for tasks ranging from inverse problems and guided image generation to reasoning. However, the methods currently deployed in practice are heuristics with a variety of failure modes -- and we have very little understanding of when these heuristics can be efficiently improved. In this paper, we consider the task of sampling from a reward-tilted diffusion model -- that is, sampling from

p^{\star}(x) \propto p(x) \exp(r(x))

-- given a reward function

r

and pre-trained diffusion oracle for

p

. We provide a fine-grained analysis of the computational tractability of this task for quadratic rewards

r(x) = x^\top A x + b^\top x

. We show that linear-reward tilts are always efficiently sampleable -- a simple result that seems to have gone unnoticed in the literature. We use this as a building block, along with a conceptually new ingredient -- the Hubbard-Stratonovich transform -- to provide an efficient algorithm for sampling from low-rank positive-definite quadratic tilts, i.e.

r(x) = x^\top A x

where

A

is positive-definite and of rank

O(1)

. For negative-definite tilts, i.e.

r(x) = - x^\top A x

where

A

is positive-definite, we prove that the problem is intractable even if

A

is of rank 1 (albeit with exponentially-large entries).

Ankur Moitra, Andrej Risteski, Dhruv Rohatgi

Diffusion Sampling Diffusion Models Inference-Time Guidance

Feb 18, 2026

Fast and Scalable Analytical Diffusion

Analytical diffusion models offer a mathematically transparent path to generative modeling by formulating the denoising score as an empirical-Bayes posterior mean. However, this interpretability comes at a prohibitive cost: the standard formulation necessitates a full-dataset scan at every timestep, scaling linearly with dataset size. In this work, we present the first systematic study addressing this scalability bottleneck. We challenge the prevailing assumption that the entire training data is necessary, uncovering the phenomenon of Posterior Progressive Concentration: the effective golden support of the denoising score is not static but shrinks asymptotically from the global manifold to a local neighborhood as the signal-to-noise ratio increases. Capitalizing on this, we propose Dynamic Time-Aware Golden Subset Diffusion (GoldDiff), a training-free framework that decouples inference complexity from dataset size. Instead of static retrieval, GoldDiff uses a coarse-to-fine mechanism to dynamically pinpoint the ''Golden Subset'' for inference. Theoretically, we derive rigorous bounds guaranteeing that our sparse approximation converges to the exact score. Empirically, GoldDiff achieves a

\bf 71 \times

speedup on AFHQ while matching or achieving even better performance than full-scan baselines. Most notably, we demonstrate the first successful scaling of analytical diffusion to ImageNet-1K, unlocking a scalable, training-free paradigm for large-scale generative modeling.

Xinyi Shang, Peng Sun, Jingyu Lin +1

University College London · Mohamed bin Zayed University of Artificial Intelligence · Zhejiang University +2

Diffusion Models Diffusion Model Inference Acceleration Diffusion Sampling

Feb 18, 2026

Training-Free Adaptation of Diffusion Models via Doob's $h$ -Transform

Adaptation methods have been a workhorse for unlocking the transformative power of pre-trained diffusion models in diverse applications. Existing approaches often abstract adaptation objectives as a reward function and steer diffusion models to generate high-reward samples. However, these approaches can incur high computational overhead due to additional training, or rely on stringent assumptions on the reward such as differentiability. Moreover, despite their empirical success, theoretical justification and guarantees are seldom established. In this paper, we propose DOIT (Doob-Oriented Inference-time Transformation), a training-free and computationally efficient adaptation method that applies to generic, non-differentiable rewards. The key framework underlying our method is a measure transport formulation that seeks to transport the pre-trained generative distribution to a high-reward target distribution. We leverage Doob's

h

-transform to realize this transport, which induces a dynamic correction to the diffusion sampling process and enables efficient simulation-based computation without modifying the pre-trained model. Theoretically, we establish a high probability convergence guarantee to the target high-reward distribution via characterizing the approximation error in the dynamic Doob's correction. Empirically, on D4RL offline RL benchmarks, our method consistently outperforms state-of-the-art baselines while preserving sampling efficiency.

Qijie Zhu, Zeqi Ye, Han Liu +2

Department of Statistics and Data Science, Northwestern University. · Department of Industrial Engineering and Management Sciences, Northwestern University. · Department of Computer Science, Northwestern University.

Diffusion Model Adaptation Diffusion Models Diffusion Model Guidance Diffusion Sampling RL for Diffusion Models Doob H-Transform

Feb 18, 2026

Discrete Stochastic Localization for Non-autoregressive Generation

Continuous diffusion is a natural framework for non-autoregressive generation but has generally lagged behind masked discrete diffusion models (MDMs) on discrete sequence generation. We argue that the bottleneck is not continuity itself, but a representation in which denoising depends on timestep-indexed noise regimes. We introduce Discrete Stochastic Localization (DSL), a continuous-state framework with unit-sphere token embeddings whose Bayes-optimal denoiser is invariant to the nominal signal-to-noise ratio (SNR) under the localization channel. One trained network then supports an entire family of per-token SNR paths, with endpoint masked-diffusion paths as a special case. Fine-tuning a pretrained MDLM checkpoint with DSL substantially improves distributional faithfulness (MAUVE) on OpenWebText across all step budgets from

T{=}128

to

T{=}1024

, and the same checkpoint supports random-order autoregressive sampling, as well as a hybrid continuous-then-discrete sampler using as few as T=48 total steps -- without distillation or retraining.

Yunshu Wu, Jiayi Cheng, Longxuan Yu +4

University of California Riverside · New York University

Non-Autoregressive Text Generation Diffusion Language Models Diffusion Sampling Continuous Diffusion Language Models

Feb 17, 2026

Steering Dynamical Regimes of Diffusion Models by Breaking Detailed Balance

We show that deliberately breaking detailed balance in generative diffusion processes can accelerate the reverse process without changing the stationary distribution. Considering the Ornstein--Uhlenbeck process, we decompose the dynamics into a symmetric component and a non-reversible anti-symmetric component that generates rotational probability currents. We then construct an exponentially optimal non-reversible perturbation that improves the long-time relaxation rate while preserving the stationary target. We analyze how such non-reversible control reshapes the macroscopic dynamical regimes of the phase transitions recently identified in generative diffusion models. We derive a general criterion for the speciation time and show that suitable non-reversible perturbations can accelerate speciation. In contrast, the collapse transition is governed by a trace-controlled phase-space contraction mechanism that is fixed by the symmetric component, and the corresponding collapse time remains unchanged under anti-symmetric perturbations. Numerical experiments on Gaussian mixture models support these findings.

Haiqi Lu, Ying Tang

Institute of Fundamental and Frontier Sciences, University of Electronic Science and Technology of China, Chengdu 611731, China · Yingcai Honor College, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Physics, University of Electronic Science and Technology of China, Chengdu 611731, China +2

Diffusion Models Diffusion Sampling Diffusion Model Acceleration Stochastic Differential Equations

Feb 16, 2026

Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees

Diffusion models over discrete spaces have recently shown striking empirical success, yet their theoretical foundations remain incomplete. In this paper, we study the sampling efficiency of score-based discrete diffusion models under a continuous-time Markov chain (CTMC) formulation, with a focus on

τ

-leaping-based samplers. We establish sharp convergence guarantees for attaining

\varepsilon

accuracy in Kullback-Leibler (KL) divergence for both uniform and masking noising processes. For uniform discrete diffusion, we show that the

τ

-leaping algorithm achieves an iteration complexity of order

\tilde O(d/\varepsilon)

, with

d

the ambient dimension of the target distribution, eliminating linear dependence on the vocabulary size

S

and improving existing bounds by a factor of

d

; moreover, we establish a matching algorithmic lower bound showing that linear dependence on the ambient dimension is unavoidable in general. For masking discrete diffusion, we introduce a modified

τ

-leaping sampler whose convergence rate is governed by an intrinsic information-theoretic quantity, termed the effective total correlation, which is bounded by

d \log S

but can be sublinear or even constant for structured data. As a consequence, the sampler provably adapts to low-dimensional structure without prior knowledge or algorithmic modification, yielding sublinear convergence rates for various practical examples (such as hidden Markov models, image data, and random graphs). Our analysis requires no boundedness or smoothness assumptions on the score estimator beyond control of the score entropy loss.

Daniil Dmitriev, Zhihan Huang, Yuting Wei

Department of Statistics and Data Science, the Wharton School, University of Pennsylvania

Sample Complexity Diffusion Model Sampling Diffusion Sampling Diffusion Models

Feb 13, 2026

Formalizing the Sampling Design Space of Diffusion-Based Generative Models via Adaptive Solvers and Wasserstein-Bounded Timesteps

Diffusion-based generative models have achieved remarkable performance across various domains, yet their practical deployment is often limited by high sampling costs. While prior work focuses on training objectives or individual solvers, the broader sampling design problem, specifically solver selection and scheduling, remains largely governed by static heuristics. We propose SDM, a principled, training-free sampling framework that adapts both the numerical solver and the timestep schedule to the intrinsic properties of the diffusion trajectory. By analyzing the PF-ODE dynamics, we show that velocity variation is small in high-noise stages and increases near the data manifold, identifying intervals where solver order is most consequential. In parallel, we introduce an offline-calibrated adaptive scheduling method that explicitly controls the local Wasserstein discretization error and projects the calibrated trajectory to a prescribed NFE budget. We further extend the formulation to a mixed-transition Wasserstein error bound, providing a unified error-propagation view of adaptive scheduling and solver selection within the overall SDM framework. Across standard benchmarks, with extensions to modern ODE samplers, high-resolution synthesis, and text-to-image generation, SDM achieves improved sample quality compared to baseline methods, attaining an FID of 1.93 on CIFAR-10, 2.41 on FFHQ, and 1.98 on AFHQv2, with a reduced number of function evaluations compared to existing samplers. Our code is available at https://github.com/aiimaginglab/sdm.

Sangwoo Jo, Sungjoon Choi

Department of Artificial Intelligence, Korea University, Seoul, South Korea.

Diffusion Model Sampling Diffusion Sampling Adaptive Sampling Diffusion Timestep Scheduling Diffusion Model Acceleration

Feb 11, 2026

Prediction--Loss Alignment for Sampler--Robust Flow Matching Training

Recent work has popularized a practical recipe in diffusion and flow matching: predict the clean signal

x

, convert it to a velocity, and train through a velocity-space loss. The conversion contains a singular endpoint amplification and therefore appears prone to unstable optimization, yet recent systems obtain strong empirical results with this recipe. We investigate this tension through the integrability of the pre-optimizer stochastic-gradient second moment. Under stated initialization conditions, the moment diverges under Uniform sampling; boundary-suppressing sampling can restore integrability under an additional upper-growth condition. We then show that prediction--loss alignment eliminates this conversion-induced source of non-integrability. Under a uniform moment bound, alignment yields a finite second moment for every timestep density, including Uniform sampling. Controlled experiments across continuous and binary settings reproduce the predicted sampler-dependent instability and show that aligned objectives remain trainable across the tested samplers. These results reconcile pointwise amplification with sampler-dependent empirical success and support alignment as a principled route to more robust flow-matching training.

Jiadong Hong, Lei Liu, Xinyu Bian +2

Zhejiang University, Hangzhou, China · Huawei Technologies Co., Ltd, Hong Kong

Flow Matching Diffusion Sampling Flow-Based Generative Modeling

Jan 26, 2026

ART for Diffusion Sampling: A Reinforcement Learning Approach to Timestep Schedule

We consider time discretization for score-based diffusion models to generate samples from a learned reverse-time dynamic on a finite grid. Uniform and hand-crafted grids can be suboptimal given a budget on the number of time steps. We introduce Adaptive Reparameterized Time (ART), which controls the clock speed of a reparameterized time variable to redistribute computation along the sampling trajectory while preserving the terminal time, with the objective of minimizing the aggregate Euler discretization error. We derive a randomized companion ART-RL that recasts ART as a continuous-time reinforcement learning problem with Gaussian policies, and prove a two-directional bridge between the two: the deterministic ART optimum lifts to an optimal Gaussian policy, and conversely any optimal Gaussian policy must recover the ART control through its mean. This bridge turns continuous-time actor--critic learning into a principled, rather than heuristic, route to the deterministic timestep optimum. Within the official EDM pipeline, ART-RL improves FID on CIFAR--10 across a wide range of budgets; after one-time offline training, the distilled deterministic schedule transfers without retraining to AFHQv2, FFHQ, and ImageNet at no extra inference cost.

Yilie Huang, Wenpin Tang, Xunyu Zhou

Department of Industrial Engineering and Operations Research, Columbia University, New York, NY 10027, USA. · Department of Industrial Engineering and Operations Research & Data Science Institute, Columbia University, New York, NY 10027, USA.

Diffusion Sampling Adaptive Sampling RL for Diffusion Models Diffusion Timestep Scheduling Diffusion Model Inference Acceleration

Jan 20, 2026

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

This paper introduces an analytical approach to quantifying and optimizing the distributional discrepancy in generative diffusion models. For a multivariate Gaussian source, we explicitly derive the closed-form evolution trajectory and the resulting Kullback-Leibler (KL) divergence between the distributions of the source data and the reversely sampled data. Asymptotic analysis via the Euler-Maclaurin expansion characterizes the convergence behavior of this KL divergence, extracting its dominant term as an explicit functional of the noise schedule. Minimizing this dominant term via the calculus of variations yields a noise schedule described by a tangent law, inherently determined by the source covariance spectrum. We further prove that the Gaussian source exhibits an extremal property for the KL divergence among general source distributions with a given covariance. We also utilize the analytical KL divergence as a principled metric to identify efficient time discretization strategies for pretrained diffusion models, and demonstrate via experiments over diverse datasets that the identified strategies consistently outperform established baselines, particularly under constrained function evaluation budgets.

Qiang Sun, H. Vincent Poor, Wenyi Zhang

Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA

Diffusion Sampling Kullback-Leibler Divergence Diffusion Model Inference Acceleration

Jan 6, 2026

Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models

Uniform-noise discrete diffusion and flow models generate sequences non-autoregressively through iterative, context-dependent token replacements. However, these models are typically formulated as time-inhomogeneous continuous- or discrete-time Markov chains (CTMC/DTMC), sampled using independent Bernoulli change decisions per discretization step. This induces Poisson-binomial variance in per-position jump counts that grows with the number of required edits, leading to the common under-editing (residual noise) and over-editing (cascading substitutions) failure modes that degrade sample quality. We identify this sampler-induced variance as an orthogonal source of degradation, distinct from model-side errors and addressable purely at inference time. We propose Systematic Hazard Sampling (SHS), a training-free, drop-in, and hyperparameter-free inference principle for any sampler that admits a stay-vs.-replace decomposition. SHS models per-token edits as events driven by cumulative hazard (CTMC) or jump mass (DTMC) and triggers an edit whenever this quantity exceeds unit-spaced thresholds with a single random phase per position. For any fixed cumulative mass, this preserves the expected jump count while achieving the minimum conditional variance possible among unbiased integer estimators (at most 1/4), without altering per-jump destination sampling. Experiments on four uniform-noise discrete diffusion and flow language models spanning ~110M to ~3B parameters show that SHS consistently improves sample quality across numbers of function evaluations (NFE). Code is available at https://github.com/Jang-seunghwan/Systematic-Hazard-Sampling.

Seunghwan Jang, Wonje Jeung, SooJean Han

Department of EE KAIST Daejeon, South Korea

Markov Models Diffusion Sampling Diffusion Language Model Inference Variance Reduction Diffusion Models

Dec 30, 2025

Fast Score-Based Sampling via Log-Concave Reductions

Sampling based on score diffusions has led to striking empirical results, and has attracted considerable attention from various research communities. It depends on availability of (approximate) Stein score functions for various levels of additive noise. We show how in some generality, the availability of scores allows the general problem to be ``reduced'' to sampling from an adaptively constructed sequence of

K

strongly log-concave (SLC) sub-problems. The reduction is simple, constructive and algorithm-independent, so that any SLC sampler can be used as a subroutine. Various bounds on score-based sampling complexity follow directly: for instance, high-accuracy SLC samplers yield

\tilde{\mathcal{O}}(K \sqrt{d} \operatorname{polylog}(1/\varepsilon))

guarantees for accuracy

\varepsilon

in dimension

d

, where randomized midpoint SLC schemes yield

\tilde{\mathcal{O}}(K d^{1/3} \operatorname{poly}(1/\varepsilon))

guarantees. When the original distribution itself is SLC, we prove that

K \leq 1 + \log_2(κ)

, thereby obtaining the first efficient procedure with logarithmic dependence on condition number

κ

; for general distributions, the quantity

K

depends on the geometry of score Hessian across the trajectory. Our analysis is direct and simple, involving techniques and insights complementary to those in standard analyses of discretized diffusions.

M. J. Wainwright

Laboratory for Information and Decision Systems · Statistics and Data Science Center · EECS & Mathematics +1

Sample Complexity Log-Concave Sampling Diffusion Sampling Score-Based Generative Modeling

Nov 11, 2025

VEDA: Generation of 3D Molecules via Variance-Exploding Diffusion with Annealing

Diffusion models show promise for 3D molecular generation, but face a fundamental trade-off between sampling efficiency and conformational accuracy. While flow-based models are fast, they often produce geometrically inaccurate structures, as they have difficulty capturing the multimodal distributions of molecular conformations. In contrast, denoising diffusion models are more accurate but suffer from slow sampling, a limitation attributed to sub-optimal integration between diffusion dynamics and SE(3)-equivariant architectures. To address this, we propose VEDA, a unified SE(3)-equivariant framework that combines variance-exploding diffusion with annealing to efficiently generate conformationally accurate 3D molecular structures. Specifically, our key technical contributions include: (1) a VE schedule that enables noise injection functionally analogous to simulated annealing, improving 3D accuracy and reducing relaxation energy; (2) a novel preconditioning scheme that reconciles the coordinate-predicting nature of SE(3)-equivariant networks with a residual-based diffusion objective, and (3) a new arcsin-based scheduler that concentrates sampling in critical intervals of the logarithmic signal-to-noise ratio. On the QM9 and GEOM-DRUGS datasets, VEDA matches the sampling efficiency of flow-based models, achieving state-of-the-art valency stability and validity with only 100 sampling steps. More importantly, VEDA's generated structures are remarkably stable, as measured by their relaxation energy during GFN2-xTB optimization. The median energy change is only 1.72 kcal/mol, significantly lower than the 32.3 kcal/mol from its architectural baseline, SemlaFlow. Our framework demonstrates that principled integration of VE diffusion with SE(3)-equivariant architectures can achieve both high chemical accuracy and computational efficiency.

Peining Zhang, Jinbo Bi, Minghu Song

University of Connecticut, Storrs, Connecticut 06269, USA · Institute of Health and Medicine, Hefei Comprehensive National Science Center, Hefei 230601, China

3D Molecular Generation Diffusion Models Diffusion Sampling Equivariant Neural Networks

Oct 6, 2025

Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusion

Masked diffusion models have shown promising performance in generating high-quality samples in a wide range of domains, but accelerating their sampling process remains relatively underexplored. To investigate efficient samplers for masked diffusion, this paper theoretically analyzes the MaskGIT sampler for image modeling, revealing its implicit temperature sampling mechanism. Through this analysis, we show that MaskGIT is asymptotically equivalent to a choose-then-sample (CTS) formulation, instantiated as the "moment sampler," which explicitly separates index selection from token sampling. This CTS reformulation is essential: it yields unbiased token sampling and exposes an algorithmic design space for index selection, both of which are inaccessible in MaskGIT's original formulation. Regarding token sampling, we reveal that MaskGIT implicitly adopts a low-temperature sampler, which explains why MaskGIT often degrades with more sampling steps. The CTS reformulation of MaskGIT allows us to fix the temperature sampling to ensure unbiasedness. We also improve the index selection in CTS through two key innovations: a partial caching technique for transformers that approximates longer sampling trajectories without proportional computational cost, and a hybrid approach formalizing the exploration-exploitation trade-off in adaptive unmasking. Experiments in image and text domains demonstrate our theory as well as the efficiency of our proposed methods, advancing both theoretical understanding and practical implementation of masked diffusion samplers.

Satoshi Hayakawa, Yuhta Takida, Masaaki Imaizumi +2

Sony Group Corporation · Sony AI · The University of Tokyo

Diffusion Model Sampling Diffusion Sampling Diffusion Language Models

Sep 15, 2025

IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed

Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.

Yongzhe Lyu, Yu Wu, Yutian Lin +1

School of Computer Science, Wuhan University, Wuhan 430072, China

Image Inpainting Diffusion Models Diffusion Sampling

Jul 29, 2025

MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE

Although GRPO substantially enhances flow matching models in human preference alignment of image generation, methods such as FlowGRPO and DanceGRPO still exhibit inefficiency due to the necessity of sampling and optimizing over all denoising steps specified by the Markov Decision Process (MDP). In this paper, we propose

\textbf{MixGRPO}

, a novel framework that leverages the flexibility of mixed sampling strategies through the integration of stochastic differential equations (SDE) and ordinary differential equations (ODE). This streamlines the optimization process within the MDP to improve efficiency and boost performance. Specifically, MixGRPO introduces a sliding window mechanism, using SDE sampling and GRPO-guided optimization only within the window, while applying ODE sampling outside. This design confines sampling randomness to the time-steps within the window, thereby reducing the optimization overhead, and allowing for more focused gradient updates to accelerate convergence. Additionally, as time-steps beyond the sliding window are not involved in optimization, higher-order solvers are supported for faster sampling. So we present a faster variant, termed

\textbf{MixGRPO-Flash}

, which further improves training efficiency while achieving comparable performance. MixGRPO exhibits substantial gains across multiple dimensions of human preference alignment, outperforming DanceGRPO in both effectiveness and efficiency, with nearly 50% lower training time. Notably, MixGRPO-Flash further reduces training time by 71%.

Junzhe Li, Yutao Cui, Tao Huang +8

Hunyuan, Tencent, China · School of Computer Science, Peking University, China · Computer Center, Peking University, China +2

Diffusion Model Alignment Flow Matching Group Relative Policy Optimization Diffusion Sampling RL for Diffusion Models

Jun 25, 2025

Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations

Diffusion models generate conditional samples by progressively denoising Gaussian noise, yet the denoising trajectory can stall at visually plausible but low-quality outcomes with conditional misalignment or structural artifacts. We interpret this behavior as local optima in a surrogate quality landscape: Once early denoising commits to a suboptimal global structure, later steps mainly sharpen details and seldom correct the underlying mistake. While existing inference-time approaches explore alternative diffusion states via re-noising with fixed strength or direction, they exhibit limited capacity to escape steep quality plateaus. We propose Controlled Random Zigzag Sampling (Ctrl-Z Sampling),a scalable sampling strategy that detects plateaus in quality landscape via a surrogate score, and allocates exploration only when a plateau is detected. Upon detection, Ctrl-Z Sampling rolls back to noisier states, samples a set of alternative continuations, and updates the trajectory when a candidate improves the score, otherwise escalating the exploration depth to escape the current plateau. The proposed method is model-agnostic and broadly compatible with existing diffusion frameworks. Experiments show that Ctrl-Z Sampling consistently improves generation quality over other inference-time scaling samplers across different NFE budgets, offering a scalable compute-quality trade-off. Code available at: https://github.com/ShunqiM/Ctrl-Z-Sampling.

Shunqi Mao, Wei Guo, Chaoyi Zhang +3

School of Computer Science, The University of Sydney, Australia

Diffusion Sampling Conditional Diffusion Models Diffusion Model Inference Acceleration Diffusion-Based Denoising

Jun 16, 2025

Fast Convergence for High-Order ODE Solvers in Diffusion Probabilistic Models

Diffusion probabilistic models generate samples by learning to reverse a noise-injection process that transforms data into noise. A key development is the reformulation of the reverse sampling process as a deterministic probability flow ordinary differential equation (ODE), which allows for efficient sampling using high-order numerical solvers. Unlike traditional time integrator analysis, the accuracy of this sampling procedure depends not only on numerical integration errors but also on the approximation quality and regularity of the learned score function, as well as their interaction. In this work, we present a rigorous convergence analysis of deterministic samplers derived from probability flow ODEs for general forward processes with arbitrary variance schedules. Specifically, we develop and analyze

p

-th order (exponential) Runge-Kutta schemes, under the practical assumption that the first and second derivatives of the learned score function are bounded. We prove that the total variation distance between the generated and target distributions can be bounded as \begin{align*} O\bigl(d^{\frac{7}{4}}\varepsilon_{\text{score}}^{\frac{1}{2}} +d(dH_{\max})^p\bigr), \end{align*} where

\varepsilon^2_{\text{score}}

denotes the

L^2

error in the score function approximation,

d

is the data dimension, and

H_{\max}

represents the maximum solver step size. Numerical experiments on benchmark datasets further confirm that the derivatives of the learned score function are bounded in practice.

Daniel Zhengyu Huang, Jiaoyang Huang, Zhengjiang Lin

Beijing International Center for Mathematical Research, Center for Machine Learning Research, Peking University, Beijing, China · Department of Statistics and Data Science, University of Pennsylvania, Philadelphia, PA, USA · Department of Mathematics, Massachusetts Institute of Technology, 77 Massachusetts Ave, 02139 Cambridge MA, USA

Diffusion Model Sampling Diffusion Sampling

Jun 13, 2025

The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysis

Sampling in score-based diffusion models can be performed by solving either a reverse-time stochastic differential equation (SDE) parameterized by an arbitrary stochasticity function or a probability flow ODE, corresponding to setting this stochasticity function to zero. In this work, we investigate the effect of this stochasticity on the generation process through the evolution of Kullback-Leibler (KL) divergences, obtaining general KL divergence bounds and a novel analysis of the impact of the time-profile of the score error on model performance. For exact score functions, stochasticity has a contractive effect, decreasing KL divergence along the sampling trajectory. For approximate scores, however, a trade-off arises between correcting accumulated errors and amplifying current score errors, meaning stochasticity can either improve or degrade generation performance. Theoretical considerations indicate that the gain from stochasticity depends on the time-localization of the trained model error. We test this in experiments on both toy and benchmark data sets, also comparing the KL divergence evolution with the obtained bounds. We also present a fully analytical example, where all the relevant quantities can be computed, and the optimal stochasticity function can be characterized via an optimal control analysis.

Bernardo P. Schaeffer, Ricardo M. S. Rosa, Glauco Valle

Instituto de Matem´atica, Universidade Federal do Rio de Janeiro

Stochastic Differential Equations Diffusion Sampling Score-Based Generative Modeling Diffusion Models

Jan 22, 2025

Low-dimensional adaptation of diffusion models: Convergence in total variation

This paper investigates how diffusion generative models leverage (unknown) low-dimensional structure to accelerate sampling. Focusing on two mainstream samplers -- the denoising diffusion implicit model (DDIM) and the denoising diffusion probabilistic model (DDPM), we prove that their iteration complexities under exact score functions are at most the order of

k/\varepsilon

(up to log factor), where

\varepsilon

is the precision in total variation distance and

k

is some intrinsic dimension of the target distribution. We further extend these convergence guarantees to the setting in which the score functions are learned from data rather than known exactly, showing that the convergence performance degrades gracefully under suitable score estimation assumptions. We then show that these assumptions are attainable via kernel-based score estimators with finite-sample guarantees that also adapt to the low-dimensional structure. Our results apply to a broad family of target distributions without requiring smoothness or log-concavity. Our findings provide the first rigorous evidence for the adaptivity of the DDIM-type samplers to unknown low-dimensional structure, and improve over the state-of-the-art DDPM theory regarding total variation convergence.

Jiadong Liang, Zhihan Huang, Yuxin Chen

Department of Statistics and Data Science, the Wharton School, University of Pennsylvania

Diffusion Sampling Diffusion Models Diffusion Model Adaptation

Nov 26, 2024

ContrastiveCFG: Guiding Diffusion Sampling by Contrasting Positive and Negative Concepts

As Classifier-Free Guidance (CFG) has proven effective in conditional diffusion model sampling for improved condition alignment, many applications use a negated CFG term as a Negative Prompting (NP) to filter out unwanted features from samples. However, simply negating CFG guidance creates an inverted probability distribution, often distorting samples away from the marginal distribution. Inspired by recent advances in conditional diffusion models for inverse problems, here we present a novel method to achieve guidance toward the given condition using contrastive loss. Specifically, our guidance term aligns or repels the denoising direction based on the given condition through contrastive loss, achieving a similar guiding effect to traditional CFG for positive conditions while overcoming the limitations of existing negative guidance methods. Experimental results demonstrate that our approach effectively injects or removes the given concepts while maintaining sample quality across diverse scenarios, from simple class conditions to complex and overlapping text prompts.

Jinho Chang, Changsun Lee, Hyungjin Chung +1

Graduate School of Artificial Intelligence, Korea Advanced Institute of Science and Technology, Seoul, South Korea · EverEx, Seoul, South Korea

Diffusion Model Guidance Classifier-Free Guidance Diffusion Guidance Diffusion Sampling