Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models

Seunghwan Jang; Wonje Jeung; SooJean Han

cs.LGJan 6, 2026

Systematic Hazard Sampling: Minimal-Variance Inference for Discrete Diffusion and Flow Models

Authors: Seunghwan Jang, Wonje Jeung, SooJean Han

Organizations: Department of EE KAIST Daejeon, South Korea

Abstract

Uniform-noise discrete diffusion and flow models generate sequences non-autoregressively through iterative, context-dependent token replacements. However, these models are typically formulated as time-inhomogeneous continuous- or discrete-time Markov chains (CTMC/DTMC), sampled using independent Bernoulli change decisions per discretization step. This induces Poisson-binomial variance in per-position jump counts that grows with the number of required edits, leading to the common under-editing (residual noise) and over-editing (cascading substitutions) failure modes that degrade sample quality. We identify this sampler-induced variance as an orthogonal source of degradation, distinct from model-side errors and addressable purely at inference time. We propose Systematic Hazard Sampling (SHS), a training-free, drop-in, and hyperparameter-free inference principle for any sampler that admits a stay-vs.-replace decomposition. SHS models per-token edits as events driven by cumulative hazard (CTMC) or jump mass (DTMC) and triggers an edit whenever this quantity exceeds unit-spaced thresholds with a single random phase per position. For any fixed cumulative mass, this preserves the expected jump count while achieving the minimum conditional variance possible among unbiased integer estimators (at most 1/4), without altering per-jump destination sampling. Experiments on four uniform-noise discrete diffusion and flow language models spanning ~110M to ~3B parameters show that SHS consistently improves sample quality across numbers of function evaluations (NFE). Code is available at https://github.com/Jang-seunghwan/Systematic-Hazard-Sampling.

Figures & tables

Figure 1 : Standard step-based sampling (left) makes independent per-step Bernoulli edit decisions, leading to under-edit (position 1) and over-edit (position 4) failure modes. SHS (right) accumulates jump mass across steps and triggers edits at unit-spaced thresholds with a single random phase per position, producing stable, low-variance edit trajectories.

	UDLM					GIDD
NFE	Standard		SHS		$\Delta$	Standard		SHS		$\Delta$
NFE	Entropy	Gen. PPL $\downarrow$	Entropy	Gen. PPL $\downarrow$	$\Delta$	Entropy	Gen. PPL $\downarrow$	Entropy	Gen. PPL $\downarrow$	$\Delta$
4	$6.633\pm 0.008$	$559.3\pm 6.3$	$\bm{6.586\pm 0.007}$	$\bm{535.6\pm 5.6}$	$-4.2\%$	$7.479\pm 0.010$	$1696.2\pm 14.5$	$\bm{7.438\pm 0.010}$	$\bm{1606.4\pm 13.6}$	$-5.3\%$
8	$6.739\pm 0.007$	$336.3\pm 2.8$	$\bm{6.670\pm 0.011}$	$\bm{316.4\pm 2.2}$	$-5.9\%$	$7.398\pm 0.005$	$528.9\pm 4.1$	$\bm{7.320\pm 0.009}$	$\bm{455.6\pm 6.6}$	$-13.9\%$
16	$6.806\pm 0.013$	$248.0\pm 2.7$	$\bm{6.742\pm 0.012}$	$\bm{232.1\pm 3.7}$	$-6.4\%$	$7.341\pm 0.009$	$227.7\pm 3.7$	$\bm{7.261\pm 0.011}$	$\bm{189.5\pm 1.9}$	$-16.7\%$
32	$6.839\pm 0.017$	$208.9\pm 4.4$	$\bm{6.791\pm 0.002}$	$\bm{198.2\pm 3.9}$	$-5.1\%$	$7.327\pm 0.009$	$150.0\pm 1.5$	$\bm{7.232\pm 0.009}$	$\bm{119.3\pm 0.3}$	$-20.5\%$

Table 1 : Generative perplexity under NFE budgets (Gen. PPL, lower is better; bold = better between Standard and SHS).

\Delta

shows SHS improvement over Standard.

\pm

: standard deviation over seeds (5 for UDLM and GIDD, 3 for DFM; single seed for 3B).

	UDLM								GIDD
NFE	Standard				SHS				Standard				SHS
NFE	D-1 $\uparrow$	D-2 $\uparrow$	D-3 $\uparrow$	S-BLEU $\downarrow$	D-1 $\uparrow$	D-2 $\uparrow$	D-3 $\uparrow$	S-BLEU $\downarrow$	D-1 $\uparrow$	D-2 $\uparrow$	D-3 $\uparrow$	S-BLEU $\downarrow$	D-1 $\uparrow$	D-2 $\uparrow$	D-3 $\uparrow$	S-BLEU $\downarrow$
4	$0.198$	$0.758$	$0.977$	$0.0000$	$0.192$	$0.752$	$0.975$	$0.0000$	$0.251$	$0.820$	$0.991$	$0.0000$	$0.245$	$0.814$	$0.990$	$0.0000$
8	$0.188$	$0.734$	$0.964$	$0.0001$	$0.181$	$0.730$	$0.965$	$0.0000$	$0.205$	$0.745$	$0.972$	$0.0000$	$0.195$	$0.735$	$0.971$	$0.0000$
16	$0.182$	$0.719$	$0.955$	$0.0001$	$0.170$	$0.704$	$0.951$	$0.0002$	$0.180$	$0.702$	$0.958$	$0.0001$	$0.173$	$0.692$	$0.956$	$0.0001$
32	$0.177$	$0.702$	$0.942$	$0.0003$	$0.168$	$0.695$	$0.942$	$0.0002$	$0.170$	$0.681$	$0.950$	$0.0001$	$0.159$	$0.664$	$0.945$	$0.0001$

Table 2 : Lexical diversity under NFE budgets (Distinct-

n

higher is more diverse, Self-BLEU lower is more diverse). Single seed, 1024 samples per cell; metric definitions in Appendix E .

NFE	Standard observed	Standard prediction $\sum_{k}p_{ik}(1-p_{ik})$	SHS observed	SHS bound
4	$0.344$	$0.405$	$0.125$	$0.25$
16	$1.254$	$1.642$	$0.134$	$0.25$
64	$2.438$	$3.096$	$0.099$	$0.25$
128	$3.087$	$3.845$	$0.086$	$0.25$

Table 3: Jump-count variance diagnostics on UDLM. We bin positions by

S_{i}^{\mathrm{tot}}

(bin width

0.5

) and report the sample-weighted average of within-bin variance, an estimator of

\mathbb{E}_{S^{\mathrm{tot}}}[\mathrm{Var}(J_{i}\mid S_{i}^{\mathrm{tot}})]

. The standard sampler’s empirical variance grows with NFE as the Poisson-binomial prediction

\sum_{k}p_{ik}(1-p_{ik})

does; SHS remains below the fixed-mass

1/4

bound (Proposition 2 ).

Figure 2 : Token-wise NLL versus count deviation

\Delta J_{i}=J_{i}-\mathrm{round}(S_{i}^{\mathrm{tot}})

on UDLM (Standard sampler) at NFE

{=}16

(left) and NFE

{=}128

(right). Orange line: mean token NLL per deviation bin (left axis). Gray bars: position count per bin (right axis).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3 : Hazard-space jump locations ( $\mathrm{NFE}=256$ ). Histograms of $S_{k}$ , the cumulative hazard at the $k$ -th jump. Standard sampling exhibits Erlang/Gamma-shaped variability, while SHS produces systematic, bounded-support locations, demonstrating reduced trajectory randomness.

Figure 4 : Quantized MNIST samples under varying step budgets. Standard sampling (top) vs. SHS (bottom) at

\mathrm{NFE}\in\{256,64,16,8\}

discretization steps. SHS reduces the under-edit regime and yields cleaner early-stage samples.

NFE	Standard	Adaptive (top-1)	SHS
4	6.758 / 936.3	8.248 / 1995.0	6.634 / 869.5
8	6.660 / 443.4	8.384 / 1483.8	6.536 / 417.2
16	6.612 / 221.4	8.459 / 1094.8	6.554 / 218.1
32	6.649 / 150.2	8.454 / 783.7	6.588 / 142.0
64	6.654 / 113.7	8.421 / 591.9	6.619 / 113.0
128	6.649 / 104.9	8.483 / 574.9	6.618 / 99.6

Appendix

Table 4: Same-axis comparison on the 3B model (single seed). Entropy / Gen. PPL.

Model	Standard	top- $p$ $0.9$	top- $p$ $0.9$ + SHS
UDLM	6.884 / 190.65	6.866 / 187.45	6.817 / 174.55
GIDD	7.350 / 111.63	7.299 / 97.56	7.206 / 78.23
GIDD, $\tau$		Temperature	Temperature + SHS
1.0		7.350 / 111.63	7.234 / 84.83
0.99		7.301 / 99.89	7.182 / 76.09
0.975		7.218 / 81.91	7.118 / 65.02

Appendix

Table 5: Composition with nucleus sampling and temperature at NFE

{=}128

(mean of 3 seeds). Entropy / Gen. PPL.

	D-1	D-2	D-3	Self-BLEU-2
Real data	0.148	0.626	0.903	0.062
NFE 16	0.180 / 0.171	0.702 / 0.688	0.958 / 0.955	0.057 / 0.059
NFE 128	0.168 / 0.153	0.677 / 0.656	0.946 / 0.941	0.053 / 0.056
NFE 512	0.169 / 0.153	0.678 / 0.654	0.946 / 0.939	0.052 / 0.054

Appendix

Table 6: GIDD lexical diversity against length-matched real text (std / SHS).

Phase $\theta$	Gen. PPL	paired $p$ vs. per-position	$\mathbb{E}[S^{\mathrm{tot}}]$
Independent per position (SHS)	174.248	–	5.0789
Shared across positions	213.477	0.0039	5.0868
Redrawn at each unit interval	173.711	0.799	–

Appendix

Table 7: Phase ablation on UDLM, NFE

{=}128

, 3 seeds.

	NFE $=16$		NFE $=128$
$S_{i}^{\mathrm{tot}}$	under-edit ( $\Delta J_{i}\leq-1$ )	over-edit ( $\Delta J_{i}\geq+1$ )	under-edit ( $\Delta J_{i}\leq-1$ )	over-edit ( $\Delta J_{i}\geq+1$ )
$[1,2)$	$+1.991$ $[+1.910,+2.070]$	$-0.259$ $[-0.312,-0.208]$	$+0.716$ $[+0.509,+0.927]$	$-0.350$ $[-0.466,-0.238]$
$[2,3)$	$+0.946$ $[+0.899,+0.995]$	$-0.445$ $[-0.478,-0.414]$	$+0.608$ $[+0.512,+0.708]$	$-0.366$ $[-0.442,-0.293]$
$[3,4)$	$+0.397$ $[+0.359,+0.436]$	$-0.155$ $[-0.189,-0.119]$	$+0.543$ $[+0.481,+0.602]$	$-0.286$ $[-0.337,-0.237]$
$[4,5)$	$+0.281$ $[+0.236,+0.324]$	$-0.051$ $[-0.099,-0.003]$	$+0.307$ $[+0.261,+0.353]$	$-0.182$ $[-0.221,-0.144]$
$[5,6)$	$+0.375$ $[+0.292,+0.457]$	$-0.110$ $[-0.206,-0.003]$	$+0.177$ $[+0.132,+0.220]$	$-0.097$ $[-0.141,-0.053]$

Appendix

Table 8: Excess token NLL (nats) relative to

\Delta J_{i}=0

within each

S_{i}^{\mathrm{tot}}

stratum (width 1), UDLM standard sampler, seeds 1–5. Brackets: 95% CI, sentence-level cluster bootstrap.

NFE	$p^{\exp}=1-\exp(-h\lambda)$	$p^{\mathrm{Euler}}=\min(h\lambda,1)$	$\Delta$
2	$1150.5\pm 157.9$	$759.2\pm 94.5$	$-34.0\%$
4	$372.7\pm 55.8$	$286.5\pm 39.5$	$-23.1\%$
8	$158.4\pm 21.8$	$141.0\pm 17.3$	$-11.0\%$
16	$86.3\pm 12.1$	$83.9\pm 10.3$	$-2.8\%$
32	$52.3\pm 10.3$	$53.4\pm 7.3$	$+2.1\%$
64	$29.5\pm 10.3$	$28.7\pm 10.6$	$-2.7\%$

Appendix

Table 9 : DFM generation quality (Gen. PPL, lower is better) under the two change-mass discretizations, standard sampler,

320

samples, single seed;

\pm

is the standard deviation over samples. The percentage column reports the relative reduction of

p^{\mathrm{Euler}}

over

p^{\exp}

Explore similar work

On the Error-Correcting Effects of Stochasticity in Discrete Diffusion
May 26, 2026William Yuan, Sungwon Jeong, Amirali AghazadehDiffusion Sampling Diffusion Models
Georgia Institute of Technology
Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
May 18, 2026Grigory Bartosh, Teodora Pandeva, Sushrut Karmalkar +1Diffusion Models Generative Models
University of Amsterdam · Microsoft Research, Cambridge
From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion
Sep 1, 2026Satoshi HayakawaTruncation Diffusion Dynamics
The University of Tokyo

May 26, 2026, cs.LGJ/K move · Enter open · S save

On the Error-Correcting Effects of Stochasticity in Discrete Diffusion

William Yuan, Sungwon Jeong, Amirali Aghazadeh

Georgia Institute of Technology

Diffusion Sampling Diffusion Models Stochastic Transitions Correction Impact Analysis

Open paper AskPDF arXiv

Discrete diffusion models achieve strong performance in text and image generation, but their inference remains slow and must inherently balance sampling efficiency and sample quality. In this work, we present a systematic study of how the \emph{degree of stochasticity} in Markov transitions governs the sampling tradeoff. We show that highly deterministic transitions converge rapidly but suffer from error accumulation, while more stochastic transitions converge more slowly yet can achieve higher final sample quality. Using an information-theoretic analysis, we identify the underlying mechanism as an error-correcting effect induced by \emph{redundant transitions} that symmetrically exchange mass between states, and show that these transitions can provably contract sampling errors. Motivated by this analysis, we propose \emph{Discrete Churn and Restart Sampling} (DCRS), a novel inference algorithm that injects controlled stochasticity by alternating between forward and reverse diffusion processes. Experiments on synthetic datasets and large-scale benchmarks show that DCRS improves the speed-quality tradeoff in the low number of function evaluations regime. On image datasets, DCRS achieves up to a

10\times

reduction in sampling steps compared to standard samplers while maintaining competitive sample quality, whereas on language benchmarks, we observe more nuanced behavior depending on the corruption process and sampling procedure.

Abstract

Explore similar work

On the Error-Correcting Effects of Stochasticity in Discrete Diffusion

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion