When Does a Diffusion Model Decide What to Draw ?

Snigdha Chandan Khilar

cs.LGOct 5, 2026

When Does a Diffusion Model Decide What to Draw ?

Authors: Snigdha Chandan Khilar

Organizations: Independent Researcher

Abstract

A diffusion model starts from pure noise and removes it step by step. Somewhere along the way it stops being able to become "anything" and becomes committed to, say, a horse rather than a truck. We measure when this happens, and what a trained model gets wrong about it, on CIFAR-10. The most direct measurement is to freeze a half-finished image, restart the generation from that point many times, and count how often each class comes out. We call this probability the committor. Measured this way, the model settles coarse questions (vehicle or animal?) at roughly twice the noise level of fine ones (which animal?). A much cheaper measurement, the noise level at which a classifier's opinion about two classes splits into two distinct groups, gets the order of these decisions right (rank correlation 0.73-0.88) but not their exact timing. We then compare pretrained models with their

Figures & tables

term	meaning
noise level $\sigma$	How much Gaussian noise is mixed into an image. Generation starts at a large $\sigma$ (pure noise) and ends near $\sigma=0$ (a clean image). Higher noise means earlier in generation.
commitment	The point in generation after which the final class of an image is effectively fixed.
committor	For a half-finished image, the probability that generation continued from it ends in a given class. Estimated by restarting generation many times from the same point.
decision noise level	The noise level below which a trajectory’s outcome is settled, i.e. its committor stays close to 0 or 1.
order parameter $m$	A single number that says how “class $a$ -like versus class $b$ -like” an image is. We use a classifier’s log-odds.
density transition $\sigma_{\rm bif}$	The noise level at which the distribution of $m$ across many images splits from one hump into two.

Table 1: Key terms, in the order they appear.

Figure 1: Two ways a population of images can split into two classes. We follow the distribution of an order parameter

m

(“how

a

-like versus

b

-like”) for a mixture of two Gaussian classes centred at

m=\pm 1

, as noise is removed. The horizontal axis is

\tau

, the noise variance in units of the class separation, so generation moves from right to left. Black lines are the peaks of the distribution, red lines its dips. (a) With equal numbers of each class, the single peak splits symmetrically into two: a pitchfork. (b) With 75% of one class, the single peak simply becomes the majority class, and the minority class appears later, suddenly, as a new peak with a dip next to it: a fold. (c) The distribution itself in case (b), before, at and after the split.

Figure 2: Can the comparison detect errors? We distort the real data on purpose, by reweighting images within each class towards more typical-looking examples. Class proportions stay unchanged. We then compare distorted with clean data exactly as we compare a model with data. The horizontal axis is the size of the distortion (KL divergence, nats per class). (a) Ratio of split noise levels, averaged over pairs, with class-block intervals: the timing of the split reacts to small distortions. (b) Agreement of the lopsidedness

h

: it barely reacts until the distortion is large. Blue: all classes distorted; red: only deer.

model	sampler and modification	images	purpose
EDM uncond.	–	14,500	main comparison (Sec. 6 )
EDM uncond.	– (replicate)	3,500	replication
EDM uncond.	–	320 (+46,080 restarts)	committors (Sec. 5 )
EDM cond.	equal labels	4,000	proportions right by design
EDM uncond.	Heun, 18 steps, full precision	10,000	standard sampler
EDM uncond.	256 / 512 / 1,024 steps, full prec.	2,000 / 1,500 / 1,000	sampler (Sec. 6.4 )

Table 2: All sampling runs. “Images” is the number of generated trajectories. Unless noted, the sampler is the 256-step stochastic sampler in half precision.

Figure 3: Decisions of individual images in the unconditional EDM model. (a) For images paused at each noise level (high noise on the left), the fraction of 12 restarts that end in the same superclass (blue) or the same class (orange) as the original. Superclass agreement rises first. (b) Median noise level at which pairs of each kind are decided, for three thresholds on the cross-over rate (blue), and the median density transition of the same pairs (black). Decisions and density transitions agree on the order, coarse before fine, but not on the absolute noise level.

	decision noise level (median)				stick: log-odds $z$		stick: $\tanh(z/2)$
threshold	veh./anim.	veh./veh.	anim./anim.	ratio	rank corr.	ratio	rank corr.	ratio
0.05	2.16	1.61	0.87	2.5	0.74 [0.47, 0.91]	1.6	0.73 [0.38, 0.93]	0.39
0.10	3.15	2.60	1.52	2.1	0.84 [0.66, 0.93]	2.8	0.88 [0.71, 0.97]	0.58
0.20	5.29	4.01	3.02	1.8	0.81 [0.54, 0.94]	4.2	0.87 [0.61, 0.96]	0.93

Table 3: When pairs are decided, and how well the cheap density transition tracks it. Left: median decision noise level by kind of pair, for three thresholds on the cross-over rate, and the ratio between vehicle/animal and animal/animal pairs. Right: rank correlation between decision levels and density transitions (class-block 95% interval), and their median ratio, under two choices of measuring stick.

Figure 4: The population view of CIFAR-10, measured with the classifier’s log-odds. (a) Noise level of the density transition for each class pair, grouped by kind of pair, with 95% bootstrap intervals; horizontal bars are medians. (b) Lopsidedness

h

of each pair under the raw log-odds and a recalibrated version. Shaded bands show the noise floor from exactly symmetric pairs. Red points are lopsided beyond the floor under both, with the same sign.

Figure 5: The unconditional EDM model, before any correction. (a) Density transition noise levels of the model against the data’s, with 95% bootstrap intervals; the dashed line is equality. (b) The model’s lopsidedness

h

for each pair against the difference in how much the model over-produces the two classes. (c) How often each class is generated, relative to the test set, under our 256-step sampler and EDM’s standard 18-step sampler, as judged by our classifier and by an independent ViT.

Figure 6: Is the class imbalance the only error? (a) For each class pair of the uncorrected EDM model, the error in split timing against the error in lopsidedness, coloured by kind of pair. (b) How well the lopsidedness agrees with the data, for EDM as is, EDM against data reweighted to its class proportions (mass-matched), the class-conditional EDM with equal labels, EDM after correcting its proportions in the sampler, and DDPM against mass-matched data. The dotted line is the agreement between two halves of the data. (c) Average ratio of split noise levels over pairs, with class-block 95% intervals. The grey band is the range expected between two halves of the data.

model	compared with	agreement of $h$	spread of timing error	timing ratio
EDM, $n{=}14{,}500$	the data	0.11 [ $-0.47$ , 0.68]	0.332 [0.16, 0.42]	1.00
	mass-matched data	0.974 [0.94, 0.99]	0.178 [0.09, 0.25]	1.115 [1.01, 1.23]
	two halves of the data	0.981 [0.93, 0.99]	0.208 [0.12, 0.30]	—
EDM cond., equal labels	the data	0.87	0.223	1.00 [0.92, 1.18]
$n{=}4{,}000$	two halves of the data	—	0.211	1.07 [0.98, 1.29]
EDM, corrected sampler	the data	0.889 [0.64, 0.98]	0.254 [0.15, 0.35]	1.017

Table 4: Models compared with their data. “Mass-matched” data have been reweighted so that their class proportions equal the model’s. “Two halves of the data” gives the agreement expected from sampling noise alone, at a matching sample size. Brackets are class-block 95% intervals.

sampler	precision	images	class imbalance	noise floor
stochastic, 256 steps	half	2,500–3,500	0.12–0.15	0.040–0.048
stochastic, 256 steps	full	2,000	0.171	0.054
stochastic, 512 steps	full	1,500	0.132	0.063
stochastic, 1,024 steps	full	1,000	0.137	0.076
deterministic, 18 steps (EDM default)	full	10,000	0.095	0.024

Table 5: Class imbalance (average

|\log|

of generated over expected frequency, by the ViT) of the same EDM network under different samplers. “Noise floor” is the value expected from random sampling alone at that sample size, with perfectly balanced classes.

Figure 7: The model’s decisions, one image at a time. (a) Along the model’s own trajectories, the class proportions that our classifier sees at each noise level, relative to uniform. Dots on the right are the final proportions. The imbalance becomes visible as superclasses are decided (dotted lines). (b) How far the model’s committors are from the data’s class probabilities, beyond what 12 restarts can resolve, with 95% bootstrap bands: as is (blue), and after adjusting for the model’s class proportions (orange), which makes no difference. The dashed line is the level reached at low noise, which reflects our classifier’s own mistakes.

run	imbalance (ours)	imbalance (ViT)	FID	$h$ vs data
no correction	0.134	0.119	12.93	$\approx 0$
first-order correction, 1 round	0.076	0.083	12.68	0.56
exact correction (Eq. 4 ), 1 round	0.093	0.107	12.95	0.54
exact correction, 2nd round	0.051	0.074	12.98	0.89

Table 6: Correcting the class proportions of the unconditional EDM model inside the sampler (2,500 images each, 256-step stochastic sampler). Class imbalance by our classifier and by the ViT; the sampling-noise floor at this size is

0.048

. FID at 2,500 images.

split point $\sigma_{1}$	0.5	1.0	2.0	4.0
correct only for $\sigma\geq\sigma_{1}$ (noisy part)	$0.96\pm 0.03$	$0.79\pm 0.02$	$0.32\pm 0.02$	$0.08\pm 0.02$
correct only for $\sigma<\sigma_{1}$ (cleaner part)	$0.08\pm 0.02$	$0.43\pm 0.02$	$0.88\pm 0.03$	$0.99\pm 0.03$
sum	1.05	1.22	1.20	1.07

Table 7: Applying the exact correction to only part of generation, on the toy six-class problem (20,000 images, mean

\pm

SD over 3 seeds): fraction of the full correction achieved when correcting only at noise levels above, or only below, a split point

\sigma_{1}

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

warp	tilting slope, raw	tilting slope, recalibrated	correlation with $z^{\ast}$	relative error
affine	1.42	0.98	0.999	0.054
$3\tanh(z/3)$	1.34	0.99	0.997	0.079
signed $\sqrt{\|z\|}$	1.65	0.99	0.998	0.060
$z+0.08z^{3}$	0.49	0.78	0.959	0.485

Appendix

Table 8: Recalibration of known warps

u=g(z^{\ast})

of an exact log-ratio (

8{,}000

samples, cross-fitted).

comparison	corr. $h$	sign agr.	median $\|\Delta h\|$	corr. $\log\sigma_{\rm bif}$	median ratio
ResNet: raw vs. recal.	0.822	0.70	0.058	0.968	1.00
VGG: raw vs. recal.	0.834	0.77	0.056	0.946	0.90
raw: ResNet vs. VGG	0.938	0.86	0.027	0.958	1.11
recal.: ResNet vs. VGG	0.903	0.93	0.039	0.944	1.00

Appendix

Table 9: Agreement between coordinates over the 44 pairs (correlation, sign agreement of

h

, and median ratio of

\sigma_{\rm bif}

Figure 8: The noise floor for lopsidedness. Filled bars:

|h|

of the 44 real class pairs (census classifier). Orange outline: 13 pairs that are exactly symmetric by construction (a class and its vertical flip or colour-swapped copy), measured through a classifier trained on them. Green dashed outline: the same symmetric pairs with the measuring stick made exactly antisymmetric, which leaves only estimation noise. The dotted line is the largest symmetric-pair value,

0.047

, used as the noise floor.

variant	corr. $h$	median $\|\Delta h\|$	corr. $\log\sigma_{\rm bif}$	median $\sigma_{\rm bif}$ ratio
bandwidth $0.75\times$	0.964	0.030	0.978	1.22
bandwidth $1.0\times$	0.984	0.020	0.989	1.22
bandwidth $2.0\times$	0.983	0.016	0.983	0.82
merge $0.06$ , prominence $0.10$	0.994	0.010	0.993	0.90
persistence over 2 levels	1.000	0.000	1.000	1.00
persistence over 3 levels	1.000	0.000	1.000	1.00

Appendix

Table 10: Sensitivity of the census to the density estimator (raw coordinate, 44 pairs), relative to the default: bandwidth

1.5\times

Silverman, merge distance

0.03

, prominence

0.05

nats, no persistence requirement.

$\sigma$	total variation	excess over 12-draw noise	above floor [95% CI]	slope
7.95	0.354	0.029	$-0.059$ [ $-0.086$ , $-0.037$ ]	1.03
5.42	0.344	0.034	$-0.054$ [ $-0.080$ , $-0.027$ ]	1.09
3.61	0.361	0.086	$-0.002$ [ $-0.040$ , $+0.015$ ]	1.11
2.42	0.359	0.118	$+0.029$ [ $-0.005$ , $+0.049$ ]	1.09
1.63	0.325	0.125	$+0.037$ [ $+0.015$ , $+0.062$ ]	1.07
1.11	0.303	0.147	$+0.058$ [ $+0.034$ , $+0.089$ ]	1.00

Appendix

Table 11: Committor fidelity of the pretrained unconditional model (320 trajectories, 12 continuations per state). “Above floor” subtracts the mean excess at the two lowest noise levels (

0.088

); the interval is from 100 bootstrap resamples of trajectories. “Slope” regresses the measured committor on the classifier posterior over all states and classes.

	airpl.	autom.	bird	cat	deer	dog	frog	horse	ship	truck
SDE, fp16 (14.5k; ours)	0.100	0.128	0.067	0.081	0.075	0.094	0.108	0.129	0.094	0.125
SDE, fp16 (3.5k; ours)	0.101	0.133	0.076	0.082	0.082	0.104	0.104	0.111	0.094	0.114
SDE, fp16 (3.5k; ViT)	0.099	0.130	0.082	0.089	0.079	0.095	0.100	0.113	0.093	0.119
Heun, fp32 (10k; ours)	0.112	0.111	0.082	0.096	0.086	0.094	0.102	0.115	0.106	0.095
Heun, fp32 (10k; ViT)	0.110	0.109	0.086	0.103	0.085	0.083	0.104	0.115	0.107	0.096

Appendix

Table 12: Generated class frequencies under two samplers (sample size; labelling classifier).

run	$\sigma_{\rm bif}$ ratio, raw	$\sigma_{\rm bif}$ ratio, rebalanced	rebalanced $h$ vs data	class-mass error	FID
baseline (self)	0.98 [0.94, 1.03]	1.00 [0.97, 1.03]	0.91	0.134	12.9
score $\times 0.995$	0.81 [0.72, 0.90]	0.94 [0.83, 1.04]	0.89	0.203	13.7
score $\times 0.98$	0.71 [0.57, 0.90]	0.84 [0.68, 1.11]	0.94	0.227	15.7
score $\times 0.95$	0.55 [0.39, 0.69]	0.87 [0.75, 0.97]	0.84	0.393	25.4
noise level $\times 1.1$	0.55 [0.44, 0.68]	0.81 [0.66, 1.05]	0.85	0.349	35.7
noise level $\times 1.25$	0.60 [0.30, 1.50]	0.66 [0.42, 1.27]	0.21	1.009	106.1

Appendix

Table 13: Corruptions of the score and band-limited class corrections. “Class-mass error” is the mean

|\log f/f_{\rm ref}|

with our classifier’s labels.

Figure 9: Graded corruptions of the score (Table 13 ). Blue: score multiplied by

\lambda

; orange: denoiser told the noise level is

\kappa\sigma

. (a) Ratio of density transitions to the uncorrupted baseline, as sampled, and (b) after rebalancing, with class-block intervals. (c) Agreement of the rebalanced lopsidedness with the data, which illustrates the low sensitivity of this statistic quantified in Section 3.5 . (d) Class imbalance. (e) FID at 2,500 images.

pair	$\tau_{\rm bif}$	jump	$h$	$a$	$b$	pair	$\tau_{\rm bif}$	jump	$h$	$a$	$b$
(0,1)	0.906	0.72	$-0.011$	$+0.042$	0.277	(2,9)	0.754	1.07	$-0.041$	$+0.143$	0.350
(0,2)	0.827	0.74	$+0.011$	$+0.045$	0.272	(3,4)	0.792	0.95	$-0.027$	$+0.101$	0.323
(0,3)	0.864	0.63	$-0.007$	$+0.026$	0.260	(3,5)	0.571	1.13	$-0.050$	$+0.167$	0.366
(0,4)	0.855	0.84	$-0.018$	$+0.069$	0.297	(3,6)	0.782	0.96	$-0.028$	$+0.104$	0.325
(0,5)	0.748	1.06	$-0.040$	$+0.142$	0.354	(3,7)	0.770	1.07	$-0.040$	$+0.140$	0.343
(0,6)	0.810	0.86	$-0.019$	$+0.074$	0.298	(3,8)	0.770	0.55	$-0.004$	$+0.014$	0.236

Appendix

Table 14: MNIST census along the LDA direction.

\tau_{\rm bif}

is in units of

\tau=\sigma^{2}/\ell^{2}

pair	data	epoch 20	epoch 80
MNIST (0,1)	0.905	$0.28\pm 0.02$	$0.74\pm 0.03$
MNIST (0,8)	0.933	$0.40\pm 0.01$	$0.64\pm 0.02$
MNIST (3,8)	0.778	not bimodal	0.58
FashionMNIST (1,7)	0.917	0.28	0.75

Appendix

Table 15: Transition of small trained denoisers along the LDA direction, in noise variance

\tau

(mean

\pm

sd over three seeds where given).

pair	sampler	none	$\tau>1.5$	$0.3\leq\tau\leq 1.5$	$\tau<0.3$	all $\tau$
(0,1)	Heun-40	$+0.213\pm 0.034$	$+0.186\pm 0.035$	$+0.188\pm 0.028$	$+0.193\pm 0.038$	$+0.138\pm 0.024$
(0,1)	SDE-40	$+0.097\pm 0.062$	$+0.079\pm 0.064$	$+0.056\pm 0.056$	$+0.071\pm 0.066$	$-0.008\pm 0.042$
(3,8)	SDE-40	$+0.101\pm 0.022$	$+0.098\pm 0.029$	$+0.077\pm 0.025$	$+0.073\pm 0.031$	$+0.030\pm 0.017$
(0,8)	Heun-40	$-0.076\pm 0.001$	$-0.065\pm 0.006$	$-0.072\pm 0.006$	$-0.079\pm 0.007$	$-0.061\pm 0.010$
(2,7)	Heun-40	$+0.184\pm 0.011$	$+0.164\pm 0.006$	$+0.172\pm 0.002$	$+0.169\pm 0.010$	$+0.149\pm 0.004$
(2,7)	SDE-40	$+0.102\pm 0.007$	$+0.107\pm 0.007$	$+0.098\pm 0.007$	$+0.101\pm 0.010$	$+0.089\pm 0.018$

Appendix

Table 16: Band-limited drift replacement (epoch 20, three seeds). Excess fraction of the first class over the real-data baseline, mean

\pm

sd.

$h$	Heun-10	Heun-40	SDE-10	SDE-40	SDE-160
$-0.0002$	$-0.014$	$-0.014$	$-0.008$	$-0.017$	$-0.015$
$+0.0080$	$+0.007$	$+0.010$	$-0.002$	$+0.015$	$+0.011$
$+0.0313$	$-0.010$	$-0.004$	$-0.014$	$+0.006$	$+0.002$
$+0.0831$	$-0.022$	$-0.018$	$-0.013$	$+0.002$	$-0.007$
$+0.2027$	$-0.020$	$-0.015$	$-0.004$	$+0.003$	$+0.000$

Appendix

Table 17: Class-fraction error towards the persisting class in

d=784

with an exact score (s.e.

\approx 0.008

pair	$\sigma_{\rm bif}$ (raw)	$h$ (raw)	$h$ (recalibrated)	robust
airplane/automobile	1.33 [1.20, 1.47]	$+0.006$ [ $-0.022$ , $+0.047$ ]	$-0.098$ [ $-0.123$ , $-0.076$ ]
airplane/bird	2.69 [1.80, 3.63]	$+0.010$ [ $-0.025$ , $+0.036$ ]	$+0.025$ [ $-0.050$ , $+0.145$ ]
airplane/cat	1.63 [1.47, 1.99]	$-0.126$ [ $-0.151$ , $-0.060$ ]	$-0.113$ [ $-0.135$ , $-0.067$ ]
airplane/deer	2.43 [1.47, 4.44]	$-0.070$ [ $-0.105$ , $-0.014$ ]	$-0.053$ [ $-0.116$ , $-0.021$ ]
airplane/dog	1.80 [1.63, 2.20]	$-0.079$ [ $-0.111$ , $+0.000$ ]	$-0.014$ [ $-0.102$ , $+0.034$ ]
airplane/frog	2.69 [2.20, 2.97]	$-0.275$ [ $-0.303$ , $-0.233$ ]	$-0.316$ [ $-0.340$ , $-0.286$ ]	yes

Appendix

Table 18: CIFAR-10 census (test set). Transition noise with 95% interval (raw coordinate); asymmetry

h

with 95% interval along the raw log-odds and along the recalibrated log-ratio. “Robust” marks pairs asymmetric beyond the through-classifier null in both coordinates, with the same sign. Cat/dog never develops two modes.

pair	$\sigma_{\rm bif}$ data	$\sigma_{\rm bif}$ model	ratio	95% interval	$h$ data	$h$ model
airplane/automobile	1.39	1.15	0.82	[0.68, 1.07]	$+0.007$	$-0.109$
airplane/bird	2.15	0.91	0.47	[0.24, 0.78]	$+0.018$	$+0.159$
airplane/cat	1.79	2.56	1.36	[1.13, 1.68]	$-0.087$	$+0.016$
airplane/deer	2.88	2.15	0.71	[0.44, 1.28]	$-0.013$	$+0.082$
airplane/dog	1.90	2.28	1.20	[0.97, 1.47]	$-0.061$	$-0.044$
airplane/frog	2.72	2.15	0.79	[0.58, 1.03]	$-0.232$	$-0.251$

Appendix

Table 19: Pretrained unconditional EDM against the data, per pair (reverse SDE, 14,500 trajectories; the reference subsampled to the same size). All values are medians of 100 paired bootstrap resamples, and the interval is the 95% interval of the ratio.

h

is raw, before rebalancing.

Explore similar work

How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation
Jul 4, 2026Saba Fathi, Fardin Ayar, Maryam Abdolali +3Diffusion Models U-Net
2Amirkabir University of Technology, Tehran, Iran · 1K. N. Toosi University of Technology, Tehran, Iran · 3The University of Tokyo, Tokyo, Japan
Test-Time Compositional Generalization in Diffusion Models via Concept Discovery
May 8, 2026Zekun Wang, Anant Gupta, Tianyi Zhu +1Diffusion Models Compositional Generalization
Georgia Institute of Technology · University of Virginia
Diffusion Models and Concept Formation
Sep 11, 2026Zekun Wang, Karthik Singaravadivelan, Christopher J. MacLellanDiffusion Models Hierarchical
College of Computing, Georgia Institute of Technology, Atlanta, GA 30332 USA

Jul 4, 2026, cs.CVJ/K move · Enter open · S save

How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

Saba Fathi, Fardin Ayar, Maryam Abdolali, Ehsan Javanmardi+2

2Amirkabir University of Technology, Tehran, Iran · 1K. N. Toosi University of Technology, Tehran, Iran · 3The University of Tokyo, Tokyo, Japan

Diffusion Models U-Net

Open paper AskPDF arXiv

Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: Attribute binding, Size-Order bias, and Background dependency. These dimensions serve not as an exhaustive taxonomy but as targeted probes of how the text-conditioned reconstruction-error score reaches a decision. Such a perspective is well studied for discriminative vision-language models, yet remains overlooked for diffusion classifiers. Extending an existing framework with five new attribute categories on newly constructed datasets, we find diffusion classifiers are less prone to attribute misbinding than an OpenCLIP baseline; on the established ComCo benchmark they are substantially more susceptible to size-order shortcuts; and on ImageNet-B they suffer far larger accuracy drops, revealing heavy reliance on background over foreground cues. Reconstruction-error heatmaps and U-Net cross-attention visualizations expose the mechanism behind each bias. Because diffusion classifiers share the same denoiser as text-to-image models, these single-pass diagnostics also point toward analogous failure modes in generation. Overall, diffusion classifiers exhibit a distinct bias profile from vision-language models, offering guidance for building more robust diffusion-based models.

Abstract

Explore similar work

How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

Test-Time Compositional Generalization in Diffusion Models via Concept Discovery

Diffusion Models and Concept Formation