Who Gets Unmasked? Sampler-Induced Bias in Diffusion Language Models

Xujun Che; Farsheed Haque; Depeng Xu; Xi Niu; Haopeng Zhang

cs.CLOct 17, 2026

Who Gets Unmasked? Sampler-Induced Bias in Diffusion Language Models

Authors: Xujun Che, Farsheed Haque, Depeng Xu, Xi Niu, Haopeng Zhang

Organizations: University of North Carolina at Charlotte, Charlotte, NC, USA

Abstract

Diffusion language models (dLLMs) generate by gradually unmasking a canvas, so their samplers decide not only which token to write but also when to commit it. We show that this second decision changes who appears in generated text, a sampler-induced bias on top of the model's own. Common commit rules, including LLaDA's default, commit the positions whose sampled tokens are most probable; this peeking favors majority options. To separate the sampler from the model, we read the model's belief about a fork, such as a protagonist's gender, in one forward pass, and show that this belief matches exact sampling from the model. Measured against it, peeking samplers on LLaDA and Dream amplify occupational gender stereotypes and collapse opinion distributions, and confidence-based ordering shifts gender-balanced occupations toward men, more so under faster parallel decoding. We propose Fork-First Sampling, which decides the fork before anything else is written: it follows the belief or any target distribution, improves fluency, and avoids the stories that remark on the protagonist's gender, which prompt-based fixes produce, at the cost of one forward pass. Fairness evaluations of dLLMs should report the commit rule.

Figures & tables

Rule	Score $s_{i}$	Peeks	Used by
Low-confidence	$p_{i}(y_{i})$	$\checkmark$	LLaDA
Threshold	$p_{i}(y_{i})\geq\lambda$	$\checkmark$	Fast-dLLM
MaskGIT+	$q_{i}(y_{i})$	$\checkmark$	Dream (opt.)
Soft low-conf.	$p_{i}(y_{i})^{1/T^{\prime}}\!+\!\epsilon$	$\checkmark$	TLC
Confidence	$\max_{v}p_{i}(v)$	$\times$	our patch
Entropy	$-H(q_{i})$	$\times$	Dream (ex.)

Table 1: Commit rules.

\checkmark

: the score peeks at the drawn token.

Figure 2: One LLaDA step. Peeking ranks the position that drew She 25th of 28; the identity-blind rule ranks it 3rd. FFS plants the fork token first.

Figure 3: (a) Belief against faithful samples. (b, c) Belief against sampler behavior, with GLM slopes

\beta

(1: faithful).

Figure 4: LLaDA over temperature (top-

p=1

): (a) slope on the belief, (b) fluency. Only FFS -faithful is faithful and fluent at every temperature.

Figure 5: (a) Majority/minority commit-rate ratio. (b) Pre-fork drift of the fork position’s female share. (c) Male share against tokens per step. Belief-balanced prompts in (b, c).

			LLaDA					Dream
Method	Edits prompt	Extra passes	$\beta$	TE	$\Delta$ Flu.	Mark.	AUC	$\beta$	TE	$\Delta$ Flu.	Mark.	AUC
Commit rules
• Greedy			2.17	.123	$-0.20$	0.0	–	3.48	.142	$-0.45$	0.0	–
• Peeking (default)			1.80	.116	0	0.3	–	2.97	.068	0	0.3	–
• Identity-blind			1.53	.087	$-0.59$	0.6	.68	2.33	.062	$-0.16$	0.3	.64
• Random order			1.49	.180	$+0.56$	0.6	.86	2.16	.048	$-0.15$	0.5	.93

Table 2: Methods in the default setting. TE: target error; Mark.: % marked narratives; AUC: separability from default outputs (0.5 is best); –: not applicable.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Sampler	$k=1$	$k=2$	$k=4$
• Peeking (low-confidence)	3.34	2.81	1.12
• Identity-blind (confidence)	0.98	1.00	1.01
• Random order	1.00	1.00	1.00
• TLC	1.72	1.64	1.35
FFS -faithful	1.00	1.00	1.01
FFS -fair	$-0.01$	$-0.01$	0.00

Appendix

Table 3: Slope

\beta

in the synthetic model (

T=1

; 95% CIs within

\pm 0.08

Labels	LLaDA $\Delta\beta$	Dream $\Delta\beta$
Primary judge	0.28 $[0.08,0.51]$	0.64 $[0.26,1.29]$
Second judge	0.21 $[0.04,0.42]$	0.64 $[0.25,1.29]$
Pronoun rules	0.15 $[-0.04,0.38]$	0.61 $[0.24,1.23]$

Appendix

Table 4: Peeking minus identity-blind slope under three label sources.

	$\beta$			$\alpha$			Flu.
	Peek	Blind	Rand.	Peek	Blind	Rand.	Blind
$T=0.3$	2.00	1.81	1.82	$-0.99$	$-0.94$	$-0.21$	3.38
$T=0.5$	1.80	1.44	1.49	$-0.90$	$-0.71$	$-0.08$	3.16
$T=0.7$	1.78	1.16	1.18	$-0.72$	$-0.64$	$-0.18$	2.75
$T=1.0$	1.73	0.76	0.77	$-0.33$	$-0.38$	$-0.19$	1.72
$T=1.3$	1.43	0.57	0.55	$-0.09$	$+0.32$	$+0.24$	1.01

Appendix

Table 5: LLaDA slope and intercept over temperature and tokens per step

k

Model	Sampler	Decided	Drift (balanced)
LLaDA	Peeking	65%	$-0.22$ $[-0.30,-0.12]$
LLaDA	Identity-blind	65%	$-0.20$ $[-0.27,-0.12]$
LLaDA	Random order	40%	$-0.04$ $[-0.07,-0.01]$
Dream	Peeking	77%	$-0.28$ $[-0.41,-0.15]$
Dream	Identity-blind	80%	$-0.26$ $[-0.37,-0.13]$
Dream	Random order	70%	$-0.25$ $[-0.35,-0.17]$

Appendix

Table 6: Fork position already decided at the first pronoun, and its pre-fork drift.

	LLaDA			Dream
	$H$ ratio	TV	Coll.	$H$ ratio	TV	Coll.
• Peeking	0.42	.252	13.6	0.45	.203	16.0
• Identity-blind	0.78	.189	2.8	0.87	.145	2.0
FFS -faithful	0.93	.111	0.8	0.91	.102	0.4
Sampling floor	0.91	.107	–	0.91	.101	–

Appendix

Table 7: Opinion distributions (

T=1

). Floor: exact sampling from the belief.

		she	he	they	xe	ey	fae
LLaDA	Belief	.05	.17	.22	.96	.92	–
	Peeking	.04	.12	.17	.95	.93	.79
	Blind	.05	.15	.17	.96	.95	.79
	Random	.05	.12	.22	.96	.96	.80
Dream	Belief	.04	.14	.31	.89	.71	–
	Peeking	.02	.15	.29	.57	.72	.89

Appendix

Table 8: TANGO misgendering rate by declared pronoun family.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 466 papers

Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models
Oct 20, 2026Koshiro Saito, Daisuke Oba, Danushka Bollegala +1Social Bias in Language Models Diffusion Language Models
Institute of Science Tokyo · University of Liverpool · Amazon +2
Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering
Oct 5, 2026Sarim Hashmi, Mukul Ranjan, Abdelrahman Elsayed +3Adversarial Attacks on Diffusion Models Diffusion Language Models
Mohamed bin Zayed University of Artificial Intelligence
Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models
Sep 30, 2026Stipe Frković, Metod Jazbec, Christian A. NaessethMasked Diffusion Models Speculative Decoding for Diffusion Language Models
University of Amsterdam · UvA-Bosch Delta Lab, University of Amsterdam

Oct 20, 2026J/K move · Enter open · S save

Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models

Koshiro Saito, Daisuke Oba, Danushka Bollegala, Naoaki Okazaki

Institute of Science Tokyo · University of Liverpool · Amazon+2

Social Bias in Language Models Diffusion Language Models Language Model Bias Evaluation Language Model Debiasing

Open paper AskPDF arXiv

Diffusion language models (DLMs) are emerging as a practical alternative to autoregressive language models (ARLMs), yet their fairness remains largely uncharacterized. DLMs introduce sampling controls with no direct autoregressive counterpart, such as block length and remasking strategy, that directly set their speed--quality operating point. This raises a basic question: does a DLM's measured social bias remain stable across these settings? Across LLaDA, LLaDA-1.5, and Dream on Open-BBQ, we find that all three models exhibit social bias, but both the magnitude of measured bias and the relative ordering of models by bias are highly sensitive to the sampling configuration. This sampling dependence also changes what successful debiasing means. While mitigation strategies established for ARLMs transfer effectively to DLMs and reduce bias without degrading downstream performance, sampling dependence persists even after mitigation. Our results show that fairness in DLMs cannot be characterized independently of the sampling configuration, even after debiasing. Project page is available at https://pakapaka333.github.io/Beyond-Model-Weights/.

Abstract

Diffusion Language Models

Explore similar work

Beyond Model Weights: Sampling-Dependent Fairness in Diffusion Language Models

Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering

Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models