Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models

Zhengyu Fang; Zhimeng Jiang; Huiyuan Chen; Xiaoge Zhang; Tianyi Li; Kaiyu Tang; Xiao Li; Jing Li

cs.LGApr 15, 2026

Dataset-Level Metrics Attenuate Non-Determinism: A Fine-Grained Non-Determinism Evaluation in Diffusion Language Models

Authors: Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiaoge Zhang, Tianyi Li, Kaiyu Tang, Xiao Li, Jing Li

Organizations: Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, USA · Department of Computer Science & Engineering, Texas A&M University, College Station, USA · Department of Biochemistry, Case Western Reserve University, Cleveland, USA · Center for RNA Science and Therapeutics, Case Western Reserve University, Cleveland, USA ·

Abstract

Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remains poorly understood. The existing non-determinism evaluations for LLMs predominantly rely on dataset-level metrics under fixed inference configurations, providing limited insight into how model behavior varies across runs and evaluation conditions. In this work, we show that dataset-level metrics systematically attenuate non-determinism in diffusion language models by aggregating sample-level prediction quality across different runs. As a result, configurations with similar aggregate performance can exhibit substantially different behaviors on individual inputs, leaving fine-grained instability and distinct error patterns uncharacterized. To address this limitation, we conduct a fine-grained evaluation of non-determinism based on sample-level prediction differences across a range of model-related factors-including guidance scale, diffusion steps, and Monte Carlo sampling-as well as system-related factors such as batch size, hardware, and numerical precision. Our analysis reveals that non-determinism in DLMs is pervasive and structured, with code generation exhibiting markedly higher sensitivity to factor-level choices than question answering. To attribute sources of non-determinism evaluation, we introduce Factor Variance Attribution (FVA), a cross-factor analysis metric that decomposes observed non-determinism into variance attributable to different evaluation factor settings. Our findings highlight the need for fine-grained, factor-aware evaluation to enable reliable non-determinism assessment of diffusion language models.

Figures & tables

Question	Answer Options	INT8	BF16	FP16	Ground Truth
In regions where prevailing winds blow from the oceans to the shore, which of these is most likely to occur?	A: heavier rainfall B: dryer conditions C: cold temperatures D: frequent hurricanes	D	C	C	A
Matt is a tall, eleven-year-old boy. He has a scar on his right cheek. He is intelligent and an excellent drummer. Which of his traits did he most likely inherit?	A: his height B: his scar on his right cheek C: his intelligence D: his ability to play the drums	B	C	C	A
Two bicyclists were riding their bikes at a rate of 20 kilometers per hour (km/h). During the next half hour, they observed that they had increased their speed to 26 km/h. What was their average acceleration?	A: 0.2 km/h 2 B: 3.0 km/h 2 C: 6.0 km/h 2 D: 12.0 km/h 2	A	C	C	D

Table 1: Examples of sample-level prediction disagreements induced by numerical precision (LLaDA). All configurations contribute identically at the dataset level, yet produce different answers for the same inputs, exposing distinct error modes that are obscured by aggregate metrics. Additional qualitative examples are provided in Appendix D .

Panel of Figure 1 — Figure 1: Sample-level prediction flip rates across inference-time configurations. Each panel varies one inference factor while holding all other factors fixed: (a) batch size, (b) classifier-free guidance (CFG) scale, and (c) number of Monte Carlo (MC) samples. For a given sample $s$ , the prediction flip rate is defined as $1-\max_{a}\lvert\{c\in\mathcal{C}:\hat{y}_{s,c}=a\}\rvert/\lvert\mathcal{C}\rvert$ , where $\hat{y}_{s,c}$ denotes the predicted label for sample $s$ under configuration $c$ , $a$ ranges over all possible predictions, and $\mathcal{C}$ is the set of configurations obtained by varying the selected factor across its settings, with all other factors held fixed. Reported values are averages of the per-sample flip rates over the dataset, with error bars indicating the standard error across samples. Larger flip rates indicate stronger configuration-induced non-determinism at the sample level, even when dataset-level accuracy remains nearly unchanged.

Figure 2: This hierarchy underlies our sample-aware and factor-aware evaluation paradigm and is formally instantiated by the definitions in Section 3.2 .

		Accuracy	Dataset-level	Sample-level	Accuracy	Dataset-level	Sample-level	Accuracy	Dataset-level	Sample-level
Backbone	Factor	PIQA			WinoGrande			ARC-Challenge
LLaDA	Precision (Std)	$0.7347$	$0.0006$	$0.4381$	$0.7340$	$0.0014$	$0.4382$	$0.4480$	$0.0023$	$0.4931$
	Precision (SE)	$0.7347$	$0.0004$	$0.0059$	$0.7340$	$0.0008$	$0.0071$	$0.4480$	$0.0013$	$0.0083$
	Batch Size of MC (Std)	$0.7383$	$0.0152$	$0.3790$	$0.7291$	$0.0148$	$0.3694$	$0.4370$	$0.0082$	$0.4644$
	Batch Size of MC (SE)	$0.7383$	$0.0068$	$0.0088$	$0.7291$	$0.0066$	$0.0104$	$0.4370$	$0.0037$	$0.0136$
	CFG (Std)	$0.7274$	$0.0068$	$0.4148$	$0.7263$	$0.0074$	$0.4011$	$0.4369$	$0.0072$	$0.4625$

Table 2: Dataset-level and sample-level evaluation results on question answering datasets for two diffusion language model backbones. For each factor, we report mean accuracy together with variability measures: dataset-level Std/SE are computed across inference configurations, while sample-level Std/SE are computed across individual samples under a fixed configuration.

		Pass@1	Dataset-level	Sample-level	Pass@1	Dataset-level	Sample-level
Backbone	Factor	HumanEval			MBPP
LLaDA	Precision (Std)	$0.3110$	$0.0061$	$0.4417$	$0.4007$	$0.0083$	$0.4649$
	Precision (SE)	$0.3110$	$0.0035$	$0.0199$	$0.4007$	$0.0048$	$0.0120$
	CFG (Std)	$0.2659$	$0.0432$	$0.3490$	$0.2864$	$0.0776$	$0.3358$
	CFG (SE)	$0.2659$	$0.0193$	$0.0272$	$0.2864$	$0.0347$	$0.0150$
	Steps (Std)	$0.1537$	$0.1258$	$0.2502$	$0.1908$	$0.1534$	$0.2592$

Table 3: Dataset-level and sample-level evaluation results on text generation datasets for two diffusion language model backbones. Within each backbone, results follow the same evaluation protocol as in Table 2 .

Figure 3: Within-factor variability across datasets and backbones. The figure reports the standard deviation (Std) of evaluation scores across different settings within each factor, aggregated by dataset. Larger Std indicates stronger sensitivity to specific settings of the same factor, reflecting pronounced setting-level non-determinism. Results are shown for both LLaDA and LLaDA-1.5 on question answering and code generation tasks.

Dataset	$\sigma^{2}_{\mathrm{between}}$	$\sigma^{2}_{\mathrm{within}}$	FVA
LLaDA
PIQA	0.000597	0.000565	0.514
WinoGrande	0.000698	0.000716	0.494
ARC-Challenge	0.000302	0.000200	0.601
HumanEval	0.018722	0.004755	0.797
MBPP	0.031495	0.007894	0.800

Table 4: Factor Variance Attribution (FVA) across datasets for two diffusion language model backbones. For each backbone, we report between-factor variance (

\sigma^{2}_{\mathrm{between}}

), within-factor variance (

\sigma^{2}_{\mathrm{within}}

), and the resulting FVA. While absolute variance magnitudes differ across backbones, task-dependent FVA patterns remain consistent.

	LLaDA			LLaDA-1.5
Factor	Mean	Std	SE	Mean	Std	SE
Batch	0.5239	0.1795	0.0359	0.6315	0.1293	0.0314
CFG	0.4886	0.2107	0.0421	0.5098	0.2095	0.0419
GPU	0.5258	0.1828	0.0578	0.5579	0.1658	0.0524
MC	0.5865	0.1552	0.0347	0.6132	0.1219	0.0272
Precision	0.5257	0.1822	0.0470	0.5585	0.1572	0.0406

Table 5: Factor-level summary statistics aggregated across datasets for two diffusion language model backbones. For each factor, we report the mean score, standard deviation (Std), and standard error (SE) across settings. Std reflects variability across settings within a factor, while SE reflects uncertainty of the mean estimate.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Uncertainty of factor-level effects across datasets and backbones. The figure reports the standard error (SE) of the mean evaluation score for each factor, aggregated across its settings. SE reflects the reliability of the estimated factor-level effect and complements within-factor variability by characterizing uncertainty at the factor level. Together with FVA, this analysis clarifies how evaluation non-determinism decomposes into between-factor effects and within-factor sensitivity.

Question	Answer Options	MC=8	MC=16	MC=32	MC=64	MC=128	MC=256	Ground Truth
Which statement best explains why the lower mantle of Earth is much more rigid and dense than the upper mantle?	A: The lower mantle is older B: The lower mantle is cooler C: The lower mantle is under more pressure D: The lower mantle is farther from the core	D	A	B	B	B	B	C
Which statement is an opinion about the heart?	A: The heart moves blood through the body B: The heart helps perform athletic activities C: The heart rate changes during the day D: The heart is composed of different types of cells	A	A	D	D	D	A	B

Appendix

Table 6: Additional examples of sample-level non-determinism under different Monte Carlo (MC) sampling.

		Accuracy	Dataset-level	Sample-level	Accuracy	Dataset-level	Sample-level	Accuracy	Dataset-level	Sample-level
Backbone	Factor	PIQA			WinoGrande			ARC-Challenge
LLaMA-2-7B	Precision (Std)	$0.7818$	$0.0005$	$0.4081$	$0.6967$	$0.0051$	$0.4442$	$0.4266$	$0.0034$	$0.4863$
	Precision (SE)	$0.7818$	$0.0003$	$0.0095$	$0.6967$	$0.0029$	$0.0125$	$0.4266$	$0.0020$	$0.0142$
	Batch size (Std)	$0.7817$	$0.0007$	$0.4127$	$0.6938$	$0.0021$	$0.4565$	$0.4232$	$0.0000$	$0.4936$
	Batch size (SE)	$0.7817$	$0.0003$	$0.0096$	$0.6938$	$0.0011$	$0.0128$	$0.4232$	$0.0000$	$0.0144$
	Temperature (Std)	$0.7824$	$0.0000$	$0.4127$	$0.6930$	$0.0000$	$0.4614$	$0.4232$	$0.0000$	$0.4943$

Appendix

Table 7: Dataset-level and sample-level evaluation results on question answering datasets for autoregressive language model backbones. For each factor, we report mean accuracy together with variability measures: dataset-level Std/SE are computed across inference configurations, while sample-level Std/SE are computed across individual samples under a fixed configuration.

		Pass@1	Dataset-level	Sample-level	Pass@1	Dataset-level	Sample-level
Backbone	Factor	HumanEval			MBPP
LLaMA-2-7B	Precision (Std)	$0.0650$	$0.0254$	$0.1517$	$0.1940$	$0.0140$	$0.3227$
	Precision (SE)	$0.0650$	$0.0147$	$0.0118$	$0.1940$	$0.0081$	$0.0144$
	Batch size (Std)	$0.0549$	$0.0122$	$0.1358$	$0.1935$	$0.0090$	$0.3172$
	Batch size (SE)	$0.0549$	$0.0061$	$0.0106$	$0.1935$	$0.0045$	$0.0142$
	Temperature (Std)	$0.0732$	$0.0198$	$0.1292$	$0.2076$	$0.0250$	$0.3224$

Appendix

Table 8: Dataset-level and sample-level evaluation results on text generation datasets for autoregressive language model backbones. Within each backbone, results follow the same evaluation protocol as in Table 7 .

Dataset	$\sigma^{2}_{\mathrm{between}}$	$\sigma^{2}_{\mathrm{within}}$	FVA
LLaMA-2-7B
PIQA	0.000002	0.000001	0.664
WinoGrande	0.000009	0.000007	0.559
ARC-Challenge	0.000009	0.000003	0.759
HumanEval	0.000307	0.000397	0.436
MBPP	0.000191	0.000314	0.378

Appendix

Table 9: Factor Variance Attribution (FVA) across datasets for autoregressive language model backbones. For each backbone, we report between-factor variance (

\sigma^{2}_{\mathrm{between}}

), within-factor variance (

\sigma^{2}_{\mathrm{within}}

), and the resulting FVA. While absolute variance magnitudes differ across backbones, task-dependent FVA patterns remain consistent.

	LLaMA-2-7B			Qwen2.5-7B
Factor	Mean	Std	SE	Mean	Std	SE
Batch	0.4294	0.2866	0.0641	0.6155	0.1322	0.0296
Temperature	0.4359	0.2779	0.0556	0.6035	0.1466	0.0293
GPU	0.4313	0.2928	0.0926	0.6181	0.1329	0.0420
Precision	0.4328	0.2870	0.0741	0.6133	0.1332	0.0344

Appendix

Table 10: Factor-level summary statistics aggregated across datasets for autoregressive language model backbones. For each factor, we report the mean score, standard deviation (Std), and standard error (SE) across settings. Std reflects variability across settings within a factor, while SE reflects uncertainty of the mean estimate.

Scope	Base	Alt	$\boldsymbol{\Delta}$
All datasets (pooled)	0.8141	0.7914	-0.0227
ARC-Challenge	0.6009	0.5874	-0.0135
HumanEval	0.7975	0.8104	+0.0130
MBPP	0.7996	0.7858	-0.0138
PIQA	0.5136	0.5206	+0.0070
Winogrande	0.4937	0.4893	-0.0044

Appendix

Table 11: Sensitivity analysis of FVA under variations in factor ranges and value selections (LLaDA). “Base” denotes the original configuration, while “Alt” expands factor ranges (e.g., scaling CFG) and uses alternative value choices (e.g., diffusion steps).

\Delta

indicates the difference (Alt

-

Base). FVA remains stable across both pooled and per-dataset evaluations, with only minor variations and no systematic shifts.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 443 papers

Diffusion Language Models: An Experimental Analysis
Jun 17, 2026Thomas Bertolani, Davide Bucciarelli, Leonardo Zini +2Diffusion Language Model Inference Diffusion Language Model Decoding
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
Generative Frontiers: Why Evaluation Matters for Diffusion Language Models
Apr 3, 2026Patrick Pynadath, Jiaxin Shi, Ruqi ZhangDiffusion Language Model Benchmarking Language Model Generation Evaluation
Department of Computer Science, Purdue University West Lafayette, Indiana, 47906, USA
TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models
Apr 15, 2026Yichuan Mo, Yukun Jiang, Yanbo Shi +4Diffusion Language Model Benchmarking LLM Safety Benchmarks
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University. · CISPA Helmholtz Center for Information Security · School of EECS, Peking University +1

Jun 17, 2026J/K move · Enter open · S save

Diffusion Language Models: An Experimental Analysis

Thomas Bertolani, Davide Bucciarelli, Leonardo Zini, Marcella Cornia+1

University of Modena and Reggio Emilia, Italy · University of Pisa, Italy

Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Language Models Efficient Language Model Inference Diffusion Language Model Benchmarking

Open paper AskPDF arXiv

Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks. Recently, Diffusion Language Models (DLMs) have emerged as an alternative paradigm that generates text through iterative denoising rather than next-token prediction, allowing parallel refinement of entire sequences. While numerous diffusion-based architectures have been proposed, differences in evaluation protocols, datasets, inference budgets, and generation hyperparameters make it difficult to compare their capabilities and understand the trade-offs they offer. In this work, we present a systematic experimental analysis of modern DLMs. Specifically, we evaluate eight state-of-the-art DLMs across eight benchmarks spanning reasoning, coding, translation, knowledge, and structured problem solving, while explicitly considering both generation quality and computational efficiency. Beyond downstream evaluation, we analyze the impact of key inference-time factors, including denoising steps, context length, block size, and parallel unmasking strategies, and complement large-scale experiments with controlled comparisons of smaller models trained under identical conditions. Our analysis highlights the strengths and limitations of diffusion-based language modeling across different tasks, architectures, and inference budgets. We show that the behavior of DLMs is strongly influenced by generation-time design choices, leading to distinct trade-offs between performance and computational efficiency. Overall, our study provides practical insights into the capabilities and deployment characteristics of contemporary DLMs.

Abstract

Diffusion Language Models

Explore similar work

Diffusion Language Models: An Experimental Analysis

Generative Frontiers: Why Evaluation Matters for Diffusion Language Models

TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models