Organizations: Department of Computer and Data Sciences, Case Western Reserve University, Cleveland, USA · Department of Computer Science & Engineering, Texas A&M University, College Station, USA · Department of Biochemistry, Case Western Reserve University, Cleveland, USA · Center for RNA Science and Therapeutics, Case Western Reserve University, Cleveland, USA · Department of Biomedical Engineering, Case Western Reserve University, Cleveland, USA
Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remains poorly understood. The existing non-determinism evaluations for LLMs predominantly rely on dataset-level metrics under fixed inference configurations, providing limited insight into how model behavior varies across runs and evaluation conditions. In this work, we show that dataset-level metrics systematically attenuate non-determinism in diffusion language models by aggregating sample-level prediction quality across different runs. As a result, configurations with similar aggregate performance can exhibit substantially different behaviors on individual inputs, leaving fine-grained instability and distinct error patterns uncharacterized. To address this limitation, we conduct a fine-grained evaluation of non-determinism based on sample-level prediction differences across a range of model-related factors-including guidance scale, diffusion steps, and Monte Carlo sampling-as well as system-related factors such as batch size, hardware, and numerical precision. Our analysis reveals that non-determinism in DLMs is pervasive and structured, with code generation exhibiting markedly higher sensitivity to factor-level choices than question answering. To attribute sources of non-determinism evaluation, we introduce Factor Variance Attribution (FVA), a cross-factor analysis metric that decomposes observed non-determinism into variance attributable to different evaluation factor settings. Our findings highlight the need for fine-grained, factor-aware evaluation to enable reliable non-determinism assessment of diffusion language models.
Figures & tables
Question
Answer Options
INT8
BF16
FP16
Ground Truth
In regions where prevailing winds blow from the oceans to the shore, which of these is most likely to occur?
Matt is a tall, eleven-year-old boy. He has a scar on his right cheek. He is intelligent and an excellent drummer. Which of his traits did he most likely inherit?
A: his height B: his scar on his right cheek C: his intelligence D: his ability to play the drums
B
C
C
A
Two bicyclists were riding their bikes at a rate of 20 kilometers per hour (km/h). During the next half hour, they observed that they had increased their speed to 26 km/h. What was their average acceleration?
A: 0.2 km/h 2 B: 3.0 km/h 2 C: 6.0 km/h 2 D: 12.0 km/h 2
A
C
C
D
Table 1: Examples of sample-level prediction disagreements induced by numerical precision (LLaDA). All configurations contribute identically at the dataset level, yet produce different answers for the same inputs, exposing distinct error modes that are obscured by aggregate metrics. Additional qualitative examples are provided in Appendix D .
Figure 1: Sample-level prediction flip rates across inference-time configurations. Each panel varies one inference factor while holding all other factors fixed: (a) batch size, (b) classifier-free guidance (CFG) scale, and (c) number of Monte Carlo (MC) samples. For a given sample s , the prediction flip rate is defined as 1−maxa∣{c∈C:y^s,c=a}∣/∣C∣ , where y^s,c denotes the predicted label for sample s under configuration c , a ranges over all possible predictions, and C is the set of configurations obtained by varying the selected factor across its settings, with all other factors held fixed. Reported values are averages of the per-sample flip rates over the dataset, with error bars indicating the standard error across samples. Larger flip rates indicate stronger configuration-induced non-determinism at the sample level, even when dataset-level accuracy remains nearly unchanged.
Figure 2: This hierarchy underlies our sample-aware and factor-aware evaluation paradigm and is formally instantiated by the definitions in Section 3.2 .
Backbone
Factor
PIQA
WinoGrande
ARC-Challenge
Accuracy
Dataset-level
Sample-level
Accuracy
Dataset-level
Sample-level
Accuracy
Dataset-level
Sample-level
LLaDA
Precision (Std)
0.7347
0.0006
0.4381
0.7340
0.0014
0.4382
0.4480
0.0023
0.4931
Precision (SE)
0.7347
0.0004
0.0059
0.7340
0.0008
0.0071
0.4480
0.0013
0.0083
Batch Size of MC (Std)
0.7383
0.0152
0.3790
0.7291
0.0148
0.3694
0.4370
0.0082
0.4644
Batch Size of MC (SE)
0.7383
0.0068
0.0088
0.7291
0.0066
0.0104
0.4370
0.0037
0.0136
CFG (Std)
0.7274
0.0068
0.4148
0.7263
0.0074
0.4011
0.4369
0.0072
0.4625
Table 2: Dataset-level and sample-level evaluation results on question answering datasets for two diffusion language model backbones. For each factor, we report mean accuracy together with variability measures: dataset-level Std/SE are computed across inference configurations, while sample-level Std/SE are computed across individual samples under a fixed configuration.
Backbone
Factor
HumanEval
MBPP
Pass@1
Dataset-level
Sample-level
Pass@1
Dataset-level
Sample-level
LLaDA
Precision (Std)
0.3110
0.0061
0.4417
0.4007
0.0083
0.4649
Precision (SE)
0.3110
0.0035
0.0199
0.4007
0.0048
0.0120
CFG (Std)
0.2659
0.0432
0.3490
0.2864
0.0776
0.3358
CFG (SE)
0.2659
0.0193
0.0272
0.2864
0.0347
0.0150
Steps (Std)
0.1537
0.1258
0.2502
0.1908
0.1534
0.2592
Table 3: Dataset-level and sample-level evaluation results on text generation datasets for two diffusion language model backbones. Within each backbone, results follow the same evaluation protocol as in Table 2 .
Figure 3: Within-factor variability across datasets and backbones. The figure reports the standard deviation (Std) of evaluation scores across different settings within each factor, aggregated by dataset. Larger Std indicates stronger sensitivity to specific settings of the same factor, reflecting pronounced setting-level non-determinism. Results are shown for both LLaDA and LLaDA-1.5 on question answering and code generation tasks.
Dataset
σbetween2
σwithin2
FVA
LLaDA
PIQA
0.000597
0.000565
0.514
WinoGrande
0.000698
0.000716
0.494
ARC-Challenge
0.000302
0.000200
0.601
HumanEval
0.018722
0.004755
0.797
MBPP
0.031495
0.007894
0.800
Table 4: Factor Variance Attribution (FVA) across datasets for two diffusion language model backbones. For each backbone, we report between-factor variance ( σbetween2 ), within-factor variance ( σwithin2 ), and the resulting FVA. While absolute variance magnitudes differ across backbones, task-dependent FVA patterns remain consistent.
LLaDA
LLaDA-1.5
Factor
Mean
Std
SE
Mean
Std
SE
Batch
0.5239
0.1795
0.0359
0.6315
0.1293
0.0314
CFG
0.4886
0.2107
0.0421
0.5098
0.2095
0.0419
GPU
0.5258
0.1828
0.0578
0.5579
0.1658
0.0524
MC
0.5865
0.1552
0.0347
0.6132
0.1219
0.0272
Precision
0.5257
0.1822
0.0470
0.5585
0.1572
0.0406
Table 5: Factor-level summary statistics aggregated across datasets for two diffusion language model backbones. For each factor, we report the mean score, standard deviation (Std), and standard error (SE) across settings. Std reflects variability across settings within a factor, while SE reflects uncertainty of the mean estimate.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Uncertainty of factor-level effects across datasets and backbones. The figure reports the standard error (SE) of the mean evaluation score for each factor, aggregated across its settings. SE reflects the reliability of the estimated factor-level effect and complements within-factor variability by characterizing uncertainty at the factor level. Together with FVA, this analysis clarifies how evaluation non-determinism decomposes into between-factor effects and within-factor sensitivity.
Question
Answer Options
MC=8
MC=16
MC=32
MC=64
MC=128
MC=256
Ground Truth
Which statement best explains why the lower mantle of Earth is much more rigid and dense than the upper mantle?
A: The lower mantle is older B: The lower mantle is cooler C: The lower mantle is under more pressure D: The lower mantle is farther from the core
D
A
B
B
B
B
C
Which statement is an opinion about the heart?
A: The heart moves blood through the body B: The heart helps perform athletic activities C: The heart rate changes during the day D: The heart is composed of different types of cells
A
A
D
D
D
A
B
Appendix
Table 6: Additional examples of sample-level non-determinism under different Monte Carlo (MC) sampling.
Backbone
Factor
PIQA
WinoGrande
ARC-Challenge
Accuracy
Dataset-level
Sample-level
Accuracy
Dataset-level
Sample-level
Accuracy
Dataset-level
Sample-level
LLaMA-2-7B
Precision (Std)
0.7818
0.0005
0.4081
0.6967
0.0051
0.4442
0.4266
0.0034
0.4863
Precision (SE)
0.7818
0.0003
0.0095
0.6967
0.0029
0.0125
0.4266
0.0020
0.0142
Batch size (Std)
0.7817
0.0007
0.4127
0.6938
0.0021
0.4565
0.4232
0.0000
0.4936
Batch size (SE)
0.7817
0.0003
0.0096
0.6938
0.0011
0.0128
0.4232
0.0000
0.0144
Temperature (Std)
0.7824
0.0000
0.4127
0.6930
0.0000
0.4614
0.4232
0.0000
0.4943
Appendix
Table 7: Dataset-level and sample-level evaluation results on question answering datasets for autoregressive language model backbones. For each factor, we report mean accuracy together with variability measures: dataset-level Std/SE are computed across inference configurations, while sample-level Std/SE are computed across individual samples under a fixed configuration.
Backbone
Factor
HumanEval
MBPP
Pass@1
Dataset-level
Sample-level
Pass@1
Dataset-level
Sample-level
LLaMA-2-7B
Precision (Std)
0.0650
0.0254
0.1517
0.1940
0.0140
0.3227
Precision (SE)
0.0650
0.0147
0.0118
0.1940
0.0081
0.0144
Batch size (Std)
0.0549
0.0122
0.1358
0.1935
0.0090
0.3172
Batch size (SE)
0.0549
0.0061
0.0106
0.1935
0.0045
0.0142
Temperature (Std)
0.0732
0.0198
0.1292
0.2076
0.0250
0.3224
Appendix
Table 8: Dataset-level and sample-level evaluation results on text generation datasets for autoregressive language model backbones. Within each backbone, results follow the same evaluation protocol as in Table 7 .
Dataset
σbetween2
σwithin2
FVA
LLaMA-2-7B
PIQA
0.000002
0.000001
0.664
WinoGrande
0.000009
0.000007
0.559
ARC-Challenge
0.000009
0.000003
0.759
HumanEval
0.000307
0.000397
0.436
MBPP
0.000191
0.000314
0.378
Appendix
Table 9: Factor Variance Attribution (FVA) across datasets for autoregressive language model backbones. For each backbone, we report between-factor variance ( σbetween2 ), within-factor variance ( σwithin2 ), and the resulting FVA. While absolute variance magnitudes differ across backbones, task-dependent FVA patterns remain consistent.
LLaMA-2-7B
Qwen2.5-7B
Factor
Mean
Std
SE
Mean
Std
SE
Batch
0.4294
0.2866
0.0641
0.6155
0.1322
0.0296
Temperature
0.4359
0.2779
0.0556
0.6035
0.1466
0.0293
GPU
0.4313
0.2928
0.0926
0.6181
0.1329
0.0420
Precision
0.4328
0.2870
0.0741
0.6133
0.1332
0.0344
Appendix
Table 10: Factor-level summary statistics aggregated across datasets for autoregressive language model backbones. For each factor, we report the mean score, standard deviation (Std), and standard error (SE) across settings. Std reflects variability across settings within a factor, while SE reflects uncertainty of the mean estimate.
Scope
Base
Alt
Δ
All datasets (pooled)
0.8141
0.7914
-0.0227
ARC-Challenge
0.6009
0.5874
-0.0135
HumanEval
0.7975
0.8104
+0.0130
MBPP
0.7996
0.7858
-0.0138
PIQA
0.5136
0.5206
+0.0070
Winogrande
0.4937
0.4893
-0.0044
Appendix
Table 11: Sensitivity analysis of FVA under variations in factor ranges and value selections (LLaDA). “Base” denotes the original configuration, while “Alt” expands factor ranges (e.g., scaling CFG) and uses alternative value choices (e.g., diffusion steps). Δ indicates the difference (Alt − Base). FVA remains stable across both pooled and per-dataset evaluations, with only minor variations and no systematic shifts.
Large Language Models (LLMs) have revolutionized language modeling through autoregressive generation, enabling strong performance across a wide range of tasks. Recently, Diffusion Language Models (DLMs) have emerged as an alternative paradigm that generates text through iterative denoising rather than next-token prediction, allowing parallel refinement of entire sequences. While numerous diffusion-based architectures have been proposed, differences in evaluation protocols, datasets, inference budgets, and generation hyperparameters make it difficult to compare their capabilities and understand the trade-offs they offer. In this work, we present a systematic experimental analysis of modern DLMs. Specifically, we evaluate eight state-of-the-art DLMs across eight benchmarks spanning reasoning, coding, translation, knowledge, and structured problem solving, while explicitly considering both generation quality and computational efficiency. Beyond downstream evaluation, we analyze the impact of key inference-time factors, including denoising steps, context length, block size, and parallel unmasking strategies, and complement large-scale experiments with controlled comparisons of smaller models trained under identical conditions. Our analysis highlights the strengths and limitations of diffusion-based language modeling across different tasks, architectures, and inference budgets. We show that the behavior of DLMs is strongly influenced by generation-time design choices, leading to distinct trade-offs between performance and computational efficiency. Overall, our study provides practical insights into the capabilities and deployment characteristics of contemporary DLMs.
Thomas Bertolani, Davide Bucciarelli, Leonardo Zini +2
University of Modena and Reggio Emilia, Italy · University of Pisa, Italy
Diffusion language models have seen exciting recent progress, offering far more flexibility in generative trajectories than autoregressive models. This flexibility has motivated a growing body of research into new approaches to diffusion language modeling, which typically begins at the scale of GPT-2 small (150 million parameters). However, these advances introduce new issues with evaluation methodology. In this technical note, we discuss the limitations of current methodology and propose principled augmentations to ensure reliable comparisons. We first discuss why OpenWebText has become the standard benchmark, and why alternatives such as LM1B are inherently less meaningful. We then discuss the limitations of likelihood evaluations for diffusion models, and explain why relying on generative perplexity alone as a metric can lead to uninformative results. To address this, we show that generative perplexity and entropy are two components of the KL divergence to a reference distribution. This decomposition explains generative perplexity's sensitivity to entropy, and naturally suggests generative frontiers as a principled method for evaluating model generative quality. We conclude with empirical observations on model quality at this scale. We include a blog post with interactive content to illustrate the argument at https://patrickpynadath1.github.io/blog/eval_methodology/.
Patrick Pynadath, Jiaxin Shi, Ruqi Zhang
Department of Computer Science, Purdue University West Lafayette, Indiana, 47906, USA
The rapid development of Language Diffusion Models (LDMs) challenges the dominant position of auto-regressive competitors in language processing. However, their flexible, any-order decoding strategies not only enable fast decoding speed but also potentially bring new trustworthiness challenges. To better understand the risks behind their pipelines, we introduce a comprehensive trustworthiness benchmark tailored to LDMs (TrustLDM), evaluating safety, privacy, and fairness across different LDM architectures with multiple categories of static post contexts. Our empirical results show that although LDMs generally exhibit strong trustworthiness with only the user prompts, their alignment behavior degrades noticeably when the malicious post contexts are attached to the masked responses. We further observe that longer contexts do not necessarily induce stronger effects, and both decoding order and generation length affect the evaluation outcomes. Finally, we propose TrustLDM-Auto, an automatic evaluation framework that leverages LDM decoding flexibility to systematically identify vulnerable configurations, revealing substantial trustworthiness weaknesses across all evaluated models and dimensions. Our work may potentially help the community build more trustworthy LDMs. Our code is available at https://github.com/PKU-ML/TrustLDM.
Yichuan Mo, Yukun Jiang, Yanbo Shi +4
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University. · CISPA Helmholtz Center for Information Security · School of EECS, Peking University +1