Rate-Resolved Text Evaluation with Masked Diffusion Language Models

Yiqi Liu; Kun Zhao; Chenghua Lin; Nafise Sadat Moosavi

cs.CLOct 11, 2026

Rate-Resolved Text Evaluation with Masked Diffusion Language Models

Authors: Yiqi Liu, Kun Zhao, Chenghua Lin, Nafise Sadat Moosavi

Organizations: The University of Manchester · University of Pittsburgh · The University of Sheffield

Abstract

A single evaluation score can rank generation systems but rarely reveals what failed. We propose rate-resolved text evaluation, which turns the masking rate of masked diffusion language models into a diagnostic axis. By reconstructing text under different fractions of masked tokens, one diffusion score becomes a vector of band-conditioned scores that reflect reconstruction difficulty at varying levels of visible context. On SummEval, preserving this rate structure yields complementary signal: the best masking-rate band improves fusion with autoregressive likelihood over a collapsed diffusion scalar, survives a sample-shuffle control that breaks rate ordering, and adds residual signal to a fixed prompted judge. A second diffusion model reproduces the effect more weakly. On WMT MQM zh-en, roundtrip mask-and-reconstruct probing produces a tunable localization curve in which ranking-oriented AUROC and span-detection F1 peak at different masking rates. Factuality benchmarks bound the interpretation: band semantics are model- and task-dependent, so each application requires empirical validation.

Figures & tables

Figure 1: Instead of reducing text evaluation to a single scalar score, masked diffusion reconstructs text under different masking rates and yields a band-conditioned score vector

\mathbf{b}(x)

. Lower masking rates retain more visible context, while higher masking rates require reconstruction from sparser evidence. The resulting masking-rate structure exposes complementary evaluation behavior beyond scalar evaluation.

Role	Model
Main diffusion	LLaDA-8B-Instruct
Replication diffusion	Dream-7B-Instruct
AR / judge	Qwen2.5-7B-Instruct
Static cloze control	XLM-R-XXL

Table 1: Model roles in the main experiments. Exact checkpoints and inference details are in Appendix A.3 .

Figure 2: SummEval relevance complementarity. Left: LLaDA masking-rate reconstruction bands remain complementary even to a strong prompted Qwen judge, with judge+AR+diffusion achieving the strongest relevance correlation. Right: for both LLaDA and Dream, combining AR likelihood with the best masking-rate band consistently outperforms either signal alone, indicating that evaluative signal is carried by reconstruction behavior across masking regimes rather than by a single scalar diffusion score.

Evaluator	SummEval relevance $\rho$
Qwen judge	0.3065
Judge + AR	0.3273
Judge + diffusion	0.3454
Judge + AR + diffusion	0.3636

Table 2: Fixed judge fusion on SummEval relevance. Diffusion bands add residual signal to the same prompted judge, and the strongest grouped-CV result uses judge, AR, and diffusion together.

Figure 3: SummEval masking-rate profiles for relevance and consistency. Both diffusion evaluators concentrate their strongest evaluative signal in high masking-rate bands, although the peak location and magnitude differ across models. Larger values indicate stronger alignment between diffusion reconstruction loss and human quality judgments.

Figure 4: Masking-rate-dependent localization behavior on WMT MQM zh-en. Different masking regimes favor different localization objectives: intermediate masking rates produce the strongest ranking behavior (AUROC), while lower masking rates favor broader thresholded error detection (F1). Dashed horizontal lines show AR token-surprisal baselines for the same severity slice.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

$s$	Med. rel. err.	IQR	Emp./theory CV	Rank $\rho$
0.1	0.0246	0.0317	0.344 / 0.338	0.9969
0.3	0.0207	0.0133	0.170 / 0.172	0.9986
0.5	0.0167	0.0078	0.112 / 0.113	0.9996
0.7	0.0102	0.0053	0.073 / 0.074	0.9994
0.9	0.0030	0.0016	0.037 / 0.038	0.9997

Appendix

Table 3: Empirical covariance probe for Eq. (11) on a 90-document SummEval subset (21 to 147 tokens). Relative error is the signed covariance term divided by the leading approximation

sN\ell(s,x)

. The empirical mask-count coefficient of variation closely matches the Binomial prediction, and document-level rankings induced by the exact and approximate quantities are nearly identical.

A. No-judge signals
Signal	Relevance $\rho$	Gain
AR likelihood	0.2106	n/a
Diffusion best band	0.2209	n/a
AR + full diffusion scalar	0.2610	ref.
AR + best diffusion band	0.2795	+0.0185

Appendix

Table 4: SummEval relevance complementarity under the fixed open-weight setting. Panel A tests whether preserving the diffusion rate axis helps beyond a collapsed diffusion scalar; Panel B tests whether diffusion adds residual signal to the same fixed prompted judge. Published SummEval scalar references from Liu et al. (2023) range from

0.312

0.547

and are used only to calibrate scale.

Item	Reproducibility detail
Code and scripts	The paper source package contains the LaTeX sources, rendered figures, and rendered tables. Experimental code and configuration files will be released after de-anonymization.
Core models	LLaDA uses GSAI-ML/LLaDA-8B-Instruct ; Dream uses Dream-org/Dream-v0-Instruct-7B ; AR and the fixed judge use Qwen/Qwen2.5-7B-Instruct ; the static cloze baseline uses facebook/xlm-roberta-xxl .
Pinned model refs	The local Qwen cache resolves main to a09a35458c702b33eeacc393d103063234e8bc28 ; diffusion checkpoints are recorded by release name in the current artifact and should be archived with commit hashes before de-anonymized release.
Environment	Python 3.11 with package lower bounds torch>=2.1 , transformers>=4.46 , accelerate>=0.34 , datasets>=2.18 , scipy>=1.11 , and pandas>=2.1 ; LLaDA and Dream configs record Transformers 4.46.3 and 4.46.2 respectively.
Benchmarks and licenses	SummEval is distributed with MIT licenses in the local artifact; WMT MQM is distributed under Apache-2.0. All are used through their released benchmark files and cited in the main text.
WMT preprocessing	The loader keeps hyp.* systems from newstest2021 zh-en, groups rows by system/document/segment, strips MQM markup, projects spans to character-level masks, derives severity slices from released segment-level counts, and aggregates rater penalties with the released severity weights.

Appendix

Table 5: Reproducibility summary for the main experiments. The table separates what is already fixed in the anonymous artifact from what should be pinned again before public release, especially diffusion checkpoint commit hashes.

Model	Dim.	Band	AR	Band $\rho$	AR+band	Gain
LLaDA	Rel.	8	0.2106	0.2209	0.2857	+0.0648
	Cons.	7	0.1022	0.1592	0.1849	+0.0257
	Flu.	0	0.1699	0.1709	0.1887	+0.0177
	Coh.	1	0.1695	0.0869	0.1695	+0.0000
Dream	Rel.	9	0.2106	0.2024	0.2638	+0.0532
	Cons.	8	0.1022	0.1239	0.1465	+0.0226

Appendix

Table 6: Full-data SummEval per-dimension AR and diffusion-band complementarity. Band correlations are sign-aligned Spearman correlations for the selected diffusion band. Gain is AR+band minus the stronger single signal. This table supplies the detailed relevance, consistency, fluency, and coherence breakdown summarized in Section 5.1.

Model	Dim.	AR	AR+scalar	AR+band	$\Delta$	$p$
LLaDA	Rel.	0.2106	0.2610	0.2795	+0.0185	.0015
	Cons.	0.1022	0.1428	0.1705	+0.0276	.0085
	Flu.	0.1699	0.1692	0.1852	+0.0160	.0895
	Coh.	0.1695	0.1590	0.1530	-0.0059	.9675
Dream	Rel.	0.2106	0.2524	0.2517	-0.0006	.6577
	Cons.	0.1022	0.1361	0.1423	+0.0062	.0055

Appendix

Table 7: SummEval scalarization controls. AR+scalar fuses AR likelihood with the full diffusion scalar from the same diffusion model, while AR+band fuses AR with the best masking-rate band selected within grouped-CV folds.

\Delta

is AR+band minus AR+scalar.

Budget	Runtime	Band $\rho$	Fusion $\rho$	Gain
$5\times 5$	3h43m	0.2263	0.2837	0.0574
$10\times 10$	9h06m	0.2204	0.2786	0.0582
$20\times 20$	2d23h21m	0.2209	0.2795	0.0586

Appendix

Table 8: Monte Carlo budget sensitivity on SummEval relevance. The

20\times 20

runtime is the original high-fidelity run; lower-budget runs were executed separately and preserve the same fusion gain.

Bands	Band $\rho$	Fusion $\rho$	Gain	Selected region
5	0.1668	0.2521	0.0414	mostly band 4
10	0.2209	0.2795	0.0586	band 8 in all folds
20	0.1799	0.2778	0.0672	bands 16 and 17

Appendix

Table 9: Band-boundary sensitivity for SummEval relevance. The high-mask region remains selected under 5, 10, and 20 equal-width partitions.

Signal structure	AR fusion $\rho$	Notes
Observed rate bands	0.2795	grouped-CV result
Full diffusion scalar	0.2610	scalarized LLaDA
Shuffled pseudo-bands	0.2098	mean over 100 shuffles
Best shuffled run	0.2431	maximum over 100 shuffles

Appendix

Table 10: Sample-shuffle control for SummEval relevance. Shuffled pseudo-bands preserve the number of reconstruction samples while removing the masking-rate ordering; the 95% shuffle range for AR+pseudo-band fusion is

[0.1911,0.2362]

Metric	Diffusion best	AR	Winner
Overall AUROC	0.6705	0.6330	Diff.
Pooled F1	0.2311	0.2636	AR
Major F1	0.3464	0.2811	Diff.
Minor F1	0.2852	0.2115	Diff.

Appendix

Table 11: WMT localization comparison at the reported operating points. Results are mixed: diffusion leads on ranking AUROC and severity-specific F1, while AR leads on pooled fixed-threshold F1.

Slice	Diff. .3	Diff. .5	AR	XLM-R
Err.	0.3251	0.3246	0.2987	0.2874
Major	0.3385	0.3381	0.3116	0.3018
Minor	0.2832	0.2838	0.2602	0.2443

Appendix

Table 12: Cross-fitted macro-F1 under threshold sensitivity analysis on WMT MQM. Diff. .3 and Diff. .5 denote diffusion at

s=0.3

and

s=0.5

; Err. denotes error-containing segments. Threshold selection leaves the comparison with AR and the bidirectional cloze baseline unchanged.

Pos.	Rec.	Diff. AUC	AR AUC	Diff. F1	AR F1
Beginning	1,215	0.6587	0.6053	0.2763	0.2344
Middle	3,168	0.6365	0.5579	0.3582	0.2437
End	1,504	0.6478	0.5263	0.2809	0.1652

Appendix

Table 13: WMT MQM error-position stress test. Diffusion AUROC uses roundtrip scores at

s=0.5

; diffusion F1 uses

s=0.3

. AR F1 uses the same per-segment top-20% token-surprisal rule as the AR token baseline.

Slice	Failure mode and excerpt
Judge + AR w/o Diff. rank err. $\downarrow$ J AR Diff. 73.5 29.0 1329.0	Long-range ordering sensitivity Eight Iranian border guards have been killed in clashes with militants near the border with Pakistan. A militant group called Jaish al Adal claimed responsibility for the attack.
Judge + Diff. w/o AR rank err. $\downarrow$ J AR Diff. 15.0 1513.0 3.0	Local fluency trap under repetition Liverpool beat Newcastle United 2 to 0. “I thought it was a penalty,” Allen said. “I thought it was a penalty. I thought it was a penalty.”
AR + Diff. w/o Judge rank err. $\downarrow$ J AR Diff. 1425.5 32.5 181.5	Possible judge preference mismatch for terse factual summaries Wales’ crunch Euro 2016 qualifier with Belgium has been declared a 33,000 sell-out. The top two sides in Group B meet in Cardiff on June 12, with Wales in their best position to qualify for a major tournament since 1958.

Appendix

Table 14: Qualitative SummEval relevance slices. Each row shows a case where one evaluator disagrees with the other two; rank errors are measured against the human ranking, with lower values better.

Probe	$s=0.1$	$s=0.5$	$s=0.9$
Token surface F1 (linear)	0.830	0.611	0.244
Major-error AUROC (linear)	0.659	0.652	0.523
Major-error F1 (linear)	0.567	0.546	0.479
Token surface F1 (MLP)	0.849	0.638	0.331
Major-error AUROC (MLP)	0.737	0.691	0.568
Major-error F1 (MLP)	0.685	0.592	0.570

Appendix

Table 15: Hidden-state probing sanity check on WMT MQM. Low masking preserves more decodable local token-surface information. Nonlinear probes partially recover major-error signal at all mask rates, but the low-to-high decodability ordering remains intact, so this table is best read as an ordering-robustness check without claiming a white-box proof of a strict information-resolution law.

Model	Maj. AUC	Maj. F1	Min. AUC	Min. F1
LLaDA	0.6774 @ .5	0.3464 @ .3	0.6506 @ .5	0.2852 @ .3
Dream	0.5871 @ .7	0.3134 @ .9	0.5418 @ .7	0.2597 @ .9

Appendix

Table 16: Dream replication on WMT MQM zh-en. Values report each model’s best masking rate for the metric. Dream is weaker and shifts toward higher masking rates.

Language Model Generation Evaluation

Language model generation evaluation measures the quality, reliability, diversity, and task fit of outputs produced by language models.

See all 395 papers

DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking
Mar 2, 2026Gilad Turok, Chris De Sa, Volodymyr KuleshovMasked Diffusion Models Diffusion Language Models
Department of Computer Science, Cornell University, New York, New York, USA.
NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models
Jun 4, 2026Andrey Fomenko, Maksim Kryzhanovskiy, Svetlana Glazyrina +1Non-Autoregressive Text Generation Masked Diffusion Models
Lomonosov Moscow State University, Moscow, Russia · Institute for Artificial Intelligence, Lomonosov Moscow State University, Moscow, Russia

May 12, 2026J/K move · Enter open · S save

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

Wen Lai, Yingli Shen, Dingnan Jin, Qing Cui+3

1Ant Group · 2Tsinghua University · 3Technical University of Munich

LLM Evaluation Language Model Generation Evaluation Automated Evaluation Diffusion Language Models Diffusion Language Model Benchmarking

Open paper AskPDF arXiv

Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.

Abstract

Language Model Generation Evaluation

Explore similar work

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking

NAVIRA: Decoupled Stochastic Remasking for Masked Diffusion Language Models