If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary. Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus. On TinyStories, WikiText-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5--40%, far short of the 75% implied by 1/n scaling. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement. A source-split jackknife that uses only reference data reduces this calibration error by 57--59%. After this correction, a simpler smoothed-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger. Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.
Figures & tables
Role
Purpose
Auxiliary
Set the unigram p and reconstruction weights.
Reference
Estimate count variation and forecast disagreement.
A and B
Supply two independent count tables for testing the forecast.
Evaluation
Supply the same masked inputs to both predictors.
Table 1: Source-disjoint data roles. Auxiliary quantities are shared by A and B and held fixed throughout.
Forecast
m<n↓
Full grid ↓
Full source
0.469
0.188
Source jackknife
0.203
0.103
Multinomial
0.205
0.115
Multinomial jackknife
0.085
0.061
Source bootstrap
0.433
0.191
Bootstrap jackknife
0.175
0.060
Table 2: Calibration on the twenty-pair, six-bank study. Lower mean absolute log error is better. The primary subset contains the three m<n cells. The full grid contains all twelve (m,n) combinations per corpus. Jackknife rows apply correction at every cell, without the m≤n restriction. Appendix B.2 gives definitions and uncertainty intervals.
Figure 1: Analytic forecast calibration at n=4096 , with 64 tokens per chunk and q=0.5 . Each point divides the six-bank mean forecast by the arithmetic mean disagreement of 20 A/B pairs. Bars show 95% crossed bootstrap intervals over banks and pairs, conditional on auxiliary and evaluation data. The dotted line marks exact calibration, and lines connect the three tested reference sizes. Stars use an uncorrected 8,192-chunk reference.
TinyStories
WikiText
CNN/DailyMail
α
Source
Mult.
Source
Mult.
Source
Mult.
1
0.525
0.220
0.803
0.307
0.677
0.260
5
0.196
0.104
0.247
0.107
0.167
0.043
20
0.052
0.108
0.057
0.114
0.024
0.024
Table 3: Effect of α on calibration for small references ( m<n ). Both forecasts use source-split jackknife correction. Entries are mean absolute log errors ( ↓ ) on the same 20 pairs and six reference banks per corpus, with weights fixed at their α=5 values. Appendix E.3 gives the protocol and interpretation.
D↓
RelImprove (%) ↑
Corpus
L
N
L
N
TinyStories
0.0223
0.0504
14.34
20.82
WikiText-103
0.0308
0.0344
5.45
3.16
CNN/DailyMail
0.0248
0.0112
4.43
1.21
Table 4: Reconstruction at n=2048 chunks and mask rate q=0.5 for the local reconstructor (L) and MDLM after 12,000 updates (N). For each corpus, both methods are fitted to the A and B corpora of three independent pairs and evaluated on the same 128 masked chunks. D is the disagreement between the A and B fits, averaged over the three pairs; lower means more stable. RelImprove is the MSE improvement over the unigram, averaged over the six fitted models; higher is better.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Forecast calibration across reference and target sizes. Rows correspond to target size n and columns to corpora; sizes count 64-token chunks. Each point is the mean forecast across six reference banks divided by the mean observed disagreement across 20 A/B pairs. The dotted line marks exact calibration, with values below one indicating underprediction and values above one indicating overprediction. Error bars are 95% intervals obtained by resampling banks and A/B pairs independently, with auxiliary data, evaluation inputs, and masks fixed. Curves use reference sizes m∈{1024,2048,4096} ; stars show the uncorrected forecast at m=8192 .
Forecast
m<n
Full grid
m>n
Full source
0.469 [0.463, 0.475]
0.188 [0.185, 0.191]
0.052 [0.051, 0.053]
Source jackknife
0.203 [0.197, 0.211]
0.103 [0.101, 0.105]
0.095 [0.088, 0.100]
Multinomial
0.205 [0.199, 0.212]
0.115 [0.112, 0.119]
0.069 [0.065, 0.073]
Multinomial jackknife
0.085 [0.077, 0.093]
0.061 [0.057, 0.065]
0.049 [0.045, 0.053]
Source bootstrap
0.433 [0.427, 0.440]
0.191 [0.186, 0.195]
0.072 [0.067, 0.076]
Bootstrap jackknife
0.175 [0.167, 0.184]
0.060 [0.056, 0.064]
0.013 [0.011, 0.016]
Appendix
Table 5: Mean absolute log prediction error across reference banks, size combinations, and corpora, with 95% bootstrap intervals. Each forecast is compared with the mean disagreement of the same twenty A/B pairs, and every method uses the same six reference banks per corpus. Lower values indicate better calibration. The m<n column averages the three combinations in P ; the full grid averages all twelve combinations; and the m>n column averages the six combinations with a larger reference bank than target corpus. Banks, combinations, and corpora receive equal weight within each column. The last two rows apply the jackknife only when m≤n and use the uncorrected law otherwise. Intervals resample whole banks and A/B pairs jointly across methods and sizes, conditional on auxiliary data, evaluation inputs, masks, and saved Monte Carlo draws.
Forecast
m<n
Full grid
m>n
Full source
0.397 [0.385, 0.410]
0.207 [0.196, 0.217]
0.053 [0.043, 0.063]
Source jackknife
0.146 [0.133, 0.159]
0.066 [0.060, 0.073]
0.071 [0.059, 0.083]
Multinomial
0.187 [0.175, 0.199]
0.126 [0.118, 0.135]
0.080 [0.073, 0.088]
Multinomial jackknife
0.077 [0.066, 0.089]
0.060 [0.051, 0.069]
0.049 [0.041, 0.059]
Multinomial ×1.25
0.041 [0.031, 0.051]
0.098 [0.090, 0.106]
0.143 [0.135, 0.150]
Multinomial, pooled factor
0.076 [0.062, 0.090]
0.045 [0.039, 0.052]
0.039 [0.031, 0.047]
Appendix
Table 6: Calibration on six A/B pairs and three reference banks per corpus. Entries are mean absolute log prediction errors with 95% bootstrap intervals; lower values indicate better calibration. Sizes count 64-token chunks. The m<n column corresponds to (m,n)=(1024,2048) , the full grid averages all four combinations of m,n∈{1024,2048} , and the m>n column corresponds to (m,n)=(2048,1024) . Each forecast is compared with the mean disagreement across the six pairs, with banks, size combinations, and corpora weighted equally. Jackknife corrections are applied at every combination. The pooled and cross-corpus multipliers are selected using only the twenty-pair results. Intervals condition on auxiliary data, evaluation inputs, masks, and, where applicable, saved Monte Carlo draws.
Figure 3: Optimal multiplicative correction for the multinomial forecast at each reference–target size combination in the twenty-pair experiment. Each factor minimizes mean absolute log error across six reference banks, using the mean observed disagreement of twenty A/B pairs. Shapes identify target size n , and lines connect combinations with the same target size. Error bars are 95% intervals obtained by resampling banks and A/B pairs and selecting the factor again in each draw. The dashed line marks the fixed factor 1.25; the dotted line marks the pooled factor 1.118 selected across all corpora and size combinations. The factors shown here are selected using the observations in their respective combinations. Table 6 reports their pooled and cross-corpus calibration alternatives on disjoint source collections.
Figure 4: Corpus-size response on the matched task at q=0.5 . The upper row shows observed mean disagreement for MDLM and the reconstructor, together with reference forecasts for the reconstructor. Both forecasts use 2,048 reference chunks. The vertical axis is logarithmic, and bars span the three observed pair means. The lower row divides each curve by its own n=512 value, with a dotted 512/n benchmark starting at that value. Sizes count 64-token chunks, and MDLM curves include both training stages.
Quantity
Image diffusion
Text reconstructor
Trained MDLM
Data sampling
Images sampled from a distribution
Sources supplying token chunks
The same source-disjoint A/B corpora
Corruption
Additive Gaussian noise with variance σ2
Independent token masking with probability q
Absorbing token masks; evaluation at t=q=0.5
Dependence on data
Empirical mean μ and covariance Σ for the linear denoiser
Lag counts and conditional rows; auxiliary p and weights fixed
Transformer parameters learned from masked chunks
Predictor
Empirical linear denoiser Dσ(x)
Weighted conditional rows in ( 4 )
Time-conditioned Transformer probabilities
Population target
Output variation over independently sampled datasets
Dn,qlag over source collections
Dn,qMDLM conditional on shared training randomness
Shared input
Noisy image for denoising; initial noise for sampling
Identical text and mask
Identical text, mask, and diffusion time
Appendix
Table 7: Comparison of the image diffusion setting of Wang et al. (2026) with the two text predictors. Each setting measures variation across datasets while holding the evaluation input fixed. The image theory analyzes a linear denoiser and its sampling map; the text theory analyzes normalized lag counts and their reconstruction scores. MDLM provides a neural test of the count-based forecasts.
Corpus
Method
MSE ↓
MSE gain (%) ↑
Top-1 (%) ↑
Top-64 (%) ↑
TinyStories
Reconstructor
0.8450
14.34
28.12
75.80
TinyStories
MDLM, 8k
0.7855
20.37
33.14
78.75
TinyStories
MDLM, 12k
0.7811
20.82
33.88
79.15
WikiText-103
Reconstructor
0.9372
5.45
15.75
48.49
WikiText-103
MDLM, 8k
0.9603
3.12
12.09
44.66
WikiText-103
MDLM, 12k
0.9599
3.16
12.37
44.39
Appendix
Table 8: Reconstruction quality at n=2048,q=0.5 on the shared 128 evaluation chunks. Each row averages six models from three A/B pairs. MSE sums squared error over the 50,257 vocabulary coordinates, averages over hidden positions within each chunk, and then weights chunks equally. MSE gain is the percentage reduction relative to the auxiliary unigram. Top-1 and top-64 accuracy are also reported as percentages. Arrows indicate the preferred direction.
Figure 5: Top- k reconstruction quality at q=0.5 . The upper row compares the radius-two reconstructor at three corpus sizes with the auxiliary unigram. The lower row gives the relative reduction in top- k error against that same unigram. Colors and markers denote corpus size in both rows. Each reconstructor curve averages A and B within each pair, then averages the three pairs. Higher values indicate better performance in both rows.
Control
n
Source
Multinomial
Independent tokens
256
1.073 [1.062, 1.085]
1.069 [1.058, 1.081]
Independent tokens
1,024
1.035 [1.024, 1.045]
1.033 [1.022, 1.043]
Dependent tokens
256
1.092 [1.076, 1.108]
1.051 [1.036, 1.067]
Dependent tokens
1,024
1.046 [1.032, 1.060]
1.010 [0.997, 1.023]
Four chunks, α=5
256
1.117 [1.099, 1.135]
1.065 [1.048, 1.082]
Four chunks, α=5
1,024
1.027 [1.015, 1.040]
0.991 [0.979, 1.003]
Appendix
Table 9: Calibration using exact population moments. Entries are forecasts divided by mean nonlinear A/B disagreement; values closer to one indicate better agreement. Brackets give 95% intervals from 5,000 bootstrap resamples of the 80 A/B pairs, with forecasts, auxiliary quantities, and evaluation inputs fixed. The first six rows use α=5 . The smoothing comparisons use four chunks per source and the same source draws across values of α .
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.
Wen Lai, Yingli Shen, Dingnan Jin +4
1Ant Group · 2Tsinghua University · 3Technical University of Munich
A single evaluation score can rank generation systems but rarely reveals what failed. We propose rate-resolved text evaluation, which turns the masking rate of masked diffusion language models into a diagnostic axis. By reconstructing text under different fractions of masked tokens, one diffusion score becomes a vector of band-conditioned scores that reflect reconstruction difficulty at varying levels of visible context. On SummEval, preserving this rate structure yields complementary signal: the best masking-rate band improves fusion with autoregressive likelihood over a collapsed diffusion scalar, survives a sample-shuffle control that breaks rate ordering, and adds residual signal to a fixed prompted judge. A second diffusion model reproduces the effect more weakly. On WMT MQM zh-en, roundtrip mask-and-reconstruct probing produces a tunable localization curve in which ranking-oriented AUROC and span-detection F1 peak at different masking rates. Factuality benchmarks bound the interpretation: band semantics are model- and task-dependent, so each application requires empirical validation.
Yiqi Liu, Kun Zhao, Chenghua Lin +1
The University of Manchester · University of Pittsburgh · The University of Sheffield
Masked diffusion language models (MDLMs) can generate text efficiently by predicting multiple masked tokens in parallel, but predictions from the same forward pass are not necessarily reliable when committed together. We study when parallel commitment is reliable. Our diagnostics show that confidence alone does not determine a reliable commitment order: confident predictions near the end of the sequence can fix an answer before its supporting computations are established, and downstream predictions become less reliable as the uncertainty of their upstream context grows. At the same time, a single forward pass can already resolve several masked tokens, and predictions that remain stable across the final layers are more likely to be correct. Based on these findings, we propose Reliable Parallel Decoding (RPD), a training-free method that selects candidates by layerwise prediction stability and final confidence, and commits them under a cumulative entropy budget over their preceding masked positions. RPD defers predictions with uncertain upstream context while committing the remaining candidates in parallel, without relying on a fixed block schedule. Across mathematical reasoning and code generation benchmarks on LLaDA and Dream, RPD achieves the highest decoding throughput among the evaluated methods while maintaining or improving accuracy.
Zhenghao He, Bohan Liu, Guangzhi Xiong +1
Department of Computer Science, University of Virginia