Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction

Hare Krishna

cs.CLOct 9, 2026

Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction

Authors: Hare Krishna

Organizations: University of Texas at Austin

Abstract

If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary. Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus. On TinyStories, WikiText-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5--40%, far short of the 75% implied by $1/n$ scaling. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement. A source-split jackknife that uses only reference data reduces this calibration error by 57--59%. After this correction, a simpler smoothed-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger. Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.

Figures & tables

Role	Purpose
Auxiliary	Set the unigram $p$ and reconstruction weights.
Reference	Estimate count variation and forecast disagreement.
A and B	Supply two independent count tables for testing the forecast.
Evaluation	Supply the same masked inputs to both predictors.

Table 1: Source-disjoint data roles. Auxiliary quantities are shared by A and B and held fixed throughout.

Forecast	$m<n\ \downarrow$	Full grid $\downarrow$
Full source	0.469	0.188
Source jackknife	0.203	0.103
Multinomial	0.205	0.115
Multinomial jackknife	0.085	0.061
Source bootstrap	0.433	0.191
Bootstrap jackknife	0.175	0.060

Table 2: Calibration on the twenty-pair, six-bank study. Lower mean absolute log error is better. The primary subset contains the three

m<n

cells. The full grid contains all twelve

(m,n)

combinations per corpus. Jackknife rows apply correction at every cell, without the

m\leq n

restriction. Appendix B.2 gives definitions and uncertainty intervals.

Figure 1: Analytic forecast calibration at

n=4096

, with 64 tokens per chunk and

q=0.5

. Each point divides the six-bank mean forecast by the arithmetic mean disagreement of 20 A/B pairs. Bars show 95% crossed bootstrap intervals over banks and pairs, conditional on auxiliary and evaluation data. The dotted line marks exact calibration, and lines connect the three tested reference sizes. Stars use an uncorrected 8,192-chunk reference.

	TinyStories		WikiText		CNN/DailyMail
$\alpha$	Source	Mult.	Source	Mult.	Source	Mult.
1	0.525	0.220	0.803	0.307	0.677	0.260
5	0.196	0.104	0.247	0.107	0.167	0.043
20	0.052	0.108	0.057	0.114	0.024	0.024

Table 3: Effect of

\alpha

on calibration for small references (

m<n

). Both forecasts use source-split jackknife correction. Entries are mean absolute log errors (

\downarrow

) on the same 20 pairs and six reference banks per corpus, with weights fixed at their

\alpha=5

values. Appendix E.3 gives the protocol and interpretation.

	$D\downarrow$		RelImprove (%) $\uparrow$
Corpus	L	N	L	N
TinyStories	0.0223	0.0504	14.34	20.82
WikiText-103	0.0308	0.0344	5.45	3.16
CNN/DailyMail	0.0248	0.0112	4.43	1.21

Table 4: Reconstruction at

n=2048

chunks and mask rate

q=0.5

for the local reconstructor (L) and MDLM after 12,000 updates (N). For each corpus, both methods are fitted to the A and B corpora of three independent pairs and evaluated on the same 128 masked chunks.

D

is the disagreement between the A and B fits, averaged over the three pairs; lower means more stable. RelImprove is the MSE improvement over the unigram, averaged over the six fitted models; higher is better.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Forecast calibration across reference and target sizes. Rows correspond to target size

n

and columns to corpora; sizes count 64-token chunks. Each point is the mean forecast across six reference banks divided by the mean observed disagreement across 20 A/B pairs. The dotted line marks exact calibration, with values below one indicating underprediction and values above one indicating overprediction. Error bars are 95% intervals obtained by resampling banks and A/B pairs independently, with auxiliary data, evaluation inputs, and masks fixed. Curves use reference sizes

m\in\{1024,2048,4096\}

; stars show the uncorrected forecast at

m=8192

Forecast	$m<n$	Full grid	$m>n$
Full source	0.469 [0.463, 0.475]	0.188 [0.185, 0.191]	0.052 [0.051, 0.053]
Source jackknife	0.203 [0.197, 0.211]	0.103 [0.101, 0.105]	0.095 [0.088, 0.100]
Multinomial	0.205 [0.199, 0.212]	0.115 [0.112, 0.119]	0.069 [0.065, 0.073]
Multinomial jackknife	0.085 [0.077, 0.093]	0.061 [0.057, 0.065]	0.049 [0.045, 0.053]
Source bootstrap	0.433 [0.427, 0.440]	0.191 [0.186, 0.195]	0.072 [0.067, 0.076]
Bootstrap jackknife	0.175 [0.167, 0.184]	0.060 [0.056, 0.064]	0.013 [0.011, 0.016]

Appendix

Table 5: Mean absolute log prediction error across reference banks, size combinations, and corpora, with 95% bootstrap intervals. Each forecast is compared with the mean disagreement of the same twenty A/B pairs, and every method uses the same six reference banks per corpus. Lower values indicate better calibration. The

m<n

column averages the three combinations in

\mathcal{P}

; the full grid averages all twelve combinations; and the

m>n

column averages the six combinations with a larger reference bank than target corpus. Banks, combinations, and corpora receive equal weight within each column. The last two rows apply the jackknife only when

m\leq n

and use the uncorrected law otherwise. Intervals resample whole banks and A/B pairs jointly across methods and sizes, conditional on auxiliary data, evaluation inputs, masks, and saved Monte Carlo draws.

Forecast	$m<n$	Full grid	$m>n$
Full source	0.397 [0.385, 0.410]	0.207 [0.196, 0.217]	0.053 [0.043, 0.063]
Source jackknife	0.146 [0.133, 0.159]	0.066 [0.060, 0.073]	0.071 [0.059, 0.083]
Multinomial	0.187 [0.175, 0.199]	0.126 [0.118, 0.135]	0.080 [0.073, 0.088]
Multinomial jackknife	0.077 [0.066, 0.089]	0.060 [0.051, 0.069]	0.049 [0.041, 0.059]
Multinomial $\times 1.25$	0.041 [0.031, 0.051]	0.098 [0.090, 0.106]	0.143 [0.135, 0.150]
Multinomial, pooled factor	0.076 [0.062, 0.090]	0.045 [0.039, 0.052]	0.039 [0.031, 0.047]

Appendix

Table 6: Calibration on six A/B pairs and three reference banks per corpus. Entries are mean absolute log prediction errors with 95% bootstrap intervals; lower values indicate better calibration. Sizes count 64-token chunks. The

m<n

column corresponds to

(m,n)=(1024,2048)

, the full grid averages all four combinations of

m,n\in\{1024,2048\}

, and the

m>n

column corresponds to

(m,n)=(2048,1024)

. Each forecast is compared with the mean disagreement across the six pairs, with banks, size combinations, and corpora weighted equally. Jackknife corrections are applied at every combination. The pooled and cross-corpus multipliers are selected using only the twenty-pair results. Intervals condition on auxiliary data, evaluation inputs, masks, and, where applicable, saved Monte Carlo draws.

Figure 3: Optimal multiplicative correction for the multinomial forecast at each reference–target size combination in the twenty-pair experiment. Each factor minimizes mean absolute log error across six reference banks, using the mean observed disagreement of twenty A/B pairs. Shapes identify target size

n

, and lines connect combinations with the same target size. Error bars are 95% intervals obtained by resampling banks and A/B pairs and selecting the factor again in each draw. The dashed line marks the fixed factor 1.25; the dotted line marks the pooled factor 1.118 selected across all corpora and size combinations. The factors shown here are selected using the observations in their respective combinations. Table 6 reports their pooled and cross-corpus calibration alternatives on disjoint source collections.

Figure 4: Corpus-size response on the matched task at

q=0.5

. The upper row shows observed mean disagreement for MDLM and the reconstructor, together with reference forecasts for the reconstructor. Both forecasts use 2,048 reference chunks. The vertical axis is logarithmic, and bars span the three observed pair means. The lower row divides each curve by its own

n=512

value, with a dotted

512/n

benchmark starting at that value. Sizes count 64-token chunks, and MDLM curves include both training stages.

Quantity	Image diffusion	Text reconstructor	Trained MDLM
Data sampling	Images sampled from a distribution	Sources supplying token chunks	The same source-disjoint A/B corpora
Corruption	Additive Gaussian noise with variance $\sigma^{2}$	Independent token masking with probability $q$	Absorbing token masks; evaluation at $t=q=0.5$
Dependence on data	Empirical mean $\widehat{\mu}$ and covariance $\widehat{\Sigma}$ for the linear denoiser	Lag counts and conditional rows; auxiliary $p$ and weights fixed	Transformer parameters learned from masked chunks
Predictor	Empirical linear denoiser $\widehat{D}_{\sigma}(x)$	Weighted conditional rows in ( 4 )	Time-conditioned Transformer probabilities
Population target	Output variation over independently sampled datasets	$\mathfrak{D}^{\rm lag}_{n,q}$ over source collections	$\mathfrak{D}^{\rm MDLM}_{n,q}$ conditional on shared training randomness
Shared input	Noisy image for denoising; initial noise for sampling	Identical text and mask	Identical text, mask, and diffusion time

Appendix

Table 7: Comparison of the image diffusion setting of Wang et al. (2026) with the two text predictors. Each setting measures variation across datasets while holding the evaluation input fixed. The image theory analyzes a linear denoiser and its sampling map; the text theory analyzes normalized lag counts and their reconstruction scores. MDLM provides a neural test of the count-based forecasts.

Corpus	Method	MSE $\downarrow$	MSE gain (%) $\uparrow$	Top-1 (%) $\uparrow$	Top-64 (%) $\uparrow$
TinyStories	Reconstructor	0.8450	14.34	28.12	75.80
TinyStories	MDLM, 8k	0.7855	20.37	33.14	78.75
TinyStories	MDLM, 12k	0.7811	20.82	33.88	79.15
WikiText-103	Reconstructor	0.9372	5.45	15.75	48.49
WikiText-103	MDLM, 8k	0.9603	3.12	12.09	44.66
WikiText-103	MDLM, 12k	0.9599	3.16	12.37	44.39

Appendix

Table 8: Reconstruction quality at

n=2048,q=0.5

on the shared 128 evaluation chunks. Each row averages six models from three A/B pairs. MSE sums squared error over the 50,257 vocabulary coordinates, averages over hidden positions within each chunk, and then weights chunks equally. MSE gain is the percentage reduction relative to the auxiliary unigram. Top-1 and top-64 accuracy are also reported as percentages. Arrows indicate the preferred direction.

Figure 5: Top-

k

reconstruction quality at

q=0.5

. The upper row compares the radius-two reconstructor at three corpus sizes with the auxiliary unigram. The lower row gives the relative reduction in top-

k

error against that same unigram. Colors and markers denote corpus size in both rows. Each reconstructor curve averages A and B within each pair, then averages the three pairs. Higher values indicate better performance in both rows.

Control	$n$	Source	Multinomial
Independent tokens	256	1.073 [1.062, 1.085]	1.069 [1.058, 1.081]
Independent tokens	1,024	1.035 [1.024, 1.045]	1.033 [1.022, 1.043]
Dependent tokens	256	1.092 [1.076, 1.108]	1.051 [1.036, 1.067]
Dependent tokens	1,024	1.046 [1.032, 1.060]	1.010 [0.997, 1.023]
Four chunks, $\alpha=5$	256	1.117 [1.099, 1.135]	1.065 [1.048, 1.082]
Four chunks, $\alpha=5$	1,024	1.027 [1.015, 1.040]	0.991 [0.979, 1.003]

Appendix

Table 9: Calibration using exact population moments. Entries are forecasts divided by mean nonlinear A/B disagreement; values closer to one indicate better agreement. Brackets give 95% intervals from 5,000 bootstrap resamples of the 80 A/B pairs, with forecasts, auxiliary quantities, and evaluation inputs fixed. The first six rows use

\alpha=5

. The smoothing comparisons use four chunks per source and the same source draws across values of

\alpha

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 802 papers

DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
Rate-Resolved Text Evaluation with Masked Diffusion Language Models
Oct 11, 2026Yiqi Liu, Kun Zhao, Chenghua Lin +1Language Model Generation Evaluation Diffusion Language Models
The University of Manchester · University of Pittsburgh · The University of Sheffield
Reliable Parallel Decoding in Masked Diffusion Language Models
Sep 29, 2026Zhenghao He, Bohan Liu, Guangzhi Xiong +1Masked Diffusion Models Diffusion Language Model Decoding
Department of Computer Science, University of Virginia

May 12, 2026J/K move · Enter open · S save

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

Wen Lai, Yingli Shen, Dingnan Jin, Qing Cui+3

1Ant Group · 2Tsinghua University · 3Technical University of Munich

LLM Evaluation Language Model Generation Evaluation Automated Evaluation Diffusion Language Models Diffusion Language Model Benchmarking

Open paper AskPDF arXiv

Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional PMI decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings. The code is released at: https://github.com/wenlai-lavine/DiffScore.

Abstract

LLM Reliability

Explore similar work

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

Rate-Resolved Text Evaluation with Masked Diffusion Language Models

Reliable Parallel Decoding in Masked Diffusion Language Models