Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

Zachary Hopton; Francesco Ignazio Re; Samuel Kiegeland; Andreas Opedal; Eleanor Chodroff; Ryan Cotterell

cs.CLOct 13, 2026

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

Authors: Zachary Hopton, Francesco Ignazio Re, Samuel Kiegeland, Andreas Opedal, Eleanor Chodroff, Ryan Cotterell

Organizations: University of Zurich · ETH Zürich · CHI-FRO · MPI for Intelligent Systems, Tübingen

Abstract

Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a single underlying distribution---capturing conditional and marginal structure respectively---this practice breaks their probabilistic link and may introduce estimation error from the mismatch. We propose three principles for estimating unigram surprisal in psycholinguistic research, including the requirement that both quantities be derived from the same language model. We operationalize this principle by introducing a Monte Carlo estimator of unigram surprisal from a language model, along with a marginalized variant that has lower variance empirically. When unigram surprisal is re-estimated under this unified framework, we find evidence that predictors change in their predictive contribution to reading time. We also confirm prior findings that frequency can be a stronger predictor of processing effort than contextual surprisal under this unified framework.

Figures & tables

Panel of Figure 2 — Figure 2: Principles for estimating unigram surprisal.

Figure 3: Comparison of

{{\color[rgb]{0,0,0}\Delta_{\text{LL}}}}

for various GAMMs of orthographic word gaze duration fit with unigram surprisals obtained with the wordfreq package ( Speer, 2022 ) or with one of our estimators using samples from mGPT. Values are averaged across languages. Dashed lines represent the

{{\color[rgb]{0,0,0}\Delta_{\text{LL}}}}

of adding contextual surprisal relative to each facet’s baseline for reference. Significance tests compare estimated unigram surprisal against the corpus-estimated values via paired permutation test on item-level

{{\color[rgb]{0,0,0}\Delta_{\text{LL}}}}

differences, with stars marking corrected significance at an

\alpha

level of 0.05. Error bars represent 95% confidence intervals.

$J$	MMC variance	NMC variance	Ratio
100	1.685e-08	1.736e-07	10.30
200	8.508e-09	9.689e-08	11.39
300	5.648e-09	6.769e-08	11.98
400	4.044e-09	5.448e-08	13.47
500	3.307e-09	4.281e-08	12.94

Table 1: Mean bootstrap variance of the Marginalized (MMC) and Naive (NMC) Monte Carlo unigram frequency estimators across subsample sizes

J

. Ratio = NMC / MMC.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Boundary	Location	Condition (satisfied when)
Left	first token of ${{{\color[rgb]{0,0,0}w}}}$	it carries a leading ␣ (word-initial marker Ġ ), or it begins the sequence
Right	token following the last token of ${{{\color[rgb]{0,0,0}w}}}$	it carries a leading ␣ (marker Ġ ), or it is eos

Appendix

Table 5

Figure 4: Visualization of the bootstrapped variance estimates of our marginalized and naive Monte Carlo estimators of unigram frequency. Each line represents a single unigram from the English MECO data. The “Mean” lines refer to the average variance across all unigrams.

Figure 5: Proportion of types and tokens in the MECO corpus for which estimates can be obtained from our Naive Monte Carlo estimator at a range of sample sizes. Samples are obtained from mGPT with the maximum length hyperparameter set to 50 tokens. Estimates cannot be obtained when the target type/token never appears in the

N

sampled strings.

Tokens	NMC (s)	MMC (s)
1	3.788 (0.003)	86.269 (0.101)
2	3.788 (0.003)	166.344 (0.161)
3	3.787 (0.002)	247.479 (0.105)
4	3.783 (0.008)	329.654 (0.153)

Appendix

Table 2: Mean (SD) wall-clock time per estimate for the Naive (NMC) and Marginalized (MMC) Monte Carlo estimators, broken down by the number of subword tokens in the target unigram (

N=500

samples, batch size 128).

Language	Participants	Word Types	Observations	Train Size	Test Size
Dutch	45	662	31680	28512	3168
English	46	930	55120	49608	5512
Finnish	49	306	14853	13368	1485
German	45	753	43942	39548	4394
Greek	45	945	39972	35975	3997
Hebrew	47	1303	49430	44487	4943

Appendix

Table 3: Per-language counts for the NMC estimator: observations entering the gaze-duration model are per participant. Train/test split refer to the size of the folds used to fit and test each of the GAMMs.

Language	Participants	Word Types	Observations	Train Size	Test Size
Dutch	45	996	44816	40334	4482
English	46	945	55792	50213	5579
Finnish	49	1042	49783	44805	4978
German	45	1037	58171	52354	5817
Greek	45	845	35558	32002	3556
Hebrew	47	954	36248	32623	3625

Appendix

Table 4: Per-language counts for the MMC estimator: observations entering the gaze-duration model are per participant. Train/test split refer to the size of the folds used to fit and test each of the GAMMs.

Language	Participants	Word Types	Observations	Train Size	Test Size
Dutch	45	1009	45314	40783	4531
English	46	980	57883	52095	5788
Finnish	49	1076	51962	46766	5196
German	45	1047	58861	52975	5886
Greek	45	975	41229	37106	4123
Hebrew	47	1319	50126	45113	5013

Appendix

Table 5: Per-language counts for the MMC (Approx.) estimator: observations entering the gaze-duration model are per participant. Train/test split refer to the size of the folds used to fit and test each of the GAMMs.

Language	Participants	Word Types	Observations	Train Size	Test Size
Dutch	45	907	41247	37122	4125
English	46	965	56924	51232	5692
Finnish	49	823	39298	35368	3930
German	45	950	53534	48181	5353
Greek	45	957	40455	36409	4046
Hebrew	47	1303	49454	44509	4945

Appendix

Table 6: Per-language counts for the Priority MMC estimator: observations entering the gaze-duration model are per participant. Train/test split refer to the size of the folds used to fit and test each of the GAMMs.

Language	Participants	Word Types	Observations	Train Size	Test Size
Dutch	45	1009	45314	40783	4531
English	46	980	57883	52095	5788
Finnish	49	1076	51962	46766	5196
German	45	1044	58707	52836	5871
Greek	45	975	41229	37106	4123
Hebrew	47	1319	50126	45113	5013

Appendix

Table 7: Per-language counts for the Priority MMC (Approx.) estimator: observations entering the gaze-duration model are per participant. Train/test split refer to the size of the folds used to fit and test each of the GAMMs.

Unigram Surprisal Metric	Model	Macro Averages	SD Across Languages
NMC Estimate	Uni. Surp.	0.0265	0.0114
	PMI	0.0071	0.0062
	PMI & Uni. Surp.	0.0286	0.0121
	Con. Surp.	0.0118	0.0049
	Con. Surp. & Uni. Surp.	0.0283	0.0119
Corpus Estimate	Uni. Surp.	0.0282	0.0126

Appendix

Table 8: Word-level

\Delta

LL for GAMMs of gaze duration, macro-averaged across languages. Values correspond to the NMC estimator versus the Corpus baseline (fit on NMC’s support).

Unigram Surprisal Metric	Model	Macro Averages	SD Across Languages
MMC Estimate	Uni. Surp.	0.0383	0.0195
	PMI	0.0122	0.0085
	PMI & Uni. Surp.	0.0415	0.0205
	Con. Surp.	0.0211	0.0100
	Con. Surp. & Uni. Surp.	0.0413	0.0204
Corpus Estimate	Uni. Surp.	0.0396	0.0211

Appendix

Table 9: Word-level

\Delta

LL for GAMMs of gaze duration, macro-averaged across languages. Values correspond to the MMC estimator versus the Corpus baseline (fit on MMC’s support).

Unigram Surprisal Metric	Model	Macro Averages	SD Across Languages
MMC (Approx.) Estimate	Uni. Surp.	0.0391	0.0178
	PMI	0.0130	0.0085
	PMI & Uni. Surp.	0.0424	0.0191
	Con. Surp.	0.0214	0.0105
	Con. Surp. & Uni. Surp.	0.0421	0.0191
Corpus Estimate	Uni. Surp.	0.0396	0.0211

Appendix

Table 10: Word-level

\Delta

LL for GAMMs of gaze duration, macro-averaged across languages. Values correspond to the MMC (Approx.) estimator versus the Corpus baseline (fit on MMC (Approx.)’s support).

Unigram Surprisal Metric	Model	Macro Averages	SD Across Languages
Priority MMC Estimate	Uni. Surp.	0.0392	0.0175
	PMI	0.0111	0.0078
	PMI & Uni. Surp.	0.0421	0.0184
	Con. Surp.	0.0211	0.0104
	Con. Surp. & Uni. Surp.	0.0416	0.0184
Corpus Estimate	Uni. Surp.	0.0387	0.0198

Appendix

Table 11: Word-level

\Delta

LL for GAMMs of gaze duration, macro-averaged across languages. Values correspond to the Priority MMC estimator versus the Corpus baseline (fit on Priority MMC’s support).

Unigram Surprisal Metric	Model	Macro Averages	SD Across Languages
Priority MMC (Approx.) Estimate	Uni. Surp.	0.0412	0.0201
	PMI	0.0110	0.0087
	PMI & Uni. Surp.	0.0436	0.0208
	Con. Surp.	0.0214	0.0105
	Con. Surp. & Uni. Surp.	0.0433	0.0207
Corpus Estimate	Uni. Surp.	0.0397	0.0211

Appendix

Table 12: Word-level

\Delta

LL for GAMMs of gaze duration, macro-averaged across languages: Values correspond to the Priority MMC (Approx.) estimator versus the Corpus baseline (fit on Priority MMC (Approx.)’s support).

	Approx.	Full
Samples ${{{\color[rgb]{0,0,0}N}}}$	$4\times 10^{7}$
Maximum length	$50$
Strata	$21$
Per-unit budget $k$	$8192$
Stratum floor	$64$
Home share	$4096$	$2048$

Appendix

Table 13: Settings of the priority-sampled variants (Priority MMC Approx. and Priority MMC). Each variant draws its own pool of samples. The full variant’s pool additionally stores the model’s own sampled tokenizations for

\widehat{H}

. Approximate-variant single-token units need no sampling, since their numerator is the exact column sum plus the exact empty-prefix term. The full estimator’s closing mass depends on the unit, so its single-token units are sampled like any other.

Computational Psycholinguistics

Computational psycholinguistics models how people learn, produce, and understand language using formal theories, algorithms, and behavioral data.

See all 86 papers

Estimating Surprisal from the Speech Signal
Oct 13, 2026Zachary Hopton, Samuel Kiegeland, Francesco Ignazio Re +4Surprisal Theory Computational Psycholinguistics
University of Zurich · ETH Zürich · CHI-FRO +1
A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese
Oct 4, 2026Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4Language Model Scaling Laws Language Modeling
Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3
From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish
Aug 23, 2026Francisco Portillo LópezSurprisal Theory Child Language Acquisition
Universidad de Navarra, Pamplona, Spain

Oct 13, 2026J/K move · Enter open · S save

Estimating Surprisal from the Speech Signal

Zachary Hopton, Samuel Kiegeland, Francesco Ignazio Re, Patrick Haller+3

University of Zurich · ETH Zürich · CHI-FRO+1

Surprisal Theory Computational Psycholinguistics Speech Language Models

Open paper AskPDF arXiv

Although language is predominantly spoken, empirical tests of surprisal theory rely almost entirely on written text. In practice, surprisal is estimated using language models (LMs) trained on text and applied to transcriptions, creating a text bottleneck that limits evaluation of the theory in the spoken modality. We remove this bottleneck by estimating surprisal directly from the speech signal using a speech LM. This raises a key question: should surprisal be defined over realization-specific acoustic inputs or over abstract linguistic units? Since a single unit can have many distinct acoustic realizations, each assigned a different probability by the model, defining surprisal over abstract units is difficult. To address this, we introduce an importance sampling estimator that approximates the probability of an abstract unit by marginalizing over realizations generated by a text-to-speech model. Using pupillometry data from a listening study, we evaluate surprisal theory in this text-free setting at multiple levels of granularity. Obtaining surprisal from several speech LMs, we consistently find that surprisal estimated via importance sampling better predicts pupil dilation than realization-specific surprisal for words. We also find that surprisal values for sub-lexical units are predictive of pupil dilation. While we find that surprisal can be estimated without recourse to text, and that marginalizing over distinct realizations of words is more predictive of lexical processing effort, surprisal as a whole is not a strong predictor of change in pupil size, regardless of its source.

Abstract

Computational Psycholinguistics

Explore similar work

Estimating Surprisal from the Speech Signal

A Systematic Analysis of the Predictive Power of LM Surprisal in Reading Chinese

From Exposure to Expectation: Frequency, Surprisal, and Language Across Development in Spanish