Estimating Surprisal from the Speech Signal

Zachary Hopton; Samuel Kiegeland; Francesco Ignazio Re; Patrick Haller; Matthew B. Winn; Ryan Cotterell; Eleanor Chodroff

cs.CLOct 13, 2026

Estimating Surprisal from the Speech Signal

Authors: Zachary Hopton, Samuel Kiegeland, Francesco Ignazio Re, Patrick Haller, Matthew B. Winn, Ryan Cotterell, Eleanor Chodroff

Organizations: University of Zurich · ETH Zürich · CHI-FRO · University of Minnesota

Abstract

Although language is predominantly spoken, empirical tests of surprisal theory rely almost entirely on written text. In practice, surprisal is estimated using language models (LMs) trained on text and applied to transcriptions, creating a text bottleneck that limits evaluation of the theory in the spoken modality. We remove this bottleneck by estimating surprisal directly from the speech signal using a speech LM. This raises a key question: should surprisal be defined over realization-specific acoustic inputs or over abstract linguistic units? Since a single unit can have many distinct acoustic realizations, each assigned a different probability by the model, defining surprisal over abstract units is difficult. To address this, we introduce an importance sampling estimator that approximates the probability of an abstract unit by marginalizing over realizations generated by a text-to-speech model. Using pupillometry data from a listening study, we evaluate surprisal theory in this text-free setting at multiple levels of granularity. Obtaining surprisal from several speech LMs, we consistently find that surprisal estimated via importance sampling better predicts pupil dilation than realization-specific surprisal for words. We also find that surprisal values for sub-lexical units are predictive of pupil dilation. While we find that surprisal can be estimated without recourse to text, and that marginalizing over distinct realizations of words is more predictive of lexical processing effort, surprisal as a whole is not a strong predictor of change in pupil size, regardless of its source.

Figures & tables

Figure 1: Pupil size trajectory and GSLM unit surprisals for participant NH–24 while listening to stimulus 5x39, “Tom has been discussing the beads.” Dotted lines represent estimated marginalized surprisals; solid lines represent unmarginalized surprisals for this specific utterance.

Figure 2: Distributions of cosine distances between speaker embeddings for the within-speaker TIMIT samples, between-speaker TIMIT samples, and the samples generated by CSM for the present study. Dotted vertical lines represent mean cosine distances.

Figure 3: Held-out delta log-likelihood relative to baseline model of pupil size (offset:

\omega=1000

ms). Each dot represents 1% of the bootstrap distribution of the mean delta log-likelihood; the point and bars below give the bootstrap mean and 95% CI. Stars correspond to significant corrected

p

-values of a model compared to the no-surprisal baseline at

\alpha

levels of 0.05 (), 0.01 (), and 0.001 (); ns = not significant.

Figure 4: Held-out delta log-likelihood relative to baseline model of pupil size at various offset times. Points give the mean delta log-likelihood and bars give 95% CI. Values at 1000 ms reflect those in Fig. 3 . See App. E for significance of delta log-likelihood values over the baseline model.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Variance of the IS estimator estimated via nonparametric bootstrap (1000 resamples per subsample size). Target speech LM is Llama-Mimi 1.3B. Each point shows the bootstrap variance for a given word at a given subsample size, and lines connect the same word across sample sizes. The top 2% of variances are excluded for visualization.

Condition	Coverage	$N$ obs.	Train	Test
Word-level
GPT-2	–	$29{,}150$	$26{,}235$	$2{,}915$
GSLM (Unmarg.)	–	$19{,}418$	$17{,}476$	$1{,}942$
GSLM (Marg.)	$959/1{,}229$ (78.0%)	$19{,}418$	$17{,}476$	$1{,}942$
SpiRit (Unmarg.)	–	$10{,}841$	$9{,}757$	$1{,}084$
SpiRit (Marg.)	$809/1{,}229$ (65.8%)	$10{,}841$	$9{,}757$	$1{,}084$

Appendix

Table 1: Per-GAMM data summary. Coverage is reported only for the marginalized word estimators, and refers to the number of corpus tokens for which the estimator produced an estimate. It is omitted (–) elsewhere, where every token is scored by construction.

N

obs. is the number of pupil samples entering the model, with the mean per-fold train/test split under 10-fold CV (at

\omega=1000

ms). For each speech LM,

N

obs. also reflects that the word with the largest marginalized surprisal estimate is excluded as an outlier, as the most extreme estimates are most affected by error in importance sampling (see App. B ). For Llama-Mimi, for example, the largest estimate was 484 nats versus 266 nats at the next-largest value. Marg. and Unmarg. word-level models of a given speech LM are fit on the same folds, so their

N

and fold sizes are identical.

Figure 6: Held-out delta log-likelihood predicting change in pupil size at three offset values

\omega

. Models are fit with word-level surprisal values obtained from GPT-2 using transcriptions of the speech heard by participants. Stars correspond to significant improvements over the no-surprisal baseline; corrected

p

-values at

\alpha

levels of 0.05 (), 0.01 (), and 0.001 ().

Condition	900 ms	1000 ms	1300 ms
GPT-2	$0.0035$ $[0.0020,\,0.0049]$ **	$0.0031$ $[0.0017,\,0.0044]$ **	$0.0019$ $[0.0007,\,0.0030]$ *
GSLM Marg.	$0.0045$ $[0.0031,\,0.0060]$ *	$0.0041$ $[0.0028,\,0.0054]$ *	$0.0021$ $[0.0011,\,0.0030]$ *
GSLM Unmarg.	$-0.0005$ $[-0.0016,\,0.0006]$ n.s.	$-0.0004$ $[-0.0014,\,0.0006]$ n.s.	$-0.0003$ $[-0.0010,\,0.0004]$ n.s.
SpiRit Marg.	$0.0024$ $[0.0009,\,0.0038]$ *	$0.0018$ $[0.0006,\,0.0030]$	$0.0015$ $[0.0005,\,0.0025]$ *
SpiRit Unmarg.	$-0.0034$ $[-0.0050,\,-0.0019]$ n.s.	$-0.0031$ $[-0.0044,\,-0.0018]$ n.s.	$-0.0029$ $[-0.0040,\,-0.0018]$ n.s.
Llama-Mimi Marg.	$0.0021$ $[0.0007,\,0.0035]$ *	$0.0018$ $[0.0004,\,0.0031]$	$0.0014$ $[0.0003,\,0.0025]$

Appendix

Table 2: Mean and 95% confidence intervals for delta log-likelihood predicting change in pupil size at three offset values

\omega

. Models are fit with word-level surprisal values obtained from four speech LMs and GPT-2. Stars correspond to significant improvements over the no-surprisal baseline; corrected

p

-values at

\alpha

levels of 0.05 (), 0.01 (), and 0.001 ().

Condition	900 ms	1000 ms	1300 ms
GSLM Unmarg.	$0.0018$ $[0.0011,\,0.0025]$ *	$0.0016$ $[0.0010,\,0.0023]$ *	$0.0011$ $[0.0007,\,0.0016]$ *
SpiRit Unmarg.	$0.0025$ $[0.0017,\,0.0033]$ *	$0.0024$ $[0.0016,\,0.0031]$ *	$0.0017$ $[0.0010,\,0.0023]$ *
Llama-Mimi Unmarg.	$0.0021$ $[0.0013,\,0.0030]$ *	$0.0018$ $[0.0011,\,0.0026]$ *	$0.0012$ $[0.0006,\,0.0018]$ *
Llama-Mimi-8B Unmarg.	$0.0023$ $[0.0014,\,0.0032]$ *	$0.0020$ $[0.0012,\,0.0028]$ *	$0.0012$ $[0.0006,\,0.0018]$ *

Appendix

Table 3: Mean and 95% confidence intervals for delta log-likelihood predicting change in pupil size at three offset values

\omega

. Models are fit with phone-level surprisal values obtained from four speech LMs. Stars correspond to significant improvements over the no-surprisal baseline; corrected

p

-values at

\alpha

levels of 0.05 (), 0.01 (), and 0.001 ().

Condition	900 ms	1000 ms	1300 ms
GSLM Unmarg.	$0.0019$ $[0.0007,\,0.0031]$ *	$0.0009$ $[0.0001,\,0.0017]$ n.s.	$0.0010$ $[0.0001,\,0.0018]$ n.s.
SpiRit Unmarg.	$0.0027$ $[0.0018,\,0.0037]$ *	$0.0027$ $[0.0017,\,0.0036]$ *	$0.0014$ $[0.0007,\,0.0021]$ *
Llama-Mimi Unmarg.	$0.0035$ $[0.0025,\,0.0044]$ *	$0.0030$ $[0.0021,\,0.0039]$ *	$0.0018$ $[0.0010,\,0.0025]$ *
Llama-Mimi-8B Unmarg.	$0.0023$ $[0.0015,\,0.0032]$ *	$0.0020$ $[0.0011,\,0.0028]$ *	$0.0011$ $[0.0004,\,0.0018]$

Appendix

Table 4: Mean and 95% confidence intervals for delta log-likelihood predicting change in pupil size at three offset values

\omega

. Models are fit with frame-level surprisal values obtained from four speech LMs. Stars correspond to significant improvements over the no-surprisal baseline; corrected

p

-values at

\alpha

levels of 0.05 (), 0.01 (), and 0.001 ().

Contrast	900 ms	1000 ms	1300 ms
GSLM	$0.0050$ $[0.0032,\,0.0068]$ $p<0.001$	$0.0045$ $[0.0029,\,0.0061]$ $p<0.001$	$0.0024$ $[0.0011,\,0.0036]$ $p<0.001$
Llama-Mimi	$0.0015$ $[0.0001,\,0.0030]$ $p=0.019$	$0.0012$ $[-0.0002,\,0.0026]$ $p=0.044$	$0.0014$ $[0.0003,\,0.0025]$ $p=0.007$
Llama-Mimi-8B	$0.0022$ $[0.0008,\,0.0035]$ $p=0.001$	$0.0019$ $[0.0006,\,0.0032]$ $p=0.002$	$0.0019$ $[0.0008,\,0.0029]$ $p<0.001$
SpiRit	$0.0058$ $[0.0038,\,0.0078]$ $p<0.001$	$0.0049$ $[0.0032,\,0.0066]$ $p<0.001$	$0.0044$ $[0.0031,\,0.0058]$ $p<0.001$

Appendix

Table 5: Paired-bootstrap tests for differences between marginalized and unmarginalized word surprisal GAMMs by model and time offset. Values represent the average difference in delta log-likelihood between GAMMs fit with marginalized and unmarginalized word surprisals derived from the same speech LMs. Positive values indicate higher delta log-likelihoods when fit with marginalized surprisals. Confidence intervals are 95% bootstrap intervals.

Figure 7: Held-out delta log-likelihood relative to baseline model of pupil size, augmented with a smooth for the time in the trial at which the pupil measurement was taken (offset:

\omega=1000

ms). Each dot represents 1% of the bootstrap distribution of the mean delta log-likelihood; the point and bars below give the bootstrap mean and 95% CI.

Surprisal Theory

See all 16 papers

Apr 30, 2026

On the Proper Treatment of Units in Surprisal Theory

Surprisal theory links human processing effort to the predictability of an upcoming linguistic unit, but empirical work often leaves the notion of a unit underspecified. In practice, experimental stimuli are segmented into linguistically motivated units (e.g., words), while pretrained language models assign probability mass to a fixed token alphabet that typically does not align with those units. As a result, surprisal-based predictors depend implicitly on ad hoc procedures that conflate two distinct modeling choices: the definition of the unit of analysis and the choice of regions of interest over which predictions are evaluated. In this paper, we disentangle these choices and give a unified framework for reasoning about surprisal over arbitrary unit inventories. We argue that surprisal-based analyses should make these choices explicit and treat tokenization as an implementation detail rather than a scientific primitive.

Samuel Kiegeland, Vésteinn Snæbjarnarson, Tim Vieira +1

ETH Zürich · CHI-FRO · University of Copenhagen

Surprisal Theory Computational Psycholinguistics

Ask PDF

Oct 13, 2026

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

Contextual and unigram surprisal are standard predictors of reading time in computational psycholinguistics. While contextual surprisal is typically derived from neural language models, unigram surprisal is usually obtained from independent corpus counts. Although both quantities are deterministic functionals of a single underlying distribution---capturing conditional and marginal structure respectively---this practice breaks their probabilistic link and may introduce estimation error from the mismatch. We propose three principles for estimating unigram surprisal in psycholinguistic research, including the requirement that both quantities be derived from the same language model. We operationalize this principle by introducing a Monte Carlo estimator of unigram surprisal from a language model, along with a marginalized variant that has lower variance empirically. When unigram surprisal is re-estimated under this unified framework, we find evidence that predictors change in their predictive contribution to reading time. We also confirm prior findings that frequency can be a stronger predictor of processing effort than contextual surprisal under this unified framework.

Zachary Hopton, Francesco Ignazio Re, Samuel Kiegeland +3

University of Zurich · ETH Zürich · CHI-FRO +1

Computational Psycholinguistics Surprisal Theory

Ask PDF

Jul 23, 2026

Surprisal Theory is Tautological (without Rational Grounding)

Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model. I argue this claim is a tautology without further constraint: for any non-negative difficulty measure over units in context, there exists a language model whose surprisal is an affine function of it under mild technical conditions. Therefore, because any pattern of difficulty is consistent with some language model, without an additional constraint on the language model, surprisal theory makes no falsifiable predictions. The tautology was long obscured by an assumption implicit in two decades of psycholinguistic work---that the relevant language model is the distribution that generated the training corpus, so that improving corpus fit improves predictions of human behavior. Recent empirical work has undermined this assumption, demonstrating that better corpus models can be worse predictors of processing difficulty. I conclude that breaking the tautology requires a rationalist intervention, i.e., the relevant language model must be derived from a non-empirically motivated model of the comprehender, which could be based on, for instance, memory constraints or processing goals, and that, thus, does not depend on the behavioral data surprisal theory is meant to explain.

Ryan Cotterell

ETH Zürich

Surprisal Theory Cognitive Modeling Computational Psycholinguistics

Ask PDF

Abstract

Surprisal Theory

Explore similar work

On the Proper Treatment of Units in Surprisal Theory

Revisiting the Estimation of Unigram Surprisal in Computational Psycholinguistics

Surprisal Theory is Tautological (without Rational Grounding)