The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time
Authors: Donato Riccio, Fabrizio Maturo
Organizations: Department of Engineering and Science, Universitas Mercatorum, Rome, Italy · Department of Economics, Statistics and Business, Universitas Mercatorum, Rome, Italy
Surprisal from a language model's final layer predicts how long people read a word, yet the model computes its prediction across many layers. We read each word's prediction at every layer with the tuned lens and summarize this layerwise surprisal trajectory by its resolution depth, the depth around which its change is centered. On eight English eye-tracking and self-paced reading corpora (81,957 words) and ten GPT-2, OPT and Pythia models, words that a model resolves later, and the words after them, are read more slowly than equally surprising words. Resolution depth improves held-out prediction beyond final-layer surprisal, entropy and the best single layer by 0.26 millinats per word (pooled p = 0.001), about a third of entropy's gain, and each model family gains, GPT-2 and Pythia through one corpus each. The net gain varies between corpora (random-effects mean 0.07, p = 0.37); against layer-shuffled depth, which adds as many coefficients, it is positive in seven of eight corpora (random-effects mean 0.23, p = 0.016), and untrained networks show no gain under the logit lens. Pooled over corpora, the effect holds with corrected word probabilities and beyond first-to-last-layer divergences, and a whole-trajectory functional regression adds no significant information beyond resolution depth.
Figures & tables
Figure 1: Mean layerwise surprisal trajectories (tuned lens) of OneStop words in the middle fifth of final surprisal (40th to 60th percentile) under GPT-2 XL, split into the fifth with the lowest and the fifth with the highest resolution depth D (Natural Stories with Pythia 2.8B in Figure B a). Dotted lines mark each group’s mean D ; n is the number of analyzed words in the band. Both groups end at the same surprisal; the late-resolved words keep a high surprisal through more of the network.
Corpus
Material
Readers
Words
Analyzed
Units
Readers/word
Provo
55 passages
84
2,745
2,115
55 passages
49
GECO
one novel
14
54,362
33,624
586 screens
7
OneStop
30 articles, 2 versions
180
35,188
25,359
30 articles
14
MECO
12 texts
46
2,107
1,732
12 texts
18
ZuCo
300 sentences
12
6,588
4,711
300 sentences
9
UCL eye tracking
205 sentences
37
1,931
1,305
205 sentences
25
Table 1: Reading corpora: six eye-tracking and two self-paced reading corpora (Natural Stories, UCL self-paced); the two UCL corpora present the same sentences (205 in eye tracking) to different readers. Words are the displayed word tokens of each release; analyzed words enter the primary analysis (go-past time for eye tracking, reading time for self-paced reading) after exclusions. Units define the cross-validation folds and the units of inference (§ 5 ). Readers/word is the median number of readers contributing to an analyzed word. Total reading time analyzes 36,382 words in GECO, 26,824 in OneStop, 1,881 in MECO and 1,316 in UCL eye tracking, and the words of the primary measure elsewhere.
Figure 2: Held-out gain in log-likelihood from adding resolution depth D(w) and D(w−1) to the control set on the primary measure (go-past time for eye tracking, reading time for self-paced reading), pooled over the eight corpora, per model, per family, for the ten trained models and for the four untrained networks; ticks mark the median gain over five layer permutations. Intervals are 95% bootstrap intervals over text units drawn within corpora.
Figure 3: Held-out gain from resolution depth on the primary measure for each corpus and model, in millinats per word. Asterisks mark one-sided permutation p<0.05 for the corpus and model alone.
Analysis
Gain
p
Main test
0.257
0.001
D(w) alone
0.094
0.015
D(w−1) alone
0.174
0.002
From readout 1
0.188
0.006
Corrected word probabilities
0.185
0.009
Single-token words
0.171
0.012
Table 2: Robustness and alternative measures: pooled held-out gain (millinats per word) on the primary measure over eight corpora and ten models, with one-sided sign-flip p . “+” rows add the named variables of w and w−1 to the controls before testing D ; “alone” rows test the alternative measure in place of D , and “beyond” rows test D with the alternative in the controls.
Comparison
Gain
95% CI
p
D
0.257
0.10, 0.40
0.001
Trajectory
0.168
− 0.00, 0.34
0.047
Trajectory beyond D
−0.020
− 0.16, 0.12
0.61
D beyond trajectory
0.069
− 0.05, 0.19
0.15
Profile beyond D
−0.010
− 0.12, 0.09
0.57
Table 3: Functional regression on the trajectories of w and w−1 : pooled held-out gain (millinats per word) on the primary measure; “beyond” rows add one term to a model that contains the other (shuffled-layer rows in Table C ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
FFD
GD
GP
TRT
Provo
GPT-2 small
−0.99
−0.73
−0.07
−0.29
GPT-2 large
+0.61
+0.96
−0.42
−0.09
GPT-2 XL
+1.98∗
+1.07
+1.03
+0.85
OPT 125M
+0.72
+1.09
+0.72
−0.41
OPT 1.3B
−0.01
−0.20
−0.28
−0.47
Appendix
Table 4: Held-out gain (millinats per word) from adding D(w) and D(w−1) to the control set, for each corpus, trained model and measure: first fixation duration (FFD), gaze duration (GD), go-past time (GP) and total reading time (TRT). The self-paced reading corpora have one measure, the reading time, listed under GP, so the table holds 260 cells. Asterisks mark one-sided permutation p<0.05 for the cell alone (10,000 sign flips over text units), daggers a significant loss (two-sided p<0.05 ).
Model
FFD
GD
GP
TRT
Provo
GPT-2 XL, random
−0.62
−0.76†
−0.20
−0.31
OPT 1.3B, random
−1.16
−1.59†
−0.19
−0.89
Pythia 410M, step 0
−0.84
−0.96
−1.07†
−1.00
Pythia 1.4B, step 0
−1.22
+0.16
−1.00†
−0.68
GECO
Appendix
Table 5: Held-out gain (millinats per word) from adding D(w) and D(w−1) to the control set for the untrained networks, read with the logit lens: GPT-2 XL and OPT 1.3B with random weights and Pythia 410M and 1.4B at initialization. Columns, asterisks and daggers as in Table B .
Figure 4: (a) Mean layerwise surprisal trajectories of Natural Stories words under Pythia 2.8B, as in Figure 1 . (b) Held-out gain from adding D(w) and D(w−1) to the control set, the six eye-tracking corpora pooled per measure and family, with 95% bootstrap intervals over text units drawn within corpora.
Comparison
Gain
95% CI
p
D
0.257
0.10, 0.40
0.001
Trajectory
0.168
− 0.00, 0.34
0.047
true vs. shuffled
0.047
− 0.03, 0.12
0.14
Trajectory beyond D
−0.020
− 0.16, 0.12
0.61
D beyond trajectory
0.069
− 0.05, 0.19
0.15
Profile beyond D
−0.010
− 0.12, 0.09
0.57
Appendix
Table 6: Functional regression on the layerwise surprisal trajectory of w and w−1 : pooled held-out gain (millinats per word) on the primary measure over eight corpora and ten models. The first two rows add D or the trajectory term to the control set; “beyond” rows add one term to a model that already contains the other. “Profile beyond D ” adds a functional term on the resolution profile, whose mean is D ; shuffled rows use the median over five layer permutations.
Corpus
D
Trajectory
Trajectory beyond D
D beyond trajectory
Profile beyond D
Provo
−0.12
+0.35
−0.08
−0.55†
−1.09
GECO
+0.08
+0.13∗
+0.09
+0.03
−0.05
OneStop
+0.51∗
+0.75∗
+0.41∗
+0.16
+0.06
Natural Stories
+0.55∗
−0.43
−0.57
+0.41
+0.87∗
MECO
−0.20
−1.16
−1.41
−0.45
−1.06
ZuCo
+0.30
−0.28
−0.65†
−0.08
−0.37
Appendix
Table 7: Functional regression per corpus on the primary measure, pooled over the ten trained models (millinats per word): D and the whole-trajectory term over the control set, each beyond the other, and the resolution-profile term beyond D . Asterisks mark one-sided sign-flip p<0.05 over the corpus’s text units, daggers a significant loss (two-sided p<0.05 ).
Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.
Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4
ETH Zürich · Toyota Technological Institute at Chicago · University College London
Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.
This study analyzes the predictive power of LM-derived, token-level surprisal on Mandarin Chinese reading times. Using a suite of Chinese-Pythia models (14M--1.4B) trained from scratch with 30B tokens, we examine how well surprisal predicts first fixation duration, gaze duration, and total reading time in three paragraph-level eye-tracking corpora of Mandarin Chinese (GECO-CN, HKP, and MECO). Contrary to previous null findings, our results show that surprisal is predictive of Chinese reading times. However, whether predictive power scales with model size and the amount of training is corpus-specific: bigger models predict reading times better in GECO-CN, whereas inverse scaling emerges in HKP and, at the largest sizes, in MECO. Subsequently, we find that checkpoints whose surprisal remains closer to n-gram statistics are better predictors of reading in HKP and MECO, providing one possible explanation for the inverse scaling behavior. All in all, the predictive power of surprisal on Chinese reading time measurements is corpus-specific, which cautions against drawing conclusions about language model scale from a single corpus.
Hongao Zhu, Muxiaoqiao Xu, Yikang Liu +4
Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3