The Depth of a Prediction: Where Language Models Resolve a Word Predicts Human Reading Time
Authors: Donato Riccio, Fabrizio Maturo
Organizations: Department of Engineering and Science, Universitas Mercatorum, Rome, Italy · Department of Economics, Statistics and Business, Universitas Mercatorum, Rome, Italy
Surprisal from a language model's final layer predicts how long people read a word, yet the model computes its prediction across many layers. We read each word's prediction at every layer with the tuned lens and summarize this layerwise surprisal trajectory by its resolution depth, the depth around which its change is centered. On eight English eye-tracking and self-paced reading corpora (81,957 words) and ten GPT-2, OPT and Pythia models, words that a model resolves later, and the words after them, are read more slowly than equally surprising words. Resolution depth improves held-out prediction beyond final-layer surprisal, entropy and the best single layer by 0.26 millinats per word (pooled p = 0.001), about a third of entropy's gain, and each model family gains, GPT-2 and Pythia through one corpus each. The net gain varies between corpora (random-effects mean 0.07, p = 0.37); against layer-shuffled depth, which adds as many coefficients, it is positive in seven of eight corpora (random-effects mean 0.23, p = 0.016), and untrained networks show no gain under the logit lens. Pooled over corpora, the effect holds with corrected word probabilities and beyond first-to-last-layer divergences, and a whole-trajectory functional regression adds no significant information beyond resolution depth.
Figures & tables
Figure 1: Mean layerwise surprisal trajectories (tuned lens) of OneStop words in the middle fifth of final surprisal (40th to 60th percentile) under GPT-2 XL, split into the fifth with the lowest and the fifth with the highest resolution depth D (Natural Stories with Pythia 2.8B in Figure B a). Dotted lines mark each group’s mean D ; n is the number of analyzed words in the band. Both groups end at the same surprisal; the late-resolved words keep a high surprisal through more of the network.
Corpus
Material
Readers
Words
Analyzed
Units
Readers/word
Provo
55 passages
84
2,745
2,115
55 passages
49
GECO
one novel
14
54,362
33,624
586 screens
7
OneStop
30 articles, 2 versions
180
35,188
25,359
30 articles
14
MECO
12 texts
46
2,107
1,732
12 texts
18
ZuCo
300 sentences
12
6,588
4,711
300 sentences
9
UCL eye tracking
205 sentences
37
1,931
1,305
205 sentences
25
Table 1: Reading corpora: six eye-tracking and two self-paced reading corpora (Natural Stories, UCL self-paced); the two UCL corpora present the same sentences (205 in eye tracking) to different readers. Words are the displayed word tokens of each release; analyzed words enter the primary analysis (go-past time for eye tracking, reading time for self-paced reading) after exclusions. Units define the cross-validation folds and the units of inference (§ 5 ). Readers/word is the median number of readers contributing to an analyzed word. Total reading time analyzes 36,382 words in GECO, 26,824 in OneStop, 1,881 in MECO and 1,316 in UCL eye tracking, and the words of the primary measure elsewhere.
Figure 2: Held-out gain in log-likelihood from adding resolution depth D(w) and D(w−1) to the control set on the primary measure (go-past time for eye tracking, reading time for self-paced reading), pooled over the eight corpora, per model, per family, for the ten trained models and for the four untrained networks; ticks mark the median gain over five layer permutations. Intervals are 95% bootstrap intervals over text units drawn within corpora.
Figure 3: Held-out gain from resolution depth on the primary measure for each corpus and model, in millinats per word. Asterisks mark one-sided permutation p<0.05 for the corpus and model alone.
Analysis
Gain
p
Main test
0.257
0.001
D(w) alone
0.094
0.015
D(w−1) alone
0.174
0.002
From readout 1
0.188
0.006
Corrected word probabilities
0.185
0.009
Single-token words
0.171
0.012
Table 2: Robustness and alternative measures: pooled held-out gain (millinats per word) on the primary measure over eight corpora and ten models, with one-sided sign-flip p . “+” rows add the named variables of w and w−1 to the controls before testing D ; “alone” rows test the alternative measure in place of D , and “beyond” rows test D with the alternative in the controls.
Comparison
Gain
95% CI
p
D
0.257
0.10, 0.40
0.001
Trajectory
0.168
− 0.00, 0.34
0.047
Trajectory beyond D
−0.020
− 0.16, 0.12
0.61
D beyond trajectory
0.069
− 0.05, 0.19
0.15
Profile beyond D
−0.010
− 0.12, 0.09
0.57
Table 3: Functional regression on the trajectories of w and w−1 : pooled held-out gain (millinats per word) on the primary measure; “beyond” rows add one term to a model that contains the other (shuffled-layer rows in Table C ).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
FFD
GD
GP
TRT
Provo
GPT-2 small
−0.99
−0.73
−0.07
−0.29
GPT-2 large
+0.61
+0.96
−0.42
−0.09
GPT-2 XL
+1.98∗
+1.07
+1.03
+0.85
OPT 125M
+0.72
+1.09
+0.72
−0.41
OPT 1.3B
−0.01
−0.20
−0.28
−0.47
Appendix
Table 4: Held-out gain (millinats per word) from adding D(w) and D(w−1) to the control set, for each corpus, trained model and measure: first fixation duration (FFD), gaze duration (GD), go-past time (GP) and total reading time (TRT). The self-paced reading corpora have one measure, the reading time, listed under GP, so the table holds 260 cells. Asterisks mark one-sided permutation p<0.05 for the cell alone (10,000 sign flips over text units), daggers a significant loss (two-sided p<0.05 ).
Model
FFD
GD
GP
TRT
Provo
GPT-2 XL, random
−0.62
−0.76†
−0.20
−0.31
OPT 1.3B, random
−1.16
−1.59†
−0.19
−0.89
Pythia 410M, step 0
−0.84
−0.96
−1.07†
−1.00
Pythia 1.4B, step 0
−1.22
+0.16
−1.00†
−0.68
GECO
Appendix
Table 5: Held-out gain (millinats per word) from adding D(w) and D(w−1) to the control set for the untrained networks, read with the logit lens: GPT-2 XL and OPT 1.3B with random weights and Pythia 410M and 1.4B at initialization. Columns, asterisks and daggers as in Table B .
Figure 4: (a) Mean layerwise surprisal trajectories of Natural Stories words under Pythia 2.8B, as in Figure 1 . (b) Held-out gain from adding D(w) and D(w−1) to the control set, the six eye-tracking corpora pooled per measure and family, with 95% bootstrap intervals over text units drawn within corpora.
Comparison
Gain
95% CI
p
D
0.257
0.10, 0.40
0.001
Trajectory
0.168
− 0.00, 0.34
0.047
true vs. shuffled
0.047
− 0.03, 0.12
0.14
Trajectory beyond D
−0.020
− 0.16, 0.12
0.61
D beyond trajectory
0.069
− 0.05, 0.19
0.15
Profile beyond D
−0.010
− 0.12, 0.09
0.57
Appendix
Table 6: Functional regression on the layerwise surprisal trajectory of w and w−1 : pooled held-out gain (millinats per word) on the primary measure over eight corpora and ten models. The first two rows add D or the trajectory term to the control set; “beyond” rows add one term to a model that already contains the other. “Profile beyond D ” adds a functional term on the resolution profile, whose mean is D ; shuffled rows use the median over five layer permutations.
Corpus
D
Trajectory
Trajectory beyond D
D beyond trajectory
Profile beyond D
Provo
−0.12
+0.35
−0.08
−0.55†
−1.09
GECO
+0.08
+0.13∗
+0.09
+0.03
−0.05
OneStop
+0.51∗
+0.75∗
+0.41∗
+0.16
+0.06
Natural Stories
+0.55∗
−0.43
−0.57
+0.41
+0.87∗
MECO
−0.20
−1.16
−1.41
−0.45
−1.06
ZuCo
+0.30
−0.28
−0.65†
−0.08
−0.37
Appendix
Table 7: Functional regression per corpus on the primary measure, pooled over the ten trained models (millinats per word): D and the whole-trajectory term over the control set, each beyond the other, and the resolution-profile term beyond D . Asterisks mark one-sided sign-flip p<0.05 over the corpus’s text units, daggers a significant loss (two-sided p<0.05 ).
Department of Linguistics, UC San Diego · School of Foreign Languages, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University +3