Reader Proficiency Shapes Layer-wise Surprisal Profiles

Akio Hayakawa; Horacio Saggion

cs.CLSep 29, 2026

Reader Proficiency Shapes Layer-wise Surprisal Profiles

Authors: Akio Hayakawa, Horacio Saggion

Organizations: Universitat Pompeu Fabra Barcelona, Spain

Abstract

Reading behaviour varies not only with linguistic input, but also with reader proficiency. In this study, we investigate whether the layer-wise relationship between surprisal from large language models (LLMs) and human gaze behaviour differs across readers with different levels of proficiency and across gaze measures. Using eye-tracking data from the MECO L2 corpus, we compare readers with high and low vocabulary proficiency on first-pass gaze duration (FPGD) and total gaze duration (TGD). We quantify the distribution of the predictive power of surprisal across model layers using Predictive Depth. Across 12 tested LLMs, we find that readers with lower vocabulary proficiency tend to show deeper Predictive Depth for FPGD, while this difference is smaller for TGD. Also, TGD itself shows deeper Predictive Depth than FPGD in both proficiency groups. These patterns suggest that where predictive power is concentrated across LLM layers may be related to the timing and breadth of the reading processes captured by different gaze measures, and that this relationship can vary with reader proficiency. Our leave-one-out analysis further shows that the advantage of informative internal layers extends to unseen texts, although the practical improvements in prediction are limited. Overall, our results show that layer-wise LLM surprisal provides a useful perspective on variation in reading behaviour across both reader groups and gaze measures.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of the distribution of predictive power along LLM layers. Readers with limited vocabulary knowledge (red) tend to show deeper Predictive Depth for FPGD compared to readers with higher vocabulary knowledge (blue). The difference in Predictive Depth is smaller for TGD, which itself shows deeper Predictive Depth than FPGD in both reader groups.

	All	High-Lex	Low-Lex
LexTALE Score	-	$\geq$ 83.75	$<$ 66.25
Group Size	1001	276	228
Total Trials	8597	2519	1845

Table 1: Statistics of reader groups from MECO L2. Trials are the number of times each participant read each text.

Figure 2: Plots of layer-wise

\Delta LL

for both FPGD (top) and TGD (bottom) from each LLM family.

\Delta LL

values on the y-axis are normalized per token and multiplied by 1000 for better visualization.

			FPGD			TGD			TGD-FPGD
LLM	Size	N	Low	High	L-H	Low	High	L-H	Low	High
Gemma 3	4B	34	$0.668$	$0.553$	+0.115	$0.629$	$0.586$	+0.043	$-0.039$	+0.033
	12B	48	$0.613$	$0.548$	+0.065	$0.678$	$0.599$	+0.079	+0.065	+0.051
	27B	62	$0.523$	$0.549$	$-0.026$	$0.674$	$0.673$	+0.001	+0.151	+0.124
Qwen 2.5	7B	28	$0.558$	$0.549$	+0.009	$0.565$	$0.587$	$-0.022$	+0.000	+0.039
	14B	48	$0.520$	$0.507$	+0.014	$0.558$	$0.559$	$-0.001$	+0.038	+0.052

Table 2: Predictive Depth for FPGD and TGD in the Low-Lex and High-Lex groups across all tested LLMs.

N

represents the number of layers in each model. The last two columns show the difference in

PD

between TGD and FPGD for each group. Positive differences are shown in bold.

Low-Lex
		MAE
		$(\times 100)$		$\rho$
LLM		Final	Selected	Final	Selected
(Surface)		8.96		-
Gemma 3	4B	8.94	8.87	$0.089$	0.117
	12B	8.95	8.87	$-0.030$	0.125

Table 3: Leave-one-out evaluation results for Low-Lex and High-Lex readers. Selected denotes the internal layer chosen based on

\Delta LL

on the training texts. MAE values are multiplied by 100 for readability.

	log-FPGD				$r_{G,w}$	Surprisal
Word	Obs.	Surf.	Fin.	Sel.	Surf.	Fin.	Sel.
related	5.82	5.73	5.73	5.71	$0.09$	1.98	$2.63$
note,	5.71	5.59	5.57	5.53	$0.12$	0.14	$0.00$
sleep	5.48	5.65	5.63	5.66	$-0.17$	1.32	$13.01$
also	5.41	5.52	5.51	5.49	$-0.11$	1.53	$0.02$
improves	5.63	5.88	5.87	5.87	$-0.25$	1.57	$8.87$

Table 4: Illustrative example for Llama 3.1 70B. The selected internal layer is layer 46 out of 80. improves is highlighted as one of the words with the largest residual

r_{G,w}

under the Surface LMM. Obs. denotes observed log-FPGD, while Surf., Fin., and Sel., denote the Surface, Final, and Selected LMM predictions, respectively. For each word, the prediction closest to the observed value is in bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3:

\Delta LL

and Predictive Depth across all tested LLMs. As in Figure 2 , the color blue refers to High-Lex, and the red refers to Low-Lex.

Explore similar work

Probing for Reading Times
Apr 20, 2026Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4Readability Eye Tracking
ETH Zürich · Toyota Technological Institute at Chicago · University College London
Dual Alignment Between Language Model Layers and Human Sentence Processing
Apr 20, 2026Tatsuki Kuribayashi, Alex Warstadt, Yohei Oseki +1Natural Language Surprisal
MBZUAI · Tohoku University · UC San Diego +2
LEXIC: Lightweight On-Device Decoding of Reading Comprehension from Eye Movements
Jul 9, 2026Sumin Lee, Kyeonghun Kim, Subeen Lee +5Readability Gaze Behavior
Seoul National University · OUTTA · NVIDIA

Apr 20, 2026, cs.CLJ/K move · Enter open · S save

Probing for Reading Times

Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu+3

ETH Zürich · Toyota Technological Institute at Chicago · University College London

Readability Eye Tracking Layer-Wise Language Modeling Surprisal

Open paper AskPDF arXiv

Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.

Abstract

Explore similar work

Probing for Reading Times

Dual Alignment Between Language Model Layers and Human Sentence Processing

LEXIC: Lightweight On-Device Decoding of Reading Comprehension from Eye Movements