Reading Time Modeling with Pixel-Based Language Models

Junlin Li; Sara Alhajeri; Tatsuki Kuribayashi

cs.CLOct 19, 2026

Reading Time Modeling with Pixel-Based Language Models

Authors: Junlin Li, Sara Alhajeri, Tatsuki Kuribayashi

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Tohoku University

Abstract

Tokenization-free, pixel-based language models (PLMs) have recently been proposed as an alternative to conventional token-based language models (TLMs), typically motivated by practical considerations, e.g., the flexibility to support new languages or vocabularies. Nevertheless, PLMs also have the potential to serve as infrastructure for computational (psycho)linguistics, where tokenization has been a source of bias in estimating information-theoretic values, such as surprisal, and has pushed the bottom-up visual orthographic features outside the scope. In this study, we first integrate PLMs into the modeling of naturalistic human reading and, in particular, examine their use to estimate next-word predictability, which can also integrate orthographic and parafoveal information. Our results demonstrate that next-word prediction difficulty, computed in pixel space, has strong predictive power for gaze duration and word skipping behavior across various languages. In particular, their advantage is complementary to existing token-level surprisal and entropy estimates. Cross-lingual analyses with script variations also clarify their distinctive visual processing biases, facilitating research into how bottom-up visual information is incrementally integrated into expectation-based reading.

Figures & tables

Figure 1: Simulating visual sentence processing with orthographic and parafoveal information using pixel-based LMs

Figure 2: We approximate next-word predictability by patch-by-patch masking and blurring of the future context, simulating parafoveal sight, using PLM .

Figure 3: Examples of visual prediction from PLMs across multiple languages. The grayed areas are the PLM predictions for the regions originally masked.

Lang.	Dataset Code	Data	Skip Rate	Avg. TRT	#Subj.	Script
German	Deu (meco po)	MECO-2 (ge_po)	0.19	380.8	40	Latin
German	Deu (meco zu)	MECO-2 (ge_zu)	0.21	390.6	45
English	Eng (meco en)	MECO-1 (en)	0.30	346.1	46
English	Eng (meco uk)	MECO-2 (en_uk)	0.28	348.2	50
Spanish	Esp (meco es)	MECO-1 (es)	0.33	406.7	48
Spanish	Esp (meco ch)	MECO-1 (es_ch)	0.29	367.7	45

Table 1: We use multilingual and multi-script eye-movement-while-reading data across 459 readers.

Figure 4: MSE-based and SSIM-based VWPD estimates show slight, but different PPP patterns across languages. The error bars in the figures indicate 95% percentile bootstrap confidence intervals of PPP across the settings.

Figure 5: PPP (larger better) of VWPD and token surprisal for fixation measures (FFD = First Fixation Duration, FPT = First-pass Reading time, TRT = Total Reading Time). All zero-duration or zero-reading-time is filtered to disentangle from skipping measures. The Y-axis is PPP. The error bars in the figures indicate standard deviation.

Figure 6: PPP (larger better) of VWPD and token surprisal for skipping measures (FSKP = First-pass skipping, SKP = Overall skipping). The Y-axis is PPP. The error bars in the figures indicate standard deviation.

Figure 7: PPP (larger better) of visual and token entropy for skipping measures (FSKP = First-pass skipping, SKP = Overall skipping). The Y-axis is PPP. The error bars in the figures indicate standard deviation.

Figure 8: Interaction between VWPD and GPT2 surprisal (+1 SD = high,

-

1 SD = low) to FFD, FPT, and FPREF.

Figure 9: Effect of Unidirectional Gaussian Blurring. The points are obtained through bootstrapping, and the blue (left) ones are from the pixel setting without blurring, while the red (right) ones are from the pixel_uni one with blurring. Y-axis is PPP.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Lang.	Unidirectional LMs	Model size (#parameters)	Bidirectional LMs	Model size (#parameters)
Deu	dbmdz/german-gpt2	510MB ( $\sim$ 137M)	google-bert/bert-base-german-cased	439MB ( $\sim$ 109M)
Eng	openai-community/gpt2	548MB ( $\sim$ 137M)	google-bert/bert-base-cased	436MB ( $\sim$ 108M)
Esp	DeepESP/gpt2-spanish	548MB ( $\sim$ 124M)	dccuchile/bert-base-spanish-wwm-cased	440MB ( $\sim$ 110M)
Far	gpt2-persian	1.31GB ( $\sim$ 328M)	HooshvareLab/bert-base-parsbert-uncased	654MB ( $\sim$ 163M)
Kor	skt/kogpt2-base-v2	513MB ( $\sim$ 125M)	kykim/bert-kor-base	476MB ( $\sim$ 118M)
Simp (zh)	uer/gpt2-chinese-cluecorpussmall	421MB ( $\sim$ 102M)	google-bert/bert-base-chinese	412MB ( $\sim$ 102M)

Appendix

Table 2: Models with their sizes in our experiment. All checkpoints of GPT and BERT are from https://huggingface.co/ .

Figure 10: We approximate next-word predictability with patch-by-patch masking and add noise for the future context, simulating parafoveal sight, using PLM .

Figure 11: Gain of PPP after introducing PLM predictability to TLM ( $\frac{\mathrm{PPP}_{\mathrm{pixel\_uni}*\mathrm{gpt}}-\mathrm{PPP}_{\mathrm{gpt}}}{\mathrm{PPP}_{\mathrm{gpt}}}$ ), which suggests multimodal prediction encodes rich metacognitive knowledge about skipping.

Figure 12: Effect of Whole Masking on PPP (Paired-t test of PPP is implemented, with matched regression model formula)

Figure 13: Main results by dataset (Reading Time)

Figure 14: Main results by dataset (Skipping)

Figure 15: Comparison of MSE and SSIM measures in 10-fold cross-validation settings.

Figure 16: Main results by dataset (Reading Time) using 10-fold cross-validation based on visual and token surprisal. (Log-likelihood change is case-wise averaged)

Figure 17: Main results by dataset (Skipping) using 10-fold cross-validation based on visual and token surprisal. (Log-likelihood change is case-wise averaged)

Figure 18: Main results by dataset (Skipping) using 10-fold cross-validation, based on visual and token entropy. (Log-likelihood change is case-wise averaged)

VWPD

Measure

Esp (es)

Esp (ch)

Deu (po)

Deu (zu)

Eng (uk)

Eng (en)

Kor

Far (PSR)

Simp

Trad

Tur

pixel_uni vs. gpt

SSIM

FFD

-23.0 ∗∗∗

-28.3 ∗∗∗

-32.7 ∗∗∗

-48.4 ∗∗∗

-68.9 ∗∗∗

-30.6 ∗∗∗

-5.6 ∗

-126.6 ∗∗∗

-27.5 ∗∗∗

-24.3 ∗∗∗

-4.1 ∗

FPT

-37.3 ∗∗∗

-74.3 ∗∗∗

-60.1 ∗∗∗

-84.1 ∗∗∗

-100.4 ∗∗∗

-49.9 ∗∗∗

-44.8 ∗∗∗

-321.2 ∗∗∗

-63.8 ∗∗∗

-24.0 ∗∗∗

-13.2 ∗∗∗

TRT

-20.8 ∗∗∗

-66.7 ∗∗∗

-214.0 ∗∗∗

-235.6 ∗∗∗

-194.3 ∗∗∗

-94.6 ∗∗∗

-68.1 ∗∗∗

-129.1 ∗∗∗

-90.6 ∗∗∗

-30.9 ∗∗∗

-15.5 ∗∗∗

FSKP

+26.4 ∗∗∗

-6.9 ∗∗

+16.4 ∗∗∗

+28.9 ∗∗∗

+41.8 ∗∗∗

-6.1 ∗

+49.2 ∗∗∗

+177.0 ∗∗∗

-23.1 ∗∗∗

-16.1 ∗∗∗

-6.3 ∗∗∗

SKP

-4.3 .

-22.1 ∗∗∗

-8.7 ∗∗∗

-55.4 ∗∗∗

-6.2

-28.0 ∗∗∗

+107.6 ∗∗∗

+214.9 ∗∗∗

-45.6 ∗∗∗

-21.1 ∗∗∗

+7.8 ∗∗∗

Appendix

Table 3: Mean paired log-likelihood difference relative to GPT. Positive values favour the reported model. The estimation is based on boostrapping. The significance is based on paired t-test. ∗

p<.05

, ∗∗

p<.01

, ∗∗∗

p<.001

, .

p<.1

VWPD	Measure	Esp (es)	Esp (ch)	Deu (po)	Deu (zu)	Eng (uk)	Eng (en)
pixel_uni vs. gpt
SSIM	FFD	-0.000033	-0.000153	+0.000029	+0.000045 .	-0.000421 .	-0.000188 .
	FPT	-0.000374	-0.001012 ∗∗	-0.000885 .	-0.000158	-0.001189 .	-0.000768 ∗∗
	TRT	-0.000125	-0.000871 ∗∗∗	-0.004425 ∗∗∗	-0.002344 ∗∗∗	-0.002263 ∗∗∗	-0.001323 ∗∗∗
	FSKP	+0.000098	+0.000007	-0.000392 ∗∗∗	-0.000599 ∗∗∗	+0.000323 ∗	+0.000003
	SKP	-0.000009	-0.000087	-0.000167 .	-0.000562 ∗∗∗	-0.000131	-0.000313 ∗∗

Appendix

Table 4: Mean paired log-likelihood difference relative to GPT. Positive values favour the reported model. ∗

p<.05

, ∗∗

p<.01

, ∗∗∗

p<.001

, .

p<.1

. The estimations are based on cross-validation. The significance is based on paired t-test.

Computational Psycholinguistics

Computational psycholinguistics models how people learn, produce, and understand language using formal theories, algorithms, and behavioral data.

See all 89 papers

Probing for Reading Times
Apr 20, 2026Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re +4Computational Psycholinguistics Language Model Probing
ETH Zürich · Toyota Technological Institute at Chicago · University College London
Reader Proficiency Shapes Layer-wise Surprisal Profiles
Sep 29, 2026Akio Hayakawa, Horacio SaggionLanguage Model Probing
Universitat Pompeu Fabra Barcelona, Spain
N-gram-like Language Models Predict Naturalistic Reading Time Best
Mar 10, 2026James A. Michaelov, Roger P. LevyLanguage Modeling Autoregressive Language Modeling
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology · MIT Libraries CREOS, Massachusetts Institute of Technology

Apr 20, 2026J/K move · Enter open · S save

Probing for Reading Times

Eleftheria Tsipidi, Samuel Kiegeland, Francesco Ignazio Re, Tianyang Xu+3

ETH Zürich · Toyota Technological Institute at Chicago · University College London

Computational Psycholinguistics Language Model Probing

Open paper AskPDF arXiv

Probing has shown that language model representations encode rich linguistic information, but it remains unclear whether they also capture cognitive signals about human processing. In this work, we probe language model representations for human reading times. Using regularized linear regression on two eye-tracking corpora spanning five languages (English, Greek, Hebrew, Russian, and Turkish), we compare the representations from every model layer against scalar predictors -- surprisal, information value, and logit-lens surprisal. We find that the representations from early layers outperform surprisal in predicting early-pass measures such as first fixation and gaze duration. The concentration of predictive power in the early layers suggests that human-like processing signatures are captured by low-level structural or lexical representations, pointing to a functional alignment between model depth and the temporal stages of human reading. In contrast, for late-pass measures such as total reading time, scalar surprisal remains superior, despite its being a much more compressed representation. We also observe performance gains when using both surprisal and early-layer representations. Overall, we find that the best-performing predictor varies strongly depending on the language and eye-tracking measure.

Abstract

Computational Psycholinguistics

Explore similar work

Probing for Reading Times

Reader Proficiency Shapes Layer-wise Surprisal Profiles

N-gram-like Language Models Predict Naturalistic Reading Time Best