On the scaling relationship between cloze probabilities and language model next-token prediction

Cassandra L. Jacobs; Morgan Grobol

cs.CLFeb 19, 2026

On the scaling relationship between cloze probabilities and language model next-token prediction

Authors: Cassandra L. Jacobs, Morgan Grobol

Organizations: Department of Linguistics, University at Buffalo, Buffalo, NY, USA · MoDyCo, Université Paris Nanterre, Paris, France

Abstract

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.

Figures & tables

Pythia-70M	-160M	-2.8B	Human
lot	swarm	swarm	hive
swarm	bee	bee	swarm
bee	new	h*	bee
new	pest	was*	nest
threat	h*	colony	wasp

Table 1: Top

5

Pythia next-subword continuations in different Pythia models for the sentence fragment, “He hated bees and feared encountering a ____” .

*

“was” possibly corresponds to the first subword of “wasp” and “h” to “hive” .

Panel of Figure 1 — Figure 1: Human-NLM cloze correlations for Pythia-2.8B-deduped showing major deficiencies in assigning probability mass to human completions. All models at all sizes showed similar patterns and are not pictured here.

Figure 2: Relationship between language model ranks and cloze ranks. Error bars represent bootstrapped confidence intervals.

	Human	$\mathrm{N}\mathrm{L}\mathrm{M}$	5-gram
NLM	.375
5-gram	.177	$\,\mathrm{4}\mathrm{8}\mathrm{3}$
Unigram	.007	$\,\mathrm{1}\mathrm{8}\mathrm{7}$	.445

Table 2: Spearman correlation coefficient estimates between different language models (5-gram vs. Pythia NLM) and human cloze completions. All estimates significant.

Figure 3: Correlation strength between Wikipedia 5-gram scores and Pythia next-token probabilities by model size and data deduplication. X axis in log scale.

Panel of Figure 4 — Figure 4: Effect of model capacity and size on cloze correlation.

Pythia-160M	Human	Pythia-160M	Pythia-2.8B	Human
Count-based PPMI + PCA		Averaging Contextual Embeddings
mosquito	fly	moth	moth	moth
sting	ant	insect	mosquito	mosquito
snake	mosquito	bug	insect	spider
spider	insect	spider	bug	bug
bug	butterfly	mouse	spider	bird

Table 3: Top five most similar words to the word bee in human and NLM cloze completion semantic spaces.

Figure 5: Correlation between model capacity, PCA dimensionality, and correlation between language model and human semantic spaces.

Figure 6: Neighborhood-based alignment between human and NLM completions as a function of model size and neighborhood size considered. X axis in log scale.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Reading Time Modeling with Pixel-Based Language Models
Oct 19, 2026Junlin Li, Sara Alhajeri, Tatsuki KuribayashiComputational Psycholinguistics Tokenization in Language Models
Mohamed bin Zayed University of Artificial Intelligence · Tohoku University
Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models
Jul 17, 2026Boi Mai Quach, Binh T. Nguyen, Cathal Gurrin +1LLM Evaluation
School of Computing, ML-Labs, Department of Computer Science, School of Computing, ADAPT Centre, Dublin City University, Ireland · VNUHCM – University of Science · School of Computing, ADAPT Centre, Dublin City University, Ireland
Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
Apr 14, 2026Dikshant Kukreja, Kshitij Sah, Gautam Gupta +5Language Model Scaling Laws Knowledge Conflicts in Language Models
IIIT Delhi, India · Singapore Institute of Technology · NVIDIA +1

Oct 19, 2026J/K move · Enter open · S save

Reading Time Modeling with Pixel-Based Language Models

Junlin Li, Sara Alhajeri, Tatsuki Kuribayashi

Mohamed bin Zayed University of Artificial Intelligence · Tohoku University

Computational Psycholinguistics Tokenization in Language Models

Open paper AskPDF arXiv

Tokenization-free, pixel-based language models (PLMs) have recently been proposed as an alternative to conventional token-based language models (TLMs), typically motivated by practical considerations, e.g., the flexibility to support new languages or vocabularies. Nevertheless, PLMs also have the potential to serve as infrastructure for computational (psycho)linguistics, where tokenization has been a source of bias in estimating information-theoretic values, such as surprisal, and has pushed the bottom-up visual orthographic features outside the scope. In this study, we first integrate PLMs into the modeling of naturalistic human reading and, in particular, examine their use to estimate next-word predictability, which can also integrate orthographic and parafoveal information. Our results demonstrate that next-word prediction difficulty, computed in pixel space, has strong predictive power for gaze duration and word skipping behavior across various languages. In particular, their advantage is complementary to existing token-level surprisal and entropy estimates. Cross-lingual analyses with script variations also clarify their distinctive visual processing biases, facilitating research into how bottom-up visual information is incrementally integrated into expectation-based reading.

Abstract

Language Modeling

Explore similar work

Reading Time Modeling with Pixel-Based Language Models

Encoding EEG Signals to Examine Human-Like Next-Word Prediction Behaviour in Language Models

Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size