To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times

Thomas Hikaru Clark; Carlos Arriaga; Javier Conde; Gonzalo Martínez; Pedro Reviriego

cs.CLMar 12, 2026

To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times

Authors: Thomas Hikaru Clark, Carlos Arriaga, Javier Conde, Gonzalo Martínez, Pedro Reviriego

Organizations: Massachusetts Institute of Technology · Universidad Politécnica de Madrid

Abstract

Large Language Models (LLMs) have recently been shown to produce estimates of psycholinguistic norms, such as valence, arousal, or concreteness, for words and multiword expressions, that correlate with human judgments. These estimates are obtained by prompting an LLM, in zero-shot fashion, with a question similar to those used in human studies. Meanwhile, for other norms such as lexical decision time or age of acquisition, LLMs require supervised fine-tuning to obtain results that align with ground-truth values. In this paper, we extend this approach to the previously unstudied features of sentence memorability and reading times, which involve the relationship between multiple words in a sentence-level context. Our results show that via fine-tuning, models can provide estimates that correlate with human-derived norms and exceed the predictive power of interpretable baseline predictors, demonstrating that LLMs contain useful information about sentence-level features. At the same time, our results show very mixed zero-shot and few-shot performance, providing further evidence that care is needed when using LLM-prompting as a proxy for human cognitive measures.

Figures & tables

Figure 1: Overview of our approach, contrasting zero-shot prompting with fine-tuning on small supervised datasets for predicting psycholinguistic norms.

Sentence	Memorability
Does olive oil work for tanning?	0.98
Scott cried, pursing his pink lips.	0.89
The weather was warm and dry.	0.79
I can’t get hold of him.	0.72
We want to make it better.	0.56

Table 1: Example sentences from the dataset of Clark et al. [2026] , with human-derived memorability scores.

Dataset	# Examples	Epochs	Batch Size
Word mem.	527	3	1
Sentence mem.	625
Self-paced	108
Eye-tracking	303

Table 2: Fine-tuning parameters used for each dataset. Note that for the reading time datasets, “# Examples” denotes the number of sentences.

Figure 2: The correlation between model predictions and ground truth norms for word memorability, across 3 model families and in 3 evaluation regimes. For comparison, the mean correlation with the predictions of interpretable baseline predictors is included.

Figure 3: The correlation between model predictions and ground truth norms for sentence memorability, across 3 model families and in 3 evaluation regimes. For comparison, the mean correlation with the predictions of interpretable baseline predictors is included.

Figure 4: The correlation between model predictions and ground truth norms for self-paced reading times (Natural Stories corpus), across 3 model families and in 3 evaluation regimes. For comparison, the mean correlation with the predictions of interpretable baseline predictors is included.

Figure 5: The correlation between model predictions and ground truth norms for eye-tracking reading times (OneStop corpus), across 3 model families and in 3 evaluation regimes. For comparison, the mean correlation with the predictions of interpretable baseline predictors is included.

Figure 6: Correlation of fine-tuned model prediction to ground truth for different word positions in a sentence.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Understanding the Interplay between Memorization and Learning in Large Language Models
Jul 20, 2025Bishwamittra Ghosh, Soumi Das, Qinyuan Wu +4Memorization in Language Models LLM Training
Max Planck Institute for Software Systems, Germany · Boston University, USA
Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing
Oct 6, 2026Jing Chen, Giulia Loca, Simona Amenta +1Language Modeling Computational Psycholinguistics
Department of Psychology, University of Milano-Bicocca, Piazza dell’Ateneo Nuovo 1, 20126 Milano, Italy · Department of Informatics, Systems and Communication – DISCo, University of Milano-Bicocca, Milano, Italy
Dual Alignment Between Language Model Layers and Human Sentence Processing
Apr 20, 2026Tatsuki Kuribayashi, Alex Warstadt, Yohei Oseki +1Language Modeling Computational Psycholinguistics
MBZUAI · Tohoku University · UC San Diego +2

Jul 20, 2025J/K move · Enter open · S save

Understanding the Interplay between Memorization and Learning in Large Language Models

Bishwamittra Ghosh, Soumi Das, Qinyuan Wu, Mohammad Aflah Khan+3

Max Planck Institute for Software Systems, Germany · Boston University, USA

Memorization in Language Models LLM Training Language Modeling

Open paper AskPDF arXiv

We investigate foundational questions about the interplay between memorization and learning in large language models (LLMs). When an LLM generates a string, can we disentangle the roles of rote memorization and contextual learning? Rote memorization results in an LLM regurgitating specific details of a training string, while contextual learning leads to the generation of new strings that follow generalizable patterns of the underlying language. Related questions of interest include: can an LLM avoid memorization when optimally learning a language?, and if not, can we design training schemes to reduce memorization and improve learning? To address these questions, we propose contextual memorization, a new measure that precisely disentangles memorization from learning. Using this measure, we establish that different training strings are memorized at different epochs in training, and that memorization of some training strings is unavoidable when optimally learning a language. Finally, we propose memorization-aware training, a scheme that dynamically down-samples training strings based on their memorization, and in doing so, equalizes learning across strings, reduces memorization, and improves learning. Our conclusions are supported by extensive experiments on multiple LLMs using formal languages as a controlled testbed, enabling precise language specification, exact string sampling, and elimination of data contamination; and further validated on natural languages.

Abstract

LLM Evaluation

Explore similar work

Understanding the Interplay between Memorization and Learning in Large Language Models

Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing

Dual Alignment Between Language Model Layers and Human Sentence Processing