cs.CLFeb 20, 2026

Information-Theoretic Storage Cost in Sentence Comprehension

Authors: Kohei Kajikawa, Shinnosuke Isono, Ethan Gotlieb Wilcox

Organizations: Department of Linguistics, Georgetown University, USA · National Institute for Japanese Language and Linguistics, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Real-time sentence comprehension imposes a significant load on working memory, as comprehenders must maintain contextual information to anticipate future input. While measures of such load have played an important role in psycholinguistic theories, they have largely been formalized using symbolic grammars, which assign discrete, uniform costs to syntactic predictions. This study proposes a measure of processing storage cost based on an information-theoretic formalization, as the amount of information previous words carry about future context, under uncertainty. Unlike previous discrete, grammar-based metrics, this measure is continuous, probabilistic, theory-neutral, and can be estimated from pre-trained neural language models. The validity of this approach is demonstrated through three analyses in English: our measure (i) recovers well-known processing asymmetries in center embeddings and relative clauses, (ii) correlates with a grammar-based storage cost in a syntactically-annotated corpus, and (iii) predicts reading-time variance in two large-scale naturalistic datasets over and above baseline models with traditional information-based predictors. Our code is available at https://github.com/kohei-kaji/info-storage.

Figures & tables

Figure 1: Illustration of the proposed storage cost measure. It quantifies the predictive potential shared between a target word wiw_{i} and the future w[k:N]\boldsymbol{w}_{[k:N]} , conditioned on the remaining context . Information-theoretic storage cost at wkw_{k} is defined as the sum of the predictive potentials across all context words, representing the load of pending information.
Figure 2: Illustration of DLT storage cost based on the predicted syntactic head hypothesis. In the center-embedded structure (top), the storage cost at Mary is five memory units, because five syntactic heads (the blue -colored words) are predicted to form a grammatical sentence at Mary . In contrast, the right-branching structure (bottom) imposes a minimal storage load. tt denotes a trace of extraction. This example is adopted from Chen et al. (2005) .
Figure 3: Mean information storage estimated by BERT at each word position. Error bars represent 95% confidence intervals across 30 items. In both cases, the more difficult structure (CE, ORC) exhibits higher storage cost, consistent with behavioral asymmetries.
Figure 4: Mean information-theoretic storage cost as a function of DLT storage cost in the UD_English-GUM corpus ( Zeldes, 2017 ) . Points represent the mean value for each DLT bin with 95% confidence intervals. For this visualization, bins with fewer than 100 observations were excluded due to data sparsity. The red dashed line represents the linear regression fitted to the raw data corresponding to the displayed bins.
Figure 5: Results of the naturalistic reading-time analysis on Natural Stories (NS) and OneStop (OS). Abbreviations: SPR = self-paced reading; FPD = first-pass duration; GPD = go-past duration; TFD = total fixation duration.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Mean predictive potential Ppred(wi;W[k:N])\mathcal{P}_{\mathrm{pred}}(w_{i};\boldsymbol{W}_{[k:N]}) as a function of distance d=k−id=k-i (up to a maximum of 30), computed over the UD_English-GUM. The monotonic decrease is consistent with the theoretical result that predictive potential is non-increasing on average.
Figure 7: Pearson correlation matrices of predictors in the naturalistic reading-time datasets.
ModelLayersHeadsHidden SizeParameters
BERT-base ( bert-base-uncased )1212768110M
BERT-Large ( bert-large-uncased )24161,024336M
RoBERTa ( roberta-base )1212768125M
Appendix
Table 1: Architectural details of the models. Layers, Heads, and Hidden Size refer to the number of layers, the number of attention heads per layer, and embedding/hidden size, respectively.
Figure 8: Mean information-theoretic storage cost as a function of DLT storage cost in the UD_English-GUM corpus. Points represent the mean value for each DLT bin with 95% confidence intervals. For this visualization, bins with fewer than 100 observations were excluded due to data sparsity. The red dashed line represents the linear regression fitted to the raw data.
Figure 9: Results of the naturalistic reading-time analysis on Natural Stories (NS) and OneStop (OS). Abbreviations: SPR = self-paced reading; FPD = first-pass duration; GPD = go-past duration; TFD = total fixation duration.
More in this topic

Computational Psycholinguistics

Computational psycholinguistics models how people learn, produce, and understand language using formal theories, algorithms, and behavioral data.

See all 89 papers

Explore similar work

CardsList

  1. Syntactically-guided Information Maintenance in Sentence Comprehension

    Apr 30, 2026Shinnosuke Isono, Kohei KajikawaComputational Psycholinguistics

    NINJAL · Georgetown University

  2. Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal

    Jun 3, 2026Elan BarenholtzSurprisal TheoryComputational Psycholinguistics

    Machine Perception & Cognitive Robotics Laboratory Department of Psychology / Center for Complex Systems Florida Atlantic University

  3. STRUCTURALCOST: A controlled reading time dataset for modeling human sentence processing difficulty

    Oct 6, 2026Nina Nusbaumer, Iria de-Dios-Flores, Corentin Bel +3Computational Psycholinguistics

    LLF, CNRS, Université Paris Cité · COLT, Universitat Pompeu Fabra · Unicog, Neurospin, CEA +2