cs.CLJul 28, 2026

A scaling law of contextual persistence in human language

Authors: Elan Barenholtz

Organizations: Machine Perception and Cognitive Robotics Laboratory, Department of Psychology and Center for Complex Systems, Florida Atlantic University

Abstract

Human language exhibits lawful structure at the level of words (frequency, vocabulary growth) and word pairs (co-occurrence across distance). Here we show that the arrangement of words in sequence -- a central determinant of meaning -- obeys a comparable law. Using large language models as probabilistic probes, we measured the reduction in target perplexity conferred by prior context at distance d beyond that of the same words scrambled; this difference, the contextual persistence function P(d), isolates the influence of arrangement. Across ten corpora spanning six language families and written and spoken modalities, P(d) decayed approximately as 1/d (P(d)dαP(d) \propto d^{-α}, mean α=1.04α= 1.04; median r2=0.96r^2 = 0.96). The effect vanished in scrambled and synthetic controls, replicated across independent probes, and did not appear in genomic or protein sequences under domain-native models. An exponent near 1 distributes contextual influence approximately uniformly across logarithmic timescales. The results establish a scaling law of contextual persistence in human language.

Explore similar work

CardsList
  1. Deriving Neural Scaling Laws from the statistics of natural language

    Feb 7, 2026Francesco Cagnetta, Allan Raventós, Surya Ganguli +1Scaling LawsLarge Models