Linguistic boundaries emerge from pressure-constrained processing dynamics
Authors: Rui He, Philipp Homan, Wolfram Hinzen
Organizations: Grammar and Cognition Lab, Department of Translation & Language Sciences, Universitat Pompeu Fabra, Barcelona 08018, Spain · Department of Adult Psychiatry and Psychotherapy, University Hospital of Psychiatry Zurich, University of Zurich, Zurich 8032, Switzerland · Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich 8057, Switzerland · Institució Catalana de Recerca i Estudis Avançats, Barcelona 08010, Spain
Human cognition organizes continuous experience into bounded units while maintaining continuity across them. Linguistic elements are also organized into bounded structural units such as words and phrases, with sentence marking the final bound of a grammatical construction process. How do such boundaries emerge? We propose boundaries emerge from interacting demands to maintain unresolved dependencies, integrate completed relations, and update semantic context, with structural completion allowing but not determining closure. First, in a generative simulation without predefined boundary positions or constituent labels, these demands produced sentence-like segment-length distributions and distinct characteristic timescales, with structural completion allowing but not determining closure. Near boundaries, unresolved dependency load declined, integration cost peaked, and accumulated pressure reset. Then, we tested whether same demands predicted human listeners' boundary choices during comprehension. Listeners were more likely to mark boundaries with fewer unresolved dependencies, higher integration cost and semantic mismatch, and elevated preceding memory load, beyond syntactic and prosodic cues. Third, in naturalistic self-paced reading, sentence-final slowing coincided with declining memory load and increasing integration cost and semantic mismatch, broadly consistent with the simulation. Processing same narratives with a large language model revealed increased local intrinsic dimensionality and straighter hidden-state trajectories at sentence endings, followed by larger representational updates. Boundary explained additional variation in reading time and hidden-state geometry beyond processing demands, but considerably less in next-token surprisal. Together, these findings suggest linguistic boundaries emerge as discrete manifestations of transitions in continuous processing dynamics.
Figures & tables
Figure 1: Pressure-constrained simulation and parameter selection. (A) Simulation model. Dependency-locality costs and contextual surprisal jointly contribute to processing pressure. Accumulated pressure is transformed by a sigmoid function into a probabilistic boundary drive. When open relations remain unresolved, pressure promotes the resolution of open relations. When no relation remains open, pressure increases the probability of a boundary, which resets accumulated pressure and the local contextual states. (B) Language-likeness score. Parameter regimes were evaluated using dependency-length decay, Zipf-like lexical rank-frequency structure, and anti-collapse lexical quality control. (C) Top 10% language-like regimes. Model families were compared within the subset of parameter combinations retained by the language-likeness score.
Figure 2: Boundary-aligned profiles in the best-performing pressure-constrained model. Analyses were performed on the highest-scoring DLT-surprisal parameter regime, rerun across 500 random seeds. Model variables were aligned to emitted boundaries and compared with non-boundary control positions sampled from the same simulations. The vertical dashed line marks the boundary crossing. Six variables of interest: (A) Processing pressure; (B) Boundary probability; (C) Contextual surprisal costs; (D) Memory costs; (E) Integration costs; and (F) Semantic mismatch.
Figure 3: Temporal scales of continuous dynamics and discrete structural events in the selected model. Analyses were performed on the highest-scoring DLT–surprisal parameter regime rerun across 500 random seeds. Points show mean period or scale across seeds. Error bars indicate 95% bootstrap confidence intervals. (A) Dominant periods of continuous model variables estimated from Welch peak periods. Boundary probability and integration cost showed similar segment-scale periods, whereas memory cost varied over a longer scale and contextual surprisal over a much slower scale. (B) Discrete structural scales estimated from median inter-event intervals or relation spans. Relation openings, relation closures, and relation spans occurred at short local scales, whereas actual boundaries occurred at a longer segmental scale.
Figure 4: Processing costs and local memory history in human boundary judgments. (A) Adjusted boundary-choice probabilities across standardized memory cost after the current word, integration cost, and local semantic mismatch. Shading indicates 95% confidence intervals. (B) Odds ratios for memory cost at the preceding word and mean memory cost across the preceding three words. Error bars indicate 95% confidence intervals; the dashed line marks an odds ratio of 1.
Figure 5: Sentence-boundary-aligned profiles of human reading time, model-derived surprisal, and LLM measures. All measures were aligned to sentence boundaries in the naturalistic stories. The vertical dashed line marks the sentence boundary position. (A) Mean log reading-time residuals, including spillover correction. (B) Memory cost from dependency parsing. (C) Integration cost from dependency parsing. (D) Semantic mismatch increased at the boundary. (E) LLM surprisal. (F) Cosine distance between consecutive hidden states. (G) Local intrinsic dimensionality. (H) Trajectory curvature among tokens. (I) Averaged layer curvature. (J) Length-to-chord ratio of layer curvature.
Figure 6: (A) Constraint sensitivity across behavioral and LLM outcomes. Each panel shows the Wald z -statistics for memory cost, integration cost, and static semantic mismatch as predictors of one outcome. Outcomes included spillover-adjusted reading-time residuals, Qwen surprisal, cosine update, local intrinsic dimensionality, trajectory curvature, mean layer curvature, and layer length-to-chord ratio (LCR). Circles show models without explicit sentence-boundary control. Squares show models including sentence-boundary status as an additional categorical predictor. Stars indicate statistical significance corrected by False Discovery Rate. (B) Boundary-control gain. Points show ΔAIC , defined as AIC with boundary control minus AIC without boundary control. More negative values indicate better model fit after adding explicit sentence-boundary status.
Parameter
Value
Interpretation
k
1.2
Steepness of the pressure-to-boundary sigmoid
Nmaxopen
6
Maximum number of simultaneously open relations
Lmin
3
Minimum segment length
Lmax
99
Maximum segment length
Preset
0.0
Pressure value after a boundary reset
σϵ
0.05
Standard deviation of pressure noise
Table 1: Fixed architectural parameters used in the parameter scan.
Parameter
Candidate values
Interpretation
Applied family
θ
5.0, 8.0, 12.0
Boundary threshold; controls how much pressure is required before boundary drive increases.
All
ρ
0.3, 0.5, 0.8, 0.95
Pressure carry-over; controls how strongly pressure persists across events.
Weight of lexical surprisal in the pressure equation.
Surprisal-only
γsurp
0.02, 0.05, 0.1, 0.3, 0.6, 0.9
Non-zero lexical surprisal weight; ensures that lexical surprisal is active.
DLT-surprisal
Table 2: Tuned parameters used in the parameter scan.
Parameter
Value
Interpretation
Model family
DLT–surprisal
Combined structural and contextual pressure model
Combination ID
13777
Parameter combination selected by language-likeness score
bopen
0.50
Baseline tendency to open dependency-like relations
bclose
0.50
Baseline tendency to close dependency-like relations
amem
0.50
Weight of memory cost from currently open relations
bint
0.20
Weight of integration cost from resolved relations
Table 3: The best parameter regime.
Figure 7: Model-family differences in component language-likeness metrics. Model families were compared among the retained top 10% parameter regimes: DLT-only, surprisal-only, and combined DLT-surprisal models. DLT stands for dependency locality theory. Panels show group differences in dependency-distance exponential fit ( R2 ; A), dependency-distance power-law fit ( R2 ; B), dependency-distance decay score (C), lexical Zipf fit ( R2 ; D), deviation of the Zipf exponent from the ideal value of 1 (E), lexical Zipf score (F), quality-control pass rate (G), and the overall language-likeness score (H). Omnibus differences were tested using Kruskal–Wallis tests, followed by Dwass–Steel–Critchlow–Fligner pairwise comparisons. Significance brackets indicate pairwise comparisons: ns, not significant; * p<0.05 ; ** p<0.01 ; *** p<0.001 . Quality-control pass rate was constant across retained regimes and was therefore not tested.
Figure 8: Parameter sensitivity of language-likeness across model families among the retained top 10% parameter regimes. Mean language-likeness scores are shown as a function of individual simulation parameters, separately for DLT-only, surprisal-only, and combined DLT-plus-surprisal models. For each panel, scores were averaged over all other parameter settings within the scanned parameter space. Panels show (A) the boundary threshold θ , (B) pressure carry-over ρ , (C) context strength βctx , (D) baseline opening tendency, (E) baseline closure tendency, (F) total DLT strength amem+bint , (G) DLT balance amem−bint , (H) memory-cost weight amem , (I) integration-cost weight bint , (J) surprisal weight γ , and (K) effective surprisal strength γ×βctx . The analyses show that language-like organization was not restricted to a single parameter setting. Models containing DLT-inspired structural costs generally achieved higher language-likeness scores than surprisal-only models across a broad range of thresholds, carry-over values, opening/closure tendencies, and structural-cost weights. Contextual parameters showed a more nonlinear profile, with strongest performance at intermediate context strengths and reduced scores when surprisal pressure became too dominant.
Figure 9: Actual sentence boundaries occur at a characteristic but variable timescale. The plot shows the empirical distribution of token intervals between consecutive sentence boundaries. Most intervals clustered around short-to-intermediate spans, while the distribution showed a pronounced right tail, indicating that sentence lengths are variable rather than periodic. This empirical distribution was used as a reference for assessing whether emergent boundaries in the simulation occurred at naturalistic sentence-like intervals.
Figure 10: Pressure-dependent versus completion-only boundary formation. (A) Survival distributions of generated segment lengths, P(L≥x) , for the pressure-driven and completion-only models, calibrated to produce approximately matched mean segment lengths. (B–E) Boundary-aligned trajectories of accumulated processing pressure (B), memory cost (C), integration cost (D), and lexical surprisal (E). Position 0 denotes the final token before boundary emission, and position +1 denotes the first token of the following segment, with the vertical dashed line marking the boundary crossing. The pressure-driven model exhibited stronger pre-boundary pressure accumulation, memory-cost reduction, and integration-cost increases, while post-boundary trajectories converged across models. Blue solid lines represent the pressure-driven model, and orange dashed lines represent the completion-only control. Shading indicates mean ± 1.96 SEM across 500 random seeds.
Figure 11: Sensitivity of processing-cost effects to unit endings. Odds ratios and 95% confidence intervals for memory cost after the current word, integration cost, and semantic mismatch in the full sample and after excluding 300 top-level unit-ending positions. The dashed vertical line indicates an odds ratio of 1.
Figure 12: Sensitivity of processing-cost effects to grammatical coding. Odds ratios and 95% confidence intervals for the three processing-cost predictors under numeric and categorical representations of grammatical structure. The dashed vertical line indicates an odds ratio of 1.
Figure 13: Sentence-boundary sensitivity differs across human behavioral and LLM representational measures. For each outcome, the plot shows the z value associated with the sentence-boundary coefficient from an ordinary least-squares regression model controlling for pressure-derived predictors, word position, and item fixed effects, with cluster-robust standard errors at the sentence level. Lines extend from zero to the estimated boundary effect. Positive values indicate increased outcome values at sentence boundaries, whereas negative values indicate decreased outcome values at sentence boundaries. Reading-time residuals and LLM local intrinsic dimensionality increased at boundaries, while LLM surprisal, cosine update, trajectory curvature, and layer-curvature measures decreased at boundaries. Reported PFDR values are corrected across outcomes using False Discovery Rate (FDR).
We introduce STRUCTURALCOST, a self-paced reading dataset of 475 participants and 40,800 observations isolating the processing cost of long-distance subject-verb dependency resolution. We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance. Different language models -- spanning n-gram models, SSMs, and transformers -- partially mirror this graded difficulty profile, yet underestimate the integration cost humans incur, with a gap that persists across architectures and model sizes. This suggests these models capture the predictive component of human processing but not the full integration cost that working memory imposes. STRUCTURALCOST provides data needed to drive progress toward evaluating the cognitive plausibility of language models.
Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).
Elan Barenholtz
Machine Perception & Cognitive Robotics Laboratory Department of Psychology / Center for Complex Systems Florida Atlantic University
A recent study (Kuribayashi et al., 2025) has shown that human sentence processing behavior, typically measured on syntactically unchallenging constructions, can be effectively modeled using surprisal from early layers of large language models (LLMs). This raises the question of whether such advantages of internal layers extend to more syntactically challenging constructions, where surprisal has been reported to underestimate human cognitive effort. In this paper, we begin by exploring internal layers that better estimate human cognitive effort observed in syntactic ambiguity processing in English. Our experiments show that, in contrast to naturalistic reading, later layers better estimate such a cognitive effort, but still underestimate the human data. This dual alignment sheds light on different modes of sentence processing in humans and LMs: naturalistic reading employs a somewhat weak prediction akin to earlier layers of LMs, while syntactically challenging processing requires more fully-contextualized representations, better modeled by later layers of LMs. Motivated by these findings, we also explore several probability-update measures using shallow and deep layers of LMs, showing a complementary advantage to single-layer's surprisal in reading time modeling.
Tatsuki Kuribayashi, Alex Warstadt, Yohei Oseki +1