cs.CLOct 27, 2026

Language Models Keep Abandoned Analyses: Residue and Adaptation after Garden-Path Sentences

Authors: Xiang Ding

Organizations: Nagoya University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reading "While the man hunted the deer ran into the woods" requires giving up the analysis in which "the deer" is the object of "hunted". People who have just read such a sentence still produce more transitive structures, so the abandoned analysis primes later language. We ask whether language models keep using an analysis they have abandoned, and what in the model carries it. In 24 open models from 124M to 70B parameters, a garden-path sentence leaves two traces in the context. A residue of the abandoned analysis raises later object attachment, by about three quarters of the effect of a sentence in which the noun phrase really is an object, and it appears in sampled text, on held-out and human-normed items and for a second type of garden path. An adaptation toward the resolved structure scales with the sentence's disambiguation error and acts mainly on comprehension-like measures. Activation patching separates the two. Later text reads the residue from the role information in the noun phrase's states, which were computed under the object analysis before the disambiguation, and replacing these states removes almost all of the residue in the 32B and 70B models. The adaptation is read from the states that processed the disambiguation. Overridden statements show the same persistence (a corrected value keeps pulling later predictions in 8 of 9 models), and in most models the pull is carried by the positions after the statement. Finally, the human paradigm transfers only with controls for punctuation, because the models also write garden-path sentences themselves, which reverses the paradigm's raw statistic.

Figures & tables

Figure 1: Two traces of a garden-path sentence in a causal language model. Later text reads a residue of the abandoned object analysis from the states of the deer , which were computed before the disambiguation (§ 6 ). The main-clause states carry an adaptation toward the resolved structure. The two traces push later processing in opposite directions, and their balance differs between production-like and comprehension-like measures (§ 5 ).
PrimeExample
gpWhile the man hunted the deer ran into the woods.
cueWhile the man hunted, the deer ran into the woods.
tbWhile the man hunted the deer it ran into the woods.
nb / ncWhile the man slept(,) the deer ran into the woods.
nomWhile the man hunted they ran into the woods.
passWhile the man was hunted the deer ran into the woods.
Table 1: Prime conditions and target quantities, illustrated with one prime and one target family. nom and pass test punctuation and local coherence (§ 4 ).
Figure 2: The residue of the abandoned object analysis. Left: ALA_{L} (Eq. 2 ) per model on the discovery, held-out and Christianson et al. (2001) item sets. Right: the same contrast on sampled completions, as a difference in the proportion of transitive continuations. Error bars are 95% intervals from a two-way bootstrap over prime and target families. Models are ordered by size.
Figure 3: Patching the garden-path prime. Left: residue on LL retained when the target reads counterfactual subject-analysis states at the noun phrase (blue) or at the main clause (orange). Right: noun-phrase states replaced only from a given fraction of the network’s depth onward. Values for both measures, with intervals, are in Appendix E .
Figure 4: Sampled production after garden-path primes. The raw contrast with comma primes used in the human study (orange) is negative in most Llama and Qwen models. Without the completions that are themselves comma-less garden-path sentences it is positive in every model (gray), and so is the controlled residue AgenA_{gen} (blue).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Item-level predictors of the residue on the discovery set. Each point is the tt value of one predictor in a regression of the family-level contrast AA on the model’s object expectation after the ambiguous verb (blue) and on the garden-path strength of the prime, the surprisal of its disambiguating word without versus with the comma (orange). Dotted lines mark ∣t∣=1.96|t|=1.96 . Object expectation raises AA on both measures. Garden-path strength lowers it on the comprehension-like measure SS in 21 of 22 models.
ExperimentPre-specified predictionOutcome
Held-out families(i) object expectation predicts the residue on SS (ii) error predicts it with the sign found in discovery (iii) sign of the net residue(i) 7/17 (not confirmed) (ii) 11/17, same sign in 16/17 (sign test p=.0001p=.0001 ) (iii) 5/9
Within-family errormore surprising disambiguation lowers the residue in the models with a discovery error effectconfirmed in 3/3, and null in the 2 models predicted null
NP/Sthe two forces generalize to NP/S garden pathsnot confirmed (1/5 and 0/4)
MV/RRerror and commitment relations in main-verb/reduced-relative garden paths(i) error 6/9 (ii) commitment 0/8
Case ( nom )(i) object expectation predicts the residue without punctuation (ii) error relation negative (iii) comma part of the predictor vanishes(i) 22/22 (ii) 20/22 (iii) not confirmed (reinterpreted as part of object expectation)
Measures(i) residue on LL positive (ii) error relation stronger on SS than on LL(i) 16/16 (ii) 14/16
Appendix
Table 4: Pre-specified predictions and outcomes. Counts are models in which the prediction held. Outcomes that did not confirm the prediction are in italics.
ModelALA_{L}ρ\rho , NPρ\rho , MCΔAS\Delta A_{S} , NPΔAS\Delta A_{S} , MC
Pythia-1.4B0.46 (0.33, 0.59)0.65 (0.56, 0.73)0.85 (0.76, 0.92)−- 0.26 ( −- 0.30, −- 0.22)0.22 (0.11, 0.34)
Pythia-2.8B0.49 (0.34, 0.65)0.50 (0.41, 0.59)0.74 (0.64, 0.83)−- 0.35 ( −- 0.42, −- 0.29)0.40 (0.25, 0.55)
Llama-3.2-3B0.82 (0.62, 1.03)0.39 (0.30, 0.48)0.83 (0.72, 0.95)−- 0.51 ( −- 0.60, −- 0.42)1.12 (0.92, 1.33)
Mistral-7B0.86 (0.64, 1.09)0.40 (0.28, 0.49)0.89 (0.74, 1.10)−- 0.55 ( −- 0.66, −- 0.46)0.93 (0.74, 1.11)
Qwen2.5-7B0.90 (0.66, 1.16)0.14 ( −- 0.05, 0.27)1.34 (1.12, 1.71)−- 0.84 ( −- 0.99, −- 0.71)1.13 (0.93, 1.33)
Llama-3.1-8B-Inst.1.09 (0.87, 1.36)0.28 (0.17, 0.38)1.03 (0.91, 1.17)−- 0.83 ( −- 0.96, −- 0.71)1.33 (1.11, 1.55)
Appendix
Table 5: Patching results with 95% two-way bootstrap intervals. ρ\rho : retained fraction of the residue on LL . ΔAS\Delta A_{S} : change of the residue on SS in nats. NP and MC: the target reads counterfactual states at the noun phrase or at the main clause.
More in this topic

Syntactic Generalization in Language Models

See all 18 papers

Explore similar work

CardsList
Feb 4, 2026

LLM surprisal is necessary but not sufficient to capture English garden-path effects: Evidence from joint latent modeling of reading paradigms

Temporarily ambiguous garden-path sentences ("While the team trained the striker wondered... ") are known to cause processing difficulty, which can manifest itself in a variety of reading behaviors (in-situ slowdowns, rereading), as well as in miscomprehension or outright rejection of the sentence as ungrammatical. Which types of reading behavior are observed critically depends on the experimental method used to collect the data, which makes comparing results between reading paradigms difficult. To address this problem, we present a latent-process multinomial processing tree (MPT) model of human reading and comprehension/judgment behavior in garden-path sentences that we fit to combined data from four different reading paradigms (eye tracking, uni- and bidirectional self-paced reading, Maze). The model distinguishes between the probability of adopting an incorrect initial analysis, the cost of encountering an incompatible continuation, and the cost of syntactic reanalysis. By taking into account trials with inattentive reading, more realistic estimates of the cost parameters are obtained. Cross-validation reveals that the MPT model has a better predictive fit to human reading patterns and end-of-trial task data than a model based solely on LLM-derived surprisal values. We also test several models that assume an influence of surprisal within the MPT architecture, and find that adding surprisal as an additional predictor or reading time and/or garden-path cost further improves predictive fit.

Dario Paape, Tal Linzen, Shravan Vasishth

Department of Linguistics, University of Potsdam · Department Linguistik, Universität Potsdam, Karl-Liebknecht-Straße 24–25, D-14476, Germany. · Center for Data Science/Department of Linguistics, New York University

Latent Variable ModelsCognitive ModelingComputational Psycholinguistics
AskPDF
May 14, 2026

Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis

Surprisal theory posits that the processing difficulty of a word is determined by its predictability in context, offering a potential link between human sentence processing and next-word predictions from language models. While language model (LM) surprisals successfully predict reading times in naturalistic text, they systematically underpredict the magnitude of difficulty observed in controlled studies of syntactic ambiguity, particularly in garden path sentences. This mismatch might arise from differences in the computational constraints between humans and LMs. Here we test one such hypothesis, specifically, that LMs may be able to simultaneously consider a greater number of distinct sentence interpretations at once, compared to humans. Using Recurrent Neural Network Grammars (RNNGs) with word-synchronous beam search, we systematically vary the number of simultaneous parses used to compute word surprisal, and then use these surprisals to predict human reading times. Reducing the number of simultaneous active parses indeed increases the magnitude of predicted garden path effects, but not nearly enough to capture the full magnitude of the effects in humans. This suggests that differences in the number of simultaneous parses available to LMs and humans cannot reconcile LM-based surprisal with human sentence processing.

William Timkey, Brian Dillon, Tal Linzen

Department of Linguistics, New York University, 10 Washington Pl, New York, NY 10003, USA · Department of Linguistics, University of Massachusetts Amherst, 650 N Pleasant St, Amherst, MA 01002, USA · Center for Data Science, New York University, 60 5th Ave, New York, NY 10011, USA

Surprisal TheoryLanguage ModelingComputational Psycholinguistics
AskPDF
Sep 30, 2026

Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction

Transformer-based large language models (LLMs) such as RoBERTa represent text using contextual word embeddings (CWEs), which alter the embeddings associated with each token based on surrounding context. We construct token-wise incremental trajectories by repeatedly recomputing a token's CWE as successive words are added to a sentence, yielding a representation of how contextualized embeddings evolve as the utterance unfolds. We evaluate this approach using garden-path sentences as a test case with characteristic features. Token-wise trajectories reproduce known features of garden-path processing, including disruption around the critical region, and reliably distinguish garden-path sentences from matched disambiguated controls. We introduce several metrics for quantifying representational displacement across contextual increments and show that trajectory information can be highly predictive of sentence type. We find that ambiguity-related information is recoverable not only from the sentence-level CLS representation but also from ordinary vocabulary tokens, suggesting that utterance-level information is distributed across multiple representational scales. In exploratory analyses, we find qualitatively similar trajectory structures in other ambiguity- and misdirection-related linguistic phenomena. Together, these results establish token-wise incremental trajectories as a promising framework for studying utterance-specific meaning construction using LLMs.

Grayson Wycliffe Storer, Julia Witte Zimmerman

Department of Mathematics and Statistics. · Department of Computer Science, University of Vermont, Burlington, 05405, Vermont, United States. · Computational Story Lab. +1

Text EmbeddingsLanguage ModelingRepresentation Geometry in Language Models
AskPDF