How Large Language Models Get Stuck: Early structure with persistent errors

Alokesh Manna; William Snyder; Whitney Tabor

cs.CLFeb 27, 2026

How Large Language Models Get Stuck: Early structure with persistent errors

Authors: Alokesh Manna, William Snyder, Whitney Tabor

Organizations: Department of Statistics University of Connecticut, Storrs, USA · Department of Linguistics University of Connecticut, Storrs, USA · Department of Psychological Sciences University of Connecticut, Storrs, USA

Abstract

Linguistic insights may help make Large Language Model (LLM) training more efficient. We trained Meta's OPT model on the 100M word BabyLM dataset, and evaluated it on the BLiMP benchmark, which consists of 67 classes, each defined by sentence pairs that differ in a targeted syntactic or semantic rule violation. We tested the model's preference for grammatical over ungrammatical sentences across training iterations and grammatical types. In nearly one-third of the BLiMP classes, OPT fails to consistently assign a higher likelihood to grammatical sentences, even after extensive training. When it fails, it often establishes a clear (erroneous) separation of the likelihoods at an early stage of processing and sustains this to the end of our training phase. We hypothesize that this mis-categorization is costly because it creates entrenched biases that must, eventually, be reversed in order for the model to perform well. We probe this phenomenon using a mixture of qualitative (based on linguistic theory and the theory of Deep Learning) and quantitative (based on numerical testing) assessments. Our qualitative assessments indicate that only some BLiMP tests are meaningful guides. We conclude by articulating a hypothesis, the Bigram Hypothesis, which claims that the learning process will exhibit erroneous entrenchment if bigram statistics bias the model toward wrong distinctions early in training, and we describe a method of testing the hypothesis on appropriately selected BLiMP classes.

Figures & tables

Figure 1: Accuracy ( 1 ) comparison among Blimp classes arranged by OPT performance.

Figure 2: Correlation plot of accuracy ( 1 ) for OPT and 5 other models, which demonstrate a positive association.

Figure 3: Erroneous early and sustained separation across checkpoints.

Figure 4: Correct early separation across checkpoints.

Figure 5: Late Correct Separation across checkpoints.

Pattern	n	Avg CUSUM	Avg Ruptures
CES	34	7,351	6,234
CLS	9	20,367	6,994
EES	24	5,542	6,798

Table 1: Summary of BLiMP change-point detection.

BLiMP case	Change point (CUSUM)	Change point (RUPTURES)	Correct (direction)	Temporal pattern
Early correct separation
adjunct_islandl	2300	4900	Yes	Early correct
animate_subject_passivel	1250	4900	Yes	Early correct
causativel	30000	8200	Yes	Early correct
coordinate_structure_constraint_object_extractionl	1250	6850	Yes	Early correct
determiner_noun_agreement_1l	20000	6850	Yes	Early correct

Table 2: BLiMP cases where the average perplexity over the last five checkpoints satisfies

\overline{\mathrm{PPL}}_{\mathrm{good}}<\overline{\mathrm{PPL}}_{\mathrm{bad}}

CASE	ANALYSIS
animate_subject_trans	Semantic constraint
complex_NP_island	Very weak separation
coordinate_structure_constraint _complex_left_branch	***Bigram: High prob of “What is/has/could/should” in Bad vs “What N” in Good
distractor_agreement_relative_clause	***Bigram: Local coherence (Bad) vs Long Distance Coherence (Good)
drop_argument	Arbitrary, distinct verb choices for Bad and Good—e.g., Good: many sentences use “buy”, “skate around”, “investigate”, “climb up”, “ascend”. Bad: many sentences use “listen to”, “talk to”, “escape from”, “boast about”. Also: some ambiguous Bad that are actually good on other readings.
existential_there_quantifiers_2	Good sentences are very semantically strange. Bad are often ambiguous and fine in their other meaning.

Table 3: Erroneous Early and Sustained (EES). “***Bigram" indicates that the Bigram hypothesis correctly predicts the model’s ordering of average perplexities for the group. “###Bigram" indicates that the Bigram hypothesis fails to predict the model’s behavior with the group.

CASE	ANALYSIS
animate_subject_passive	Separation is very slight
causative	Separation is not very early
passive_1	***Bigram: Good: BE + participle of transitive verb
passive_2	***Bigram: Same situation as passive_1 (difference follows from lexical frequencies)
transitive	***Bigram: Good: “V[Trans] NP”, Bad: “V[Intrans]”
principle_A_case_1	***Bigram: Good: that[Comp] + Nominative pronoun

Table 4: Correct Early and Sustained (CES). “***Bigram" indicates that the Bigram hypothesis correctly predicts the model’s ordering of average perplexities for the group. “###Bigram" indicates that the Bigram hypothesis fails to predict the model’s behavior with the group.

BLiMP case	Change point (CUSUM)	Change point (Ruptures)	Correct (direction)
Early Erroneous Separation
animate_subject_transitive	1250	3450	No
complex_NP_island	28000	9750	No
coordinate_structure_constraint_complex_left_branch	1250	6850	No
distractor_agreement_relative_clause	2300	4900	No
drop_argument	1250	4900	No

Table 5: BLiMP paradigms where the average perplexity over the last five checkpoints does not satisfy

\overline{\mathrm{PPL}}_{\mathrm{good}}<\overline{\mathrm{PPL}}_{\mathrm{bad}}

, grouped by temporal separation pattern.

Figure 6: Q (Query) matrix analysis across different heads

Figure 7: K (Key) matrix analysis across different heads

Figure 8: V (Vector) matrix analysis across different heads

Figure 9: O (output) matrix analysis across different heads

Figure 10: Q (Query) matrix cosine similarity analysis across different heads

Figure 11: K (Key) matrix cosine similarity analysis across different heads

Figure 12: V (Vector) matrix cosine similarity analysis across different heads

Figure 13: Q,K,V,O matrix cosine similarity analysis across the first layer

Figure 14: Ratio $\|W_{Q}\|_{F}/\|W_{K}\|_{F}$ per layer across training. All layers remain within 5% of unity throughout, indicating well-balanced attention logit scaling.

Figure 15: Spectral diagnostics for $W_{Q}$ : condition number $\sigma_{\max}/\sigma_{\min}$ , effective rank (99% energy threshold), largest singular value $\sigma_{\max}$ , and smallest singular value $\sigma_{\min}$ .

Figure 16: Spectral diagnostics for $W_{K}$ .

Figure 17: Spectral diagnostics for $W_{V}$ .

Figure 18: Spectral diagnostics for $W_{O}$ .

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 519 papers

Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?
Apr 20, 2026H S V N S Kowndinya Renduchintala, Sumit BhatiaLanguage Model Pretraining Language Modeling
Media and Data Science Research Adobe Inc., India
Do Language Models Encode Knowledge of Linguistic Constraint Violations?
May 12, 2026Hardy, Sebastian PadóLLM Interpretability Mechanistic Interpretability
IMS, University of Stuttgart, Stuttgart, Germany · Universitas Mikroskil, Medan, Indonesia.
Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models
Feb 24, 2026Paola Merlo, Chunyang Jiang, Giuseppe Samo +1LLM Evaluation LLM Interpretability
Idiap Research Institute · University of Geneva

Apr 20, 2026J/K move · Enter open · S save

Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

H S V N S Kowndinya Renduchintala, Sumit Bhatia

Media and Data Science Research Adobe Inc., India

Language Model Pretraining Language Modeling

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit a puzzling disparity in their formal linguistic competence: while they learn some linguistic phenomena with near-perfect mastery, they often perform below chance on others, even after training on trillions of tokens. In this work, we investigate whether these failures stem from inherent architectural limitations or simply the scarcity of these specific grammatical constructions in web-scale corpora. We pre-train simple GPT-2 Small (124M) models on a 100M-token random sample of the FineWeb corpus and intervene by injecting a minimal amount (1%) of synthetic data targeting specific linguistic phenomena. We find that this targeted intervention substantially improves model performance in 8 out of the 9 worst-performing BLiMP paradigms - notably the accuracy on a specific paradigm, only_npi_scope, surges from 20.9% to 69.4%. Furthermore, we observe that these interventions generally preserve or slightly improve aggregate performance. However, while we also identify a resistant phenomenon, principle_A_c_command, whose performance remains below chance even after our data augmentation, our findings do serve as an optimistic existence proof that even small language models can substantially improve on those linguistic phenomena on which models typically perform poorly, provided the pre-training data contains sufficient exposure to them. This suggests that efforts towards human-scale language modeling may benefit greatly by focusing on data composition. The code to reproduce our results is open-sourced at https://github.com/kowndinya-renduchintala/heterogeneity-in-formal-linguistic-competence.

Abstract

Language Modeling

Explore similar work

Heterogeneity in Formal Linguistic Competence of Language Models: Is Data the Real Bottleneck?

Do Language Models Encode Knowledge of Linguistic Constraint Violations?

Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models