Humans and transformer LMs: Abstraction drives language learning

Jasper Jian; Christopher D. Manning

cs.CLMar 18, 2026

Humans and transformer LMs: Abstraction drives language learning

Authors: Jasper Jian, Christopher D. Manning

Organizations: Stanford University

Abstract

Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.

Figures & tables

Figure 1: (a) Categories, like verb classes determined by argument structure, are pervasive in language: to Dative verbs take recipient arguments, whereas Motion verbs take goal locations. (b) Divergence metrics compare LM prediction distributions conditioned on different categories. (c) Tracking divergences over training reveals that categories like verb classes are differentiated early by LMs, before gradual item-level learning.

Class	Example Verbs	Example Sentences	Total
(i) to Dative	give , sell , grant ( n =35)	Chipotle gave away free burritos to the __	1841
(ii) Motion	go , walk , flee ( n =36)	Toby accordingly goes to the __	1942
(iii) Reciprocal	speak , meet , talk ( n =16)	Cherry recalled Orr had refused to speak with the __	465
(iv) spray-load	spray , load , smear ( n =16)	Ray and Devon then sprayed the table with the __	404

Table 1: Example verbs and sentence prefixes from the four verb classes used in the experiments.

Figure 2: Pairwise $D_{JS}$ between verbs in two classes at three early steps in training. Each row and column represents a single verb. Differentiation into verb classes is evident when any change has occurred – between-class quadrants have higher $D_{JS}$ (top-right, bottom-left) than in-class quadrants (top-left, bottom-right).

Figure 3: Mann-Whitney U-test between $U_{A}=\{D_{JS}(P_{v_{t}}(x)||P_{v}(x)):v,v_{t}\in C_{1}\}$ (same class), $U_{B}=\{D_{JS}(P_{v_{t}}(x)||P_{v}(x)):v_{t}\in C_{1},v\in C_{2}\}$ (different class), for verb classes $C_{1},C_{2}$ . Mean of $D_{JS}$ across verbs in the same class. Shading is standard deviation.

Figure 4: Mean pairwise Spearman correlations between prototype noun trajectories for each verb across training. Within-class and between-class means are reported with 95% CI.

Panel of Figure 5 — Figure 5: Top: Pairwise $D_{JS}$ across two classes in exemplar-first baseline show verb-specific patterns emerge without clear class distinctions. Middle and bottom: Exemplar-first baseline (middle) shows increasing divergence between verbs driven by verb-specific learning, before decrease, while GPT-2 is monotonic (bottom); baseline between-class mean divergences are not consistently greater than in-class means, unlike GPT-2. Shading is 95% CI.

Structure	Example Sentences
Transitivity ( n =1000)	The truck hadn’t astounded __
	The truck hadn’t fallen __
relative clause ( to Dative; n =824)	The person that they sold the land to __
	The person thinks that they sold the land to __
relative clause (Reciprocal; n =288)	The person that Optimus spoke with __
	The person thinks that Optimus spoke with __

Table 2: Example structures for Experiment 2. Transitivity is from BLiMP ( Warstadt et al., 2020 ) .

Figure 6: Comparing class-level and item-based behaviour for Transitivity dataset.

Figure 7: Comparing class-level and item-based behaviour for Relative Clause datasets.

Figure 8: Learning curves for each abstraction plotted for three models trained with different random seeds.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Comparing class-level and item-based behaviour on Relative Clause datasets with varied embedding verbs.

LLM Training

Large language model training studies the data, objectives, optimizers, and systems used to build and adapt language models.

See all 342 papers

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning
Aug 1, 2026Zachary Nicholas Houghton, Vsevolod KapatsinskiMemorization in Language Models Language Modeling
University of Oregon · Vail Systems, Inc.
Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns
Jun 25, 2026Wang Bojun, Holly Jenkins, Elizabeth WonnacottAutoregressive Language Modeling Computational Psycholinguistics
Department of Education, University of Oxford
Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorization
Sep 26, 2026Wang Bojun, Holly Jenkins, Elizabeth WonnacottMemorization in Language Models Language Modeling

Aug 1, 2026J/K move · Enter open · S save

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Zachary Nicholas Houghton, Vsevolod Kapatsinski

University of Oregon · Vail Systems, Inc.

Memorization in Language Models Language Modeling

Open paper AskPDF arXiv

Whether idiosyncratic, item-specific knowledge is learned before abstract class-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction-based theories making opposite predictions. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item-specific or class-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input. We further argue that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations, as a word's class-level properties may not be separable from its item-specific properties.

Abstract

LLM Training

Explore similar work

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns

Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorization