Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.
Figures & tables
Figure 1: (a) Categories, like verb classes determined by argument structure, are pervasive in language: to Dative verbs take recipient arguments, whereas Motion verbs take goal locations. (b) Divergence metrics compare LM prediction distributions conditioned on different categories. (c) Tracking divergences over training reveals that categories like verb classes are differentiated early by LMs, before gradual item-level learning.
Class
Example Verbs
Example Sentences
Total
(i) to Dative
give , sell , grant ( n =35)
Chipotle gave away free burritos to the __
1841
(ii) Motion
go , walk , flee ( n =36)
Toby accordingly goes to the __
1942
(iii) Reciprocal
speak , meet , talk ( n =16)
Cherry recalled Orr had refused to speak with the __
465
(iv) spray-load
spray , load , smear ( n =16)
Ray and Devon then sprayed the table with the __
404
Table 1: Example verbs and sentence prefixes from the four verb classes used in the experiments.
Figure 2: Pairwise DJS between verbs in two classes at three early steps in training. Each row and column represents a single verb. Differentiation into verb classes is evident when any change has occurred – between-class quadrants have higher DJS (top-right, bottom-left) than in-class quadrants (top-left, bottom-right).
Figure 3: Mann-Whitney U-test between UA={DJS(Pvt(x)∣∣Pv(x)):v,vt∈C1} (same class), UB={DJS(Pvt(x)∣∣Pv(x)):vt∈C1,v∈C2} (different class), for verb classes C1,C2 . Mean of DJS across verbs in the same class. Shading is standard deviation.
Figure 4: Mean pairwise Spearman correlations between prototype noun trajectories for each verb across training. Within-class and between-class means are reported with 95% CI.
Figure 5: Top: Pairwise DJS across two classes in exemplar-first baseline show verb-specific patterns emerge without clear class distinctions. Middle and bottom: Exemplar-first baseline (middle) shows increasing divergence between verbs driven by verb-specific learning, before decrease, while GPT-2 is monotonic (bottom); baseline between-class mean divergences are not consistently greater than in-class means, unlike GPT-2. Shading is 95% CI.
Structure
Example Sentences
Transitivity ( n =1000)
The truck hadn’t astounded __
The truck hadn’t fallen __
relative clause ( to Dative; n =824)
The person that they sold the land to __
The person thinks that they sold the land to __
relative clause (Reciprocal; n =288)
The person that Optimus spoke with __
The person thinks that Optimus spoke with __
Table 2: Example structures for Experiment 2. Transitivity is from BLiMP ( Warstadt et al., 2020 ) .
Figure 6: Comparing class-level and item-based behaviour for Transitivity dataset.
Figure 7: Comparing class-level and item-based behaviour for Relative Clause datasets.
Figure 8: Learning curves for each abstraction plotted for three models trained with different random seeds.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Comparing class-level and item-based behaviour on Relative Clause datasets with varied embedding verbs.
Whether idiosyncratic, item-specific knowledge is learned before abstract class-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction-based theories making opposite predictions. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item-specific or class-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input. We further argue that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations, as a word's class-level properties may not be separable from its item-specific properties.
In this study, we use a developmental approach to investigate the statistical learning and mental representation of neural language models (NLM). A series of Generative Transformer models are trained on a synthetic grammar. The model states are saved at multiple stages in the course of training. Through analyzing how the internal representations of these models change in the developmental path, we found that NLMs acquire the most abstract global statistical knowledge at the beginning of learning and later acquire the relatively local statistical dependencies. This learning path contains many over-generalizations from the very beginning and these over-generalizations are gradually constrained in the later stage of learning. Based on this observation, we propose a new framework to explain the statistical learning and language cognition of NLMs.
We investigate how generalization and memorization develop along the learning trajectory of neural language models. Using controlled synthetic grammars, we examine both the geometry of representation space and model behaviour over training. We find that continuous regions of representation space not occupied by observed tokens become systematically structured from the earliest stages of learning, forming category-level geometric organization that supports generalization to unattested combinations. Generalization therefore emerges from the beginning of learning rather than only after extensive memorization. With prolonged training, larger models increasingly distinguish observed from unobserved grammatical combinations. At the same time, the continuous geometric structure supporting category-level generalization is gradually destructed. Together, these results suggest that neural language models initially learn through categorization-based generalization, followed by a gradual transition toward more exemplar-specific memorization.