cs.LGOct 13, 2026

Tracing Morphology, Semantics, and Divergence in Language Acquisition

Authors: Forough Shirin Abkenar, Haoshuo Fu, Liang Feng, Ning Yan, Masood S. Mortazavi

Organizations: Futurewei Technologies, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.

Figures & tables

Figure 1: Without a closing token, target probabilities shift due to valid alternate continuations after the presumably unfavorable inflection.
Figure 2: Morphological inflections across time for Pythia-70M (top), Pythia-12B (middle), and OLMo3-Stage1 (bottom). The horizontal axis represents the number and percentage of total training tokens elapsed in the training steps; the vertical axis shows the log scale of target words’ probabilities.
Figure 3: Morphological inflections in Pythia suite and OLMo3 w.r.t. all unit-test cases. Shadows represent the standard deviation of results.
Figure 4: Semantic embedding stability for Pythia-70M (top), Pythia-12B (middle), and OLMo3-Stage1 (bottom): left (Verbs), middle (Nouns), and right (Pronouns).
Test TypePrompt Format
rDAT“““Pick NtwN_{tw} very different words from the list below: {list of 10 words}:”””
RAT“““Given three remote stimulus words “word_1”, “word_2”, “word_3”, name the single word that connects all three.”””
Table 1: Prompt formats for creativity tests, rDAT and RAT.
Figure 5: Our rDAT results across time for Pythia-70M, Pythia-12B, and OLMo3with a reference 10-word list “male, female, cat, dog, chalk, cheese, fish, bird, animal, vegetable”.
Figure 6: Performance of Pythia and OLMo3: DR and MR for top-10 triplets of 16 lists in rDAT test.
Figure 7: RAT results across time for Pythia-70M (top), Pythia-12B (middle), and OLMo3 (bottom): (a) Easy; (b) Medium; and (c) Hard cases.
Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

ModelSizeLayersHidden DimensionHeads
Pythia70M65128
160M1276812
410M24102416
1B1620488
6.9B32409632
12B36512040
Appendix
Table A.1 : Architectures of under-study LLMs
Prompt TypeVerbsNounsPronouns
Incipit PromptTarget WordIncipit PromptTarget WordIncipit PromptTarget Word
FavorableIam␣I lovebooks.Every Wednesday, he visitsher.
Unfavorableare␣book.she.
FavorableYouare␣I eatbananas.This book ismine.
Unfavorableis␣banana.my.
FavorableI amgoing␣He wearsboots.His father liked to talk tohim.
Appendix
Table D.1 : List of unit-test cases in various lexical categories for some common morphological inflections. Here, ␣ shows a space character.
Figure D.1 : Morphological inflections in GPT-2 and OLMo2variants w.r.t. all unit-test cases.
Figure D.2 : Morphological structures across time for Pythia suite: 160M, 410M, 1B, and 6.9B—first to fourth rows, respectively.
Figure D.3 : Morphological structures across time for OLMo2 (top) and OLMo2-Instruct (bottom).
Figure D.4 : Morphological structures across time for GPT-2 variations: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively.
Figure D.5 : Training corpus strongly affects morphological structure.
Figure D.6 : Semantic embedding stability for Pythia suite: 160M, 410M, 1B, and 6.9B—first to fourth rows, respectively; left (Verbs), middle (Nouns), and right (Pronouns).
Figure D.7 : Semantic embedding stability for OLMo2 and OLMo2-Instruct: left (Verbs), middle (Nouns), and right (Pronouns).
Figure D.8 : Semantic embedding stability for GPT-2 variations: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively; left (Verbs), middle (Nouns), and right (Pronouns).
Figure D.9 : Rank evolution of of words in The Pile dataset over checkpoints.
Figure D.10 : Performance of Pythia suite and OLMo3: baseline and maturity likelihoods for 16 lists in rDAT test—top row (a-c), top-1 triplets; bottom row (d-f), top-5 triplets.
Figure D.11 : Performance of Pythia suite, GPT-2 variations, OLMo2 variations, and OLMo3: baseline and maturity likelihoods for top-10 triplets of 32 lists in rDAT test.
Figure D.12 : RAT results across time for Pythia suit: 160M, 410M, 1B, and 6.9B—first to fourth rows, respectively; (a) Easy; (b) Medium; and (c) Hard cases.
Figure D.13 : RAT results across time for GPT-2 seeds: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively; (a) Easy; (b) Medium; and (c) Hard cases.
Figure D.14 : RAT results across time for OLMo2 (top row) and OLMo2-Instruct (bottom row): (a) Easy; (b) Medium; and (c) Hard cases.

Explore similar work

CardsList

  1. What do Language Models Learn and When? The Implicit Curriculum Hypothesis

    Apr 9, 2026Emmy Liu, Kaiser Sun, Millicent Li +4Language Model Pretraining

    Language Technologies Institute, Carnegie Mellon University · Department of Computer Science, Data Science and AI Institute, Johns Hopkins University · Khoury College of Computer Science, Northeastern University +1

  2. Language Acquisition Device in Large Language Models

    May 16, 2026Masato Mita, Taiga Someya, Ryo Yoshida +1Language Model PretrainingSynthetic Data Pretraining

    The University of Tokyo

  3. Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

    Aug 4, 2026Jo-Ku Cheng, Nikolaos Aletras, Marco ValentinoLanguage Model PretrainingRepresentation Geometry in Language Models

    School of Computer Science, University of Sheffield, United Kingdom