For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.
Figures & tables
Figure 1: The diagram contrasts Wh-Questions (left, green) and Topicalization structures (right, orange).
Figure 2: To create a DAS vector, we learn a direction a to align neural representations with the binary variable FILLER_PRESENT . We intervene by projecting the difference between the source and base representations onto a and injecting it into the base sentence.
Figure 3: Developmental trajectory of all filler-gap mechanisms across training. Error bands show ± 1 SE across a minimum of 6 seeds. All four conditions show a monotonic increase with training tokens.
Contrast
Est.
SE
t
p
d
Within–Across
1.33
0.08
17.4
<.001
0.52
Wh→Topic–Topic→Wh
-0.57
0.11
-5.3
<.001
-0.22
Animate–Inanimate
0.68
0.08
8.6
<.001
0.26
Table 1: Post-hoc comparisons from linear model. Negative asymmetry results suggest Topic → Wh outperformed Wh → Topic.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Hyperparameter sweep for DAS training. max odds increases with training samples and stabilizes around 2000 samples (batch size 25 × 80 steps).
Figure 5: max odds as a function of total training samples, collapsed across batch sizes. Performance plateaus around 2000–2500 samples.
Figure 6: Developmental trajectory of lexical boost across training. Error bands show ± 1 SE across a minimum of 2 seeds.
Figure 7: Full developmental trajectory from 1M to 1000M tokens. Filler-gap mechanisms continue to improve but begin to plateau around 500M–700M tokens.
Figure 8: Transfer asymmetry across full training range. The asymmetry (Topic → Wh > Wh → Topic) persists and slightly increases at later checkpoints.
Children's acquisition of filler-gap dependencies has been argued by some to depend on innate grammatical knowledge, while others suggest that the distributional evidence available in child-directed speech suffices. Unfortunately, the relevant input is difficult to quantify at scale with fine granularity, making this question difficult to resolve. We present a system that identifies three core filler-gap constructions in spoken English corpora -- matrix wh-questions, embedded wh-questions, and relative clauses -- and further identifies the extraction site (i.e., subject vs. object vs. adjunct). Our approach combines constituency and dependency parsing, leveraging their complementary strengths for construction classification and extraction site identification. We validate the system on human-annotated data and find that it scores well across most categories. Applying the system to 57 English CHILDES corpora, we are able to characterize children's filler-gap input and their filler-gap production trajectories over the course of development, including construction-specific frequencies and extraction-site asymmetries. The resulting fine-grained labels enable future work in both acquisition and computational studies, which we demonstrate with a case study using filtered corpus training with language models.
Zhenghao Herbert Zhou, William Dai, Maya Viswanathan +3
Department of Linguistics, Yale University · Department of Computer Science, Yale University · Wu Tsai Institute, Yale University
Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim memorization remains underexplored. We study the memorization dynamics of FIM in a controlled setting by pretraining matched Llama 3.2 models with FIM and standard left-to-right (LTR) objectives on a FineWeb-Gutenberg corpus containing repeated Gutenberg excerpts. With prefix-based probes, FIM more often recovers short or partially matching spans, while LTR more often assigns high confidence to long exact continuations. We observe that verbatim extraction under FIM-training grows approximately linearly with repetitions over the tested range. Evaluating native FIM-format probes reveals that suffix context is not sufficient: verbatim recall under FIM-training remains strongly anchored in prefix context. Our results also show that evaluating only one span length or probing format can miss important nuances in memorization behavior.
Tobias von Arx, Tanguy Dieudonné
Department of Computer Science, ETH Zurich, Zurich, Switzerland.
Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.