cs.CLApr 15, 2026

Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?

Authors: Atrey Desai, Sathvik Nair

Organizations: University of Maryland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.

Figures & tables

Figure 1: The diagram contrasts Wh-Questions (left, green) and Topicalization structures (right, orange).
Figure 2: To create a DAS vector, we learn a direction a\mathbf{a} to align neural representations with the binary variable FILLER_PRESENT . We intervene by projecting the difference between the source and base representations onto a\mathbf{a} and injecting it into the base sentence.
Figure 3: Developmental trajectory of all filler-gap mechanisms across training. Error bands show ±\pm 1 SE across a minimum of 6 seeds. All four conditions show a monotonic increase with training tokens.
ContrastEst.SEttppdd
Within–Across1.330.0817.4<.0010.52
Wh→Topic–Topic→Wh-0.570.11-5.3<.001-0.22
Animate–Inanimate0.680.088.6<.0010.26
Table 1: Post-hoc comparisons from linear model. Negative asymmetry results suggest Topic →\rightarrow Wh outperformed Wh →\rightarrow Topic.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Hyperparameter sweep for DAS training. max odds increases with training samples and stabilizes around 2000 samples (batch size 25 ×\times 80 steps).
Figure 5: max odds as a function of total training samples, collapsed across batch sizes. Performance plateaus around 2000–2500 samples.
Figure 6: Developmental trajectory of lexical boost across training. Error bands show ±\pm 1 SE across a minimum of 2 seeds.
Figure 7: Full developmental trajectory from 1M to 1000M tokens. Filler-gap mechanisms continue to improve but begin to plateau around 500M–700M tokens.
Figure 8: Transfer asymmetry across full training range. The asymmetry (Topic →\rightarrow Wh >> Wh →\rightarrow Topic) persists and slightly increases at later checkpoints.
More in this topic

Syntactic Generalization in Language Models

See all 13 papers

Explore similar work

CardsList

  1. What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies

    Mar 2, 2026Zhenghao Herbert Zhou, William Dai, Maya Viswanathan +3Constituency ParsingChild Language Acquisition

    Department of Linguistics, Yale University · Department of Computer Science, Yale University · Wu Tsai Institute, Yale University

  2. Memorization Dynamics of Fill-in-the-Middle Pretraining

    May 21, 2026Tobias von Arx, Tanguy DieudonnéLanguage Model PretrainingMemorization in Language Models

    Department of Computer Science, ETH Zurich, Zurich, Switzerland.

  3. Tracing Morphology, Semantics, and Divergence in Language Acquisition

    Oct 13, 2026Forough Shirin Abkenar, Haoshuo Fu, Liang Feng +2Language Modeling

    Futurewei Technologies, USA