cs.CLOct 27, 2026

Specialization, Diversity, and Scaling in Child-Scale Language Models

Authors: Steven Y. Feng, Michael C. Frank

Organizations: Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child's language environment predict that environment substantially better than other children's environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child's input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.

Figures & tables

Figure 1: BabyView examples and two central findings. (A) A representative BabyView egocentric frame and a separate excerpt of automatically transcribed child–adult conversation. The egocentric frame and transcript excerpt come from separate BabyView recordings and are presented as illustrative examples. (B) Models specialize to local environments, and a 5-gram reproduces much of the cross-child pattern ( r=.91r=.91 after excluding same-child comparisons). (C) With 100k training words, more contributing children improve prediction of pooled held-out data; Zorro (grammaticality) shows no clear trend. The dashed diagonal in (B) indicates equality between the adjusted model scores; the horizontal dashed line in (C) indicates theoretical chance performance on Zorro.
Figure 2: Specialization is concentrated in rare vocabulary and remains detectable months later. (A) Words unseen in the source child’s 100k-word training corpus account for 94.8% of the same-environment advantage while comprising 6.2% of evaluated words. (B) The advantage remains on later recordings but becomes weaker with time.
Figure 3: At a fixed word budget, language from more children improves pooled held-out prediction more clearly than grammatical-benchmark performance. Each point is one 100k-word training dataset (mean across three model seeds); diamonds show the mean for each number of contributing children. (A) Pooled held-out loss decreases as the training set has more children (slope −0.124-0.124 BPW per ln⁡k\ln k ; permutation p<.001p<.001 ). (B) Zorro shows no clear monotonic trend at the same budget ( p=.28p=.28 ). The dashed line is theoretical 50% chance on Zorro.
Figure 4: Scaling of Zorro and WordSim performance (GPT-2 small). (A) Zorro accuracy (50% is chance; color-matched dashed lines show random-initialization baselines: 48.4% for BabyView and 52.0% for age-2 TinyDialogues). (B) Mean WordSim Spearman correlation across five datasets (the same 7,917 word pairs for both corpora). Lines connect the shared 25k–3.2M-word conditions (three corpus draws ×\times three seeds each), with pointwise 95% two-stage bootstrap intervals. Unconnected hollow markers show 4.67M-word results: full BabyView (one draw, three seeds; seed-only range) and size-matched age-2 TinyDialogues (three draws; bootstrap interval). Both use 20 training epochs and validation-selected checkpoints. Appendix H shows GPT-2 micro and larger age-2-only conditions; Appendix J mixed-age comparisons; Appendix I.1 the common-word-pair analysis.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus/modelFixed setupTuned setup
BabyView micro800k–1.6M400k–800k
BabyView small800k–1.6M400k–800k
Age-2 TD micro400k–800k200k–400k
Age-2 TD small200k–400k200k–400k
Appendix
Table 1: Vocabulary-aligned KN-5/GPT-2 crossover ranges at the primary checkpoint. Tuning moves the crossover one tested data size earlier in three of the four comparisons.
Figure 5: GPT-2 small versus KN-5 on the same held-out in-vocabulary units. Positive values mean KN-5 predicts the evaluated units better; negative values mean GPT-2 is better.
Figure 6: Zorro accuracy differences by grammatical phenomenon (BabyView minus age-2 TinyDialogues) for GPT-2 small and micro, pooled across 800k–3.2M training words. Positive values favor BabyView. Error bars are pointwise 95% two-stage bootstrap intervals across three corpus draws and three model seeds per arm; the overall difference is +4.51+4.51 percentage points for small and +2.36+2.36 for micro.
Figure 7: Cross-child held-out prediction patterns for GPT-2 and KN-5 after the same row/column adjustment. Diagonal entries are masked and colors are clipped to [−0.40,0.40][-0.40,0.40] BPW to reveal the remaining 240 comparisons, whose patterns correlate at r=.91r=.91 . The color clipping affects display only, not the correlation.
MonthsCellsChildrenBPW [95% CI]
0–361200.640 [0.574,0.705]0.640\ [0.574,0.705]
3–648170.525 [0.450,0.600]0.525\ [0.450,0.600]
6–1235100.456 [0.354,0.556]0.456\ [0.354,0.556]
12+1960.326 [0.206,0.429]0.326\ [0.206,0.429]
Appendix
Table 2: Own-environment prediction advantage for later speech from other speakers in the child’s environment. Intervals resample children, not individual evaluation windows.
Figure 8: Own-environment prediction advantage over time in the main analysis and an independent chronological-prefix replication. Points show averages for future evaluation windows and error bars show 95% child-cluster bootstrap intervals. The open diamond is a descriptive pre-cutoff reference, not part of the estimated time trend; exact window counts and slopes appear in Table 2 and the text.
Conditionα\alphaJSDBPW advantage95% CI
Primary0.0345−.002-.002[−.030,.028][-.030,.028]
Primary.25.0426.097.097[.010,.193][.010,.193]
Primary.50.0650.366.366[.205,.575][.205,.575]
Primary.75.1012.826.826[.590,1.098][.590,1.098]
Primary1.0.15242.1072.107[1.690,2.465][1.690,2.465]
Noun/verb1.0.0572.289.289[.232,.347][.232,.347]
Appendix
Table 3: Controlled synthetic-environment specialization. The advantage is other-environment minus own-environment content BPW; larger values indicate stronger specialization. JSD is calculated over the top 5,000 unigram types with add-0.5 smoothing. Intervals resample environments.
Figure 9: Controlled synthetic specialization. (A) Increasing the synthetic mixing parameter α\alpha increases the own-environment prediction advantage. (B) The same results plotted against measured excess unigram divergence, alongside three alternative synthetic manipulations and BabyView as a descriptive reference. Excess divergence is the median other-environment minus own-environment JSD. Error bars indicate 95% environment-cluster bootstrap intervals. The BabyView marker is not part of the controlled synthetic experiment. Different types of changes to the language input can produce different levels of specialization, even when the overall differences between environments are similar.
Figure 10: Raw benchmark scaling for GPT-2 micro: (A) Zorro accuracy and (B) mean WordSim Spearman correlation over all 7,917 word pairs in five datasets. Lines connect the shared 25k–3.2M training sizes (pointwise 95% two-stage bootstrap intervals over three corpus draws and three seeds). Hollow markers show the full 4.67M-word BabyView corpus (one draw and a seed-only range), the size-matched age-2 TinyDialogues sample (three draws), and age-2-only extensions at 6.4M and 12.8M words. Color-matched horizontal dashed lines give tokenizer-matched random-initialization Zorro baselines (46.1% for BabyView and 54.8% for age-2 TinyDialogues).
Representation/layer conventionMean ρ\rho
Final layer (five tests)0.050
Two-thirds depth (five tests)0.082
MEN-dev-selected (four tests)0.044
Best layer per test (five tests)0.128
Previous work ( Feng et al., 2026 ) (five tests)0.153
Appendix
Table 4: WordSim sensitivity to layer selection. The best-layer-per-test-set procedure uses the evaluation targets to choose a layer, so its score is optimistic. The MEN-dev-selected result covers four held-out datasets, unlike the five-dataset means. The v1 result also differs in model, tokenizer, pooling, and input format.
Corpus / benchmarkTrainedInitialGain
BabyView: Zorro (%)59.4448.41+11.03
Age-2 TD: Zorro (%)54.7851.98+2.80
BabyView: per-token Zorro (%)60.6347.71+12.91
Age-2 TD: per-token Zorro (%)56.5951.22+5.37
BabyView: BLiMP (%)54.8649.43+5.43
Age-2 TD: BLiMP (%)51.3551.93−0.58-0.58
Appendix
Table 5: 25k-word trained versus matched random-initialization performance (GPT-2 small). Gains are in percentage points. The random-initialization reference averages the matched seeds and tokenizer; repeated corpus draws do not create independent initializations.
Figure 11: Mixed-age minus age-2 TinyDialogues across training sizes: (A) Zorro accuracy differences (percentage points) and (B) word-similarity differences (Spearman). Error bars resample corpus draws and seeds.
Experiment familyRunsGPU-h
Main scaling and closeout checks39035.4
Mixed-age TinyDialogues scaling636239.6
Large-data replication (plus 24 replays)2414.0
Recipe sensitivity grid60020.4
Individual environments and prefixes52520.9
Diversity, reference data, and AoA49423.8
Appendix
Table 6: Approximate GPU-work accounting. GPU-hours combine sampled utilization and process-time proxies; training-run counts exclude the separate replay jobs. Evaluation time is included. Allocated RunPod time was approximately 807 GPU-hours.
More in this topic

Small Language Models

Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.

See all 277 papers

Explore similar work

CardsList
Mar 31, 2026

Baby Scale: Investigating Models Trained on Individual Children's Language Input

Modern language models (LMs) must be trained on many orders of magnitude more words of training data than human children receive before they begin to produce useful behavior. Assessing the nature and origins of this "data gap" requires benchmarking LMs on human-scale datasets to understand how linguistic knowledge emerges from children's natural training data. Using transcripts from the BabyView dataset (videos from children ages 6-36 months), we investigate (1) scaling performance at child-scale data regimes, (2) variability in model performance across datasets from different children's experiences and linguistic predictors of dataset quality, and (3) relationships between model and child language learning outcomes. LMs trained on child data show acceptable scaling for grammar tasks, but lower scaling on semantic and world knowledge tasks than models trained on synthetic data; we also observe substantial variability on data from different children. Beyond dataset size, performance is most associated with a combination of distributional and interactional linguistic features, broadly consistent with what makes high-quality input for child language development. Finally, model likelihoods for individual words correlate with children's learning of those words, suggesting that properties of child-directed input may influence both model learning and human language development. Overall, understanding what properties make language data efficient for learning can enable more powerful small-scale language models while also shedding light on human language acquisition.

Steven Y. Feng, Alvin W. M. Tan, Michael C. Frank

Stanford University

Language Model ScalingSmall Language ModelsChild Language Acquisition
AskPDF
Mar 13, 2026

Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study

Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.

Liel Binyamin, Elior Sulem

Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev

Language Model PretrainingLanguage ModelingMultilingual Language ModelsMultilingual Language Model EvaluationCross-Lingual Transfer
AskPDF
Aug 17, 2026

Children, but not language models, show accelerating returns in word learning

Children learn hundreds of words over the first years of their lives, in a process that begins slowly but quickly picks up speed. Prior models describe vocabulary growth as evidence accumulation over time. Here we show that the process is best characterized as accelerating accumulation: children learn more from each additional unit of linguistic experience than they did from the one before. In contrast to children, language models -- even those trained on child-directed speech -- do not accelerate. Instead, they show constant proportional returns on new data, consistent with scaling laws. Children learn using many orders of magnitude less training data than language models; their increasingly efficient use of their learning input is a candidate explanation.

Michael C. Frank

Child Language Acquisition
AskPDF