cs.CLOct 27, 2026
SaveSpecialization, Diversity, and Scaling in Child-Scale Language Models
Organizations: Stanford University
Abstract
Modern language models are trained on far more language than children experience, but we know much less about learning from small, naturalistic datasets. We train small GPT-2 models on transcripts of speech with children from dense home recordings (BabyView) and on synthetic child-directed dialogue (TinyDialogues), varying the amount, source, and diversity of training data. Models trained on one child's language environment predict that environment substantially better than other children's environments. Much of this specialization is lexical and distributional: a simple 5-gram model reproduces the cross-child pattern (r=.91). Training on data from multiple children clearly improves prediction of pooled held-out language relative to training on a single child's input without a similarly clear improvement on grammatical benchmarks. Across data sizes, naturalistic and synthetic training inputs also produce different grammatical and lexical-semantic patterns, with important differences between grammatical phenomena. Together, these results show that small-data language learning depends not only on how much language a model sees, but on which environments it comes from and how well that input matches what the model is asked to predict.
Figures & tables
Figure 1: BabyView examples and two central findings. (A) A representative BabyView egocentric frame and a separate excerpt of automatically transcribed child–adult conversation. The egocentric frame and transcript excerpt come from separate BabyView recordings and are presented as illustrative examples. (B) Models specialize to local environments, and a 5-gram reproduces much of the cross-child pattern ( after excluding same-child comparisons). (C) With 100k training words, more contributing children improve prediction of pooled held-out data; Zorro (grammaticality) shows no clear trend. The dashed diagonal in (B) indicates equality between the adjusted model scores; the horizontal dashed line in (C) indicates theoretical chance performance on Zorro.
Figure 2: Specialization is concentrated in rare vocabulary and remains detectable months later. (A) Words unseen in the source child’s 100k-word training corpus account for 94.8% of the same-environment advantage while comprising 6.2% of evaluated words. (B) The advantage remains on later recordings but becomes weaker with time.
Figure 3: At a fixed word budget, language from more children improves pooled held-out prediction more clearly than grammatical-benchmark performance. Each point is one 100k-word training dataset (mean across three model seeds); diamonds show the mean for each number of contributing children. (A) Pooled held-out loss decreases as the training set has more children (slope BPW per ; permutation ). (B) Zorro shows no clear monotonic trend at the same budget ( ). The dashed line is theoretical 50% chance on Zorro.
Figure 4: Scaling of Zorro and WordSim performance (GPT-2 small). (A) Zorro accuracy (50% is chance; color-matched dashed lines show random-initialization baselines: 48.4% for BabyView and 52.0% for age-2 TinyDialogues). (B) Mean WordSim Spearman correlation across five datasets (the same 7,917 word pairs for both corpora). Lines connect the shared 25k–3.2M-word conditions (three corpus draws three seeds each), with pointwise 95% two-stage bootstrap intervals. Unconnected hollow markers show 4.67M-word results: full BabyView (one draw, three seeds; seed-only range) and size-matched age-2 TinyDialogues (three draws; bootstrap interval). Both use 20 training epochs and validation-selected checkpoints. Appendix H shows GPT-2 micro and larger age-2-only conditions; Appendix J mixed-age comparisons; Appendix I.1 the common-word-pair analysis.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Corpus/model | Fixed setup | Tuned setup |
|---|---|---|
| BabyView micro | 800k–1.6M | 400k–800k |
| BabyView small | 800k–1.6M | 400k–800k |
| Age-2 TD micro | 400k–800k | 200k–400k |
| Age-2 TD small | 200k–400k | 200k–400k |
Table 1: Vocabulary-aligned KN-5/GPT-2 crossover ranges at the primary checkpoint. Tuning moves the crossover one tested data size earlier in three of the four comparisons.
Figure 5: GPT-2 small versus KN-5 on the same held-out in-vocabulary units. Positive values mean KN-5 predicts the evaluated units better; negative values mean GPT-2 is better.
Figure 6: Zorro accuracy differences by grammatical phenomenon (BabyView minus age-2 TinyDialogues) for GPT-2 small and micro, pooled across 800k–3.2M training words. Positive values favor BabyView. Error bars are pointwise 95% two-stage bootstrap intervals across three corpus draws and three model seeds per arm; the overall difference is percentage points for small and for micro.
Figure 7: Cross-child held-out prediction patterns for GPT-2 and KN-5 after the same row/column adjustment. Diagonal entries are masked and colors are clipped to BPW to reveal the remaining 240 comparisons, whose patterns correlate at . The color clipping affects display only, not the correlation.
| Months | Cells | Children | BPW [95% CI] |
|---|---|---|---|
| 0–3 | 61 | 20 | |
| 3–6 | 48 | 17 | |
| 6–12 | 35 | 10 | |
| 12+ | 19 | 6 |
Table 2: Own-environment prediction advantage for later speech from other speakers in the child’s environment. Intervals resample children, not individual evaluation windows.
Figure 8: Own-environment prediction advantage over time in the main analysis and an independent chronological-prefix replication. Points show averages for future evaluation windows and error bars show 95% child-cluster bootstrap intervals. The open diamond is a descriptive pre-cutoff reference, not part of the estimated time trend; exact window counts and slopes appear in Table 2 and the text.
| Condition | JSD | BPW advantage | 95% CI | |
|---|---|---|---|---|
| Primary | 0 | .0345 | ||
| Primary | .25 | .0426 | ||
| Primary | .50 | .0650 | ||
| Primary | .75 | .1012 | ||
| Primary | 1.0 | .1524 | ||
| Noun/verb | 1.0 | .0572 |
Table 3: Controlled synthetic-environment specialization. The advantage is other-environment minus own-environment content BPW; larger values indicate stronger specialization. JSD is calculated over the top 5,000 unigram types with add-0.5 smoothing. Intervals resample environments.
Figure 9: Controlled synthetic specialization. (A) Increasing the synthetic mixing parameter increases the own-environment prediction advantage. (B) The same results plotted against measured excess unigram divergence, alongside three alternative synthetic manipulations and BabyView as a descriptive reference. Excess divergence is the median other-environment minus own-environment JSD. Error bars indicate 95% environment-cluster bootstrap intervals. The BabyView marker is not part of the controlled synthetic experiment. Different types of changes to the language input can produce different levels of specialization, even when the overall differences between environments are similar.
Figure 10: Raw benchmark scaling for GPT-2 micro: (A) Zorro accuracy and (B) mean WordSim Spearman correlation over all 7,917 word pairs in five datasets. Lines connect the shared 25k–3.2M training sizes (pointwise 95% two-stage bootstrap intervals over three corpus draws and three seeds). Hollow markers show the full 4.67M-word BabyView corpus (one draw and a seed-only range), the size-matched age-2 TinyDialogues sample (three draws), and age-2-only extensions at 6.4M and 12.8M words. Color-matched horizontal dashed lines give tokenizer-matched random-initialization Zorro baselines (46.1% for BabyView and 54.8% for age-2 TinyDialogues).
| Representation/layer convention | Mean |
|---|---|
| Final layer (five tests) | 0.050 |
| Two-thirds depth (five tests) | 0.082 |
| MEN-dev-selected (four tests) | 0.044 |
| Best layer per test (five tests) | 0.128 |
| Previous work ( Feng et al., 2026 ) (five tests) | 0.153 |
Table 4: WordSim sensitivity to layer selection. The best-layer-per-test-set procedure uses the evaluation targets to choose a layer, so its score is optimistic. The MEN-dev-selected result covers four held-out datasets, unlike the five-dataset means. The v1 result also differs in model, tokenizer, pooling, and input format.
| Corpus / benchmark | Trained | Initial | Gain |
|---|---|---|---|
| BabyView: Zorro (%) | 59.44 | 48.41 | +11.03 |
| Age-2 TD: Zorro (%) | 54.78 | 51.98 | +2.80 |
| BabyView: per-token Zorro (%) | 60.63 | 47.71 | +12.91 |
| Age-2 TD: per-token Zorro (%) | 56.59 | 51.22 | +5.37 |
| BabyView: BLiMP (%) | 54.86 | 49.43 | +5.43 |
| Age-2 TD: BLiMP (%) | 51.35 | 51.93 |
Table 5: 25k-word trained versus matched random-initialization performance (GPT-2 small). Gains are in percentage points. The random-initialization reference averages the matched seeds and tokenizer; repeated corpus draws do not create independent initializations.
Figure 11: Mixed-age minus age-2 TinyDialogues across training sizes: (A) Zorro accuracy differences (percentage points) and (B) word-similarity differences (Spearman). Error bars resample corpus draws and seeds.
| Experiment family | Runs | GPU-h |
|---|---|---|
| Main scaling and closeout checks | 390 | 35.4 |
| Mixed-age TinyDialogues scaling | 636 | 239.6 |
| Large-data replication (plus 24 replays) | 24 | 14.0 |
| Recipe sensitivity grid | 600 | 20.4 |
| Individual environments and prefixes | 525 | 20.9 |
| Diversity, reference data, and AoA | 494 | 23.8 |
Table 6: Approximate GPU-work accounting. GPU-hours combine sampled utilization and process-time proxies; training-run counts exclude the separate replay jobs. Evaluation time is included. Allocated RunPod time was approximately 807 GPU-hours.
More in this topic
Small Language Models
Small language models are compact neural language systems designed to deliver useful text and reasoning capabilities with limited compute, memory, and energy.
See all 277 papers