Modern language models (LMs) must be trained on many orders of magnitude more words of training data than human children receive before they begin to produce useful behavior. Assessing the nature and origins of this "data gap" requires benchmarking LMs on human-scale datasets to understand how linguistic knowledge emerges from children's natural training data. Using transcripts from the BabyView dataset (videos from children ages 6-36 months), we investigate (1) scaling performance at child-scale data regimes, (2) variability in model performance across datasets from different children's experiences and linguistic predictors of dataset quality, and (3) relationships between model and child language learning outcomes. LMs trained on child data show acceptable scaling for grammar tasks, but lower scaling on semantic and world knowledge tasks than models trained on synthetic data; we also observe substantial variability on data from different children. Beyond dataset size, performance is most associated with a combination of distributional and interactional linguistic features, broadly consistent with what makes high-quality input for child language development. Finally, model likelihoods for individual words correlate with children's learning of those words, suggesting that properties of child-directed input may influence both model learning and human language development. Overall, understanding what properties make language data efficient for learning can enable more powerful small-scale language models while also shedding light on human language acquisition.
Figures & tables
Figure 1: Broader performance scaling with data quantity across all four benchmarks. Each color denotes a separate dataset, with different markers showing different models. Gray curves show spline-smoothed average relationships across all data.
Model
Condition
# of Fams
Tokens
TTR
Zorro
WordSim
COMPS
EWoK
GPT-2 small
Mean of fams
1
142,094
–
55.19 ± 2.89
0.049 ± 0.007
50.09 ± 0.14
50.18 ± 0.15
GPT-2 small
Top-1
1
725,600
0.026
58.09 ± 3.36
0.042 ± 0.025
50.19 ± 0.49
50.07 ± 0.36
GPT-2 small
Top-2
2
727,986
0.025
59.58 ± 1.85
0.064 ± 0.022
50.37 ± 0.33
50.12 ± 0.13
GPT-2 small
Top-5
5
727,883
0.026
56.70 ± 2.88
0.062 ± 0.008
50.22 ± 0.27
50.32 ± 0.05
GPT-2 small
Top-10
10
735,309
0.027
57.14 ± 2.79
0.048 ± 0.020
50.22 ± 0.20
50.13 ± 0.21
GPT-2 small
Top-20
20
734,234
0.026
56.59 ± 2.18
0.044 ± 0.015
50.03 ± 0.23
50.31 ± 0.29
Table 1: Major BabyView results (averaged across 5 seeds per experiment). Mean of fams reports mean and standard deviation across the 20 individual-family datasets. Pooled rows (Top-2/5/10/20) are proportional mixtures with seed-level variability. Top-1 is the largest single family; All fams is the all-families model trained on the concatenated corpus. Zorro, COMPS, and EWoK are percentages (higher is better); WordSim is Spearman correlation (higher is better). TTR is type–token ratio (unique word types divided by total tokens).
Figure 2: Performance scaling of (A) BabyView experiments and (B) TinyDialogues experiments across all four benchmarks. Each color denotes a separate model; different markers show different experimental conditions. Lines of best fit show average linear relationships.
Feature
Top10
Meth.
Spear.
Lasso
XGB
Mean rank
Mean ∣ρ∣
Mean ∣β∣
Mean imp.
Bigram mutual information
9
3
3
1
5
5.56
0.72
0.153
0.060
Caregiver POS-tagged token count
7
3
4
1
2
4.29
0.74
0.016
0.062
Child POS-tagged token count
7
3
1
4
2
2.00
0.86
0.604
0.356
Child-to-caregiver semantic pair count
6
3
2
2
2
6.17
0.80
0.005
0.050
POS bigram entropy
6
3
1
1
4
3.33
0.56
0.263
0.069
Mean KL divergence from other datasets
5
3
3
1
1
3.60
0.71
0.500
0.053
Table 2: Top linguistic predictors across statistical analyses, identified and sorted by Top10 hit. Top10 counts the number of times a feature appears in top-10 lists across all methods and experimental conditions. Meth. is the number of distinct methods (Spearman, Lasso, XGBoost) that select the feature at least once. Spear. , Lasso , and XGB report method-specific top-10 counts. The following mean columns compute average across top-10 appearances. Mean rank is the average within-list rank when selected. Mean ∣ρ∣ is the average absolute Spearman correlation. Mean ∣β∣ is the average absolute lasso coefficient (linear effect size under regularization). Mean imp. is the average XGBoost feature importance (nonlinear contribution to predictive performance). Full results (all linguistic features) in Table 9 .
Figure 3: (A) Relationship between mean token NLL and AoA for CDI words by lexical category, for models trained on data from all families. (B) AIC of the regressions predicting AoA from mean token NLL. Dashed line indicates AIC of the base frequency model.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Family
Age (months)
Tokens
Zorro
WordSim
COMPS
EWoK
BV-fam-S00400001
31
725,600
58.09 ± 3.36
0.042 ± 0.025
50.19 ± 0.49
50.07 ± 0.36
BV-fam-S00510002
11
569,788
55.95 ± 2.74
0.054 ± 0.015
50.05 ± 0.42
50.14 ± 0.20
BV-fam-S00360001
8
257,817
59.53 ± 1.83
0.048 ± 0.022
50.07 ± 0.15
50.17 ± 0.30
BV-fam-S00220001
5
242,954
58.02 ± 2.19
0.058 ± 0.029
50.12 ± 0.34
50.18 ± 0.36
BV-fam-S00400002
28
181,317
57.95 ± 2.19
0.054 ± 0.031
50.07 ± 0.35
50.42 ± 0.80
BV-fam-S00320002
7
171,656
55.98 ± 2.31
0.028 ± 0.024
50.25 ± 0.32
50.21 ± 0.27
Appendix
Table 3: Individual-family BabyView results for GPT-2 small with seed-level variability, sorted by training tokens. Age is months at onboarding time of that family’s given child.
Family
Age (months)
Tokens
Zorro
WordSim
COMPS
EWoK
BV-fam-S00400001
31
725,600
57.54 ± 1.98
0.044 ± 0.018
50.10 ± 0.16
50.03 ± 0.26
BV-fam-S00510002
11
569,788
57.65 ± 2.52
0.054 ± 0.027
49.97 ± 0.20
49.91 ± 0.10
BV-fam-S00360001
8
257,817
59.29 ± 0.95
0.032 ± 0.033
50.13 ± 0.29
49.94 ± 0.36
BV-fam-S00220001
5
242,954
59.43 ± 1.08
0.058 ± 0.018
49.74 ± 0.45
50.19 ± 0.30
BV-fam-S00400002
28
181,317
56.25 ± 3.03
0.014 ± 0.024
50.07 ± 0.32
50.16 ± 0.25
BV-fam-S00320002
7
171,656
56.49 ± 2.74
0.014 ± 0.019
50.42 ± 0.27
50.17 ± 0.38
Appendix
Table 4: Individual-family BabyView results for GPT-2 mini with seed-level variability, sorted by training tokens. Age is months at onboarding time of that family’s given child.
Family
Age (months)
Tokens
Zorro
WordSim
COMPS
EWoK
BV-fam-S00400001
31
725,600
59.10 ± 1.58
0.068 ± 0.026
50.22 ± 0.27
50.31 ± 0.35
BV-fam-S00510002
11
569,788
57.13 ± 1.69
0.062 ± 0.013
50.17 ± 0.20
50.13 ± 0.17
BV-fam-S00360001
8
257,817
58.26 ± 0.85
0.058 ± 0.015
50.18 ± 0.29
50.08 ± 0.27
BV-fam-S00220001
5
242,954
59.88 ± 1.82
0.056 ± 0.009
49.93 ± 0.23
50.10 ± 0.17
BV-fam-S00400002
28
181,317
57.57 ± 1.58
0.054 ± 0.015
49.82 ± 0.28
49.80 ± 0.27
BV-fam-S00320002
7
171,656
56.68 ± 0.94
0.026 ± 0.011
50.24 ± 0.34
50.14 ± 0.35
Appendix
Table 5: Individual-family BabyView results for GPT-BERT base with seed-level variability, sorted by training tokens. Age is months at onboarding time of that family’s given child.
Family
Age (months)
Tokens
Zorro
WordSim
COMPS
EWoK
BV-fam-S00400001
31
725,600
54.37 ± 1.63
0.072 ± 0.016
50.10 ± 0.10
50.00 ± 0.11
BV-fam-S00510002
11
569,788
55.65 ± 0.85
0.068 ± 0.047
50.12 ± 0.33
49.96 ± 0.20
BV-fam-S00360001
8
257,817
57.97 ± 1.34
0.066 ± 0.021
50.22 ± 0.23
49.88 ± 0.39
BV-fam-S00220001
5
242,954
57.24 ± 1.12
0.074 ± 0.018
49.98 ± 0.36
50.17 ± 0.34
BV-fam-S00400002
28
181,317
59.01 ± 1.77
0.058 ± 0.016
49.77 ± 0.23
50.06 ± 0.28
BV-fam-S00320002
7
171,656
56.21 ± 1.77
0.056 ± 0.019
50.07 ± 0.27
49.99 ± 0.26
Appendix
Table 6: Individual-family BabyView results for GPT-BERT small with seed-level variability, sorted by training tokens. Age is months at onboarding time of that family’s given child.
Model
Target
α
NZ feat.
Lasso R2
Lasso RMSE
XGB R2
XGB RMSE
GPT-2 Mini
Zorro
0.0488
21
0.657
1.784
0.395
2.369
GPT-2 Mini
WordSim
0.0026
13
-0.208
0.029
-0.166
0.029
GPT-2 Mini
COMPS
0.0668
0
-0.144
0.166
-1.076
0.223
GPT-2 Small
Zorro
0.5230
6
0.289
2.974
0.172
3.208
GPT-2 Small
WordSim
0.0173
1
-1.047
0.031
-1.943
0.037
GPT-2 Small
COMPS
0.0713
3
-1.479
0.257
-0.601
0.207
Appendix
Table 7: Cross-validated predictive performance of lasso regression and XGBoost models for each model–target pair. α denotes the lasso regularization strength selected via cross-validation. NZ feat. is the number of features with nonzero coefficients in the fitted lasso model, reflecting model sparsity. Lasso R2 and XGB R2 report the proportion of variance explained on held-out data (higher is better), while Lasso RMSE and XGB RMSE report root mean squared error (lower is better). Together, these metrics characterize both predictive performance and model complexity across methods.
Feature name
Brief description
Lexical / Distributional / Scale
avg_conversation_len_tokens
This feature measures the average number of tokens per conversation, capturing how much language is typically contained in a single interaction.
avg_sentence_len_tokens
This feature measures the average number of tokens per sentence, providing a simple proxy for sentence-level length and complexity.
avg_utterance_len_tokens
This feature measures the average number of tokens per utterance, summarizing how long individual turns tend to be.
bigram_entropy
This feature measures the entropy of the bigram distribution, so higher values indicate greater diversity in adjacent token sequences.
bigram_mutual_information
This feature measures the average pointwise mutual information of adjacent token pairs, so higher values indicate that neighboring words are more strongly associated than expected by chance.
Appendix
Table 8: Comprehensive catalog of the 175 linguistic features considered in our BabyView analysis. The table is grouped by feature family and provides a one-sentence description of each feature. Note that all 175 features were kept after restricting to numeric variables and applying missingness ( ≤0.4 across the 25 experiments) and zero-variance filtering. Hence, all 175 features were used in the final statistical analyses (Spearman, lasso, and XGBoost).
Feature
Top10
Meth.
Spear.
Lasso
XGB
Mean rank
Mean ∣ρ∣
Mean ∣β∣
Mean imp.
Lexical / Distributional / Scale
bigram_mutual_information
9
3
3
1
5
5.56
0.72
0.153
0.060
hapax_ratio
5
3
1
1
3
6.60
0.48
0.264
0.067
trigram_entropy
5
2
4
0
1
5.00
0.74
0.000
0.075
num_conversations
5
1
0
5
0
2.00
0.00
0.178
0.000
avg_conversation_len_tokens
4
3
2
1
1
8.25
0.57
0.005
0.036
Appendix
Table 9: Full linguistic feature results grouped by category, sorted by Top10 hit count within each category. Top10 is the total number of model–target analyses in which a feature appeared in a method’s top-10 list; Meth. is the number of methods (Spearman, lasso, XGBoost) for which the feature appeared at least once; Spear. , Lasso , and XGB give the corresponding method-specific hit counts; Mean rank is the average within-method top-10 rank across all hits; Mean ∣ρ∣ is the mean absolute Spearman correlation across the feature’s Spearman top-10 appearances; Mean ∣β∣ is the mean absolute lasso coefficient across the feature’s lasso top-10 appearances; Mean imp. is the mean XGBoost feature-importance score across the feature’s XGBoost top-10 appearances.
Children learn hundreds of words over the first years of their lives, in a process that begins slowly but quickly picks up speed. Prior models describe vocabulary growth as evidence accumulation over time. Here we show that the process is best characterized as accelerating accumulation: children learn more from each additional unit of linguistic experience than they did from the one before. In contrast to children, language models -- even those trained on child-directed speech -- do not accelerate. Instead, they show constant proportional returns on new data, consistent with scaling laws. Children learn using many orders of magnitude less training data than language models; their increasingly efficient use of their learning input is a candidate explanation.
Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.
Liel Binyamin, Elior Sulem
Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev
Recent studies suggest that child-directed speech is not conducive to language learning in BabyLMs. However, current evaluations focus predominantly on comprehension and not production, which is central to usage-based theories of language acquisition which argue how CDS facilitates early language use through constructional ''frames'' (frequent lexical patterns with open slots). We introduce a novel generation-based evaluation inspired by such theories in form of a frame-completion task, and compare Llama models trained with CDS, the BabyLM corpus, and web-crawl data (FineWeb-edu) on comprehension benchmarks and our novel framework. Our results reveal a clear dissociation between models' comprehension and production capabilities: while FineWeb-trained models excel at minimal pairs, CDS-trained models produce grammatical completions substantially earlier in training and concentrate probability mass on appropriate slot-fillers. These findings show that comprehension benchmarks underestimate what CDS affords to BabyLMs.
Bastian Bunzeck, Sina Zarrieß
Computational Linguistics, Department of Linguistics Bielefeld University, Germany