Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Study
Authors: Liel Binyamin, Elior Sulem
Organizations: Faculty of Computer and Information Science, Institute for Applied AI Research Data Science Research Center Ben-Gurion University of the Negev
Research on developmentally plausible language models has largely focused on English, leaving open questions about multilingual settings. We present a systematic study of compact language models by extending BabyBERTa to English-French scenarios under strictly size-matched data conditions, covering monolingual, bilingual, and cross-lingual settings. Our design contrasts two types of training corpora: (i) child-directed speech (about 2.5M tokens), following BabyBERTa and related work, and (ii) multi-domain corpora (about 10M tokens), extending the BabyLM framework to French. To enable fair evaluation, we also introduce new resources, including French versions of QAMR and QASRL, as well as English and French multi-domain corpora. We evaluate the models on both syntactic and semantic tasks and compare them with models trained on Wikipedia-only data. The results reveal context-dependent effects: training on Wikipedia consistently benefits semantic tasks, whereas child-directed speech improves grammatical judgments in monolingual settings. Bilingual pretraining yields notable gains for textual entailment, with particularly strong improvements for French. Importantly, similar patterns emerge across BabyBERTa, RoBERTa, and LTG-BERT, suggesting consistent trends across architectures.
Figures & tables
Figure 1: Monolingual, bilingual, and cross-lingual settings. Left column shows the pretraining language(s); right column shows the testing language, and these labels apply to all three settings. For downstream QA/TE experiments, fine-tuning is performed in the same language as testing.
Dataset (French and English)
Domain
# Words
Proportion
CHILDES ( MacWhinney 2000 )
Child-directed speech
0.50M
5%
CFDD (FR) 1 / BNC (EN) 1
Dialogue
0.90M
9%
Children’s Book Corpus ( Hill et al. 2016 )
Children’s books
0.90M
9%
Gutenberg Corpus 2
Written prose
1.00M
10%
OpenSubtitles ( Lison and Tiedemann 2016 )
Movie subtitles
3.00M
31%
QED ( Abdelali et al. 2014 )
Educational video subtitles
1.10M
11%
Table 1: Composition of our 10M-word pre-training corpora in French and English. Sentence boundaries were respected, so totals fall slightly below 10M words. Domain proportions follow the BabyLM Strict-Small configuration.
SQuAD
QAMR
QASRL
XNLI
CLAMS
Model (pretrain)
BB
RB
BB
RB
BB
RB
BB
RB
BB
RB
Tested in English
Monolingual
En-CHILDES
23.35
23.88
38.54
49.01
84.80
88.16
55.71
61.27
58.77
54.36
En-Wikipedia (baseline)
45.59
34.72
56.41
62.66
88.09
88.42
61.70
66.83
50.32
52.71
Bilingual
Table 2: F1 scores for BabyBERTa (BB) vs. RoBERTa (RB) across tasks using child-directed speech and Wikipedia corpora ( ≈ 2.5M words) . Sections indicate the language in which the model was tested , with downstream fine-tuning always performed in that same language. ∗ indicates statistically significant improvements over the corresponding monolingual baseline ( p<0.05 , paired bootstrap test across seeds).
SQuAD
QAMR
QASRL
XNLI
CLAMS
Model (pretrain)
BB
RB
BB
RB
BB
RB
BB
RB
BB
RB
Tested in English
Monolingual
En-Multidomain
30.93
43.70
52.66
66.45
85.87
89.06
64.03
68.44
55.90
58.19
En-Wikipedia (baseline)
40.42
46.12
55.58
68.93
88.21
89.41
65.04
69.32
53.12
56.71
Bilingual
Table 3: F1 scores for BabyBERTa (BB) vs. RoBERTa (RB) across tasks using multi-domain pretraining corpora ( ≈ 10M words) . Sections indicate the language in which the model was tested ; downstream fine-tuning was always performed in that same language. In the bilingual setup, ≈ 1.25M words are used in each of the languages. ∗ indicates statistically significant improvements over the corresponding monolingual baseline ( p<0.05 , paired bootstrap test across seeds).
Model (Baseline)
SQuAD
QAMR
QASRL
XNLI
CLAMS
Tested in English
RoBERTa-base (EN)
80.12
86.54
91.11
88.52
77.68
CamemBERT-base (FR)
14.28
18.84
24.23
62.12
58.34
Tested in French
RoBERTa-base (EN)
16.35
21.32
43.12
53.23
62.47
CamemBERT-base (FR)
67.94
72.92
80.85
79.56
76.92
Table 4: Baseline results for the original pretrained RoBERTa-base (English) and CamemBERT-base (French) models. These off-the-shelf baselines are distinct from the RoBERTa variants retrained on our 2.5M-token datasets. All downstream tasks are fine-tuned and tested in the same language.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Question Answering
Model
SQuAD
QAMR
QASRL
XNLI
Fine-tuned in English
Cross-lingual
Fr-CHILDES
6.48
10.36
19.81
17.32
Fr-Wikipedia
10.06
16.39
21.45
19.54
Fr-CHILDES-Wikipedia
7.30
15.45
20.41
18.88
Appendix
Table 5: F1 scores for T5 tiny models across QA tasks. Results are grouped by fine-tuning language and model training setup (monolingual, cross-lingual, and bilingual). This represents an additional analysis using a different encoder-decoder architecture to complement our main BabyBERTa findings.
Question Answering
TE
Grammar
Model
SQuAD
QAMR
QASRL
XNLI
CLAMS
Fine-tuned in English
Monolingual
En-Multidomain Corpus words
32.15
54.02
86.45
65.21
57.14
En-Wikipedia
41.83
56.92
88.77
66.42
51.20
Bilingual
Appendix
Table 6: F1 scores for LTG-BERT across all tasks, using 10M pretraining data (including Wikipedia and mixed-domain corpora). LTG-BERT was the winning model from the BabyLM Challenge that achieved the best results through architectural innovations.
Model (pretrain)
SQuAD
QAMR
QASRL
XNLI
CLAMS
Tested in English
Monolingual
En-CHILDES-Wikipedia
31.42
51.95
87.01
61.19
58.42
Cross-lingual
Fr-CHILDES-Wikipedia
27.18
41.52
79.93
58.43
48.30
Tested in French
Appendix
Table 7: Results of Interaction Between Child-Directed Speech and Wikipedia on BabyBERTa (BB) across all evaluation tasks.