Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale

Bill K. S. Ku; Janet B. Pierrehumbert

cs.CLOct 16, 2026

Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale

Authors: Bill K. S. Ku, Janet B. Pierrehumbert

Organizations: Department of Engineering Science, University of Oxford

Abstract

Human children learn language from continuous, prosodically structured speech long before they can read, yet almost all language models train on orthographic text. Building on previous phonemic BabyLM training (Goriely et al., 2024; arXiv:2410.22906), we ask whether supplying an estimate of the prosody infants exploit improves phonemic language modelling at BabyLM scale. We compare an orthographic baseline to three conditions forming a ladder of phonological richness. All conditions are trained as BERT MLMs at 10M and 100M words. Addressing the ecological validity of existing metrics in the small-data regime, we build a lexical decision task and two semantic organisation tasks. Our new lexical decision task offers improved insights into the distinction between wordlikeness, token familiarity, and lexicality as factors. Triphone phonotactic control and pseudo-log-likelihood scoring shows that these scores reflect phonotactic familiarity rather than wordhood: every model sits near or below chance on an adversarial inverted condition, revealing prior results of this kind as partly artefactual. On our new semantic tasks, a prosodically annotated phoneme stream gives the strongest taxonomic organisation at both the 10M and 100M word scales. Phonemic models with boundary structure are competitive with the orthographic baseline. These findings suggest that prosodic structure is a key component of phonemic language modelling in the small-data regime most analogous to infant learning.

Figures & tables

Condition	Added structure
Ortho	orthographic text (baseline)
P-Goriely	continuous IPA; [UTT] at utterance ends
P-breaks	$+$ [B2] , [B4] phrase boundaries
Prosody-NN	$+$ [H%] , [L%] boundary tones

Table 1: The four conditions, organised as a baseline plus three phonemic conditions of increasing richness. The phonemic conditions carry no whitespace.

S1: “The dog sat on the mat.”
Condition	Rendered output
Ortho	the dog sat on the mat .
P-Goriely	\ipafont ðədɒɡsætɒnðəmæt [UTT]
P-breaks	\ipafont ðədɒɡsætɒnðəmæt [B4]
Prosody-NN	\ipafont ðədɒɡsætɒnðəmæt [B4] [L%]
S2: “Did she eat?”

Table 2: Three example sentences rendered under each condition. S3 shows a [B2] inserted at a subordinate-clause onset with no comma; S2 shows the [H%] / [L%] sentence-type distinction.

Target	Thematic foil	Relation	CDI
Script / event
cat	milk	typical food	✓
fish	water	habitat	✓
dog	bone	typical food	†
Production / source
cow	milk	product	✓

Table 3: Representative thematic foil pairs by relation family. Each foil is a different-category word tied to the target by a non-taxonomic relation and, to have CHILDES excerpts, must itself be a CDI target. Entries marked † have no CDI-resident foil and contribute no probes.

10M
Condition	LexMem	Comp	Inv.
Ortho	0.991	0.768	0.499
P-Goriely	0.942	0.819	0.506
P-breaks	0.948	0.799	0.489
Prosody-NN	0.930	0.800	0.479
100M

Table 4: PLL accuracy on the lexical decision task (

n=4{,}129

pairs) by pair category. Chance is

0.50

. Bold marks the best condition in each column within a scale.

Condition	3-way	Tv-them	Tv-unrel	Adv
10M
Ortho	0.463	0.537	0.693	$+0.019$
P-Goriely	0.420	0.560	0.613	$+0.028$
P-breaks	0.400	0.490	0.600	$+0.009$
Prosody-NN	0.483	0.607	0.670	$\mathbf{+0.040}$
100M

Table 5: CBD accuracy (

n=300

probes;

\mathrm{SE}\approx 0.028

on Tv-them). Tv-them : taxonomic vs. thematic (primary); Tv-unrel : taxonomic vs. unrelated; Adv : mean taxonomic advantage. Chance is

\tfrac{1}{3}

for 3-way and

0.50

for the pairwise columns. Bold marks the best condition in each column within a scale.

Figure 1: Taxonomic-versus-thematic accuracy at 10M and 100M for the four conditions. The dashed line marks chance (

0.50

10M
Condition	K-ARI	K-NMI	A-ARI
Ortho	0.483	0.601	0.337
P-Goriely	0.279	0.439	0.257
P-breaks	0.290	0.417	0.260
Prosody-NN	0.303	0.454	0.259
100M

Table 6: SCI cluster-recovery scores (

n=68

words,

k=6

). K-ARI , K-NMI :

k

-means ARI and NMI (K-ARI primary); A-ARI : agglomerative ARI. Bold marks the best condition in each column within a scale.

Figure 2:

K

-means ARI at 10M and 100M for the four conditions. P-breaks and Prosody-NN cross above Ortho at 100M; the dashed line marks ARI

=0

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
Hyperparameter	10M	100M
Variant	BERT-small	BERT-base
Hidden size	256	768
Layers	8	12
Attention heads	8	12
FFN size	1,024	3,072

Appendix

Table 7: Model architecture and training hyperparameters at the two scales. Architecture dimensions differ by scale; training settings are shared except batch size, gradient accumulation, and peak learning rate.

Source	10M	100M	Share
CHILDES (child-directed)	2.8M	28.9M	29%
Project Gutenberg (prose)	2.5M	26.4M	26%
OpenSubtitles (dialogue)	2.0M	20.0M	20%
Simple Wikipedia	1.5M	14.7M	15%
BNC Spoken (conversation)	0.9M	7.8M	8%
Switchboard (telephone)	0.1M	1.3M	1%

Appendix

Table 8: BabyLM training corpus composition (approximate word counts).

Rule	Trigger
A	Coordinating clause junction: a coordinating conjunction heads a clause-level cc dependency on a verbal or auxiliary root.
B	Subordinate or complement clause onset: a subordinating conjunction or complementiser marks an adverbial, complement, or embedded clause.
C	Relative clause onset: a relative pronoun or the relcl arc.
D	Minor surface punctuation: comma, semicolon, colon, dash, or ellipsis.
E	Appositive or parenthetical nominal: a token bearing the appos dependency and not preceded by punctuation.

Appendix

Table 9: Syntactic cases that trigger a [B2] token in P-breaks .

Head	$F_{1}$	Ship threshold
[B2]	0.58	$\geq 0.50$
[B4]	0.99	$\geq 0.80$
[L%]	0.69	$\geq 0.60$
[H%]	0.56	$\geq 0.30$

Appendix

Table 10: Held-out

F_{1}

of the neural annotator against the shipping thresholds. All four clear their gate, so the full annotator is used.

	Biphone	Triphone
	(controlled)	(uncontrolled)
Real word favoured	58.1%	70.3%
Mean difference	$+0.31$	$+1.21$
Median difference	$+0.21$	$+0.89$
Skew	$+0.56$	$+0.75$
Mean $-$ median	$+0.10$	$+0.32$

Appendix

Table 11: Phonotactic asymmetry between the real words and the Wuggy pseudowords of the BabySLM lexical development set (

n=17{,}706

pairs), under a biphone model, which Wuggy controls, and a triphone model, which it does not.

Category	Filter B IPA collisions removed
LexicalMemory	959
Composition	1,053
Inverted	3,244

Appendix

Table 12: Pseudowords rejected by the phonemic screen (Filter B) because their IPA form matched a real corpus word.

LexicalMemory
Real	Pseudoword	$\mathbf{S_{\text{real}}}$	$\mathbf{S_{\text{fake}}}$
pencil \ipafont /pɛnsəl/	blaver \ipafont /bleɪvɚ/	$-18.7$	$-18.7$
heart \ipafont /hɑːɹt/	pred \ipafont /pɹɛd/	$-11.9$	$-12.0$
Composition
reissued \ipafont /ɹiːɪʃuːd/	pophini \ipafont /pəfiːni/	$-29.1$	$-29.1$
foppish \ipafont /fɑːpɪʃ/	cenfse \ipafont /sɛnfs/	$-25.0$	$-25.0$

Appendix

Table 13: Example real-word / pseudoword pairs with Witten-Bell triphone phonotactic scores

S

. Standard pairs are score-matched; Inverted pairs satisfy

S_{\text{fake}}>S_{\text{real}}

Category	Words
Animal	dog, cat, bird, fish, bear, duck, horse, cow, pig, frog, mouse, sheep
Food	milk, juice, water, cookie, apple, banana, bread, cheese, cake, egg, soup, rice
Body part	eye, nose, mouth, ear, hand, foot, head, hair, finger, knee, arm, leg
Clothing	shoe, sock, hat, shirt, coat, dress, boot, sweater, diaper, button, pocket
Vehicle	car, bus, truck, bike, boat, plane, wagon, stroller, tractor
Household	cup, spoon, bowl, plate, bottle, chair, table, bed, blanket, door, phone, clock

Appendix

Table 14: The 68 CDI target words by semantic category ( Fenson et al., 1993 ) .

Condition	Tokens	Per utt.	Structural	% struct.	vs P-Goriely	vs Ortho
10M
Ortho	14,979,765	12.85	0	0.00%	$+22.0\%$	n/a
P-Goriely	12,276,352	10.55	1,010,527	8.23%	n/a	$-18.0\%$
P-breaks	13,498,252	11.59	2,152,618	15.95%	$+10.0\%$	$-9.9\%$
Prosody-NN	13,417,556	11.52	2,104,219	15.68%	$+9.3\%$	$-10.4\%$
100M

Appendix

Table 15: BPE token counts per condition and scale, with the share contributed by structural tokens and the budget relative to P-Goriely and to Ortho .

Figure 3: UMAP projections of the 68 CDI word embeddings, coloured by semantic category, for Ortho and Prosody-NN at both scales;

k

-means ARI is annotated in each panel.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 521 papers

Sep 29, 2026

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

Cross-linguistic effects are a central topic in bilingual first-language acquisition. Artificial learners can help investigate L1-L2 interactions by enabling controlled comparisons across language combinations and learning conditions. Recent work explores this direction by training bilingual language models under developmentally plausible constraints. However, human and model learners still diverge in fundamental ways, with one major difference being input modality: children learn primarily from spoken input, whereas language models are typically trained on orthographic text. To reduce this gap, researchers have trained models on phonemic representations of speech. In this work, we combine these research directions to train bilingual BabyLMs with phonemic input. We keep English fixed as the L2 and vary the L1 across German, Swedish, Persian, and Basque, selected to represent contrasting combinations of syntactic and phoneme-inventory distance from English. Our results show stronger L1-related variation in grammatical learning trajectories under phonemic than orthographic input, while early lexical differences align with phoneme-inventory similarity.

Nikitas Theodoropoulos, Maria Lymperaiou, Giorgos Filandrianos

Independent Researcher · National Technical University of Athens · Instituto de Telecomunicações, Portugal

Multilingual Language Models Child Language Acquisition

Ask PDF

Sep 22, 2026

A retrospective analysis on the use of LLMs to study infant syntax learning

Large language models (LLMs) have increasingly been used to investigate how children acquire syntax at an early stage of development. This is notably the central scientific goal of the BabyLM challenge, a community-wide effort to develop models that achieve human-level syntactic performance while being trained on developmentally realistic corpora. In this paper, we reflect on the use of LLMs in the study of infant syntax learning by providing an epistemological assessment of several studies from this research program. We discuss how datasets are built, which models are implemented, how they are trained and syntactically evaluated. We observe significant assumptions in the methodology of BabyLM and related studies, thus mitigating their theoretical scope. We additionally observe that using developmentally-realistic corpora have limited effects on models performance on commonly-used benchmarks, which suggest important computational differences between LLMs and the infant syntax learner.

Hélie Bazin, Anouk Barberousse, François Yvon

SCAI, SND, ISIR · SND · MLIA

LLM Evaluation Language Modeling Child Language Acquisition

Ask PDF

May 31, 2026

Child-directed speech facilitates production, not comprehension, in BabyLMs

Recent studies suggest that child-directed speech is not conducive to language learning in BabyLMs. However, current evaluations focus predominantly on comprehension and not production, which is central to usage-based theories of language acquisition which argue how CDS facilitates early language use through constructional ''frames'' (frequent lexical patterns with open slots). We introduce a novel generation-based evaluation inspired by such theories in form of a frame-completion task, and compare Llama models trained with CDS, the BabyLM corpus, and web-crawl data (FineWeb-edu) on comprehension benchmarks and our novel framework. Our results reveal a clear dissociation between models' comprehension and production capabilities: while FineWeb-trained models excel at minimal pairs, CDS-trained models produce grammatical completions substantially earlier in training and concentrate probability mass on appropriate slot-fillers. These findings show that comprehension benchmarks underestimate what CDS affords to BabyLMs.

Bastian Bunzeck, Sina Zarrieß

Computational Linguistics, Department of Linguistics Bielefeld University, Germany

Language Model Pretraining LLM Evaluation Language Model Generation Evaluation Small Language Models Child Language Acquisition Generative AI Evaluation

Ask PDF

Abstract

Language Modeling

Explore similar work

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

A retrospective analysis on the use of LLMs to study infant syntax learning

Child-directed speech facilitates production, not comprehension, in BabyLMs