Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale

Bill K. S. Ku; Janet B. Pierrehumbert

cs.CLOct 16, 2026

Prosodic Structure as a Critical Component in Phonemic Language Modelling at BabyLM Scale

Authors: Bill K. S. Ku, Janet B. Pierrehumbert

Organizations: Department of Engineering Science, University of Oxford

Abstract

Human children learn language from continuous, prosodically structured speech long before they can read, yet almost all language models train on orthographic text. Building on previous phonemic BabyLM training (Goriely et al., 2024; arXiv:2410.22906), we ask whether supplying an estimate of the prosody infants exploit improves phonemic language modelling at BabyLM scale. We compare an orthographic baseline to three conditions forming a ladder of phonological richness. All conditions are trained as BERT MLMs at 10M and 100M words. Addressing the ecological validity of existing metrics in the small-data regime, we build a lexical decision task and two semantic organisation tasks. Our new lexical decision task offers improved insights into the distinction between wordlikeness, token familiarity, and lexicality as factors. Triphone phonotactic control and pseudo-log-likelihood scoring shows that these scores reflect phonotactic familiarity rather than wordhood: every model sits near or below chance on an adversarial inverted condition, revealing prior results of this kind as partly artefactual. On our new semantic tasks, a prosodically annotated phoneme stream gives the strongest taxonomic organisation at both the 10M and 100M word scales. Phonemic models with boundary structure are competitive with the orthographic baseline. These findings suggest that prosodic structure is a key component of phonemic language modelling in the small-data regime most analogous to infant learning.

Figures & tables

Condition	Added structure
Ortho	orthographic text (baseline)
P-Goriely	continuous IPA; [UTT] at utterance ends
P-breaks	$+$ [B2] , [B4] phrase boundaries
Prosody-NN	$+$ [H%] , [L%] boundary tones

Table 1: The four conditions, organised as a baseline plus three phonemic conditions of increasing richness. The phonemic conditions carry no whitespace.

S1: “The dog sat on the mat.”
Condition	Rendered output
Ortho	the dog sat on the mat .
P-Goriely	\ipafont ðədɒɡsætɒnðəmæt [UTT]
P-breaks	\ipafont ðədɒɡsætɒnðəmæt [B4]
Prosody-NN	\ipafont ðədɒɡsætɒnðəmæt [B4] [L%]
S2: “Did she eat?”

Table 2: Three example sentences rendered under each condition. S3 shows a [B2] inserted at a subordinate-clause onset with no comma; S2 shows the [H%] / [L%] sentence-type distinction.

Target	Thematic foil	Relation	CDI
Script / event
cat	milk	typical food	✓
fish	water	habitat	✓
dog	bone	typical food	†
Production / source
cow	milk	product	✓

Table 3: Representative thematic foil pairs by relation family. Each foil is a different-category word tied to the target by a non-taxonomic relation and, to have CHILDES excerpts, must itself be a CDI target. Entries marked † have no CDI-resident foil and contribute no probes.

10M
Condition	LexMem	Comp	Inv.
Ortho	0.991	0.768	0.499
P-Goriely	0.942	0.819	0.506
P-breaks	0.948	0.799	0.489
Prosody-NN	0.930	0.800	0.479
100M

Table 4: PLL accuracy on the lexical decision task (

n=4{,}129

pairs) by pair category. Chance is

0.50

. Bold marks the best condition in each column within a scale.

Condition	3-way	Tv-them	Tv-unrel	Adv
10M
Ortho	0.463	0.537	0.693	$+0.019$
P-Goriely	0.420	0.560	0.613	$+0.028$
P-breaks	0.400	0.490	0.600	$+0.009$
Prosody-NN	0.483	0.607	0.670	$\mathbf{+0.040}$
100M

Table 5: CBD accuracy (

n=300

probes;

\mathrm{SE}\approx 0.028

on Tv-them). Tv-them : taxonomic vs. thematic (primary); Tv-unrel : taxonomic vs. unrelated; Adv : mean taxonomic advantage. Chance is

\tfrac{1}{3}

for 3-way and

0.50

for the pairwise columns. Bold marks the best condition in each column within a scale.

Figure 1: Taxonomic-versus-thematic accuracy at 10M and 100M for the four conditions. The dashed line marks chance (

0.50

10M
Condition	K-ARI	K-NMI	A-ARI
Ortho	0.483	0.601	0.337
P-Goriely	0.279	0.439	0.257
P-breaks	0.290	0.417	0.260
Prosody-NN	0.303	0.454	0.259
100M

Table 6: SCI cluster-recovery scores (

n=68

words,

k=6

). K-ARI , K-NMI :

k

-means ARI and NMI (K-ARI primary); A-ARI : agglomerative ARI. Bold marks the best condition in each column within a scale.

Figure 2:

K

-means ARI at 10M and 100M for the four conditions. P-breaks and Prosody-NN cross above Ortho at 100M; the dashed line marks ARI

=0

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture
Hyperparameter	10M	100M
Variant	BERT-small	BERT-base
Hidden size	256	768
Layers	8	12
Attention heads	8	12
FFN size	1,024	3,072

Appendix

Table 7: Model architecture and training hyperparameters at the two scales. Architecture dimensions differ by scale; training settings are shared except batch size, gradient accumulation, and peak learning rate.

Source	10M	100M	Share
CHILDES (child-directed)	2.8M	28.9M	29%
Project Gutenberg (prose)	2.5M	26.4M	26%
OpenSubtitles (dialogue)	2.0M	20.0M	20%
Simple Wikipedia	1.5M	14.7M	15%
BNC Spoken (conversation)	0.9M	7.8M	8%
Switchboard (telephone)	0.1M	1.3M	1%

Appendix

Table 8: BabyLM training corpus composition (approximate word counts).

Rule	Trigger
A	Coordinating clause junction: a coordinating conjunction heads a clause-level cc dependency on a verbal or auxiliary root.
B	Subordinate or complement clause onset: a subordinating conjunction or complementiser marks an adverbial, complement, or embedded clause.
C	Relative clause onset: a relative pronoun or the relcl arc.
D	Minor surface punctuation: comma, semicolon, colon, dash, or ellipsis.
E	Appositive or parenthetical nominal: a token bearing the appos dependency and not preceded by punctuation.

Appendix

Table 9: Syntactic cases that trigger a [B2] token in P-breaks .

Head	$F_{1}$	Ship threshold
[B2]	0.58	$\geq 0.50$
[B4]	0.99	$\geq 0.80$
[L%]	0.69	$\geq 0.60$
[H%]	0.56	$\geq 0.30$

Appendix

Table 10: Held-out

F_{1}

of the neural annotator against the shipping thresholds. All four clear their gate, so the full annotator is used.

	Biphone	Triphone
	(controlled)	(uncontrolled)
Real word favoured	58.1%	70.3%
Mean difference	$+0.31$	$+1.21$
Median difference	$+0.21$	$+0.89$
Skew	$+0.56$	$+0.75$
Mean $-$ median	$+0.10$	$+0.32$

Appendix

Table 11: Phonotactic asymmetry between the real words and the Wuggy pseudowords of the BabySLM lexical development set (

n=17{,}706

pairs), under a biphone model, which Wuggy controls, and a triphone model, which it does not.

Category	Filter B IPA collisions removed
LexicalMemory	959
Composition	1,053
Inverted	3,244

Appendix

Table 12: Pseudowords rejected by the phonemic screen (Filter B) because their IPA form matched a real corpus word.

LexicalMemory
Real	Pseudoword	$\mathbf{S_{\text{real}}}$	$\mathbf{S_{\text{fake}}}$
pencil \ipafont /pɛnsəl/	blaver \ipafont /bleɪvɚ/	$-18.7$	$-18.7$
heart \ipafont /hɑːɹt/	pred \ipafont /pɹɛd/	$-11.9$	$-12.0$
Composition
reissued \ipafont /ɹiːɪʃuːd/	pophini \ipafont /pəfiːni/	$-29.1$	$-29.1$
foppish \ipafont /fɑːpɪʃ/	cenfse \ipafont /sɛnfs/	$-25.0$	$-25.0$

Appendix

Table 13: Example real-word / pseudoword pairs with Witten-Bell triphone phonotactic scores

S

. Standard pairs are score-matched; Inverted pairs satisfy

S_{\text{fake}}>S_{\text{real}}

Category	Words
Animal	dog, cat, bird, fish, bear, duck, horse, cow, pig, frog, mouse, sheep
Food	milk, juice, water, cookie, apple, banana, bread, cheese, cake, egg, soup, rice
Body part	eye, nose, mouth, ear, hand, foot, head, hair, finger, knee, arm, leg
Clothing	shoe, sock, hat, shirt, coat, dress, boot, sweater, diaper, button, pocket
Vehicle	car, bus, truck, bike, boat, plane, wagon, stroller, tractor
Household	cup, spoon, bowl, plate, bottle, chair, table, bed, blanket, door, phone, clock

Appendix

Table 14: The 68 CDI target words by semantic category ( Fenson et al., 1993 ) .

Condition	Tokens	Per utt.	Structural	% struct.	vs P-Goriely	vs Ortho
10M
Ortho	14,979,765	12.85	0	0.00%	$+22.0\%$	n/a
P-Goriely	12,276,352	10.55	1,010,527	8.23%	n/a	$-18.0\%$
P-breaks	13,498,252	11.59	2,152,618	15.95%	$+10.0\%$	$-9.9\%$
Prosody-NN	13,417,556	11.52	2,104,219	15.68%	$+9.3\%$	$-10.4\%$
100M

Appendix

Table 15: BPE token counts per condition and scale, with the share contributed by structural tokens and the budget relative to P-Goriely and to Ortho .

Figure 3: UMAP projections of the 68 CDI word embeddings, coloured by semantic category, for Ortho and Prosody-NN at both scales;

k

-means ARI is annotated in each panel.

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 522 papers

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs
Sep 29, 2026Nikitas Theodoropoulos, Maria Lymperaiou, Giorgos FilandrianosMultilingual Language Models Child Language Acquisition
Independent Researcher · National Technical University of Athens · Instituto de Telecomunicações, Portugal
A retrospective analysis on the use of LLMs to study infant syntax learning
Sep 22, 2026Hélie Bazin, Anouk Barberousse, François YvonLLM Evaluation Language Modeling
SCAI, SND, ISIR · SND · MLIA
Child-directed speech facilitates production, not comprehension, in BabyLMs
May 31, 2026Bastian Bunzeck, Sina ZarrießLanguage Model Pretraining LLM Evaluation
Computational Linguistics, Department of Linguistics Bielefeld University, Germany

Sep 29, 2026J/K move · Enter open · S save

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

Nikitas Theodoropoulos, Maria Lymperaiou, Giorgos Filandrianos

Independent Researcher · National Technical University of Athens · Instituto de Telecomunicações, Portugal

Multilingual Language Models Child Language Acquisition

Open paper AskPDF arXiv

Cross-linguistic effects are a central topic in bilingual first-language acquisition. Artificial learners can help investigate L1-L2 interactions by enabling controlled comparisons across language combinations and learning conditions. Recent work explores this direction by training bilingual language models under developmentally plausible constraints. However, human and model learners still diverge in fundamental ways, with one major difference being input modality: children learn primarily from spoken input, whereas language models are typically trained on orthographic text. To reduce this gap, researchers have trained models on phonemic representations of speech. In this work, we combine these research directions to train bilingual BabyLMs with phonemic input. We keep English fixed as the L2 and vary the L1 across German, Swedish, Persian, and Basque, selected to represent contrasting combinations of syntactic and phoneme-inventory distance from English. Our results show stronger L1-related variation in grammatical learning trajectories under phonemic than orthographic input, while early lexical differences align with phoneme-inventory similarity.

Abstract

Language Modeling

Explore similar work

Cross-Linguistic Effects in Bilingual Phoneme BabyLMs

A retrospective analysis on the use of LLMs to study infant syntax learning

Child-directed speech facilitates production, not comprehension, in BabyLMs