Automatic speech recognition (ASR) has the potential to substantially reduce manual annotation effort in child speech research by generating automatic transcriptions. However, obtaining reliably high-quality ASR transcriptions for child speech remains challenging in low-resource languages due to limited child-specific pre-trained models and highly diverse noise conditions. This study investigates the effectiveness of state-of-the-art ASR models on child speech through two research questions, by evaluating nine ASR models from three model families (Whisper, Parakeet, and Wav2Vec2) on two Dutch child speech datasets, JASMIN and DART. Research question 1 examines the performance of ASR-models applied to child speech. The fine-tuned Whisper-medium model achieves the best overall performance, with a WER of 5.54% on JASMIN and 70.37% on DART, showing that the noisy DART data are clearly more challenging. Research question 2 examines to what extent it is possible to select a subset for which reliable orthographic transcriptions can be obtained automatically, without the need for manual verification. We use an utterance-level selection method that compares ASR output with the original read prompt to identify correctly pronounced recordings. Using the proposed selection method, 42.0% [for JASMIN] and 18.1% [for DART] of the utterances can be automatically identified as correctly pronounced with high confidence, resulting in very low error rates on an utterance level (precisions of 98.3% and higher) and reducing the need for manual verification.
Figures & tables
Property
JASMIN
DART
DART (subset)
Total speech files (n)
10,642
2,343
1,049
Testing or Evaluation set
Train / Evaluate
Evaluate
Evaluate
Total Duration
9 hours 51 minutes
2 hours 54 minutes
0 hours 44 minutes
Sentences / Words ratio (%)
90.3 / 9.7
12.4 / 87.6
8.5 / 91.5
Reading Mistakes Manual (%)
53.5
45.4-53.9
19.8
Table 1: Overview of the datasets used in this study and their key properties. Reading Mistakes Manual (RMM) describes the total amount of utterances in the full datasets in which the children made at least one error pronouncing a prompt.
Model Name
Base Model
Params (M)
Finetuned
whisper-large
openai/whisper-large-v2
1550
-
whisper-large-prompted
openai/whisper-large-v2
1550
-
whisper-small-FT
openai/whisper-small
244
v
whisper-medium-FT
openai/whisper-medium
769
v
whisper-large-FT
openai/whisper-large-v2
1550
v
Wav2Vec2-gronlp
GroNLP/Wav2Vec2-dutch-large
317
-
Table 2: Overview of ASR models used in the experiments, including base model, number of parameters, and whether the model was custom finetuned on Dutch child speech in the current study.
PR
MO
AO
RMM
RMA
Outcome
zij heeft een klacht
zij heeft een klant
we heeft een klans
0
0
TN
duikplank
duikplank
duikplank
1
1
TP
schrapen
schrappen
schrapen
0
1
FP
hoge bomen
hoge bomen
handgebaren
1
0
FN
Table 3: Example comparisons for true positive, true negative, false positive, false negative for research question 2. PR refers to the original prompt the child read, which are included in the JASMIN and DART corpora. MO refers to the manual orthographic transcription, also included in the two corpora used. AO refers to the ASR-prediction generated by a model. RMA (Reading Mistakes Automatic) represents whether AO matches PR in its entirety, if so denoted with a 1. RMM (Reading Mistakes Manual) represents whether MO matches PR in its entirety.
Figure 1: WER and UER of all models for JASMIN (top graph) and DART (bottom graph). The horizontal dotted line represents the baseline measure (a non-modified version of whisper-large-v2).
JASMIN (n=1550)
DART (n=1047)
Model
WER
CER
UER
WER
CER
UER
whisper-large (baseline)
16.94
9.34
58.03
77.57
45.99
76.41
whisper-large-prompted
15.65
7.11
56.61
73.37
41.29
73.45
whisper-small-FT
8.94
3.33
37.72
75.62
35.44
70.01
whisper-medium-FT
5.54
2.08
27.14
70.37
33.27
63.13
whisper-large-FT
5.46
2.03
27.08
78.40
38.33
67.72
Table 4: ASR performance on the JASMIN and DART datasets measured in WER, CER, and UER. Bolded values represent the best performance for a metric/dataset combination. The -FT suffix denotes that the model has been finetuned on child audio.
JASMIN (n=1550)
DART (n=1047)
DART (n=2343)
Model
P
R
F1
MCC
P
R
F1
MCC
P
R
F1
MCC
whisper-large
80.1
59.5
68.3
0.49
99.0
23.8
38.3
0.23
94.6
19.0
31.7
0.28
whisper-prompted
89.2
59.7
71.7
0.59
99.6
27.3
42.9
0.25
96.2
21.6
35.2
0.31
whisper-small-FT
97.9
77.8
86.7
0.79
99.6
29.6
45.6
0.27
95.4
25.6
40.3
0.32
whisper-medium-FT
98.3
88.9
93.4
0.88
100.0
36.6
53.6
0.32
99.3
32.9
49.4
0.42
whisper-large-FT
96.8
89.5
93.0
0.88
99.3
31.9
48.3
0.28
97.8
27.2
42.6
0.37
Table 5: Precision (P), Recall (R), F1-score (F1), and Matthews Correlation Coefficient (MCC) for utterance selection across datasets. Best performance for the individual models is displayed in bold (top 9 rows). Conditions in which output of the three finetuned models is combined (whisper-medium-FT, Wav2Vec2-FT, and parkeet-FT) are displayed in the bottom two rows, with the best value for combined strategies also bolded. † For DART n=1047, comb-and-condition agreement is a case in which at least two out of three models of [whisper-medium-FT, Wav2Vec2-FT, parakeet-FT] match the PR. For DART n=2343 and JASMIN n=1550, comb-and-condition are cases in which all three match the PR.
Error
MO → AO
Description
Frequency
S
hij → haar
AO transcribes ”haar” instead of ’hij’ for 3 cases in which child hesitated during pronouncing ’hij’.
3
S
een → hun
There are multiple case in which ”hun” is transcibed by ASR as ”een” and vice versa.
3
S
d’r → haar
”d’r” is a short-form common contraction of ”haar”, in multiple sentences instances of ”er”, ”d’r”, and ”haar” produce errors between MO and AO.
2
I
(none) → de
Insertion of article ”de”, in two instances the MO forgot to transcribe these.
4
I
(none) → een
Insertion of ”een”, due to hallucination (2 times) or MO error (1 time).
3
I
(none) → naar
Insertion of ’naar’ in cases in which child spoke very rapidly.
2
Table 6: Overview of common errors made for the JASMIN dataset.
Error
MO → AO
Description
Frequency
S
pluimpje → blijmpje
Failure to transcribe ”plui”, instead transcribing ”blij” in the word ”pluimpje”.
4
S
besproei → bestproe
Consistently transcribing regularly pronounced ’besproei’ as ’bestproe’
3
S
fraai → vrij
Three cases, of wich 1 recording has poor speech quality, one fringe case with deviant ’fraai’ pronunciation, and one regular case in which ASR-model failed to detect ’raai’.
3
I
(none) → de
Many cases in which normally pronounced word was split by ASR into two subwords (e.g., ”verhaal” transcribed by ASR-model as ”de” + ”haa”).
8
I
(none) → op
Cases in which ASR incorrectly split one spoken word into 2 subwords.
6
I
(none) → ja
A number of cases where caused by the ASR-model transcribing background noise of children yelling, or small partial pronunciations of the word ”ja”.
4
Table 7: Overview of common errors made for the DART dataset.
Reference (MO)
Hypothesis (AO)
Error
Explanation
net als ze zich willen omdraaien blijft de lichtstraal hangen bij een zoort rek
net als ze zich willen omdraaien blijft de lichtstraal hangen bij een soort rek
Automatic Speech Recognition (ASR) is increasingly used in applications involving child speech, such as language learning and literacy acquisition. However, the effectiveness of such applications is limited by high ASR error rates. The negative effects can be mitigated by identifying in advance which ASR-outputs are reliable. This work aims to develop two novel approaches for selecting reliable ASR-output at the utterance level, one for selecting reliable read speech and one for dialogue speech material. Evaluations were done on an English and a Dutch dataset, each with a baseline and finetuned model. The results show that utterance-level selection methods for identifying reliably transcribed speech recordings have high precision for the best strategy (P > 97.4) for both read speech and dialogue material, for both languages. Using the current optimal strategy allows 21.0% to 55.9% of dialogue/read speech datasets to be automatically selected with low (UER of < 2.6) error rates.
Gus Lathouwers, Lingyun Gao, Catia Cucchiarini +1
Centre for Language Studies, Radboud University, Netherlands · Department of Language and Communication, Radboud University, Netherlands
Longform audio recordings obtained with microphones worn by children-also known as child-centered daylong recordings-have become a standard method for studying children's language experiences and their impact on subsequent language development. Transcripts of longform speech audio would enable rich analyses at various linguistic levels, yet the massive scale of typical longform corpora prohibits comprehensive manual annotation. Meanwhile, automatic speech recognition (ASR)-based transcription faces significant challenges due to the noisy, unconstrained nature of real-world audio. Previous attempts have assumed that ASR must process each longform recording in its entirety. In this work, we present an approach to automatically detect those utterances in longform audio that can be reliably transcribed with modern ASR systems, allowing automatic and relatively accurate transcription of a notable proportion of all speech in typical longform data. We validate the approach on four English longform corpora, showing that it achieves a median word error rate (WER) of 0% and a mean WER of 16% when transcribing 30% of the total speech in the dataset. In contrast, transcribing all speech without any filtering yields a median WER of 52% and a mean WER of 51%. We also compare word log-frequencies derived from the automatic transcripts with those from manual annotations and show that the frequencies correlate at r = 0.94 (Pearson) for all transcribed words and r = 0.99 for words that appear at least five times in the automatic transcripts. Overall, the work provides a concrete step toward increasingly detailed automated linguistic analyses of child-centered longform audio.
Daniil Kocharov, Azarias Galama, Okko Räsänen
Signal Processing Research Centre, Tampere University, Finland, Faculty of Information Technology and Communication Sciences, Department of Computing Sciences, P.O. Box 553, FI-33101 Tampere, Finland
Automatic Speech Recognition (ASR) systems often underperform for children and non-native speakers, while adapting adult ASR models to child speech can cause adult-speech forgetting. We study child ASR adaptation with adult retention across Arabic and English. We compare full fine-tuning, LoRA, and post-hoc weight-space merging across encoder--decoder, encoder--CTC, and AudioLLM-based ASR systems. Experiments use Arabic native and non-native child speech, English MyST child speech, and adult benchmarks from MGB-2 and LibriSpeech test-clean. We evaluate recognition quality with WER and quantify the adaptation--retention trade-off using Retention Index, Child Adaptation Gain, and Adaptation Recovery. Results show that child adaptation is necessary, especially for non-native Arabic and English child speech, but direct adaptation often reduces adult ASR performance. Bilingual adaptation is more stable than language-specific adaptation. Weight-space merging often improves the trade-off, especially for encoder--CTC, Whisper, and AudioLLM-based ASR, with LERP favoring adult retention and TIES recovering stronger child gains. For the encoder--decoder model, direct bilingual fine-tuning remains strongest in raw WER.\footnote{Code, and models are available at https://github.com/qcri/Child-ASR-Adaptation.