Elderly-Contextual Data Augmentation via Speech Synthesis for Elderly ASR
Authors: Minsik Lee, Seoi Hong, Chongmin Lee, Sieun Choi, Jian Kim, Jua Han, Jihie Kim
Organizations: Department of Computer Science and Artificial Intelligence, Dongguk University, Seoul, South Korea · Department of Biostatistics, Harvard University, Boston, MA, USA
Despite recent progress in automatic speech recognition (ASR), elderly ASR (EASR) remains challenging due to limited training data and the distinct acoustic and linguistic characteristics of elderly speech. In this work, we address data scarcity in EASR through a data augmentation pipeline that combines large language model (LLM)-based transcript paraphrasing with text-to-speech (TTS) synthesis. Given an elderly speech dataset, the LLM first generates elderly-contextual paraphrases of the original transcripts, and the TTS model then synthesizes corresponding speech using elderly reference speakers. The resulting synthetic audio-text pairs are merged with the original data to fine-tune Whisper without architectural modification. We further analyze the effects of augmentation ratio and reference-speaker composition in low-resource EASR. Experiments on English and Korean elderly speech datasets from speakers aged 70 and above show that the proposed method consistently improves performance over conventional augmentation baselines, achieving up to a 58.2% reduction in word error rate (WER) compared with the Whisper baseline.
Figures & tables
Fig. 1: Age distribution of the CommonVoice 18.0 English corpus.
Fig. 2: Overview of the proposed EASR augmentation pipeline. An LLM first paraphrases each transcript into an elderly-contextual form, and a TTS model then generates one synthetic audio sample using a reference speaker selected from the elderly speaker pool. The resulting augmented dataset is merged with the original dataset to fine-tune Whisper.
Step
LLM Instructions and Description
1. Preparation
We first present the instruction, ”Make a sentence into a CSV file,” allowing the LLM to generate sentences related to the elderly based on the transcription data.
2. Sentence Paraphrasing
Specific paraphrasing guidelines are provided, such as “Write a sentence, related to the CSV file” and “Do not repeat the original sentence.” A word limit is also set: “Use a minimum of 3 words and a maximum of 20 words.”
3. Elderly-Contextual Paraphrasing
The final instruction guides the LLM to adjust the tone and content. For instance: “Change the given sentence into a sentence frequently used by elderly people.”
TABLE I: Prompt structure for LLM-based elderly-contextual paraphrasing, including formatting, paraphrasing constraints, and contextual transformation.
Models
CV18 (%)
VOTE400 (%)
WER
CER
WER
CER
Baseline
4.1
1.8
11.6
5.5
+ TTS [ 23 ] (W/o LLM)
3.1
1.4
8.8
4.5
+ TTS [ 23 ] (W/ LLM)
2.7
1.2
6.5
3.5
Proposed (LLM+TTS, ECT)
2.2
1.0
5.2
2.9
TABLE II: Performances according to the proposed method combined with SpecAugment on CV18 and VOTE400.
Dataset
Aug %
Spk.
Whisper Model Performance (WER / CER)
Small
Medium
Large-v2
Large-v3
CV18
Baseline
-
5.8 / 2.6
4.3 / 1.9
4.2 / 1.8
4.1 / 1.8
+10%
2
5.4 / 2.4
4.0 / 1.8
3.8 / 1.7
3.8 / 1.6
+30%
4
4.8 / 2.1
3.4 / 1.5
3.2 / 1.4
3.1 / 1.4
+50%
8
3.9 / 1.7
2.7 / 1.2
2.5 / 1.1
2.5 / 1.1
+70%
8
3.7 / 1.6
2.5 / 1.1
2.4 / 1.0
2.3 / 1.0
TABLE III: Ablation of augmentation ratio using ECT-based synthetic data across Whisper model sizes.
Method / Factor
CV18 (WER / CER)
VOTE400 (WER / CER)
(1) Comparison of Data Augmentation Methods
Baseline
4.1 / 1.8
11.6 / 5.5
+ Speed perturbation [ 39 ]
3.9 / 1.7
10.7 / 5.1
+ SpecAugment [ 40 ]
3.6 / 1.5
10.1 / 4.8
Proposed (LLM+TTS, ECT)
2.2 / 1.0
6.1 / 2.9
Proposed + Speed perturbation
2.1 / 0.9
5.9 / 2.9
TABLE IV: Results of Data Augmentation Methods, Speaker Balance, and LLM on EASR Performance.
Part A: ASR Transcription Correction Examples
CV18 (English Example)
Reference
My grandchild has grown up so much.
Proposed
My grandchild has grown up so much. (Correct)
Baseline
My and child has grown up so much. (Incorrect)
VOTE400 (Korean Example)
Reference
오늘 병원에 가서 혈압 약(hyeorap yak) 타왔어.
TABLE V: Qualitative Analysis Examples, (A) Examples of ASR Outputs, Highlighting Corrections for Phonetic Ambiguity in English (CV18) and Domain-Specific Lexicon in Korean (VOTE400). (B) Examples of Elderly-Contextual Paraphrasing From CV18.
Dept. of Telecommunications and Artificial Intelligence, Budapest University of Technology and Economics, Budapest, Hungary · SpeechTex Ltd., Budapest, Hungary · ELTE Research Centre for Linguistics, Budapest, Hungary