Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation

Natsuo Yamashita; Koichi Nagatsuka; Hiroaki Kokubo; Kota Dohi; Tuan Vu Ho

eess.ASMar 11, 2026

Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation

Authors: Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu Ho

Organizations: Hitachi, Ltd.

Abstract

End-to-end automatic speech recognition often degrades on domain-specific data due to scarce in-domain resources. We propose a synthetic-data-based domain adaptation framework with two contributions: (1) a large language model (LLM)-based text augmentation pipeline with a filtering strategy that balances lexical diversity, perplexity, and domain-term coverage, and (2) phonetic respelling augmentation (PRA), a novel method that introduces pronunciation variability through LLM-generated orthographic pseudo-spellings. Unlike conventional acoustic-level methods such as SpecAugment, PRA provides phonetic diversity before speech synthesis, enabling synthetic speech to better approximate real-world variability. Experimental results across four domain-specific datasets demonstrate consistent reductions in word error rate, confirming that combining domain-specific lexical coverage with realistic pronunciation variation significantly improves ASR robustness.

Figures & tables

Panel of Figure 1 — Figure 1: Proposed methods overview. Italics denote placeholders.

Dataset	Utts.	Terms	Term Ratio	Scenarios	Synthetic utts.	Filtered utts.	Filtered duration
ATCOSIM [ 12 ]	1,901	44	8.4 %	176	165K	26.7K	50h
ATCO2 [ 34 ]	871	27	3.6 %	108	121K	22.4K	50h
Court [ 27 ]	3,639	33	0.5 %	132	148K	21.9K	50h
MedSyn [ 5 ]	7,906	605	4.9 %	2,420	2900K	27.5K	50h

Table 1: Summary of evaluation subsets and generated data. Term Ratio is the proportion of test words that are domain-specific terms.

	ATCOSIM					ATCO2					Court					MedSyn
		Ours					Ours					Ours					Ours
	DAS	Pipeline	Rand.	VCM	PPLmin	DAS	Pipeline	Rand.	VCM	PPLmin	DAS	Pipeline	Rand.	VCM	PPLmin	DAS	Pipeline	Rand.	VCM	PPLmin
MATTR	0.784	0.884	0.875	0.895	0.842	0.763	0.879	0.869	0.882	0.844	0.818	0.801	0.778	0.806	0.737	0.902	0.916	0.902	0.917	0.862
Distinct-2	0.226	0.269	0.253	0.336	0.203	0.220	0.246	0.238	0.300	0.203	0.291	0.300	0.256	0.328	0.171	0.340	0.416	0.384	0.534	0.187
Perplexity	819.6	1168.0	614.8	610.5	117.7	822.2	970.0	464.4	464.8	121.9	137.1	198.8	106.5	124.0	33.0	118.4	507.3	122.9	165.1	14.6
Avg. Term	0.3	248.5	111.2	110.5	72.2	5.2	345.4	136.2	120.2	100.5	13.4	280.2	124.5	82.8	66.9	1.6	20.4	8.3	3.9	6.2

Table 2: Results of varied filtering methods across datasets. The best and second best scores are bolded and underlined , respectively.

Large Language Model-Based Data Augmentation

See all 22 papers

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR
Apr 7, 2026Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello +9Domain Adaptation for ASR Cross-Modal Alignment
Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland +2
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
Jun 27, 2026Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso +9Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Idiap Research Institute, Martigny, Switzerland · Universite de Toulon, Aix Marseille Univ, LIS, CNRS, Toulon, France · EPFL, Lausanne, Switzerland +3
Few-Shot Synthetic Accented Speech for ASR Fine-Tuning: What Helps and When?
Apr 30, 2026Yurii Halychanskyi, Nimet Beyza Bozdag, Mark Hasegawa-Johnson +2Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
University of Illinois Urbana-Champaign, USA · National Center for Supercomputing Applications, USA

Apr 7, 2026J/K move · Enter open · S save

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

Thibault Bañeras-Roux, Sergio Burdisso, Esaú Villatoro-Tello, Dairazalia Sánchez-Cortés+8

Idiap Research Institute, Switzerland · Laboratoire d’Informatique et des Syst`emes, France · EPFL, Switzerland+2

Domain Adaptation for ASR Cross-Modal Alignment Automatic Speech Recognition Speech Language Models

Open paper AskPDF arXiv

Conventional end-to-end automatic speech recognition (ASR) systems rely on paired speech-text data for domain adaptation. Recent LLM-based ASR architectures connect a speech encoder to a large language model via a projection module, enabling adaptation with text-only data. However, this introduces a modality gap, as the LLM is not exposed to the noisy representations produced by the speech projector. We investigate whether small amounts of speech can mitigate this mismatch. We compare three strategies: text-only adaptation, paired speech-text adaptation, and mixed batching (MB), which combines both. Experiments in in-domain and out-of-domain settings show that even limited speech consistently improves performance. Notably, MB using only 10% of the target-domain (less than 4 hours) speech achieves word error rates comparable to, or better than, conventional ASR fine-tuning with the full dataset, indicating that small amounts of speech provide a strong modality-alignment signal.

Abstract

Large Language Model-Based Data Augmentation

Explore similar work

Closing the Speech-Text Gap with Limited Audio for Effective Domain Adaptation in LLM-Based ASR

How to Leverage Synthetic Speech for LLM-Based ASR Systems?

Few-Shot Synthetic Accented Speech for ASR Fine-Tuning: What Helps and When?