Enhancing Speech Recognition through Text-to-Speech and Voice Conversion Augmentation

Yunus Emre Ozkose; Ali Haznedaroglu

cs.SDOct 9, 2026

Enhancing Speech Recognition through Text-to-Speech and Voice Conversion Augmentation

Authors: Yunus Emre Ozkose, Ali Haznedaroglu

Organizations: Sestek, Istanbul, Turkey

Abstract

Given the challenges associated with obtaining large volumes of real-world audio, the advancement of Automatic Speech Recognition (ASR) systems increasingly depends on synthetic data. This study focuses on the call center-banking domain, providing a targeted analysis of ASR models in this context. We evaluate real and synthetic datasets, using Word Error Rate (WER) and Character Error Rate (CER) as metrics, and employ a speech quality model to assess the impact on ASR performance. Our research compares Text-to-Speech (TTS) and advanced voice conversion methods, including KNNVC, Seed-VC, and Vec2Wav2, revealing significant improvements in speech quality and ASR accuracy, particularly with Seed-VC. Additionally, our domain-specific experimentation provides insight into the unique challenges and opportunities that arise when applying ASR technologies to industry-relevant settings. This highlights the potential of voice conversion technologies to enhance ASR systems, guiding future research in diverse linguistic scenarios, and paving the way for the broader application of ASR innovations across various fields.

Figures & tables

Figure 1: Histogram of NISQA-predicted MOS (Mean Opinion Score) values for real and synthetic speech data. The "Real Data" (blue) shows a broad distribution of perceived quality, reflecting diverse acoustic conditions. The TTS system (orange) yields consistently high scores with a sharp peak near 5.0, indicating very high synthetic quality but limited variability. Voice conversion methods (knn-VC in green, sedd-VC in red, and vec2wav2 in purple) produce more naturally distributed scores, with sedd-VC and vec2wav2 achieving high MOS values while preserving greater variability than TTS. These results highlight the trade-off between synthetic clarity and realism across systems.

Train Data	CER $\downarrow$	WER $\downarrow$
Real Data	15.33	34.40
TTS	80.18	97.10
KNNVC	55.08	83.06
Seed-VC	43.97	75.53
Vec2Wav2	46.62	78.42

Table 1: Word Error Rate (WER) comparison for different types of training data in an Automatic Speech Recognition (ASR) system, evaluated over training duration of 20 hours. The results highlight the effectiveness of real-world audio data and various synthetic speech generation methods, including TTS and voice conversion techniques such as KNNVC, Seed-VC, and Vec2Wav2.

Train Data	CER $\downarrow$	WER $\downarrow$
KNNVC $\times$ 5	52.97	82.25
Seed-VC $\times$ 5	41.47	73.99
Vec2Wav2 $\times$ 5	44.4	75.77
TTS + KNNVC	52.19	81.8
TTS + Seed-VC	43.95	74.62
TTS + Vec2Wav2	46.42	76.82

Table 2: Word Error Rate (WER) analysis of speech recognition performance using enhanced voice conversion models and combinations with Text-to-Speech (TTS). Evaluated over training duration of 20 hours, this table presents results for models trained on synthetic speech generated through repeated applications of KNNVC, Seed-VC, and Vec2Wav2, as well as the integration of TTS-generated speech with these voice conversion techniques.

Train Data	CER $\downarrow$	WER $\downarrow$
Real Data + TTS + KNNVC	12.32	28.98
Real Data + TTS + KNNVC $\times$ 5	13.71	30.26
Real Data + TTS + Seed-VC	13.71	32.30
Real Data + TTS + Seed-VC $\times$ 5	13.25	29.71
Real Data + TTS + Vec2Wav2	12.12	28.65
Real Data + TTS + Vec2Wav2 $\times$ 5	13.51	30.26

Table 3: Performance comparison of Word Error Rate (WER) when blending real and synthetic training data in Automatic Speech Recognition (ASR) systems. The table highlights WER improvements with real data alone and in combination with TTS and multiple iterations of voice conversion techniques (KNNVC, Seed-VC, Vec2Wav2). Evaluated over 20 hours of training data, configurations include single and multiple applications (denoted as

\times

5) of voice conversion methods

	Amount of Hours in Training
	10 hours		20 hours		50 hours
Train Data	CER	WER	CER	WER	CER	WER
Real Data	35.57	58.71	15.33	34.40	10.44	25.48
Real Data + TTS + KNNVC	24.38	47.08	12.32	28.98	10.18	25.47
Real Data + TTS + Seed-VC	23.89	46.56	13.71	32.30	10.19	25.19
Real Data + TTS + Vec2Wav2	24.5	46.72	12.12	28.65	10.04	25.26

Table 4: An In-Depth Analysis of Character Error Rate (CER) and Word Error Rate (WER) for Various Voice Conversion Models, Evaluating Performance Across Different Training Durations: A Comprehensive Comparison for 10, 20, and 50 Hours of Training on Real Data, Augmented with TTS and Different Voice Conversion Techniques Including KNN-VC, Seed-VC, and Vec2Wav2.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation
Mar 11, 2026Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo +2Large Language Model-Based Data Augmentation Domain Adaptation for ASR
Hitachi, Ltd.
How to Leverage Synthetic Speech for LLM-Based ASR Systems?
Jun 27, 2026Yanis Labrak, Dairazalia Sanchez-Cortes, Sergio Burdisso +9Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Idiap Research Institute, Martigny, Switzerland · Universite de Toulon, Aix Marseille Univ, LIS, CNRS, Toulon, France · EPFL, Lausanne, Switzerland +3
Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition
Jun 19, 2026Raphaël Bagat, Zhe Zhang, Junichi Yamagishi +2Synthetic Data Generation Speech Processing
Universit´e de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France · National Institute of Informatics, Tokyo, Japan

Mar 11, 2026J/K move · Enter open · S save

Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation

Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi+1

Hitachi, Ltd.

Large Language Model-Based Data Augmentation Domain Adaptation for ASR Synthetic Data Augmentation for Speech Recognition Data Augmentation Synthetic-to-Real Domain Adaptation

Open paper AskPDF arXiv

End-to-end automatic speech recognition often degrades on domain-specific data due to scarce in-domain resources. We propose a synthetic-data-based domain adaptation framework with two contributions: (1) a large language model (LLM)-based text augmentation pipeline with a filtering strategy that balances lexical diversity, perplexity, and domain-term coverage, and (2) phonetic respelling augmentation (PRA), a novel method that introduces pronunciation variability through LLM-generated orthographic pseudo-spellings. Unlike conventional acoustic-level methods such as SpecAugment, PRA provides phonetic diversity before speech synthesis, enabling synthetic speech to better approximate real-world variability. Experimental results across four domain-specific datasets demonstrate consistent reductions in word error rate, confirming that combining domain-specific lexical coverage with realistic pronunciation variation significantly improves ASR robustness.

Abstract

Automatic Speech Recognition

Explore similar work

Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation

How to Leverage Synthetic Speech for LLM-Based ASR Systems?

Synthetic Audio Generation Framework for Air Traffic Control Speech Recognition