When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Anupam Purwar; Aditya Choudhary

cs.SDMar 11, 2026

When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Authors: Anupam Purwar, Aditya Choudhary

Organizations: Sprinklr AI Gurugram, India

Abstract

Large language models are increasingly adopted as semantic backbones for neural text-to-speech systems. However, frozen LLM representations are insufficient for modeling speaker specific acoustic and perceptual characteristics. Our experiments involving fine tuning of the Language Model backbone of TTS show promise in improving the voice consistency and Signal to Noise ratio SNR in voice cloning task. Across multiple speakers LoRA finetuning consistently outperforms the non-finetuned base Qwen-0.5B model across three complementary dimensions of speech quality. First, perceptual quality improves significantly with DNS-MOS gains of up to 0.42 points for speakers whose training data exhibits sufficient acoustic variability. Second, speaker fidelity improves for all evaluated speakers with consistent increases in voice similarity indicating that LoRA effectively adapts speaker identity representations without degrading linguistic modeling. Third, signal level quality improves in most cases with signal to noise ratio increasing by as much as 34 percent. Crucially these improvements are strongly governed by the characteristics of the training data. Speakers with high variability in acoustic energy and perceptual quality achieve simultaneous gains in DNS-MOS voice similarity and SNR. Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems. When supported by sufficiently diverse training data LoRA adapted Qwen-0.5B consistently surpasses its frozen base model in perceptual quality speaker similarity with low latency using GGUF model hosted in quantized form.

Figures & tables

Paper	What was implemented (2-line summary)	LoRA on LLM TTS Backbone
LoRP-TTS Bondaruk and Kubiak (2025) Bondaruk & Kubiak (2025)	Applies LoRA to TTS model layers for rapid speaker personalization from a single noisy reference utterance. Low-rank updates adapt speaker characteristics while keeping the synthesis backbone frozen.	No
TTS-Hub Anonymous (2025) OpenReview (ICLR submission)	Introduces modular LoRA adapters trained for individual speech attributes and composed arithmetically. Enables controllable TTS without modifying the underlying language or synthesis backbone.	No
StyleSpeech Lou et al. (2024) Lou et al. (ACM MM Asia 2024)	Employs parameter-efficient LoRA fine-tuning on pre-trained TTS components for phoneme and style control. Focuses on efficiency and controllability rather than LM adaptation.	No
LoRA-based Multi-Speaker TTS IEEE Access (2024)	Integrates LoRA and residual adapters into VITS-based multi-speaker TTS architectures. Achieves efficient speaker adaptation with substantially fewer trainable parameters.	No
EELE Qi et al. (2024) arXiv:2408.10852	Applies LoRA modules to emotional and expressive control layers in TTS systems. Targets parameter-efficient emotion adaptation without altering linguistic modeling layers.	No
UtterTune Kato (2025) (2025)	Injects LoRA directly into the Transformer-based language model backbone of an LLM-TTS system. Enables pronunciation and pitch-accent control by adapting semantic/phoneme prediction layers.	Yes

Table 1: Comparison of LoRA-based Methods in Text-to-Speech Systems

Figure 1: End-to-End Pipeline of a cascading voice agent comprising of a ASR module to transcribe voice to text, LLM module to reason over the text and generate text output and lastly a TTS module to synthesize speech for the reasoned text. Our work to the pipeline focuses on finetuning the LLM backbone (enclosed in red) responsible for token prediction, which are subsequently decoded by the neural codec to generate the waveform

	Data Len (hr.)	Avg Len (s)	Std. Dev. (s)	Unique Words	Avg Words
1	3.523	2.536	1.239	4492	6.717
2	3.907	2.813	1.633	7981	7.521
11614	3.860	2.778	1.471	7460	8.069
1401	17.590	14.314	6.209	16064	41.399
1212	18.492	15.543	6.446	13067	42.575
1259	13.652	14.675	6.202	11943	42.978

Table 2: Summary of the datasets used

Speaker	Dataset	Dataset Len.	Energy	Data MOS	Base	LoRA 1000	$\Delta$ vs	LoRA 5
ID		used (hours)	Spread	(Mean $\pm$ Std)	MOS	Train Steps	Base	Epochs
1	HiFiTTS	3.523	-29.73 $\pm$ 12.93	3.48 $\pm$ 0.32	3.832	3.861	$+0.029$	3.813
2	HiFiTTS	3.907	-29.34 $\pm$ 13.11	3.51 $\pm$ 0.37	3.717	4.141	$+0.424$	4.106
11614	HiFiTTS	3.860	-32.20 $\pm$ 13.34	3.30 $\pm$ 0.38	3.680	3.818	$+0.138$	3.768
1401	LibriHeavy	17.590	-32.11 $\pm$ 11.89	3.44 $\pm$ 0.26	3.659	3.385	$-0.274$	3.435
1212	LibriHeavy	18.492	-28.41 $\pm$ 9.91	3.42 $\pm$ 0.25	3.647	3.233	$-0.414$	3.350

Table 3: Detailed Speaker Fine-tuning Results across HiFiTTS and LibriHeavy Datasets

Table 5

Panel of Figure 2 — Figure 2: Plots of Training and Validation Loss at various training steps for each speaker ID. The curves indicate stochastic decrease of training loss with training steps with saturation in validation loss as we approach 4 epochs

Table 7

Panel of Figure 3 — Figure 3: Plots indicating MOS Score (Y axis) of Audio Generated with varying lengths of cloning audio provided (X axis). Base refers to the standard NeuTTS model, Ref refers to the audio used for cloning. "Long" refers to generation of a longer sentence (180 characters) whereas other audios have 70 characters.

Metric / Speaker ID		1	2	11614	1401	1212	1259
Dataset		HiFi TTS			LibriHeavy-HQ
Dataset Length (Hr)		3.523	3.907	3.860	17.590	18.492	13.652
SNR	Base	41.747	31.562	33.403	32.163	20.397	16.895
	LoRA	55.854	39.567	36.217	40.778	12.412	15.395
	Ref	31.900	27.047	24.905	26.791	22.032	17.630
	Inc. Base/Ref %	30.868	16.690	34.124	20.049	-7.420	-4.170

Table 8: Results of LoRA Finetuning

Speaker ID	Energy Mean	Energy Std (Training data)	DNS-MOS Std (Training data)	Fine-tuning Outcome	Observed Pattern
2	-29.34	13.11	0.370	+0.424 $\times$	High energy variability + high DNS-MOS variability
11614	-32.20	13.34	0.383	+0.138 $\times$	Highest energy variability + highest DNS-MOS variability
1	-29.73	12.93	0.321	+0.029	Medium energy and DNS-MOS variability
1401	-32.11	11.89	0.255	-0.274 $\times$	Low energy variability + low DNS-MOS variability
1212	-28.41	9.91	0.248	-0.414 $\times$	Second-lowest energy variability + lowest DNS-MOS variability
1259	-31.18	8.25	0.293	-0.001	Lowest energy variability

Table 9: Impact of training-data energy statistics on DNS-MOS variability and fine-tuning outcome for speaker-specific LoRA adaptation of a Qwen-0.5B TTS backbone. While absolute energy levels (mean) show no consistent influence, higher energy variability in the training data strongly correlates with increased DNS-MOS variability and improved perceptual gains after fine-tuning.

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	3.76	4.01	3.86	3.50	3.32	3.72
Pure FT 1000	3.86	4.14	3.82	3.39	3.23	3.66
2+2+2 FT final	3.96	3.95	3.738	3.83	3.74	3.84
2+2+2 FT 1000	3.95	4.09	3.77	3.56	3.69	3.70
2+2+2 FT 2000	3.79	4.09	3.75	3.63	3.71	3.83
1+1+1 FT final	3.81	3.92	3.73	3.51	3.69	3.67

Table 10: Impact of different audio training data combinations used for Fine tuning Language model on MOS

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	0.790	0.794	0.737	0.503	0.678	0.481
2+2+2 FT final	0.749	0.780	0.686	0.412	0.569	0.292
2+2+2 FT 1000	0.740	0.813	0.728	0.354	0.569	0.292
2+2+2 FT 2000	0.752	0.799	0.698	0.438	0.546	0.423
1+1+1 FT final	0.740	0.786	0.679	0.396	0.566	0.399
1+1+1 FT 2000	0.756	0.785	0.672	0.422	0.539	0.401

Table 11: Impact of different audio training data combinations used for Fine tuning Language model on Similarity

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	57.265	41.640	51.437	35.504	19.995	22.704
2+2+2 FT final	51.957	55.632	45.849	32.065	76.746	23.612
2+2+2 FT 1000	41.104	67.328	65.867	36.580	32.736	22.195
2+2+2 FT 2000	43.791	45.714	37.693	33.385	26.819	23.006
2+2+2 FT final	28.871	50.757	54.411	32.979	27.112	21.942
1+1+1 FT 2000	62.695	35.731	67.628	38.310	37.083	19.507

Table 12: Impact of different audio training data combinations used for Fine tuning Language model on SNR

Sentence	S1		S2		S3		S4		S5
Env.	CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU
1st Chunk Latency (s)	0.3988	0.3067	0.4032	0.311	0.4134	0.3189	0.4195	0.3237	0.4167	0.3212
1st Chunk RTF	0.7386	0.568	0.7466	0.576	0.7656	0.5906	0.7768	0.5994	0.7717	0.5949
2nd Chunk RTF	0.5195	0.3155	0.5211	0.3124	0.524	0.3111	0.5217	0.313	0.5189	0.3194
MOS	3.19489	3.19882	3.48858	3.48964	3.78335	3.78371	3.89385	3.89359	3.73711	3.73772
Speaker Similarity	0.68481	0.68464	0.60134	0.60058	0.62643	0.62618	0.74191	0.74214	0.71144	0.71538

Table 13: Impact of running codec (NeuCodec) in onnx-runtime running on CPU vs GPU for different input text sentences ( Language model in GGUF format running on GPU in all cases)

-	Base Q8	LoRAF32	LoRAQ8	2+2+2 1000q8	2+2+2 2000q8	2+2+2 FullQ8	1+1+1 1000Q8	1+1+1 FullQ8	Mix 1000Q8	Mix FullQ8
Total Time (s)	5.33	10.24	7.57	5.43	6.26	5.65	5.94	5.38	5.97	6.18
1st Chunk Latency (s)	0.52	0.69	0.50	0.53	0.53	0.53	0.50	0.54	0.54	0.54
1st Chunk RTF	0.96	1.27	0.92	0.98	0.98	0.98	0.92	1.00	0.99	1.00
2nd Chunk RTF	0.34	0.62	0.34	0.33	0.34	0.34	0.34	0.34	0.34	0.34
MOS	3.89	4.03	3.88	3.95	3.69	4.21	3.67	4.05	3.61	3.72
Speaker Similarity	0.76	0.73	0.74	0.75	0.78	0.73	0.79	0.80	0.59	0.66

Table 14: Evaluation of Real time voice cloning for Speaker 2 using different quantised models: Response times, RTF, MOS, SNR, Speaker Similarity and RTF

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Jun 25, 2026

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

Large pretrained text-to-speech (TTS) models sound almost human for well-resourced languages, but much worse for languages that are rare in their training data. We study this quality gap for Khmer and Korean using VoxCPM2, a 2.4B parameter, tokenizer-free TTS model that joins a MiniCPM-4 language-model backbone with a flow-matching diffusion decoder. We build one shared, language-tagged corpus of 25.5 hours after cleaning and adapt VoxCPM2 with a single Low-Rank Adaptation (LoRA) adapter, trained on both languages at once and added to both the language model and the decoder. The adapter is zero-initialized, so training starts exactly at the original zero-shot model. In native-speaker listening tests, the Khmer Mean Opinion Score (MOS) rises from 3.85 to 4.23 with the best adapter, rank 64. This gain is highly significant under a paired Wilcoxon test with p < 0.001, and it is achieved while training only 0.19 to 3.03 percent of the parameters. Two findings stand out. First, the training loss and human ratings disagree on the best rank. The loss is lowest at rank 128, but MOS peaks at rank 64. Second, the same adapter gives no significant gain for Korean, which the base model already covers well, and a high rank even hurts quality. This shows that adaptation helps mainly where the base model is truly weak.

Phannet Pov, Hyun Woo Park, Voneat Pen +3

Department of Big Data, Chungbuk National University, Cheongju, South Korea · Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology, Phnom Penh, Cambodia · Department of Management Information Systems, Chungbuk National University, Cheongju, South Korea +1

Low-Resource TTS Synthesis Low-Resource Language Processing Text-to-Speech Parameter-Efficient Fine-Tuning

Ask PDF

Aug 29, 2026

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.

Zeyang Song, Tianchi Liu, Tianrui Wang +3

National University of Singapore · Tencent · LIGHTSPEED +4

TTS Synthesis Text-to-Speech TTS Evaluation

Ask PDF

Oct 10, 2026

Training a Compact, Strong, and Realistic Text-to-Speech Model

Text-to-speech (TTS) systems are conventionally optimized for intelligibility and speaker similarity. However, neither objective captures the prosodic detail that distinguishes human performances from synthetic speech, leaving a realism gap that larger models and higher-quality data can narrow but not close. In the absence of an objective that directly targets this gap, realism remains a persistent challenge in TTS. We present Audio8-TTS, a compact 0.6B-parameter model for zero-shot voice cloning that achieves state-of-the-art performance in intelligibility and speaker similarity, with particularly strong results in perceptual naturalness. To align the training signal with human perception rather than signal-level fidelity, we construct a quality ladder from widely used open-source TTS systems and train a realism critic to assess generated speech against matched human recordings of the same sentences. An identity-erasure objective discourages the critic from relying on generator fingerprints. We train the model in three stages: multilingual supervised fine-tuning; group relative policy optimization (GRPO) with word error rate (WER) and speaker similarity (SIM) rewards; and GRPO guided by the realism critic. The final stage improves perceived prosodic naturalness and speaker similarity while preserving WER performance.

Runyuan Cai, Yu Lin, Yiming Wang +1

AutoArk

TTS Synthesis Zero-Shot TTS Speech Generation Evaluation Speech Quality Assessment

Ask PDF

Abstract

TTS Synthesis

Explore similar work

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Training a Compact, Strong, and Realistic Text-to-Speech Model