When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Anupam Purwar; Aditya Choudhary

cs.SDMar 11, 2026

When Fine-Tuning Fails and when it Generalises: Role of Data Diversity and Mixed Training in LLM-based TTS

Authors: Anupam Purwar, Aditya Choudhary

Organizations: Sprinklr AI Gurugram, India

Abstract

Large language models are increasingly adopted as semantic backbones for neural text-to-speech systems. However, frozen LLM representations are insufficient for modeling speaker specific acoustic and perceptual characteristics. Our experiments involving fine tuning of the Language Model backbone of TTS show promise in improving the voice consistency and Signal to Noise ratio SNR in voice cloning task. Across multiple speakers LoRA finetuning consistently outperforms the non-finetuned base Qwen-0.5B model across three complementary dimensions of speech quality. First, perceptual quality improves significantly with DNS-MOS gains of up to 0.42 points for speakers whose training data exhibits sufficient acoustic variability. Second, speaker fidelity improves for all evaluated speakers with consistent increases in voice similarity indicating that LoRA effectively adapts speaker identity representations without degrading linguistic modeling. Third, signal level quality improves in most cases with signal to noise ratio increasing by as much as 34 percent. Crucially these improvements are strongly governed by the characteristics of the training data. Speakers with high variability in acoustic energy and perceptual quality achieve simultaneous gains in DNS-MOS voice similarity and SNR. Overall this work establishes that LoRA finetuning is not merely a parameter efficient optimization technique but an effective mechanism for better speaker level adaptation in compact LLM-based TTS systems. When supported by sufficiently diverse training data LoRA adapted Qwen-0.5B consistently surpasses its frozen base model in perceptual quality speaker similarity with low latency using GGUF model hosted in quantized form.

Figures & tables

Paper	What was implemented (2-line summary)	LoRA on LLM TTS Backbone
LoRP-TTS Bondaruk and Kubiak (2025) Bondaruk & Kubiak (2025)	Applies LoRA to TTS model layers for rapid speaker personalization from a single noisy reference utterance. Low-rank updates adapt speaker characteristics while keeping the synthesis backbone frozen.	No
TTS-Hub Anonymous (2025) OpenReview (ICLR submission)	Introduces modular LoRA adapters trained for individual speech attributes and composed arithmetically. Enables controllable TTS without modifying the underlying language or synthesis backbone.	No
StyleSpeech Lou et al. (2024) Lou et al. (ACM MM Asia 2024)	Employs parameter-efficient LoRA fine-tuning on pre-trained TTS components for phoneme and style control. Focuses on efficiency and controllability rather than LM adaptation.	No
LoRA-based Multi-Speaker TTS IEEE Access (2024)	Integrates LoRA and residual adapters into VITS-based multi-speaker TTS architectures. Achieves efficient speaker adaptation with substantially fewer trainable parameters.	No
EELE Qi et al. (2024) arXiv:2408.10852	Applies LoRA modules to emotional and expressive control layers in TTS systems. Targets parameter-efficient emotion adaptation without altering linguistic modeling layers.	No
UtterTune Kato (2025) (2025)	Injects LoRA directly into the Transformer-based language model backbone of an LLM-TTS system. Enables pronunciation and pitch-accent control by adapting semantic/phoneme prediction layers.	Yes

Table 1: Comparison of LoRA-based Methods in Text-to-Speech Systems

Figure 1: End-to-End Pipeline of a cascading voice agent comprising of a ASR module to transcribe voice to text, LLM module to reason over the text and generate text output and lastly a TTS module to synthesize speech for the reasoned text. Our work to the pipeline focuses on finetuning the LLM backbone (enclosed in red) responsible for token prediction, which are subsequently decoded by the neural codec to generate the waveform

	Data Len (hr.)	Avg Len (s)	Std. Dev. (s)	Unique Words	Avg Words
1	3.523	2.536	1.239	4492	6.717
2	3.907	2.813	1.633	7981	7.521
11614	3.860	2.778	1.471	7460	8.069
1401	17.590	14.314	6.209	16064	41.399
1212	18.492	15.543	6.446	13067	42.575
1259	13.652	14.675	6.202	11943	42.978

Table 2: Summary of the datasets used

Speaker	Dataset	Dataset Len.	Energy	Data MOS	Base	LoRA 1000	$\Delta$ vs	LoRA 5
ID		used (hours)	Spread	(Mean $\pm$ Std)	MOS	Train Steps	Base	Epochs
1	HiFiTTS	3.523	-29.73 $\pm$ 12.93	3.48 $\pm$ 0.32	3.832	3.861	$+0.029$	3.813
2	HiFiTTS	3.907	-29.34 $\pm$ 13.11	3.51 $\pm$ 0.37	3.717	4.141	$+0.424$	4.106
11614	HiFiTTS	3.860	-32.20 $\pm$ 13.34	3.30 $\pm$ 0.38	3.680	3.818	$+0.138$	3.768
1401	LibriHeavy	17.590	-32.11 $\pm$ 11.89	3.44 $\pm$ 0.26	3.659	3.385	$-0.274$	3.435
1212	LibriHeavy	18.492	-28.41 $\pm$ 9.91	3.42 $\pm$ 0.25	3.647	3.233	$-0.414$	3.350

Table 3: Detailed Speaker Fine-tuning Results across HiFiTTS and LibriHeavy Datasets

Table 5

Panel of Figure 2 — Figure 2: Plots of Training and Validation Loss at various training steps for each speaker ID. The curves indicate stochastic decrease of training loss with training steps with saturation in validation loss as we approach 4 epochs

Table 7

Panel of Figure 3 — Figure 3: Plots indicating MOS Score (Y axis) of Audio Generated with varying lengths of cloning audio provided (X axis). Base refers to the standard NeuTTS model, Ref refers to the audio used for cloning. "Long" refers to generation of a longer sentence (180 characters) whereas other audios have 70 characters.

Metric / Speaker ID		1	2	11614	1401	1212	1259
Dataset		HiFi TTS			LibriHeavy-HQ
Dataset Length (Hr)		3.523	3.907	3.860	17.590	18.492	13.652
SNR	Base	41.747	31.562	33.403	32.163	20.397	16.895
	LoRA	55.854	39.567	36.217	40.778	12.412	15.395
	Ref	31.900	27.047	24.905	26.791	22.032	17.630
	Inc. Base/Ref %	30.868	16.690	34.124	20.049	-7.420	-4.170

Table 8: Results of LoRA Finetuning

Speaker ID	Energy Mean	Energy Std (Training data)	DNS-MOS Std (Training data)	Fine-tuning Outcome	Observed Pattern
2	-29.34	13.11	0.370	+0.424 $\times$	High energy variability + high DNS-MOS variability
11614	-32.20	13.34	0.383	+0.138 $\times$	Highest energy variability + highest DNS-MOS variability
1	-29.73	12.93	0.321	+0.029	Medium energy and DNS-MOS variability
1401	-32.11	11.89	0.255	-0.274 $\times$	Low energy variability + low DNS-MOS variability
1212	-28.41	9.91	0.248	-0.414 $\times$	Second-lowest energy variability + lowest DNS-MOS variability
1259	-31.18	8.25	0.293	-0.001	Lowest energy variability

Table 9: Impact of training-data energy statistics on DNS-MOS variability and fine-tuning outcome for speaker-specific LoRA adaptation of a Qwen-0.5B TTS backbone. While absolute energy levels (mean) show no consistent influence, higher energy variability in the training data strongly correlates with increased DNS-MOS variability and improved perceptual gains after fine-tuning.

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	3.76	4.01	3.86	3.50	3.32	3.72
Pure FT 1000	3.86	4.14	3.82	3.39	3.23	3.66
2+2+2 FT final	3.96	3.95	3.738	3.83	3.74	3.84
2+2+2 FT 1000	3.95	4.09	3.77	3.56	3.69	3.70
2+2+2 FT 2000	3.79	4.09	3.75	3.63	3.71	3.83
1+1+1 FT final	3.81	3.92	3.73	3.51	3.69	3.67

Table 10: Impact of different audio training data combinations used for Fine tuning Language model on MOS

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	0.790	0.794	0.737	0.503	0.678	0.481
2+2+2 FT final	0.749	0.780	0.686	0.412	0.569	0.292
2+2+2 FT 1000	0.740	0.813	0.728	0.354	0.569	0.292
2+2+2 FT 2000	0.752	0.799	0.698	0.438	0.546	0.423
1+1+1 FT final	0.740	0.786	0.679	0.396	0.566	0.399
1+1+1 FT 2000	0.756	0.785	0.672	0.422	0.539	0.401

Table 11: Impact of different audio training data combinations used for Fine tuning Language model on Similarity

Speaker ID	1	2	11614	1401	1212	1259
Pure FT final	57.265	41.640	51.437	35.504	19.995	22.704
2+2+2 FT final	51.957	55.632	45.849	32.065	76.746	23.612
2+2+2 FT 1000	41.104	67.328	65.867	36.580	32.736	22.195
2+2+2 FT 2000	43.791	45.714	37.693	33.385	26.819	23.006
2+2+2 FT final	28.871	50.757	54.411	32.979	27.112	21.942
1+1+1 FT 2000	62.695	35.731	67.628	38.310	37.083	19.507

Table 12: Impact of different audio training data combinations used for Fine tuning Language model on SNR

Sentence	S1		S2		S3		S4		S5
Env.	CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU	CPU	GPU
1st Chunk Latency (s)	0.3988	0.3067	0.4032	0.311	0.4134	0.3189	0.4195	0.3237	0.4167	0.3212
1st Chunk RTF	0.7386	0.568	0.7466	0.576	0.7656	0.5906	0.7768	0.5994	0.7717	0.5949
2nd Chunk RTF	0.5195	0.3155	0.5211	0.3124	0.524	0.3111	0.5217	0.313	0.5189	0.3194
MOS	3.19489	3.19882	3.48858	3.48964	3.78335	3.78371	3.89385	3.89359	3.73711	3.73772
Speaker Similarity	0.68481	0.68464	0.60134	0.60058	0.62643	0.62618	0.74191	0.74214	0.71144	0.71538

Table 13: Impact of running codec (NeuCodec) in onnx-runtime running on CPU vs GPU for different input text sentences ( Language model in GGUF format running on GPU in all cases)

-	Base Q8	LoRAF32	LoRAQ8	2+2+2 1000q8	2+2+2 2000q8	2+2+2 FullQ8	1+1+1 1000Q8	1+1+1 FullQ8	Mix 1000Q8	Mix FullQ8
Total Time (s)	5.33	10.24	7.57	5.43	6.26	5.65	5.94	5.38	5.97	6.18
1st Chunk Latency (s)	0.52	0.69	0.50	0.53	0.53	0.53	0.50	0.54	0.54	0.54
1st Chunk RTF	0.96	1.27	0.92	0.98	0.98	0.98	0.92	1.00	0.99	1.00
2nd Chunk RTF	0.34	0.62	0.34	0.33	0.34	0.34	0.34	0.34	0.34	0.34
MOS	3.89	4.03	3.88	3.95	3.69	4.21	3.67	4.05	3.61	3.72
Speaker Similarity	0.76	0.73	0.74	0.75	0.78	0.73	0.79	0.80	0.59	0.66

Table 14: Evaluation of Real time voice cloning for Speaker 2 using different quantised models: Response times, RTF, MOS, SNR, Speaker Similarity and RTF

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean
Jun 25, 2026Phannet Pov, Hyun Woo Park, Voneat Pen +3Low-Resource TTS Synthesis Low-Resource Language Processing
Department of Big Data, Chungbuk National University, Cheongju, South Korea · Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology, Phnom Penh, Cambodia · Department of Management Information Systems, Chungbuk National University, Cheongju, South Korea +1
Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction
Aug 29, 2026Zeyang Song, Tianchi Liu, Tianrui Wang +3TTS Synthesis Text-to-Speech
National University of Singapore · Tencent · LIGHTSPEED +4
Training a Compact, Strong, and Realistic Text-to-Speech Model
Oct 10, 2026Runyuan Cai, Yu Lin, Yiming Wang +1TTS Synthesis Zero-Shot TTS
AutoArk

Jun 25, 2026J/K move · Enter open · S save

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

Phannet Pov, Hyun Woo Park, Voneat Pen, Sovandara Chhoun+2

Department of Big Data, Chungbuk National University, Cheongju, South Korea · Institute of Digital Research and Innovation, Cambodia Academy of Digital Technology, Phnom Penh, Cambodia · Department of Management Information Systems, Chungbuk National University, Cheongju, South Korea+1

Low-Resource TTS Synthesis Low-Resource Language Processing Text-to-Speech Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Large pretrained text-to-speech (TTS) models sound almost human for well-resourced languages, but much worse for languages that are rare in their training data. We study this quality gap for Khmer and Korean using VoxCPM2, a 2.4B parameter, tokenizer-free TTS model that joins a MiniCPM-4 language-model backbone with a flow-matching diffusion decoder. We build one shared, language-tagged corpus of 25.5 hours after cleaning and adapt VoxCPM2 with a single Low-Rank Adaptation (LoRA) adapter, trained on both languages at once and added to both the language model and the decoder. The adapter is zero-initialized, so training starts exactly at the original zero-shot model. In native-speaker listening tests, the Khmer Mean Opinion Score (MOS) rises from 3.85 to 4.23 with the best adapter, rank 64. This gain is highly significant under a paired Wilcoxon test with p < 0.001, and it is achieved while training only 0.19 to 3.03 percent of the parameters. Two findings stand out. First, the training loss and human ratings disagree on the best rank. The loss is lowest at rank 128, but MOS peaks at rank 64. Second, the same adapter gives no significant gain for Korean, which the base model already covers well, and a high rank even hurts quality. This shows that adaptation helps mainly where the base model is truly weak.

Abstract

TTS Synthesis

Explore similar work

Closing the Quality Gap in Low-Resource Text-to-Speech: LoRA Fine-Tuning of VoxCPM2 for Khmer and Korean

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Training a Compact, Strong, and Realistic Text-to-Speech Model