Does Expert Supervision Improve Text-to-Speech? An Empirical Study of Preference Optimization

Minh Duc Vu; Junchuan Zhao; Ye Wang

cs.SDOct 18, 2026

Does Expert Supervision Improve Text-to-Speech? An Empirical Study of Preference Optimization

Authors: Minh Duc Vu, Junchuan Zhao, Ye Wang

Organizations: Department of Statistics & Data Science, National University of Singapore, Singapore · School of Computing, National University of Singapore, Singapore

Abstract

Preference optimization promises to improve text-to-speech (TTS) systems by learning which speech outputs are better. External expert speech seems a natural source of preferred responses, especially when student generations contain errors. Yet demonstration quality does not establish whether the student can benefit from that supervision, or whether better target fitting will produce better speech. We examine this gap across two TTS students using real and synthetic expert speech, comparing preference objectives and direct versus bridge supervision while tracking fixed-sequence likelihoods and generation metrics. With expert supervision held fixed, objectives differ markedly in their sensitivity to preference-score scaling and the resulting generation quality. Likelihood diagnostics characterize these responses, but cannot establish their benefit: expert likelihood can increase while intelligibility worsens, or decrease while it improves. Bridge supervision highlights a distinction between mitigating degradation and improving the student: it can recover performance lost under direct training without consistently surpassing the pretrained base. Route benefits vary across objectives, students, and expert sources; neither route consistently dominates. Together, these findings distinguish demonstration quality, target fitting, and generated-speech quality as complementary criteria for assessing expert supervision.

Figures & tables

Figure 1: Motivating questions for expert-guided preference optimization. Schematic panels H1–H3 correspond to RQ1–RQ3: objective sensitivity, likelihood responses, and supervision routes.

Speech source	WER (%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$
Pretrained base	4.2	0.709	4.357
Real: original	3.2	0.672	4.066
Real: reconstructed	4.1	0.721	4.349
Qwen: original	2.7	0.753	4.358
Qwen: reconstructed	2.9	0.798	4.368

Table 1: Target quality before CosyVoice training. WER is corpus-level; SIM uses a common speaker reference per input. Reconstruction uses the student’s speech representation.

Figure 2: Bridge construction and support-weighted training (schematic). Token symbols $z$ correspond to $y$ in the text; experiments use a discrete ratio grid and two fixed training phases.

Figure 3: CosyVoice constructed targets across ratios. Error reduction uses

r=0

; SIM/NISQA show absolute changes. Dashed lines mark rule-selected ratios

r_{t}^{\mathrm{sel}}

, distinct from training ratios. Shaded quality bands show 95% bootstrap intervals.

Objective	$\beta$	LibriTTS test			SEED TTS Eval			Fixed probes: $\Delta\ell$
Objective	$\beta$	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	Expert	Base
Pretrained base	–	3.87	0.731	4.484	4.27	0.739	4.169	$0.000$	$0.000$
LibriTTS expert	–	3.40	0.696	4.151	–	–	–	–	–
DPO	0.1	4.59	0.706	4.427	6.33	0.721	4.057	$-0.348$	$-1.448$
DPO	0.01	29.78	0.611	4.119	63.02	0.496	3.462	$-3.473$	$-22.301$
WPO	0.1	4.81	0.708	4.429	6.58	0.722	4.072	$-0.295$	$-1.191$

Table 2: CosyVoice with LibriTTS supervision under Direct training. Green/red marks likelihood increases/decreases. Bold marks the best student result across objective–

\beta

settings.

Objective	Route	LibriTTS test			SEED TTS Eval			Fixed probes: $\Delta\ell$
Objective	Route	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	Expert	Base
Pretrained base	–	3.87	0.731	4.484	4.27	0.739	4.169	$0.000$	$0.000$
LibriTTS	Expert	3.40	0.696	4.151	–	–	–	–	–
Qwen3-TTS	Expert	2.80	0.780	4.441	–	–	–	–	–
Expert: LibriTTS recordings
DPO	Direct	4.59	0.706	4.427	6.33	0.721	4.057	$-0.348$	$-1.448$

Table 3: CosyVoice: Direct versus Bridge at

\beta=0.1

. Green/red marks likelihood increases/decreases. Bold marks the best student result within each expert block.

Objective	Route	LibriTTS test			SEED TTS Eval			Fixed probes: $\Delta\ell$
Objective	Route	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	Expert	Base
Pretrained base	–	9.50	0.387	3.767	4.27	0.401	3.986	$0.000$	$0.000$
LibriTTS	Expert	3.40	0.696	4.151	–	–	–	–	–
Qwen3-TTS	Expert	2.80	0.780	4.441	–	–	–	–	–
Expert: LibriTTS recordings
DPO	Direct	18.12	0.381	3.577	5.25	0.406	3.908	$-0.685$	$-2.666$

Table 4: GPT-SoVITS: Direct versus Bridge at

\beta=0.1

. Green/red marks likelihood increases/decreases. Bold marks the best student result within each expert block.

Figure 4: DPO log-probability distributions under real-speech supervision. Positive/negative sequences denote fixed expert/initial-student probes. Dotted lines mark means;

\Delta\mu=\mu_{\mathrm{post}}-\mu_{\mathrm{base}}

. Panel headings report macro WER.

Figure 5: Paired macro-WER differences (Phase 2 minus comparator, in percentage points) with 95% speaker-bootstrap confidence intervals. Negative values favor Phase 2.

Objective	Phase 1		Phase 2		Direct
Objective	$E$	$S_{0}$	$E$	$S_{0}$	$E$	$S_{0}$
APO-zero	0.088	0.165	0.113	0.331	0.120	0.299
DPO	0.204	0.308	0.265	0.453	0.391	0.640
WPO	0.159	0.229	0.226	0.354	0.345	0.509
NCA	0.019	0.058	0.016	0.098	0.018	0.099

Table 5: Conditional KL to the pretrained base (nats/token) on fixed expert (

E

) and initial-student (

S_{0}

) contexts, averaged over tokens within each sequence, then over utterances. See Appendix E.5 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Student	Expert	$r$	Utterances	Speakers	Duration (h)
Student	Expert	$r$	Utterances	Speakers	Total	Prefix	Continuation
CosyVoice-1	LibriTTS	0.6	36,000	274	57.40	34.79	22.61
GPT-SoVITS	LibriTTS	0.4	35,729	270	57.91	22.81	35.10
CosyVoice-1	Qwen3-TTS	0.8	33,931	267	57.05	44.72	12.33
GPT-SoVITS	Qwen3-TTS	0.8	35,459	267	57.79	45.72	12.07

Appendix

Table 6: Composition of the retained Bridge datasets. The copied token fraction

r

specifies the proportion of expert tokens retained as the prefix. Total speech duration is partitioned into copied expert prefixes and student-generated continuations.

Figure 6: Normalized duration histograms of the retained Bridge samples across student–expert configurations. The Qwen3-TTS source distribution is shown in gray. Ratios denote the copied fraction of expert tokens.

Figure 7: GPT-SoVITS constructed targets across ratios. Error reduction uses

r=0

; SIM/NISQA show absolute changes. Dashed lines mark rule-selected ratios

r_{t}^{\mathrm{sel}}

, distinct from training ratios. Shaded quality bands show 95% bootstrap intervals.

Objective	$\beta$	NLL		Probes (%)
Objective	$\beta$	Expert	Base	Expert $\uparrow$	Base $\downarrow$
Pretrained base		2.729	2.509	–	–
DPO	0.1	$3.077$	$3.957$	0.0	100.0
DPO	0.01	$6.202$	$24.810$	0.0	100.0
WPO	0.1	$3.024$	$3.700$	0.0	100.0
WPO	0.01	$6.196$	$10.955$	0.0	100.0

Appendix

Table 7: CosyVoice: fixed-probe diagnostics across

\beta

under LibriTTS supervision.

Objective	Route	NLL		Probes (%)
Objective	Route	Expert	Base	Expert $\uparrow$	Base $\downarrow$
Expert: LibriTTS recordings
Pretrained base		2.729	2.509	–	–
DPO	Direct	$3.077$	$3.957$	0.0	100.0
DPO	Bridge	$2.954$	$3.586$	0.1	100.0
WPO	Direct	$3.024$	$3.700$	0.0	100.0

Appendix

Table 8: CosyVoice: fixed-probe diagnostics by route (

\beta=0.1

Objective	Route	NLL		Probes (%)
Objective	Route	Expert	Base	Expert $\uparrow$	Base $\downarrow$
Expert: LibriTTS recordings
Pretrained base		5.518	4.876	–	–
DPO	Direct	$6.203$	$7.542$	6.2	100.0
DPO	Bridge	$5.416$	$5.246$	84.8	93.5
WPO	Direct	$5.548$	$5.380$	49.8	99.2

Appendix

Table 9: GPT-SoVITS: fixed-probe diagnostics by route (

\beta=0.1

Objective	Weighting	WER $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$
DPO	Constant	4.1	0.712	4.468
	Linear	4.1	0.710	4.464
	Parabolic	4.3	0.708	4.458
APO-zero	Constant	3.6	0.722	4.505
	Linear	3.7	0.720	4.498
	Parabolic	3.6	0.721	4.496

Appendix

Table 10: Ablation of policy-support weighting with fixed preference targets and training budgets. WER is macro-averaged over utterances and reported in percent.

Figure 8: WPO fixed-probe log-probability distributions under real-speech supervision.

Figure 9: APO-zero fixed-probe log-probability distributions under real-speech supervision.

Figure 10: NCA fixed-probe log-probability distributions under real-speech supervision.

Objective	Stage	WER(%) $\downarrow$	SIM $\uparrow$	NISQA $\uparrow$	Chosen NLL	Rejected NLL	Chosen gain (%)	Rejected suppression (%)
Pretrained base	Reference	3.91	0.730	4.482	–	2.527	–	–
Qwen3-TTS	Expert	3.08	0.826	4.500	2.263	–	–	–
DPO	Phase 1	3.19	0.721	4.522	$2.227\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}2.391}$	$2.517\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}3.292}$	2.7	100.0
	Phase 2	4.23	0.723	4.498	$2.423\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}2.427}$	$2.759\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}3.301}$	47.4	100.0
	Direct	3.30	0.713	4.542	$2.263\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}2.581}$	$2.527\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}3.927}$	0.8	100.0
WPO	Phase 1	3.31	0.721	4.517	$2.227\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}2.348}$	$2.517\!\rightarrow\!{\color[rgb]{0.7266,0.1758,0.1758}3.113}$	3.3	100.0

Appendix

Table 11: CosyVoice checkpoints under Qwen3-TTS supervision. NLL entries show before-to-after measurements on each stage’s chosen/rejected pairs. Pairs and stage-start diagnostic baselines differ across stages; the training reference remains the original pretrained student. Gain/suppression reports the percentages of sequences whose likelihood increases/decreases within that stage. Phases 1 and 2 use Bridge and full-expert targets, respectively.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation
Sep 1, 2026Thibaut Thonet, Jos Rozen, Laurent BesacierLLM Alignment TTS Evaluation
NAVER LABS Europe
Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Aug 10, 2026Oluwanifemi Bamgbose, Jash Shah, Simon Rosen +6Audio-Language Model Evaluation Human Preference Evaluation
ServiceNow
Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech
Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS Synthesis TTS Synthesis
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea

Sep 1, 2026J/K move · Enter open · S save

Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation

Thibaut Thonet, Jos Rozen, Laurent Besacier

NAVER LABS Europe

LLM Alignment TTS Evaluation Preference Optimization Text Generation

Open paper AskPDF arXiv

Current Large Language Models (LLMs) are primarily optimized for written text, often producing outputs that are grammatically correct and helpful yet poorly suited for spoken delivery via Text-to-Speech (TTS). In this work, we study how to make LLMs natively generate TTS-friendly text, which we frame as a preference alignment problem: instead of relying on downstream rewriting modules, we directly align LLMs to generate text optimized for spoken delivery. We introduce two preference datasets spanning different target domains, CORA and Recipe, which contain paired TTS-friendly and TTS-unfriendly responses. We further propose an evaluation suite combining a pattern-based heuristic metric, a TTS

\to

ASR evaluation pipeline, and a MUSHRA listening study with human judges. Our experiments compare the recently proposed Feature-aware Sampling and Tuning (FaST) framework -- leveraging interpretable features instead of a black-box reward model -- against an array of alignment baselines on the TTS-friendly generation task. Notably, we found that FaST achieves the best overall tradeoff between TTS-friendliness and helpfulness across various settings. We also identified a strong correlation between our different metrics, highlighting the ability to reliably assess TTS-friendliness via an efficient heuristic.

Abstract

TTS Synthesis

Explore similar work

Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech