Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Zeyang Song; Tianchi Liu; Tianrui Wang; Chenglin Xu; Steven Y. Guo; Haizhou Li

cs.SDAug 29, 2026

Diagnose, Then Refine: A Closed-Loop TTS System with AudioLLM-Guided Correction

Authors: Zeyang Song, Tianchi Liu, Tianrui Wang, Chenglin Xu, Steven Y. Guo, Haizhou Li

Organizations: National University of Singapore · Tencent · LIGHTSPEED · Nanyang Technological University · Independent Researcher · The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute

Abstract

Current TTS systems typically rely on open-loop, single-pass generation and can produce sporadic local prosodic defects, such as misplaced stress, unnatural pauses, or flattened intonation, that utterance-level metrics often fail to expose. We present LoopTTS, a judge-guided Filter-Judge-Refiner framework for recovering low-quality TTS outputs diagnosed by an AudioLLM. Given an initial utterance from a base TTS model, an AudioLLM Judge identifies salient prosodic issues and generates structured refine instructions; a Refiner, our fine-grained instruction-following TTS model, then performs guided expressive re-synthesis conditioned on the initial utterance, target text, and instruction. To train the Refiner, we construct Refiner-DB, a 42K-example AudioLLM-annotated dataset with word-level prosodic weak supervision. Human evaluation on diagnosed low-quality utterances shows that LoopTTS can detect perceptually salient errors and correct them with the Refiner, outperforming raw generated audio and practical open-loop re-generation baselines in recovery quality. The Refiner also demonstrates stronger instruction-following ability for stress and pause control in targeted prosody modification.

Figures & tables

Figure 1: Overview of the LoopTTS pipeline. Stage 1 (Filter): coarse-grained metrics (WER, UTMOS) discard catastrophically degraded outputs and trigger full re-generation. Stage 2 (Judge): an AudioLLM evaluates prosody naturalness and emotional fidelity, producing an overall score (1–10) along with structured refine instructions that specify both global attributes (emotion, speed, pitch) and local operations (stress, pause). Utterances meeting the threshold are accepted as final output. Stage 3 (Refiner): the Refiner takes the initial utterance together with the refine instructions and performs guided expressive re-synthesis, producing corrected speech with optional iterative refinement for residual defects.

Figure 2: Contrastive data construction for Refiner-DB . Each expressive recording $y_{\text{tgt}}$ is paired with a zero-shot neutral clone $y_{\text{neu}}$ synthesized from the same text and speaker with neutral emotion. Global style comes from dataset metadata; an AudioLLM compares the two utterances and outputs word-level prosodic differences (stress, pause) as structured JSON. Both sources are merged into a natural-language refine instruction $I$ , yielding tuples $(y_{\text{neu}},I,y_{\text{tgt}},T)$ . Full prompts are in Appendix I .

Figure 3: Refiner architecture and position-aware training objective. The Refiner takes a refine instruction $I$ , the initial utterance $y_{\text{init}}$ , and the target text $T$ as input. Two losses are applied in training: a position-weighted CE loss $\mathcal{L}^{\text{CE}}$ that amplifies gradients at stressed and pause positions, and a structural alignment loss $\mathcal{L}_{\text{struct}}$ that aligns span-level consistency.

AudioLLM	Regular	Emotional
Gemini-3-flash Team et al. (2023)	3.1	3.2
Gemini-3-pro Team et al. (2023)	3.3	3.4
Kimi-Audio KimiTeam et al. (2025)	2.2	2.5
Qwen3-Omni Xu et al. (2025)	2.6	2.7
Qwen3-Omni Thinking Xu et al. (2025)	2.9	3.0

Table 1: Mean human–AudioLLM alignment scores (1–4). Higher is better.

	Stress		Pause
AudioLLM	Rec.	F1	Rec.	F1
Gemini-3-flash	0.861	0.535	0.573	0.432
Gemini-3-pro	0.884	0.673	0.697	0.532
Kimi-Audio	0.588	0.438	0.440	0.316
Qwen3-Omni	0.613	0.441	0.487	0.330
Qwen3-Omni Thinking	0.667	0.465	0.478	0.325

Table 2: AudioLLM annotation quality (micro Recall and F1) against professional human labels. The labels mark salient stress and pause differences rather than exhaustive prosodic structure.

Figure 4: Blind paired preference evaluation on flagged utterances. Each paired comparison uses 30 samples.

Baseline
	Neutral			Emotional
Condition	WER $\downarrow$	SIM $\uparrow$	MOS $\uparrow$	WER $\downarrow$	SIM $\uparrow$	MOS $\uparrow$
Before refinement	2.04	0.73	3.21 ±.15	2.11	0.69	3.01 ±.16
Re-generation (open-loop)
CosyVoice2 Du et al. (2024b)	1.95	0.71	3.45 ±.14	2.03	0.70	3.15 ±.15
EmoVoice Yang et al. (2025a)	1.83	0.68	3.90 ±.13	1.72	0.65	3.85 ±.12

Table 3: Pipeline recovery evaluation on Stage 2 flagged utterances under Neutral and Emotional conditions. LoopTTS uses Gemini-3-pro as the inference-time Judge, while LoopTTS -Q uses the same Refiner with the inference-time Judge changed to Qwen3-Omni Thinking.

	Objective			Subjective
Model	WER $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	MOS $\uparrow$	MOS-I $\uparrow$
Model	WER $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$	MOS $\uparrow$	Emo.	Pitch	Speed	Stress	Pause
CosyVoice2 †	–	0.72	3.40	2.21 ±.11	2.25 ±.12	2.31 ±.10	2.54 ±.09	2.15 ±.10	2.12 ±.09
CosyVoice2-marker	4.74	0.69	3.27	2.95 ±.16	3.02 ±.11	3.78 ±.12	3.46 ±.09	4.32 ±.07	3.88 ±.08
EmoVoice	2.81	0.67	3.86	4.32 ±.13	4.13 ±.11	3.45 ±.09	3.76 ±.08	3.83 ±.08	2.92 ±.09
Refiner	2.90	0.68	3.60	4.21 ±.14	4.04 ±.10	4.01 ±.10	4.13 ±.08	4.57 ±.07	4.28 ±.08

Table 4: Instruction-following comparison. † CosyVoice2 frequently reads the instruction text verbatim, making its WER not comparable.

Variant	WER $\downarrow$	MOS $\uparrow$	Avg. MOS-I $\uparrow$
Refiner	2.90	4.21 ±.13	4.21 ±.09
w/o $\mathcal{L}_{\text{struct}}$	2.78	4.06 ±.15	4.01 ±.09
w/o Position-Weighting	2.92	4.17 ±.14	4.13 ±.08
w/o Both	2.87	4.03 ±.14	3.75 ±.10

Table 5: Ablation: effect of loss components.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

System	WER $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$
CosyVoice2	3.69	0.70	3.33
EmoVoice	3.65	0.65	3.31
LoopTTS (Stage 1)	2.94	0.74	3.76
CosyVoice2 re-generation	2.90	0.74	3.81
EmoVoice re-generation	2.81	0.72	3.79
Judge-reranked best-of-5	2.90	0.74	3.82

Appendix

Table 6: Objective evaluation on 3,000 emotional synthesis utterances. WER is computed via Whisper-large-v3-turbo; SIM is measured against the neutral speaker reference; UTMOS estimates perceptual quality.

Stage	# Utt.	Rate
Total	3,000	100.00%
Initial pass	2,367	78.90%
After regen.	2,753	91.76%
Stage 2 flagged	237	7.90%
After 1st refine	103	3.43%

Appendix

Table 7: Stage-level accounting for the large-scale LoopTTS cascade.

Corpus	# Tuples	Type
ESD	14,000	Emotional
RAVDESS	1,440	Emotional
SAVEE	480	Emotional
MESS	6,800	Emotional
EmoVoice-DB	17,280	Emotional
LibriTTS	2,000	Regular

Appendix

Table 8: Refiner-DB composition by source corpus.

Variant	WER $\downarrow$	SIM $\uparrow$	MOS $\uparrow$	Avg. MOS-I $\uparrow$
Refiner	2.90	0.68	4.21 ±.14	4.21 ±.09
w/o $y_{\text{init}}$	2.91	0.62	4.17 ±.12	3.94 ±.11

Appendix

Table 9: Ablation of conditioning on the initial utterance.

Setting	WER $\downarrow$	SIM $\uparrow$
Before refinement	1.92	0.68
With GT instruction	1.72	0.67
With counterfactual instruction	1.87	0.68

Appendix

Table 10: Objective results under counterfactual local stress/pause instructions.

Type	Wrong Exe. Rate	Corr. Pres. Rate
Stress	10%	34%
Pause	28%	52%

Appendix

Table 11: Professional annotation of counterfactual instruction execution. Exe. and Pres. denote execution and preservation.

Setting	Neutral		Emotional
Setting	WER $\downarrow$	SIM $\uparrow$	WER $\downarrow$	SIM $\uparrow$
1-round	1.86	0.66	1.79	0.65
2-round	1.85	0.64	1.80	0.61
3-round	1.85	0.60	1.80	0.58

Appendix

Table 12: Objective results with additional refinement rounds.

System	MOS	Emotion	Pause	Stress	Pitch	Speed
CosyVoice2	0.784	0.562	0.915	0.496	0.728	0.848
CosyVoice2-marker	0.549	0.588	0.943	0.598	0.729	0.865
EmoVoice	0.496	0.706	0.899	0.694	0.683	0.823
Refiner	0.791	0.679	0.886	0.677	0.744	0.835

Appendix

Table 13: ICC(2,k) reliability for MOS and MOS-I ratings.

System	Neutral SIM $\uparrow$	SIM-EXP $\uparrow$
CosyVoice2	0.72	0.65
EmoVoice	0.67	0.75
Refiner	0.68	0.77

Appendix

Table 14: Speaker similarity against a neutral reference and five same-speaker, same-emotion expressive references.

Figure 5: Screenshot of the web-based human annotation interface used for collecting prosodic ground-truth labels (Section 5.2 ). Annotators toggle between Stress and Pause modes, clicking words or inter-word gaps to mark stressed words (red) and pause positions (vertical bars). A live preview and free-text note field are provided below.

Explore similar work

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis
Sep 21, 2026Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang +4TTS Synthesis Audio Reasoning
Graduate Institute of Electrical Engineering, National Taiwan University, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NVIDIA Research 4 Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2TTS Synthesis TTS Evaluation
Argmax, Inc. · University of Virginia · Bilkent University
TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis
Apr 24, 2026Xi Wang, Jie Wang, Xingchen Song +8TTS Synthesis Speech Generation Evaluation
Tsinghua University, China · MiLM Plus, Xiaomi Inc., China · The University of Tokyo, Japan

Sep 21, 2026J/K move · Enter open · S save

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis

Chee-En Yu, Yi-Cheng Lin, Sung-Feng Huang, Yun-Shao Tsai+3

Graduate Institute of Electrical Engineering, National Taiwan University, Taiwan · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NVIDIA Research 4 Artificial Intelligence Center of Research Excellence, National Taiwan University, Taiwan

TTS Synthesis Audio Reasoning Speech Processing Audio-Language Models

Open paper AskPDF arXiv

Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate "thinking" tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.

Abstract

Explore similar work

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis

Taming Long-form Text-to-Speech

TTS-PRISM: A Perceptual Reasoning and Interpretable Speech Model for Fine-Grained Diagnosis