RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning

Maxim Maslov; Kirill Borodin; Vasilii Kudryavtsev; Nikita Vasiliev; Grach Mkrtchian

cs.SDSep 29, 2026

RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning

Authors: Maxim Maslov, Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian

Organizations: lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia

Abstract

Zero-shot text-to-speech synthesizes new utterances in a speaker's voice from a short reference recording. Voice cloning requires accurate content and preserved speaker identity, but supervised acoustic-token prediction does not directly optimize these waveform-level properties. Reward-based post-training addresses this mismatch, but in discrete diffusion, token choices and reveal positions jointly define the sampling trajectory, complicating alignment. We introduce RAWD-TTS (Ratio-free Advantage-Weighted Denoising), which scores decoded samples with recognition and speaker rewards and uses group-relative advantages to weight masked-token reconstruction of those samples, without reverse-trajectory likelihoods or target audio. On 500 Russian CV3-Eval voice-cloning prompts, joint alignment reduces word error rate from 3.18% to 2.42% at the reward-selected checkpoint (24.0% relative) and to 2.58% at the final checkpoint (19.0%), while WavLM speaker cosine rises from 0.733 to 0.748 and 0.755. Controlled experiments characterize recognition-identity trade-offs and the effects of corruption count, group composition, and weighting.

Figures & tables

	Training scorers		Independent		Quality
System	WER $\downarrow$	Cos $\uparrow$	WER $\downarrow$	Cos $\uparrow$	MOS $\uparrow$
CosyVoice3	5.34	0.738	7.87	0.770	3.41
Qwen3-TTS	4.69	0.734	6.18	0.701	3.67
OmniVoice	3.18	0.733	4.38	0.704	3.18
OmniVoice+SFT	2.94	0.765	4.11	0.726	3.20
ASR-only	2.35	0.705	3.72	0.688	3.29

Table 1: CV3-Eval Multilingual Voice Cloning (Russian, 500 prompts). Adapters use 2,000 updates; the lower block contains RAWD-TTS variants. Cos: raw cosine. Adapter rows not marked final are checkpoints selected by joint training-scorer reward on these prompts (updates 1,300 for Joint

K=3

, 1,500 for

K=5

); final rows use update 2,000 without selection. Dashes: not scored. Bold: column optimum; shading: highest joint training-scorer reward.

$K$	$B$	$G$	WER $\downarrow$	Cos $\uparrow$
1	4	8	2.82	0.736
3	4	8	2.55	0.733
5	4	8	2.47	0.738
3	8	4	2.72	0.733
3	2	16	2.59	0.725

Table 2: Joint-reward ablations: 1,000 updates, 500 evaluation prompts, and

BG=32

. The shared control uses

K=3

B=4

, and

G=8

. Checkpoint selection and highlighting follow Table 1 .

Explore similar work

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
Sep 29, 2026Jian Chen, You Zhang, Mark VintonCross-Lingual Voice Cloning Transcript
Dolby Laboratories
VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation
Jun 25, 2026Tianxin Xie, Chenxing Li, Dong Yu +1Flow-Matching Text-To-Speech Test-Time Adaptation
The Hong Kong University of Science and Technology (Guangzhou) · Tencent
X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning
May 7, 2026Rixi Xu, Qingyu Liu, Haitao Li +10Cross-Lingual Voice Cloning Multilingual Automatic Speech Recognition
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4

Sep 29, 2026, eess.ASJ/K move · Enter open · S save

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

Jian Chen, You Zhang, Mark Vinton

Dolby Laboratories

Cross-Lingual Voice Cloning Transcript Autoregressive Decoding Autoregressive Model

Open paper AskPDF arXiv

TTS systems with autoregressive semantic modeling have demonstrated strong zero-shot voice cloning performance and rich expressive variation, but their sequential decoding incurs substantial latency. Non-autoregressive alternatives offer much faster generation, yet often rely on more restrictive reference conditioning, such as requiring transcripts of the reference speech during inference. We present Tacit-TTS, an efficient transcript-free zero-shot voice cloning system distilled from IndexTTS2. Our model replaces autoregressive text-to-semantic decoding with masked non-autoregressive generation, introduces training-free acoustic length estimation, and accelerates the flow-matching renderer through ReFlow distillation. Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality while generating speech over 10x faster than IndexTTS2 for utterances longer than 5 seconds. Its transcript-free conditioning further supports cross-lingual and non-lexical references. We validate this capability using references from eight other languages, infant babble, and synthetic gibberish, where transcript-dependent systems often degrade or fail due to unreliable ASR transcripts.

Abstract

Explore similar work

Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

VoiceTTA: Enhancing Zero-Shot Text-to-Speech via Reinforcement Learning-Based Test-Time Adaptation

X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning