Large language model (LLM)-based text-to-speech (TTS) systems achieve remarkable naturalness via autoregressive (AR) decoding, but require N sequential steps to generate N speech tokens. We present LLaDA-TTS, which replaces the AR LLM with a masked diffusion model that completes generation in a fixed number of parallel steps, decoupling inference latency from sequence length. Remarkably, using only 50 hours of fine-tuning data, we successfully transfer a pretrained AR checkpoint to the masked diffusion paradigm via bidirectional attention. At 64 steps, LLaDA-TTS achieves 0.98% CER (zh) and 1.96% WER (en) on Seed-TTS-Eval, matching the original CosyVoice 3 baseline performance while delivering a 2x LLM-stage speedup--a notable acceleration achieved despite the absence of KV cache, an optimization the AR baseline heavily relies on. Beyond acceleration, the bidirectional architecture naturally enables zero-shot speech editing--including word-level insertion, deletion, and substitution--without any additional training. Theoretically, we prove that AR-pretrained weights are near-optimal for bidirectional masked prediction under the locality property of acoustic tokens, explaining this rapid convergence. This general method modifies only the attention mask and objective, applying seamlessly to any LLM-based AR TTS system. Code and audio samples will be available at https://deft-piroshki-b652b5.netlify.app/.
Figures & tables
Figure 1: LLaDA-TTS architecture overview. A bidirectional Transformer (Qwen2) iteratively unmasks speech tokens in T steps. The text encoder, sequence format, and downstream flow matching vocoder remain identical to the AR baseline.
Figure 2: Speech editing pipeline: (1) align text → speech via attention, (2) mask affected region with context margins, (3) regenerate via iterative unmasking. The surrounding tokens remain frozen, providing bidirectional conditioning throughout.
test-zh
test-en
test-zh-hard
Model
Type
CER ↓
SS ↑
WER ↓
SS ↑
CER ↓
SS ↑
Human
—
1.26
75.5
2.14
73.4
—
—
Seed-TTS
AR
1.12
79.6
2.25
76.2
7.59
77.6
VoxCPM ( Li et al., 2025a )
AR
0.93
77.2
1.85
72.9
8.87
73.0
Index-TTS 2 ( Deng et al., 2025 )
AR
1.03
76.5
2.23
70.6
7.12
75.5
MaskGCT
NAR
2.27
77.4
2.62
71.4
10.27
74.8
Table 1: Comparison on Seed-TTS-Eval benchmark (open-source leaderboard). SS: speaker similarity (%, WavLM-based). ↓ : lower is better; ↑ : higher is better.
Figure 3: Speed–quality tradeoff. Left axis: test-zh CER (%) vs. denoising steps; right axis: LLM-stage RTF on A100. The dashed green line marks the CosyVoice 3 AR baseline CER (1.21%). LLaDA-TTS surpasses the AR baseline at 48 steps ( ∼2.6× speedup) and achieves 0.74% CER at 96 steps.
Configuration
zh CER ↓
zh-hard CER ↓
en WER ↓
AR weights only (epoch 0, no training)
99.97
99.94
99.97
From scratch
45.27
68.25
62.58
AR init, no label shift
1.45
12.33
2.30
AR init + label shift (full)
0.98
7.04
1.96
Table 2: Ablation of initialization strategy and label shift on Seed-TTS-Eval mini set at matched training compute.
Figure 4: Unmasking process for a Chinese utterance (64 steps). Each column is a speech token position; color indicates the step at which the token was unmasked (blue=early, red=late, gray=still masked). The generation exhibits a predominantly left-to-right progression with confidence-based deviations.
Figure 5: Emergent text-to-speech alignment in LLaDA-TTS. Axes: text tokens ( → ) vs. speech tokens ( ↓ ).
Autoregressive (AR) text-to-speech (TTS) models generate discrete speech tokens sequentially, which makes inference slow and can degrade robustness, since local errors propagate to later positions and can escalate into hallucination. This limitation stems from their left-to-right AR commitment: each token must be determined before future speech-token context is available. However, such ordering is not an inherent requirement for TTS, since the model receives the full input text before synthesis. In this paper, we introduce DELTA-TTS, a lightweight LoRA-based adaptation framework that converts a pretrained AR TTS model into a discrete diffusion language model (dLLM) for confidence-ordered speech-token decoding. To better capture the local structure of speech, DELTA-TTS incorporates a convolution module that injects local acoustic context, together with a 1/t-weighted training objective and a time-shifted inference schedule that together defer low-confidence positions to later steps. Trained on only 585 hours of LibriTTS, DELTA-TTS achieves a 1.75% WER on Seed-TTS test-en, outperforming its AR backbone while generating tokens 3.3 times faster. Further analysis shows that DELTA-TTS produces sharper text-speech alignment, increases overall decoding confidence, and mitigates the hallucinations observed in AR generation. Audio samples, inference code, and model weights are available at https://slpr-lab.github.io/DELTA-TTS.
Streaming text-to-speech synthesis in cascaded LLM-TTS systems still faces latency challenges as most TTS models require full context before initiating generation. We present S5-TTS, a streaming variant of T5-TTS that enables low-latency, word-by-word incremental speech synthesis through encoder-decoder language modeling and monotonic alignment learning. S5-TTS begins generating speech immediately after receiving the first few words, substantially reducing end-to-end response latency. To maintain quality under limited lookahead, we introduce a lookahead-causal masking mechanism with Conv-based auxiliary attention that preserves intelligibility and speaker similarity, and employ interleaved multi-source distillation to further restore naturalness. Experiments show that S5-TTS achieves comparable quality to full-context T5-TTS, supports zero-shot synthesis with high speaker similarity, and significantly reduces end-to-end latency for practical conversational AI systems.
Unlike autoregressive models, discrete diffusion-based models for zero-shot text-to-speech generate speech tokens in parallel and can revisit earlier predictions. Mask-and-replace training extends mask-only training by randomly replacing some tokens, and its gains are commonly attributed to self-correction, the ability to revise previously generated tokens. However, exposure to randomly perturbed context during training may itself improve generation, raising the question of whether these gains require inference-time token revision. To investigate this question, we use DeMaR, which combines mask-and-replace training with confidence-ranked mask-only sampling while preserving the total training corruption probability. Trained from scratch on LibriTTS, DeMaR achieves lower word error rates (WER) than autoregressive and mask-only diffusion baselines using the same speech tokenizer. This advantage persists when each token remains unchanged after first being unmasked. Matched training conditions on two heterogeneous speech tokenizers show that both noisy-context augmentation and replacement supervision improve WER under this restriction. These findings demonstrate training-side benefits of replacement beyond enabling inference-time token revision.
Hounsu Kim, Joonyong Park, Yuki Saito +2
KAIST, Republic of Korea · The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan