We present LongCat-AudioDiT, a novel, non-autoregressive diffusion-based text-to-speech (TTS) model that achieves state-of-the-art (SOTA) performance. Unlike previous methods that rely on intermediate acoustic representations such as mel-spectrograms, the core innovation of LongCat-AudioDiT lies in operating directly within the waveform latent space. This approach effectively mitigates compounding errors and drastically simplifies the TTS pipeline, requiring only a waveform variational autoencoder (Wav-VAE) and a diffusion backbone. Furthermore, we introduce two critical improvements to the inference process: first, we identify and rectify a long-standing training-inference mismatch; second, we replace traditional classifier-free guidance with adaptive projection guidance to elevate generation quality. Experimental results demonstrate that, despite the absence of complex multi-stage training pipelines or high-quality human-annotated datasets, LongCat-AudioDiT achieves SOTA zero-shot voice cloning performance on the Seed benchmark while maintaining competitive intelligibility. Specifically, our largest variant, LongCat-AudioDiT-3.5B, outperforms the previous SOTA model (Seed-TTS), improving the speaker similarity (SIM) scores from 0.809 to 0.818 on Seed-ZH, and from 0.776 to 0.797 on Seed-Hard. Finally, through comprehensive ablation studies and systematic analysis, we validate the effectiveness of our proposed modules. Notably, we investigate the interplay between the Wav-VAE and the TTS backbone, revealing the counterintuitive finding that superior reconstruction fidelity in the Wav-VAE does not necessarily lead to better overall TTS performance. Code and model weights are released to foster further research within the speech community.
Figures & tables
Figure 1: Overview of LongCat-AudioDiT. Our architecture generates continuous waveform latents directly, thereby avoiding the compounding errors that inherently arise when predicting and subsequently converting intermediate representations (e.g., mel-spectrograms) into waveforms.
Figure 2: Architecture of LongCat-AudioDiT. Middle : The overall architecture. Left : Detailed structure of the DiT block. Right : Detailed structure of the text encoder.
Model
ZH
EN
ZH-Hard
CER (%) ↓
SIM ↑
WER (%) ↓
SIM ↑
CER (%) ↓
SIM ↑
GT
1.26
0.755
2.14
0.734
-
-
NAR Models
Seed-DiT ( Anastassiou et al., 2024 )
1.18
0.809
1.73
0.790
-
-
MaskGCT ( Wang et al., 2024 )
2.27
0.774
2.62
0.714
10.27
0.748
E2 TTS ( Eskimez et al., 2024b )
1.97
0.730
2.19
0.710
-
-
Table 1: Objective evaluation results of LongCat-AudioDiT on the Seed benchmark ( Anastassiou et al., 2024 ) . The results of other methods are taken from the original paper or, if open-sourced, evaluated by us. Bold indicates the best score. Underline indicates the second-best score.
Model
Nq
FPS
PESQ ↑
STOI ↑
UTMOS ↑
GT
–
–
4.644
1.0
4.056
Discrete Codecs
DAC ( Kumar et al., 2023 )
9
900
3.908
0.970
3.910
Encodec ( Défossez et al., 2022 )
8
600
2.720
0.939
3.040
Vocos ( Siuzdak, 2023 )
8
600
2.807
0.943
3.695
WavTokenizer ( Ji et al., 2024 )
1
75
2.373
0.914
4.049
Table 2: Objective evaluation results of the proposed Wav-VAE on the LibriTTS Zen et al. (2019) test-clean subset. Bold indicates the best score among continuous VAEs. Nq is the number of codebooks for discrete codecs. For codecs, frame per second (FPS) denotes the number of tokens per second.
Figure 3: Objective evaluation results for both Wav-VAE reconstruction and TTS synthesis under varying latent dimensions . For ease of reading, we negate WER-TTS.
TTS Latent Model
ZH
EN
ZH-Hard
CER (%) ↓
SIM ↑
WER (%) ↓
SIM ↑
CER (%) ↓
SIM ↑
Mel-VAE
1.29
0.706
2.20
0.714
7.70
0.696
Wav-VAE
1.18
0.812
1.78
0.762
6.33
0.787
Table 3: Objective evaluation results of TTS models based on Wav-VAE and Mel-VAE on the Seed benchmark ( Anastassiou et al., 2024 ) . Bold indicates the best score.
Figure 4: Objective evaluation results for both Wav-VAE reconstruction and TTS synthesis across varying latent frame rates (FPS) . For ease of reading, we negate WER-TTS.
Experiment
CER (%) ↓
SIM ↑
UTMOS ↑
DNSMOS ↑
LongCat-AudioDiT-1B
1.18
0.812
3.16
3.40
training-inference mismatch
1.21
0.769
2.83
3.34
w/o APG
1.18
0.812
3.06
3.38
Table 4: Objective evaluation results of the ablation studies on noise-prompt dual masking and APG on the Seed-ZH benchmark ( Anastassiou et al., 2024 ) . Bold indicates the best score.
Recently, diffusion models operating on VAE latents or mel-spectrograms have become the dominant paradigm for zero-shot TTS. Although these compressed representations improve generation efficiency, they inevitably suffer from information loss and non-end-to-end training. Theoretically, directly modeling raw waveforms circumvents these issues; however, this direction remains underexplored and is often deemed difficult due to the extremely long sequence length of audio signals. To overcome this, we propose WavTTS, the first raw waveform generative TTS model that substantially narrows the gap with latent-space generative models. Built upon the flow matching with Diffusion Transformer (DiT), WavTTS directly models speech waveforms via a simple patchification strategy, while integrating multi-scale mel-spectrogram supervision to provide perceptual guidance during training. Furthermore, we investigate the impact of prediction targets and noise scheduling in waveform diffusion, and develop an effective schedule design to improve generation quality. Evaluations on open-source benchmarks demonstrate that WavTTS closely approaches the performance of current state-of-the-art latent generative zero-shot TTS models, while substantially outperforming previous end-to-end speech generation models. Our findings demonstrate the feasibility of scaling diffusion-based TTS directly in the waveform space, opening a new direction for end-to-end speech generation.
Wenxi Chen, Dongya Jia, Yushen Chen +11
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2
Argmax, Inc. · University of Virginia · Bilkent University
Zero-shot text-to-speech (TTS) now supports interactive assistants, personalized media, and accessibility tools. All TTS systems require faithful linguistic rendering, consistent speaker identity, and low-latency response. Yet compact streaming systems must preserve sufficient acoustic detail in a predictable low-rate latent sequence, while iterative diffusion sampling and classifier-free guidance multiply inference cost at every autoregressive step. To strike a balance between high-fidelity synthesis and low-latency inference, we present CuteTTS, a compact continuous-autoregressive TTS system. It combines semantically aligned causal VAE latents with patch-level autoregression, explicit speaker conditioning, and a bidirectional flow-matching head. We further introduce guidance-step distillation, which absorbs classifier-free guidance and multiple solver steps into a single interval-conditioned student. Evaluations on LibriSpeech and Seed-TTS-Eval demonstrate competitive intelligibility and speaker similarity in zero-shot voice cloning, while distillation lowers first-audio latency by 23.3% and real-time factor by 40.8% relative to the base model with comparable objective and subjective quality. These results provide a practical path toward continuous-autoregressive TTS that reconciles high-fidelity generation with the latency demands of real-time interaction.
Yuqian Zhang, Yao Shi, Kexin Huang +6
Work done during internship at OPPO. · OPPO AI Center · Fudan University +1