This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral v-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
Figures & tables
Model
Qwen3-TTS
VoiceBox
E2/F5-TTS
CosyVoice3
VoxCPM
VITS
E3-TTS
WaveGen
End-to-End (T → W)
✗
✗
✗
✗
✗
✓
✗
✓
Single-stage
✗
✗
✗
✗
✗
✓
✗
✓
Zero-shot
✓
✓
✓
✓
✓
✗
✓
✓
Dependency on Intermediate Representations (Explicit Waveform Decoder)
Codec
✓
✗
✗
✓
✗
✗
✗
✗
Vocoder
✗
✓
✓
✓
✗
✗
✗
✗
Table 1: Comparison of WaveGen with recent TTS systems. WaveGen uniquely achieves a fully E2E single-stage zero-shot TTS without relying on intermediate representations or auxiliary modules. T, C, W, M, L, and B denote text, codec, waveform, Mel, latent, and BERT, respectively.
Figure 1: E2E text-to-waveform diffusion training details. We reshape high-resolution waveform signals (24,000 Hz) into a low-resolution space (100 Hz) for efficient E2E waveform diffusion. Then, we introduce Spectral Forcing for internal spectral guidance and simply concatenate average-upsampled text embeddings. We use shared DiT blocks for text-speech alignment and disentangled DiT heads for representation refinement. We adopt x -prediction as the prediction space, and v -loss as the target loss space, and introduce a new multi-scale v -space spectral loss
Model
E2E
Params
Data
Vocoder
Codec
VAE
SPK Enc.
LLM
GAN
Diffusion
WER ↓
SIM ↑
GT
-
-
-
-
-
-
-
-
-
-
2.14
0.73
SparkTTS [ 59 ]
✗
0.5B
100k
✗
✓
✗
✓
✓
✓
✗
1.98
0.58
TadiCodec-AR [ 60 ]
✗
4B
100k
✓
✓
✗
✗
✓
✓
✓
2.28
0.65
F5-TTS [ 5 ]
✗
0.3B
100k
✓
✗
✗
✗
✗
✓
✓
2.00
0.67
ZipVoice [ 77 ]
✗
0.1B
100k
✓
✗
✗
✗
✗
✓
✓
1.70
0.69
CosyVoice2 [ 14 ]
✗
0.5B
166k
✓
✓
✗
✓
✓
✓
✓
3.09
0.65
Table 2: Zero-shot TTS results on Seed-en Benchmark. Note that parameter size (Params) of baseline models only indicates their TTS module, but they require additional modules for waveform decoding.
Model
GT
F5-TTS
CosyVoice3
Qwen3-TTS-0.6B
Qwen3-TTS-1.7B
WaveGen-S
WaveGen-B
nMOS
4.117 ± 0.112
3.938 ± 0.119
3.917 ± 0.128
3.883 ± 0.123
3.954 ± 0.117
3.958 ± 0.129
4.000 ± 0.126
sMOS
3.932 ± 0.038
3.989 ± 0.037
3.972 ± 0.038
3.968 ± 0.037
4.008 ± 0.037
3.952 ± 0.037
3.988 ± 0.037
Table 3: Subjective evaluation results on Seed-en benchmark
Type
Bin
WER
SIM
DNSMOS
Analysis
No SF
-
123.67
0.19
3.20
Fails to align
Mel
256
126.50
0.35
3.17
Denoising
Mel
128
18.67
0.52
3.55
Reconstruction
Mel
80
6.80
0.50
3.62
Mel Bins ↓↑
Mel
64
4.96
0.47
3.72
Faster
Mel
32
4.26
0.48
3.64
Convergence
Table 4: Comparison of Spectral Forcing
Method
Params.
NFE
RTF
WER
SIM
DNSMOS
End-to-End
407M
32
0.24
2.91
0.53
3.67
End-to-End
210M
32
0.12
4.68
0.50
3.71
End-to-End
113M
32
0.08
4.96
0.47
3.72
Vocoding (GT Mel)
113M
32
0.08
1.47
0.70
3.32
Vocoding (GT Mel)
113M
16
0.04
1.57
0.70
3.33
Two-Stage
226M
32 + 32
0.16
4.39
0.46
3.53
Table 5: Comparison of E2E and cascaded models
# Shared DiT
# Disentangled DiT
WER
SIM
DNSMOS
18
0
7.83
0.44
3.60
14
2 (4)
4.96
0.47
3.72
10
4 (8)
7.64
0.46
3.65
Table 6: Disentangled DiT Heads
Pred
Loss
SF
WER
SIM
DNSMOS
x -pred
v -loss
✗
123.67
0.19
3.20
v -pred
v -loss
✗
119.52
0.16
2.65
x -pred
v -loss
✓
5.57
0.46
3.45
v -pred
v -loss
✓
3.93
0.48
2.94
Table 7: Comparison of prediction space
Method
Loss
WER
SIM
DNSMOS
Orig. CFM [ 37 ]
Lv
5.57
0.46
3.45
Modified CFM [ 69 ]
Lv-spec
68.82
0.46
3.49
Ours (Ablation)
Lms-v-spec
16.98
0.51
3.57
Ours (Final)
Lv + λmsLms-v-spec
4.96
0.47
3.72
Table 8: Loss Comparison
Method
Loss
WER
SIM
DNSMOS
Ours (Final)
Lv + λmsLms-v-spec
4.96
0.47
3.72
+REPA (Optional)
+ λrepaLrepa (Whisper)
3.54
0.54
3.70
Table 9: REPA result
Model
Params.
Data (hours)
Training Steps
WER ↓
Basic CFG (2.0) SIM ↑
DNSMOS ↑
WER ↓
Modified CFG (2.0) SIM ↑
DNSMOS ↑
WaveGen-S
113M
0.5k
1.1M
4.96
0.47
3.72
4.64
0.49
3.57
WaveGen-B
407M
0.5k
1.1M
2.91
0.53
3.67
2.93
0.55
3.59
WaveGen-S
113M
120k
1.1M
4.27
0.49
3.60
4.42
0.53
3.56
WaveGen-B
407M
120k
1.1M
2.37
0.59
3.60
1.77
0.63
3.60
Table 10: Model comparison with respect to model size and dataset scale
Model
NFE
WER
SIM
DNSMOS
4
52.0
0.39
3.06
8
4.85
0.61
3.48
16
2.16
0.63
3.56
WaveGen-B
32
1.77
0.63
3.60
50
1.87
0.64
3.61
100
1.84
0.64
3.63
Table 11: Analysis of NFE (Seed-en)
Model
E2E
SF (Drop)
Anchoring
Hz
Hop (Frame)
Params
Data
WER ↓
seed-en SIM ↑
UTMOS ↑
CER ↓
seed-zh SIM ↑
UTMOS ↑
GT
-
-
-
24k
-
-
-
2.06
0.73
3.53
1.26
0.76
2.78
Seed-TTS [ 1 ]
✗
✗
-
24k
-
-
-
2.25
0.76
-
1.12
0.79
-
VoxCPM-Emilia [ 74 ]
✗
✗
-
16k
-
0.5B
100K
2.34
0.68
-
1.11
0.74
-
MaskGCT [ 62 ]
✗
✗
-
24k
-
2.2B
100K
2.88
0.71
3.55
2.40
0.77
2.63
OmniVoice-Emilia [ 78 ]
✗
✗
-
24k
-
0.8B
100K
1.72
0.71
3.88
0.89
0.76
3.05
ZipVoice [ 77 ]
✗
✗
-
24k
256 (93.75)
123M
100K
1.70
0.69
3.82
1.40
0.75
3.15
Table 12: Multilingual TTS results using F5-TTS architecture and large-scale training configuration. For a fair comparison, all models are trained with the open-source Emilia dataset about 100k hours (English and Chinese). With end-to-end waveform generation, our models achieve the state-of-the-art performance on Seed-TTS-Eval benchmarks using public-available datasets.
Figure 2: Spectral analysis with respect to audio format and prediction space
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Module
Hyperparameter
WaveGen-S
WaveGen-B
WaveGen
Waveform
Sampling Rate
24000
24000
24000
Reshape
(100, 240)
(100, 240)
(50,480)
Diffusion Resolution
100Hz
100Hz
50Hz
Spectral Forcing
Hop
240
240
480
Window
960
960
1920
FFT
960
960
1920
Appendix
Table 13: Hyperparameters of WaveGen
Figure 3: MOS interfaces
Model
Params.
Data
Training Steps
WER( ↓ )
Basic CFG (2.0) SIM( ↑ )
DNSMOS( ↑ )
WER( ↓ )
No CFG SIM( ↑ )
DNSMOS( ↑ )
WaveGen-S
113M
0.5k
0.3M
104.23
0.32
3.53
129.85
0.25
3.31
WaveGen-S
113M
0.5k
0.5M
26.33
0.40
3.55
92.88
0.27
3.22
WaveGen-S
113M
0.5k
0.7M
8.75
0.45
3.68
76.66
0.33
3.38
WaveGen-S
113M
0.5k
0.9M
6.02
0.47
3.65
58.53
0.35
3.37
WaveGen-S
113M
0.5k
1.1M
4.96
0.47
3.72
56.58
0.36
3.36
WaveGen-B
407M
0.5k
0.3M
28.19
0.42
3.60
93.01
0.31
3.33
Appendix
Table 14: Comparison with respect to training steps.
Recently, diffusion models operating on VAE latents or mel-spectrograms have become the dominant paradigm for zero-shot TTS. Although these compressed representations improve generation efficiency, they inevitably suffer from information loss and non-end-to-end training. Theoretically, directly modeling raw waveforms circumvents these issues; however, this direction remains underexplored and is often deemed difficult due to the extremely long sequence length of audio signals. To overcome this, we propose WavTTS, the first raw waveform generative TTS model that substantially narrows the gap with latent-space generative models. Built upon the flow matching with Diffusion Transformer (DiT), WavTTS directly models speech waveforms via a simple patchification strategy, while integrating multi-scale mel-spectrogram supervision to provide perceptual guidance during training. Furthermore, we investigate the impact of prediction targets and noise scheduling in waveform diffusion, and develop an effective schedule design to improve generation quality. Evaluations on open-source benchmarks demonstrate that WavTTS closely approaches the performance of current state-of-the-art latent generative zero-shot TTS models, while substantially outperforming previous end-to-end speech generation models. Our findings demonstrate the feasibility of scaling diffusion-based TTS directly in the waveform space, opening a new direction for end-to-end speech generation.
Wenxi Chen, Dongya Jia, Yushen Chen +11
Shanghai Jiao Tong University · Shanghai Innovation Institute · ByteDance Seed
Removing intermediate representations and separately trained decoding stages has become an important direction in generative modeling. In text-to-speech, however, high-quality systems are still commonly built through an intermediate acoustic representation before waveform synthesis. In this work, we present BareWave, a fully waveform-native framework for direct text-to-wave generation in flow-matching TTS. We consider this setting to raise three training challenges: raw-waveform modeling lacks a strong pretrained representational scaffold, different stages of training benefit from different noise schedules, and data-space perceptual objectives do not automatically share the temporal structure of the velocity-space flow objective. As a result, direct waveform training is hard to optimize efficiently, hard to push toward a strong final operating point with a fixed recipe, and hard to integrate effective perceptual refinement. Guided by this view, we develop a direct text-to-wave training framework that combines training-time representation alignment, staged noise scheduling, and velocity-aware perceptual alignment (VAPA), while preserving a single waveform-native inference path without pretrained components at test time. Experiments on zero-shot voice cloning show that strong intelligibility, speaker similarity, and naturalness can be achieved under a fully waveform-native inference path, supporting waveform-native flow-matching TTS as a practical direction. Project page with audio demos is available at https://barewave.github.io/.
Wei Fan, Chao-Hong Tan, Qian Chen +5
1Anhui Province Key Laboratory of Digital Security · 2Tongyi Fun Team, Alibaba Group
Recent full-stream text-to-speech (TTS) models enable low-latency personalized synthesis as input text arrives. However, sustaining generation beyond the lengths seen during training without additional training remains largely unexplored. We systematically analyze VoXtream2, which extends generation by periodically discarding past context while retaining the acoustic prompt. We find that this strategy does not prevent degradation: pronunciation accuracy deteriorates near the end of the training range due to the range of positions in which the model operates rather than the loss of past context, while speaker similarity gradually declines throughout generation. Based on these findings, we propose two training-free modifications that reduce the attention window and strengthen attention to the acoustic prompt. Experiments show that these modifications preserve pronunciation accuracy and speaker similarity throughout long-form generation while retaining the naturalness of the original model. Ablations further show complementary effects of the two modifications.
Jeseung Lim, Jiseong Woo, Heeseung Kim
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea