DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting

Jaehyun Ahn; Dohyun Kim; Heeseung Kim

cs.SDOct 16, 2026

DriftTTS: One-stage Single Step Text-to-Speech via Generative Drifting

Authors: Jaehyun Ahn, Dohyun Kim, Heeseung Kim

Organizations: Department of AI, University of Seoul, Seoul, South Korea

Abstract

Text-to-speech models built on score matching and flow matching produce natural speech, but only by calling the decoder many times, which makes synthesis slow. Existing ways to cut the number of calls either distill from a multi-step teacher or add a stage on top of a pretrained model, leaving single-step TTS without a pretrained TTS model unexplored. We question whether pretraining a TTS model is necessary, and propose DriftTTS, which adopts generative drifting, an objective that moves iterative sampling from inference into training. Drifting assumes samples of fixed shape, whereas speech has variable length; we address this issue with segment drifting. On LJSpeech, DriftTTS matches the multi-step quality of Matcha-TTS and RapFlow-TTS in a single decoder call, while being faster. To our knowledge, this is the first application of generative drifting to text-to-speech.

Figures & tables

Figure 1: Overview of DriftTTS during training. The text encoder and MAS construct the prior sample ${\bm{\epsilon}}=\mu$ by repeating each token representation according to its duration, and the decoder maps it to a generated segment ${\mathbf{x}}\sim f_{\#}p_{\mu}$ in a single call. Segment drifting compares the generated segment with a target segment ${\color[rgb]{0,0,1}\mathbf{y}^{+}}\sim p$ in the MAE feature space.

Figure 2: Evolution of the generated distribution toward the target distribution during training. Each panel flattens a fixed 80-frame log-mel segment from every evaluation utterance and projects it using a single PCA basis fitted to the targets.

Model	NFE	WER	MOS	RTF ( $10^{-3}$ )
Ground truth	–	1.92	$4.53\pm 0.06$	–
Vocoded GT	–	1.99	$4.48\pm 0.06$	–
Matcha-TTS	10	1.93	$4.38\pm 0.08$	18.7
RapFlow-TTS	2	1.82	$4.42\pm 0.08$	5.85
Matcha-TTS	1	1.86	$4.09\pm 0.10$	4.15
+ MeanFlow	1	1.83	$4.39\pm 0.07$	4.19

Table 1: Comparison on the LJSpeech evaluation set. MOS is computed from 290 responses of 153 listeners, and RTF is measured on a single NVIDIA A10 GPU.

Type	WER	UTMOSv2	DNSMOS
Baseline	1.80	3.931	3.263
w/o MAE	20.03	0.517	1.394
frame, w/o MAE	2.72	2.137	1.724
$\tau=0.02$	2.00	3.829	3.149
$\tau=0.05$	2.00	3.982	3.244
$\tau=0.2$	1.90	3.936	3.273

Table 2: Ablation on the feature extractor, drifting unit, and temperature. Each row lists its change from the baseline, and the

\tau

rows use a single temperature in place of the multi-temperature loss. DNSMOS is the overall score.

Figure 3: False convergence of frame drifting. The target is the mel-spectrogram of a validation utterance, and the frame-wise model predicts it using the alignment found by MAS, so the two panels have the same number of frames.

TTS Synthesis

Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.

See all 275 papers

DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift
Oct 2, 2026Mohammad Nur Hossain Khan, Subrata Biswas, Bashima IslamTTS Synthesis Distribution Matching
University of Massachusetts Amherst, MA, USA · Worcester Polytechnic Institute, MA, USA
DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech
Jul 5, 2026Junwon Moon, Yejin Lee, Seungbeom Kim +4TTS Synthesis Diffusion Model Fine-Tuning
Sungkyunkwan University · University of Seoul
Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech
Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS Synthesis TTS Synthesis
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea

Oct 2, 2026J/K move · Enter open · S save

DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift

Mohammad Nur Hossain Khan, Subrata Biswas, Bashima Islam

University of Massachusetts Amherst, MA, USA · Worcester Polytechnic Institute, MA, USA

TTS Synthesis Distribution Matching

Open paper AskPDF arXiv

Few-step neural text-to-speech models often rely on short- ened diffusion or flow-matching schedules, or on distillation from pretrained multi-step teachers. To avoid these depen- dencies, we present DriftTTS, a few-step mel-spectrogram generator trained without a generative teacher, distillation, or adversarial discrimination. DriftTTS uses a distribution- matching drift objective in a mel-domain feature space defined by raw mels and a frozen masked-autoencoder encoder pretrained on the same LJSpeech training split. On-policy rollout trains the decoder on its own interme- diate states and supports inference up to the trained roll- out depth. On LJSpeech, DriftTTS at NFE=4 achieves 3.87 dB MCD and 3.7% WER, compared with 3.85 dB and 3.4% for Matcha-TTS. In a fully paired blind listen- ing test, DriftTTS obtains 4.18 MOS, compared with 3.96 for Matcha-TTS and 4.22 for ground truth. These results demonstrate competitive few-step synthesis without a pre- trained generative teacher. Code can be found at https: //github.com/BASHLab/driftTTS.git

Abstract

TTS Synthesis

Explore similar work

DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift

DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech