Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by formulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similarity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured representations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high-quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the viability of this embedding-to-reasoning pipeline. Our work establishes speaker drift as a standalone research problem and bridges geometric signal analysis with LLM-based perceptual reasoning in modern TTS.
Figures & tables
Figure 1: Our proposed LLM-based framework for detecting speaker drift in synthesized speech. Further details are provided in Section IV.
Task
Goal
Key Characteristics
Speaker Change Detection
Identify speaker boundaries in natural multi-speaker speech streams
Assumes abrupt, discrete speaker transitions; aims at accurate change-point detection
Speaker Verification or Identification
Confirm or classify speaker identity using labeled reference data
Requires ground-truth speaker labels; evaluates similarity across separate utterances
Speaker Diarization
Segment and cluster speech by speaker in multi-speaker audio
Operates on known or unknown speakers; assumes clear speaker turns; insensitive to subtle intra-speaker drift
Voice Cloning Consistency Evaluation
Evaluate preservation of speaker identity in synthetic speech
Primarily relies on subjective human assessments or limited ABX testing; lacks automated, fine-grained detection methods
Detect anomalous speaker embeddings outside known distribution
Depends on large reference datasets; unsuitable for fine-grained temporal consistency within utterances
Voice Style Transfer Consistency
Assess preservation of non-identity attributes such as prosody, emotion, or accent
Focuses on style or affective features; does not explicitly target speaker identity stability
Table 1: Comparison of speaker drift detection with related tasks in speaker and TTS research.
Figure 2: Smooth speaker morphing across an utterance. The morphing region (3s–6s) involves audio-level cross-fading from Speaker A to Speaker B.
Embedding Type
GPT-4o [ 26 ]
Gemini-Pro-2.5 [ 34 ]
Claude-4 [ 2 ]
Qwen-3 [ 4 ]
DeepSeek-R1 [ 7 ]
PCA-based Baseline
Fixed-threshold Baseline
Wav2Vec2 [ 3 ]
90.70%
82.90%
88.20%
72.70%
80.00%
71.30%
61.70%
MFCC [ 6 ]
39.00%
38.64%
76.00%
71.40%
80.00%
65.60%
57.40%
Whisper [ 28 ]
89.41%
80.00%
84.38%
72.70%
90.60%
67.30%
61.30%
Thresholds
(0.960, 0.950, 0.995)
(0.970, 0.950, 0.998)
(0.950, 0.950, 0.995)
(0.910, 0.950, 0.990)
(0.970, 0.990, 0.995)
(0.950, 0.950, 0.950)
(0.900, 0.900, 0.900)
Table 2: Performance comparison with baselines and ablation studies using different audio embedding methods, evaluated using F1 score and pairwise cosine similarity.
Method
Input Type
Accuracy
F1 Score
GPT-4o
PCA Embeddings (8)
50.3%
66.7%
GPT-4o
PCA Embeddings (16)
73.4%
73.6%
GPT-4o
Cosine Scores
89.5%
90.7%
Gemini-Pro-2.5
PCA Embeddings (8)
50.8%
58.3%
Gemini-Pro-2.5
PCA Embeddings (16)
52.3%
59.8%
Gemini-Pro-2.5
Cosine Scores
79.7%
82.9%
Table 3: Ablation study of the proposed LLM-driven speaker drift detection framework, evaluating the impact of different LLM backbones and input feature formats.
Text-to-speech models built on score matching and flow matching produce natural speech, but only by calling the decoder many times, which makes synthesis slow. Existing ways to cut the number of calls either distill from a multi-step teacher or add a stage on top of a pretrained model, leaving single-step TTS without a pretrained TTS model unexplored. We question whether pretraining a TTS model is necessary, and propose DriftTTS, which adopts generative drifting, an objective that moves iterative sampling from inference into training. Drifting assumes samples of fixed shape, whereas speech has variable length; we address this issue with segment drifting. On LJSpeech, DriftTTS matches the multi-step quality of Matcha-TTS and RapFlow-TTS in a single decoder call, while being faster. To our knowledge, this is the first application of generative drifting to text-to-speech.
Jaehyun Ahn, Dohyun Kim, Heeseung Kim
Department of AI, University of Seoul, Seoul, South Korea
Few-step neural text-to-speech models often rely on short- ened diffusion or flow-matching schedules, or on distillation from pretrained multi-step teachers. To avoid these depen- dencies, we present DriftTTS, a few-step mel-spectrogram generator trained without a generative teacher, distillation, or adversarial discrimination. DriftTTS uses a distribution- matching drift objective in a mel-domain feature space defined by raw mels and a frozen masked-autoencoder encoder pretrained on the same LJSpeech training split. On-policy rollout trains the decoder on its own interme- diate states and supports inference up to the trained roll- out depth. On LJSpeech, DriftTTS at NFE=4 achieves 3.87 dB MCD and 3.7% WER, compared with 3.85 dB and 3.4% for Matcha-TTS. In a fully paired blind listen- ing test, DriftTTS obtains 4.18 MOS, compared with 3.96 for Matcha-TTS and 4.22 for ground truth. These results demonstrate competitive few-step synthesis without a pre- trained generative teacher. Code can be found at https: //github.com/BASHLab/driftTTS.git
Mohammad Nur Hossain Khan, Subrata Biswas, Bashima Islam
University of Massachusetts Amherst, MA, USA · Worcester Polytechnic Institute, MA, USA
Personalized text-to-speech (TTS) aims to clone the target speaker in the synthesized speech, imitating both the voice and speaking style. Current large language model (LLM)-based TTS methods ignore the style-specific prosodic patterns in generated speech, resulting in deficient style learning and thus limiting speaker similarity in synthesized speech. To this end, we investigate the prosody learning conditioned on the synthesized speech, and propose to predict the prosody of the current syllable based on previously predicted speech. Experimental results obtained on three datasets demonstrated the efficacy of the proposed dynamic prosody prediction method in enhancing the prosody learning capability, thereby improving the speaker similarity of the generated speech. Audio samples are available at https://muzw.github.io/dynapros/.
Zhenwei Mou, Liping Chen, Yajun Hu +3
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China