Organizations: PhD Program in Mathematics, Yeshiva University, New York, NY 10033, USA · Department of Computer Science & Engineering, Yeshiva University, New York, NY 10033, USA · School of Artificial Intelligence, Chuzhou University, Anhui, 239000, China
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.
Figures & tables
Figure 1: Overview of MambaVoiceCloning (MVC). The framework uses Bi-Mamba Text Encoders for phoneme modeling, a Temporal Bi-Mamba for rhythmic alignment, and an Expressive Mamba for prosodic control. A lightweight aligner (dotted box) provides phoneme–frame supervision only during training, ensuring an SSM-only encoder at inference. Conditioning features drive a diffusion decoder and vocoder for waveform synthesis.
Model
MOS-N ↑
MOS-S ↑
Ground Truth
4.60
4.35
VITS
3.69
3.54
StyleTTS2
4.15
4.03
MVC (ours)
4.22
4.07
Table 1: Subjective evaluation on unseen LibriTTS speakers.
Model
MOS-ID
MOS-OOD
GT
3.81
3.70
StyleTTS2
3.83
3.87
VITS
3.44
3.21
JETS
3.57
3.21
MVC
3.87
3.88
Table 2: MOS on in-distribution (ID) and OOD texts.
Model
MOS-short
MOS-long
RTF-short
RTF-long
StyleTTS2
4.15
3.91
0.0185
0.0200
MVC
4.22
4.16
0.0177
0.0170
Table 3: Short- vs. long-form performance on LJSpeech.
Model
F0 RMSE ↓
MCD ↓
WER ↓
PESQ ↑
RTF ↓
VITS
0.667 ± 0.011
4.97 ± 0.09
7.23%
3.64 ± 0.08
0.0211
StyleTTS2
0.651 ± 0.013
4.93 ± 0.06
6.50%
3.79 ± 0.07
0.0174
MVC (ours)
0.653 ± 0.014
4.91 ± 0.07
6.52%
3.85 ± 0.06
0.0169
Table 4: Objective metrics on LJSpeech. Arrows indicate the desired direction of improvement (higher is better for PESQ, lower is better for others). Values are averaged over three seeds.
L (s)
WER
MOS
0.25
11.2%
3.74
0.50
9.4%
3.81
1.00
7.8%
3.89
2.00
7.3%
3.91
Table 5: Streaming performance with look-ahead L on 2–6 min Gutenberg passages.
Removed component
CMOS-N drop
Bi-Mamba text encoder
-0.38
Expressive Mamba predictor
-0.41
Temporal Bi-Mamba encoder
-0.36
Table 6: Component removal on the OOD set, reported as CMOS-N drop relative to full MVC.
Encoder
MOS ID ↑
RTF ↓
Pitch RMSE (Hz) ↓
BiLSTM (no Mamba)
3.61 ± 0.13
0.0268
29.2
2 Mamba layers
3.65 ± 0.12
0.0215
27.5
3 Mamba layers
3.72 ± 0.11
0.0203
25.4
4 Mamba layers
3.78 ± 0.10
0.0198
24.1
5 Mamba layers
3.85 ± 0.10
0.0195
23.7
6 Mamba layers
3.87 ± 0.07
0.0189
23.2
Table 7: Depth ablation for the text encoder on LJSpeech (in-distribution). Includes a BiLSTM baseline; results are averaged over three seeds. Lower RTF is better.
Variant
MOS long ↑
Pitch RMSE (Hz) ↓
RTF ↓
MVC (gated + AdaLN)
4.16 ± 0.07
1.92 ± 0.05
0.0177
Gated only (no AdaLN)
4.02 ± 0.08
2.04 ± 0.06
0.0186
AdaLN only (no gating)
3.95 ± 0.04
2.22 ± 0.05
0.0198
Concat (no gating, no AdaLN)
3.64 ± 0.09
2.89 ± 0.07
0.0216
Table 8: Fusion and conditioning ablation on LJSpeech long-form utterances. Removing gated fusion or AdaLN reduces MOS and increases pitch RMSE. Values are averaged over three seeds.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Encoder Params
Encoder speedup ( × ) ↑
Peak Memory ↓
StyleTTS2
42M
1.00
100%
Mamba-hybrid
32M
1.15
86%
MVC (ours)
21M
1.60
72%
Appendix
Table 9: Encoder-only throughput and peak memory, normalized to StyleTTS2. All models share the same mel–diffusion–vocoder stack. MVC’s SSM-only conditioning achieves the best encoder efficiency and memory usage, enabling larger-batch and longer-context synthesis while leaving the diffusion latency profile unchanged.
System
Inference attention?
Rhythm/ duration
Prosody/ style mech.
Fusion/ modulation
SSM-only?
StyleTTS2
Yes (Transformer)
Variance/ duration pred.
Ref./style encoder (attn)
Attention / concat
No
Miyazaki’24
Hybrid (SSM+attn)
Mixed (keeps attn)
Mixed (keeps attn)
Concat
No
Jiang’25 (Speech Slytherin)
Hybrid (SSM+attn)
Attn/var. pred.
Ref./style enc. (attn)
Concat
No
Zhang’24 (Mamba in Speech)
Hybrid (SSM+attn)
Mixed (keeps attn)
Mixed (keeps attn)
Concat
No
MVC (ours)
No (SSM-only)
Temporal Bi-Mamba
Mamba + AdaLN
Gated bi-dir. fusion + AdaLN
Yes
Appendix
Table 10: Inference-time comparison with attention-centric and Mamba-based TTS systems. “Hybrid” denotes that attention or recurrence is retained in at least one conditioning module (duration, rhythm, or prosody); only MVC deploys an SSM-only conditioning stack across all of them.
Symbol
Description
Input & Dimensions
Tx
Number of text tokens
Tm
Number of mel frames
F
Number of mel bins (80)
d
Text embedding dimension
dh
SSM hidden dimension
Appendix
Table 11: Notation summary covering all variables used in the Methodology section.
Model
F0 RMSE ↓
MCD ↓
WER ↓
PESQ ↑
RTF ↓
Hybrid-Mamba (Concat)
0.659 ± 0.013
4.95 ± 0.07
6.68%
3.79 ± 0.06
0.0189
Bi-Mamba (Concat-only)
0.656 ± 0.014
4.93 ± 0.06
6.58%
3.82 ± 0.06
0.0181
MVC (gated + AdaLN)
0.653 ± 0.014
4.91 ± 0.07
6.52%
3.85 ± 0.06
0.0177
Appendix
Table 12: Protocol-matched Mamba–TTS baselines under our mel/diffusion/vocoder pipeline on LJSpeech. Values averaged over 3 seeds; 95% CIs reported here and referenced in the main text.
# Steps
MOS-N ↑
RTF ↓
3
3.62 ± 0.12
0.0151
4
3.74 ± 0.09
0.0164
5 (used)
3.87 ± 0.07
0.0177
6
3.88 ± 0.08
0.0190
7
3.89 ± 0.08
0.0205
9
3.89 ± 0.08
0.0221
Appendix
Table 13: Diffusion step ablation on the LJSpeech validation set. Increasing steps improves quality but degrades synthesis speed. Five steps yield the best quality–efficiency trade-off and are used in the main experiments.
Model
Encoder / Conditioning Path
StyleTTS2
Transformer acoustic text encoder with reference style encoder; attention-based duration and prosody predictors; diffusion decoder conditioned via attention and style embeddings.
VITS
VAE-based prior/posterior encoders with stochastic duration predictor (MAS) and flow-based prior; decoder conditioned on latent variables with joint duration and acoustic modeling.
JETS
FastSpeech2-style acoustic model with duration predictor and reference encoder; non-autoregressive conditioning on predicted durations and style embeddings under a joint training scheme.
Hybrid-Mamba (Concat)
Mamba text encoder paired with attention-based duration and style modules; diffusion decoder receives concatenated SSM features and attention-derived style signals.
Bi-Mamba (Concat-only)
SSM-only encoder path with bidirectional Uni-Mamba scans for text, temporal, and expressive streams; conditioning formed by simple concatenation of SSM features without gating or AdaLN.
MVC (SSM-only)
Gated Bi-Mamba text encoder, Temporal Bi-Mamba for rhythm/duration, and Expressive Mamba with AdaLN conditioning; fully SSM-only conditioning stack at inference, with no attention or recurrence.
Appendix
Table 14: Protocol-matched baseline configurations under the shared mel/diffusion/vocoder pipeline. The table highlights only encoder and conditioning-path differences; training schedule and vocoders are identical across all models (Sec. 4.1, Appendix C.2 ).
Figure 2: Waveform comparison of synthesized speech from different TTS models on LJSpeech, evaluated using MOS (95% CI). MVC closely aligns with the ground truth, capturing finer prosodic variations and outperforming StyleTTS2 and JETS in expressiveness and naturalness.
Figure 3: Validation MOS and F0 RMSE curves over training epochs for MVC and StyleTTS2 on LJSpeech. MVC reaches strong validation quality and stable pitch error in fewer epochs under a matched optimization schedule.
Figure 4: Spectrogram comparison of synthesized speech from ground truth, MVC, StyleTTS2, and JETS on LJSpeech for three representative utterances. Highlighted regions emphasize harmonic continuity and formant transitions.
Module
Avg. time (ms)
Proportion (%)
Bi-Mamba encoder stack
42.5
31.4
Diffusion decoder
73.4
54.2
Vocoder (HiFi-GAN / iSTFTNet)
19.5
14.4
Total
135.4
100.0
Appendix
Table 15: Average inference time per utterance (milliseconds) and proportion of total runtime, measured on 500 LJSpeech utterances on a single A100 with FP16 inference.
Model
MOS-N ↑
MOS-S ↑
VITS
3.66 ± 0.12
3.53 ± 0.13
StyleTTS2
4.12 ± 0.11
4.01 ± 0.10
MVC (ours)
4.18 ± 0.10
4.09 ± 0.11
Appendix
Table 16: VCTK zero-shot speaker generalization (MOS with 95% confidence intervals). Higher is better.
Model
ES ↑
DE ↑
FR ↑
VITS
3.48 ± 0.12
3.39 ± 0.13
3.46 ± 0.12
StyleTTS2
3.84 ± 0.12
3.76 ± 0.11
3.85 ± 0.11
MVC (ours)
3.91 ± 0.11
3.82 ± 0.10
3.93 ± 0.10
Appendix
Table 17: CSS10 cross-lingual naturalness (MOS-N with 95% confidence intervals). Higher is better.
Reference length
MOS-S ↑
MOS-N ↑
2 seconds
3.87 ± 0.10
4.03 ± 0.11
4 seconds
3.94 ± 0.09
4.11 ± 0.10
6 seconds (main)
4.02 ± 0.09
4.18 ± 0.10
8 seconds
4.03 ± 0.09
4.19 ± 0.10
Appendix
Table 18: Effect of reference length on zero-shot VCTK speaker similarity (MOS-S) and naturalness (MOS-N).
State dimension
MOS (in-dist.) ↑
MOS (OOD) ↑
RTF ↓
64
3.96 ± 0.09
3.88 ± 0.09
0.0164
96
4.02 ± 0.08
3.92 ± 0.09
0.0169
128
4.03 ± 0.08
3.93 ± 0.08
0.0176
160
4.04 ± 0.09
3.94 ± 0.09
0.0184
Appendix
Table 19: Sensitivity to state dimension dssm on LJSpeech. MOS values are averaged over three seeds with 95% confidence intervals; lower RTF is better. The configuration used in the main paper is in bold.
Kernel size
MOS (long) ↑
Pitch RMSE (Hz) ↓
RTF ↓
3
4.08 ± 0.08
2.06 ± 0.06
0.0172
5
4.16 ± 0.07
1.92 ± 0.05
0.0177
7
4.17 ± 0.07
1.90 ± 0.05
0.0184
Appendix
Table 20: Sensitivity to convolution kernel size kconv in the Mamba block. Pitch RMSE is computed on long-form LJSpeech utterances; lower is better.
Temperature
MOS (OOD) ↑
CMOS-N ↑
WER ↓
0.7
3.83 ± 0.09
-0.06
7.12%
1.0
3.88 ± 0.09
0.00
6.89%
1.3
3.86 ± 0.09
-0.02
6.97%
Appendix
Table 21: Sensitivity to gating temperature τgate on the Gutenberg OOD set. CMOS-N is measured relative to the default configuration with τgate=1.0 .
System
Training data
Languages
Main architecture / setting
Representative reported metrics
Scale
NaturalSpeech 3
∼ 200k h multi-speaker, multi-style speech (public + private)
Multilingual
Factorized diffusion TTS with FACodec; separate prosody/content/acoustic/timbre modules; zero-shot and long-form generation.
On LibriSpeech test-clean: Sim-O 0.67, Sim-R 0.76, WER 1.81, SMOS 4.01.
Industry (Multi-hundred thousand hours, proprietary)
On SEED-TTS EVAL: CER 1.27 (zh), WER 2.46 (en), WER 6.96 (hard) with strong Sim-O.
Industry (Large multilingual corpus, proprietary)
HiggsAudio-V2
∼ 10M h AudioVerse (speech, music, SFX)
Multilingual / multi-domain
Audio language model (5.8B params) with 12-codebook RVQ tokenizer and DualFFN adapter; unified speech/music/SFX.
SeedTTS-Eval: WER 2.44%, 67.7% speaker similarity; dialogue WER 18.88%.
Industry (Large-scale, multi-domain, proprietary)
MVC (ours)
24 h LJSpeech + 245 h LibriTTS (public)
English
Fully SSM-based conditioning stack (bi-Mamba text, Temporal Bi-Mamba, Expressive Mamba + AdaLN) under a fixed diffusion/vocoder pipeline.
Improves MOS/CMOS and WER over StyleTTS2, VITS, and hybrid Mamba-attention baselines under identical preprocessing and vocoders.
Academic (24h + 245h, public datasets)
Appendix
Table 22: Qualitative positioning of MVC relative to recent industrial-scale systems. NaturalSpeech 3, CosyVoice 3, and HiggsAudio-V2 operate at much larger data and model scales, with different objectives and proprietary evaluation pipelines. MVC is an open-data encoder study under a unified mel–diffusion–vocoder setup, and thus numeric comparisons would be misleading.
Conditional-flow text-to-speech (TTS) models can achieve high synthesis quality with relatively few sampling steps, but collapsing the remaining trajectory to a single update can still cause substantial quality degradation. We propose MixiSpeech, a progressive state distillation framework that converts a multi-step conditional-flow acoustic model into a one-step generator while preserving its architecture and speaker conditioning. Rather than training the one-step model directly, MixiSpeech progressively reduces the global sampling budget through a 10-4-2-1 curriculum, with each student trained to match local teacher-integration endpoints using a masked state-matching objective. We instantiate the framework on a 710-speaker Vietnamese Matcha-TTS model trained from scratch on PhoAudiobook. On 100 shared test text-speaker pairs, direct one-step inference substantially degrades synthesis quality, obtaining 1.6846 UTMOS22-Strong and 6.21% WER compared with 2.2923 and 4.65% for the 10-step teacher. In contrast, the distilled one-step model achieves 2.2678 UTMOS22-Strong, 4.63% WER, and 2.83% CER, recovering most of the quality lost at the one-step operating point while requiring only a single acoustic vector-field evaluation. End-to-end inference time decreases from 0.0662 s for the 10-step teacher to 0.0312 s for MixiSpeech, corresponding to a 2.12 measured speedup. These results show that progressive state distillation can make one-step conditional-flow TTS practical without relying on a large multi-step sampling budget. Project page: https://mixispeech.solanai.us/
Tran Quoc Bao Le, Binh Mai, Hung Dinh +2
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam
Recent full-stream text-to-speech (TTS) models enable low-latency personalized synthesis as input text arrives. However, sustaining generation beyond the lengths seen during training without additional training remains largely unexplored. We systematically analyze VoXtream2, which extends generation by periodically discarding past context while retaining the acoustic prompt. We find that this strategy does not prevent degradation: pronunciation accuracy deteriorates near the end of the training range due to the range of positions in which the model operates rather than the loss of past context, while speaker similarity gradually declines throughout generation. Based on these findings, we propose two training-free modifications that reduce the attention window and strengthen attention to the acoustic prompt. Experiments show that these modifications preserve pronunciation accuracy and speaker similarity throughout long-form generation while retaining the naturalness of the original model. Ablations further show complementary effects of the two modifications.
Jeseung Lim, Jiseong Woo, Heeseung Kim
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko +4