MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control

Sahil Kumar; Namrataben Patel; Honggang Wang; Youshan Zhang

cs.SDMar 31, 2026

MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control

Authors: Sahil Kumar, Namrataben Patel, Honggang Wang, Youshan Zhang

Organizations: PhD Program in Mathematics, Yeshiva University, New York, NY 10033, USA · Department of Computer Science & Engineering, Yeshiva University, New York, NY 10033, USA · School of Artificial Intelligence, Chuzhou University, Anhui, 239000, China

Abstract

MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.

Figures & tables

Figure 1: Overview of MambaVoiceCloning (MVC). The framework uses Bi-Mamba Text Encoders for phoneme modeling, a Temporal Bi-Mamba for rhythmic alignment, and an Expressive Mamba for prosodic control. A lightweight aligner (dotted box) provides phoneme–frame supervision only during training, ensuring an SSM-only encoder at inference. Conditioning features drive a diffusion decoder and vocoder for waveform synthesis.

Model	MOS-N $\uparrow$	MOS-S $\uparrow$
Ground Truth	4.60	4.35
VITS	3.69	3.54
StyleTTS2	4.15	4.03
MVC (ours)	4.22	4.07

Table 1: Subjective evaluation on unseen LibriTTS speakers.

Model	MOS-ID	MOS-OOD
GT	3.81	3.70
StyleTTS2	3.83	3.87
VITS	3.44	3.21
JETS	3.57	3.21
MVC	3.87	3.88

Table 2: MOS on in-distribution (ID) and OOD texts.

Model	MOS-short	MOS-long	RTF-short	RTF-long
StyleTTS2	4.15	3.91	0.0185	0.0200
MVC	4.22	4.16	0.0177	0.0170

Table 3: Short- vs. long-form performance on LJSpeech.

Model	$F_{0}$ RMSE $\downarrow$	MCD $\downarrow$	WER $\downarrow$	PESQ $\uparrow$	RTF $\downarrow$
VITS	0.667 $\pm$ 0.011	4.97 $\pm$ 0.09	7.23%	3.64 $\pm$ 0.08	0.0211
StyleTTS2	0.651 $\pm$ 0.013	4.93 $\pm$ 0.06	6.50%	3.79 $\pm$ 0.07	0.0174
MVC (ours)	0.653 $\pm$ 0.014	4.91 $\pm$ 0.07	6.52%	3.85 $\pm$ 0.06	0.0169

Table 4: Objective metrics on LJSpeech. Arrows indicate the desired direction of improvement (higher is better for PESQ, lower is better for others). Values are averaged over three seeds.

$L$ (s)	WER	MOS
0.25	11.2%	3.74
0.50	9.4%	3.81
1.00	7.8%	3.89
2.00	7.3%	3.91

Table 5: Streaming performance with look-ahead

L

on 2–6 min Gutenberg passages.

Removed component	CMOS-N drop
Bi-Mamba text encoder	-0.38
Expressive Mamba predictor	-0.41
Temporal Bi-Mamba encoder	-0.36

Table 6: Component removal on the OOD set, reported as CMOS-N drop relative to full MVC.

Encoder	MOS ID $\uparrow$	RTF $\downarrow$	Pitch RMSE (Hz) $\downarrow$
BiLSTM (no Mamba)	3.61 $\pm$ 0.13	0.0268	29.2
2 Mamba layers	3.65 $\pm$ 0.12	0.0215	27.5
3 Mamba layers	3.72 $\pm$ 0.11	0.0203	25.4
4 Mamba layers	3.78 $\pm$ 0.10	0.0198	24.1
5 Mamba layers	3.85 $\pm$ 0.10	0.0195	23.7
6 Mamba layers	3.87 $\pm$ 0.07	0.0189	23.2

Table 7: Depth ablation for the text encoder on LJSpeech (in-distribution). Includes a BiLSTM baseline; results are averaged over three seeds. Lower RTF is better.

Variant	MOS long $\uparrow$	Pitch RMSE (Hz) $\downarrow$	RTF $\downarrow$
MVC (gated + AdaLN)	4.16 $\pm$ 0.07	1.92 $\pm$ 0.05	0.0177
Gated only (no AdaLN)	4.02 $\pm$ 0.08	2.04 $\pm$ 0.06	0.0186
AdaLN only (no gating)	3.95 $\pm$ 0.04	2.22 $\pm$ 0.05	0.0198
Concat (no gating, no AdaLN)	3.64 $\pm$ 0.09	2.89 $\pm$ 0.07	0.0216

Table 8: Fusion and conditioning ablation on LJSpeech long-form utterances. Removing gated fusion or AdaLN reduces MOS and increases pitch RMSE. Values are averaged over three seeds.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Encoder Params	Encoder speedup ( $\times$ ) $\uparrow$	Peak Memory $\downarrow$
StyleTTS2	42M	1.00	100%
Mamba-hybrid	32M	1.15	86%
MVC (ours)	21M	1.60	72%

Appendix

Table 9: Encoder-only throughput and peak memory, normalized to StyleTTS2. All models share the same mel–diffusion–vocoder stack. MVC’s SSM-only conditioning achieves the best encoder efficiency and memory usage, enabling larger-batch and longer-context synthesis while leaving the diffusion latency profile unchanged.

System	Inference attention?	Rhythm/ duration	Prosody/ style mech.	Fusion/ modulation	SSM-only?
StyleTTS2	Yes (Transformer)	Variance/ duration pred.	Ref./style encoder (attn)	Attention / concat	No
Miyazaki’24	Hybrid (SSM+attn)	Mixed (keeps attn)	Mixed (keeps attn)	Concat	No
Jiang’25 (Speech Slytherin)	Hybrid (SSM+attn)	Attn/var. pred.	Ref./style enc. (attn)	Concat	No
Zhang’24 (Mamba in Speech)	Hybrid (SSM+attn)	Mixed (keeps attn)	Mixed (keeps attn)	Concat	No
MVC (ours)	No (SSM-only)	Temporal Bi-Mamba	Mamba + AdaLN	Gated bi-dir. fusion + AdaLN	Yes

Appendix

Table 10: Inference-time comparison with attention-centric and Mamba-based TTS systems. “Hybrid” denotes that attention or recurrence is retained in at least one conditioning module (duration, rhythm, or prosody); only MVC deploys an SSM-only conditioning stack across all of them.

Input & Dimensions
Symbol	Description
$T_{x}$	Number of text tokens
$T_{m}$	Number of mel frames
$F$	Number of mel bins (80)
$d$	Text embedding dimension
$d_{h}$	SSM hidden dimension

Appendix

Table 11: Notation summary covering all variables used in the Methodology section.

Model	F0 RMSE $\downarrow$	MCD $\downarrow$	WER $\downarrow$	PESQ $\uparrow$	RTF $\downarrow$
Hybrid-Mamba (Concat)	0.659 $\pm$ 0.013	4.95 $\pm$ 0.07	6.68%	3.79 $\pm$ 0.06	0.0189
Bi-Mamba (Concat-only)	0.656 $\pm$ 0.014	4.93 $\pm$ 0.06	6.58%	3.82 $\pm$ 0.06	0.0181
MVC (gated + AdaLN)	0.653 $\pm$ 0.014	4.91 $\pm$ 0.07	6.52%	3.85 $\pm$ 0.06	0.0177

Appendix

Table 12: Protocol-matched Mamba–TTS baselines under our mel/diffusion/vocoder pipeline on LJSpeech. Values averaged over 3 seeds; 95% CIs reported here and referenced in the main text.

# Steps	MOS-N $\uparrow$	RTF $\downarrow$
3	3.62 $\pm$ 0.12	0.0151
4	3.74 $\pm$ 0.09	0.0164
5 (used)	3.87 $\pm$ 0.07	0.0177
6	3.88 $\pm$ 0.08	0.0190
7	3.89 $\pm$ 0.08	0.0205
9	3.89 $\pm$ 0.08	0.0221

Appendix

Table 13: Diffusion step ablation on the LJSpeech validation set. Increasing steps improves quality but degrades synthesis speed. Five steps yield the best quality–efficiency trade-off and are used in the main experiments.

Model	Encoder / Conditioning Path
StyleTTS2	Transformer acoustic text encoder with reference style encoder; attention-based duration and prosody predictors; diffusion decoder conditioned via attention and style embeddings.
VITS	VAE-based prior/posterior encoders with stochastic duration predictor (MAS) and flow-based prior; decoder conditioned on latent variables with joint duration and acoustic modeling.
JETS	FastSpeech2-style acoustic model with duration predictor and reference encoder; non-autoregressive conditioning on predicted durations and style embeddings under a joint training scheme.
Hybrid-Mamba (Concat)	Mamba text encoder paired with attention-based duration and style modules; diffusion decoder receives concatenated SSM features and attention-derived style signals.
Bi-Mamba (Concat-only)	SSM-only encoder path with bidirectional Uni-Mamba scans for text, temporal, and expressive streams; conditioning formed by simple concatenation of SSM features without gating or AdaLN.
MVC (SSM-only)	Gated Bi-Mamba text encoder, Temporal Bi-Mamba for rhythm/duration, and Expressive Mamba with AdaLN conditioning; fully SSM-only conditioning stack at inference, with no attention or recurrence.

Appendix

Table 14: Protocol-matched baseline configurations under the shared mel/diffusion/vocoder pipeline. The table highlights only encoder and conditioning-path differences; training schedule and vocoders are identical across all models (Sec. 4.1, Appendix C.2 ).

Figure 2: Waveform comparison of synthesized speech from different TTS models on LJSpeech, evaluated using MOS (95% CI). MVC closely aligns with the ground truth, capturing finer prosodic variations and outperforming StyleTTS2 and JETS in expressiveness and naturalness.

Figure 3: Validation MOS and $F_{0}$ RMSE curves over training epochs for MVC and StyleTTS2 on LJSpeech. MVC reaches strong validation quality and stable pitch error in fewer epochs under a matched optimization schedule.

Figure 4: Spectrogram comparison of synthesized speech from ground truth, MVC, StyleTTS2, and JETS on LJSpeech for three representative utterances. Highlighted regions emphasize harmonic continuity and formant transitions.

Module	Avg. time (ms)	Proportion (%)
Bi-Mamba encoder stack	42.5	31.4
Diffusion decoder	73.4	54.2
Vocoder (HiFi-GAN / iSTFTNet)	19.5	14.4
Total	135.4	100.0

Appendix

Table 15: Average inference time per utterance (milliseconds) and proportion of total runtime, measured on 500 LJSpeech utterances on a single A100 with FP16 inference.

Model	MOS-N $\uparrow$	MOS-S $\uparrow$
VITS	3.66 $\pm$ 0.12	3.53 $\pm$ 0.13
StyleTTS2	4.12 $\pm$ 0.11	4.01 $\pm$ 0.10
MVC (ours)	4.18 $\pm$ 0.10	4.09 $\pm$ 0.11

Appendix

Table 16: VCTK zero-shot speaker generalization (MOS with 95% confidence intervals). Higher is better.

Model	ES $\uparrow$	DE $\uparrow$	FR $\uparrow$
VITS	3.48 $\pm$ 0.12	3.39 $\pm$ 0.13	3.46 $\pm$ 0.12
StyleTTS2	3.84 $\pm$ 0.12	3.76 $\pm$ 0.11	3.85 $\pm$ 0.11
MVC (ours)	3.91 $\pm$ 0.11	3.82 $\pm$ 0.10	3.93 $\pm$ 0.10

Appendix

Table 17: CSS10 cross-lingual naturalness (MOS-N with 95% confidence intervals). Higher is better.

Reference length	MOS-S $\uparrow$	MOS-N $\uparrow$
2 seconds	3.87 $\pm$ 0.10	4.03 $\pm$ 0.11
4 seconds	3.94 $\pm$ 0.09	4.11 $\pm$ 0.10
6 seconds (main)	4.02 $\pm$ 0.09	4.18 $\pm$ 0.10
8 seconds	4.03 $\pm$ 0.09	4.19 $\pm$ 0.10

Appendix

Table 18: Effect of reference length on zero-shot VCTK speaker similarity (MOS-S) and naturalness (MOS-N).

State dimension	MOS (in-dist.) $\uparrow$	MOS (OOD) $\uparrow$	RTF $\downarrow$
64	3.96 $\pm$ 0.09	3.88 $\pm$ 0.09	0.0164
96	4.02 $\pm$ 0.08	3.92 $\pm$ 0.09	0.0169
128	4.03 $\pm$ 0.08	3.93 $\pm$ 0.08	0.0176
160	4.04 $\pm$ 0.09	3.94 $\pm$ 0.09	0.0184

Appendix

Table 19: Sensitivity to state dimension

d_{\text{ssm}}

on LJSpeech. MOS values are averaged over three seeds with 95% confidence intervals; lower RTF is better. The configuration used in the main paper is in bold.

Kernel size	MOS (long) $\uparrow$	Pitch RMSE (Hz) $\downarrow$	RTF $\downarrow$
3	4.08 $\pm$ 0.08	2.06 $\pm$ 0.06	0.0172
5	4.16 $\pm$ 0.07	1.92 $\pm$ 0.05	0.0177
7	4.17 $\pm$ 0.07	1.90 $\pm$ 0.05	0.0184

Appendix

Table 20: Sensitivity to convolution kernel size

k_{\text{conv}}

in the Mamba block. Pitch RMSE is computed on long-form LJSpeech utterances; lower is better.

Temperature	MOS (OOD) $\uparrow$	CMOS-N $\uparrow$	WER $\downarrow$
0.7	3.83 $\pm$ 0.09	-0.06	7.12%
1.0	3.88 $\pm$ 0.09	0.00	6.89%
1.3	3.86 $\pm$ 0.09	-0.02	6.97%

Appendix

Table 21: Sensitivity to gating temperature

\tau_{\text{gate}}

on the Gutenberg OOD set. CMOS-N is measured relative to the default configuration with

\tau_{\text{gate}}{=}1.0

System	Training data	Languages	Main architecture / setting	Representative reported metrics	Scale
NaturalSpeech 3	$\sim$ 200k h multi-speaker, multi-style speech (public + private)	Multilingual	Factorized diffusion TTS with FACodec; separate prosody/content/acoustic/timbre modules; zero-shot and long-form generation.	On LibriSpeech test-clean: Sim-O 0.67, Sim-R 0.76, WER 1.81, SMOS 4.01.	Industry (Multi-hundred thousand hours, proprietary)
CosyVoice 3	3k h + 170k h multilingual corpora	Multilingual (zh/en + CV3-Eval)	MinMo-based acoustic tokenizer; TTS LM + conditional flow matching (CFM) decoder; multi-task TTS/S2S.	On SEED-TTS EVAL: CER 1.27 (zh), WER 2.46 (en), WER 6.96 (hard) with strong Sim-O.	Industry (Large multilingual corpus, proprietary)
HiggsAudio-V2	$\sim$ 10M h AudioVerse (speech, music, SFX)	Multilingual / multi-domain	Audio language model (5.8B params) with 12-codebook RVQ tokenizer and DualFFN adapter; unified speech/music/SFX.	SeedTTS-Eval: WER 2.44%, 67.7% speaker similarity; dialogue WER 18.88%.	Industry (Large-scale, multi-domain, proprietary)
MVC (ours)	24 h LJSpeech + 245 h LibriTTS (public)	English	Fully SSM-based conditioning stack (bi-Mamba text, Temporal Bi-Mamba, Expressive Mamba + AdaLN) under a fixed diffusion/vocoder pipeline.	Improves MOS/CMOS and WER over StyleTTS2, VITS, and hybrid Mamba-attention baselines under identical preprocessing and vocoders.	Academic (24h + 245h, public datasets)

Appendix

Table 22: Qualitative positioning of MVC relative to recent industrial-scale systems. NaturalSpeech 3, CosyVoice 3, and HiggsAudio-V2 operate at much larger data and model scales, with different objectives and proprietary evaluation pipelines. MVC is an open-data encoder study under a unified mel–diffusion–vocoder setup, and thus numeric comparisons would be misleading.

Expressive Speech Synthesis

See all 23 papers

MixiSpeech: Progressive State Distillation for One-Step Vietnamese TTS
Oct 18, 2026Tran Quoc Bao Le, Binh Mai, Hung Dinh +2TTS Synthesis Multi-Speaker TTS
Posts and Telecommunications Institute of Technology, Hanoi, Vietnam
Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech
Oct 14, 2026Jeseung Lim, Jiseong Woo, Heeseung KimLong-Form TTS Synthesis TTS Synthesis
Department of Physics, University of Seoul, Seoul, Republic of Korea · Department of AI, University of Seoul, Seoul, Republic of Korea
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Aug 16, 2026Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko +4Voice Cloning Audio-Video Generation
Kandinsky Lab, Moscow, Russia

Oct 18, 2026J/K move · Enter open · S save

MixiSpeech: Progressive State Distillation for One-Step Vietnamese TTS

Tran Quoc Bao Le, Binh Mai, Hung Dinh, Lam Nguyen+1

Posts and Telecommunications Institute of Technology, Hanoi, Vietnam

TTS Synthesis Multi-Speaker TTS

Open paper AskPDF arXiv

Conditional-flow text-to-speech (TTS) models can achieve high synthesis quality with relatively few sampling steps, but collapsing the remaining trajectory to a single update can still cause substantial quality degradation. We propose MixiSpeech, a progressive state distillation framework that converts a multi-step conditional-flow acoustic model into a one-step generator while preserving its architecture and speaker conditioning. Rather than training the one-step model directly, MixiSpeech progressively reduces the global sampling budget through a 10-4-2-1 curriculum, with each student trained to match local teacher-integration endpoints using a masked state-matching objective. We instantiate the framework on a 710-speaker Vietnamese Matcha-TTS model trained from scratch on PhoAudiobook. On 100 shared test text-speaker pairs, direct one-step inference substantially degrades synthesis quality, obtaining 1.6846 UTMOS22-Strong and 6.21% WER compared with 2.2923 and 4.65% for the 10-step teacher. In contrast, the distilled one-step model achieves 2.2678 UTMOS22-Strong, 4.63% WER, and 2.83% CER, recovering most of the quality lost at the one-step operating point while requiring only a single acoustic vector-field evaluation. End-to-end inference time decreases from 0.0662 s for the 10-step teacher to 0.0312 s for MixiSpeech, corresponding to a 2.12 measured speedup. These results show that progressive state distillation can make one-step conditional-flow TTS practical without relying on a large multi-step sampling budget. Project page: https://mixispeech.solanai.us/

Abstract

Expressive Speech Synthesis

Explore similar work

MixiSpeech: Progressive State Distillation for One-Step Vietnamese TTS

Speaking Longer Than Learned: Training-Free Long-Form Generation for Full-Stream Text-to-Speech

Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer