cs.SDNov 11, 2026

RSE-TTS: Compact Text-to-Speech through Shared Text-Aware Reference Encoding and Generation

Authors: Ming Ma, Mingjie Chen, Yuanzhong Zheng, Hengyao Bao, Hailin Yu, Shijing Yu, Hao Qiu, Gongfan Cao, 

Organizations: VoxNexus

Abstract

Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LeanTTS: Efficient Full-Stack Bilingual TTS for Zero-Shot Cloning and Voice Customization

    Oct 25, 2026Yuheng Lu, Qiulin Li, Xi Wang +5TTS SynthesisZero-Shot TTS

  2. PilotTTS: A Disciplined Modular Recipe for Competitive Speech Synthesis

    May 26, 2026Bowen Li, Shaotong Guo, Zhen Wang +11TTS SynthesisZero-Shot TTS

  3. TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech

    Jun 8, 2026Yejin Lee, Junwon Moon, Hyoeun Kim +3Efficient Transformer InferenceAudio Token Compression