RSE-TTS: Compact Text-to-Speech through Shared Text-Aware Reference Encoding and Generation
Organizations: VoxNexus
Abstract
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.
Figures & tables
| Reference representation | Cosine |
|---|---|
| 192-dim. speaker embedding | 0.438 |
| Eight reference vectors | 0.439 |
| Both combined | 0.469 |
| Patched positions | Layer | Restore | Disrupt |
|---|---|---|---|
| No patch | — | 0.497 | 0.538 |
| Reference audio | 3 | 0.545 | 0.504 |
| Reference audio | 6 | 0.538 | 0.512 |
| Reference audio | 9 | 0.499 | 0.538 |
| Eight queries | 3 | 0.503 | 0.539 |
| Eight queries | 6 | 0.512 | 0.541 |
| Benchmark | Continuation | RSE |
|---|---|---|
| Seed-TTS | 1.45 / 0.591 | 1.33 / 0.552 |
| CV3-eval | 3.48 / 0.526 | 2.63 / 0.499 |
| MiniMax | 1.77 / 0.684 | 0.64 / 0.615 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Uses | Unique | SIM |
|---|---|---|---|
| Seed-TTS | 1,088 | 666 | 0.894 |
| CV3-eval | 500 | 500 | 0.873 |
| MiniMax | 100 | 2 | 0.948 |
TTS Synthesis
Text-to-speech synthesis converts written language into intelligible, natural-sounding speech with control over voice, prosody, style, and timing.
See all 291 papers