Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.
Figures & tables
Figure 1: RSE-TTS uses the same Global transformer to encode reference text and speech and to generate a new utterance. Eight learned queries produce Z , which is combined with a projected speaker embedding to form Zˉ . During generation, Global consumes frame embeddings E(ci) , while Local produces discrete codec frames ci . Repeated Local and decoder blocks illustrate successive generation steps, not separate parameters.
Figure 2: Longer reference speech improves similarity even though target generation always receives eight compressed vectors. Points show measured means. The auxiliary speaker vector is fixed across durations.
Reference representation
Cosine ↑
192-dim. speaker embedding
0.438
Eight reference vectors
0.439
Both combined
0.469
Table 2: Predicting the speaker embedding of another real utterance from the same person. Results are averaged over 63 unseen speakers.
Figure 3: Model size versus intelligibility and speaker similarity. WER and SIM are macro-averaged with equal weight across Seed-TTS, CV3-eval, and MiniMax. Parameter counts include the generator and active waveform decoder. Lower WER and higher SIM are better; the horizontal axis uses a logarithmic scale. “Ref.” denotes reference conditioning and “cont.” denotes acoustic continuation.
Patched positions
Layer
Restore
Disrupt
No patch
—
0.497
0.538
Reference audio
3
0.545
0.504
Reference audio
6
0.538
0.512
Reference audio
9
0.499
0.538
Eight queries
3
0.503
0.539
Eight queries
6
0.512
0.541
Table 3: Speaker similarity after replacing reference-audio or query representations at layers 3, 6, and 9. Restoration inserts correct-text representations into the reversed-text run; disruption makes the opposite replacement. Each condition generates the same 120 targets from 30 speakers.
Benchmark
Continuation
RSE
Seed-TTS
1.45 / 0.591
1.33 / 0.552
CV3-eval
3.48 / 0.526
2.63 / 0.499
MiniMax
1.77 / 0.684
0.64 / 0.615
Table 4: Matched continuation-only and RSE systems. Cells report WER (%) ↓ / SIM ↑ as in Table 1.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Mean per-position gradient norm for reference text, reference speech, and the eight queries, using the fixed-history soft-backward SIM estimator at temperature 0.4. The vertical axis is logarithmic. Layer 0 denotes input embeddings. Final-layer text and audio gradients are exactly zero because only query states are read out; these structural zeros are omitted from the log plot.
While recent advances in text-to-speech (TTS) synthesis have substantially improved speech quality and modeling flexibility, the development of lightweight and computationally efficient systems remains comparatively underexplored. We present LeanTTS, a bilingual complete synthesis stack that supports two complementary regimes: reference-prompt zero-shot cloning for unseen speakers and token-conditioned customization for recurring voices. LeanTTS combines a compact Mandarin polyphone front end, a structurally compressed flow-matching acoustic model, and a lightweight Vocos decoder with four-step MeanFlow generation. The resulting system contains 18.9M learned parameters, 86.12% fewer than the 136.2M-parameter ZipVoice-Vocos pipeline. On Seed-TTS-Eval, its zero-shot mode obtains 2.02 CER/62.3 SIM on Mandarin and 2.32 WER/48.3 SIM on English, exposing the quality-size trade-off under aggressive compression. With token-conditioned customized voices, the same system reaches 1.57 CER/77.2 SIM on Mandarin and 1.93 WER/76.6 SIM on English, with UTMOS scores of 3.03 and 4.33, respectively. It further achieves mean/P90 CPU RTFs of 0.049/0.061, establishing a sub-20M operating point for bilingual personalization and real-time CPU deployment. Audio demos are available at https://github.com/notlu/leantts.
Yuheng Lu, Qiulin Li, Xi Wang +5
Tianjin University, Tianjin, China · Didichuxing Co. Ltd., Beijing, China · Tsinghua University, Beijing, China
Building state-of-the-art text-to-speech (TTS) systems typically demands millions of hours of proprietary data and complex multi-stage architectures, creating substantial barriers for resource-constrained research teams. In this report, we present PilotTTS, a lightweight autoregressive TTS system that achieves competitive performance through minimalist architecture and rigorous data engineering. PilotTTS is trained on only 200K hours of data processed entirely with open-source tools. Specifically, our contributions are: (1) a reproducible multi-stage data processing pipeline covering quality assessment, label annotation, and filtering, and (2) a compact model architecture that employs Q-Former-based conditioning to decouple speaker identity from speaking style via cross-sample paired training. Within a unified framework, PilotTTS supports zero-shot voice cloning, emotion synthesis (11 categories), paralinguistic synthesis (4 categories), and Chinese dialect synthesis (14 dialects). On the Seed-TTS Eval benchmark, PilotTTS achieves the lowest WER of 1.50% on test-en, a CER of 0.87% on test-zh, and the highest speaker similarity on both test sets (0.862 and 0.815), outperforming systems trained on significantly larger datasets. We release the complete data pipeline recipe, pretrained weights, and code at https://github.com/AMAPVOICE/PilotTTS.
Bowen Li, Shaotong Guo, Zhen Wang +11
Amap, Alibaba Group · The Chinese University of Hong Kong, Shenzhen
Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones. However, this token-level formulation creates a structural efficiency bottleneck: speech-token sequences are much longer than text sequences, requiring the AR backbone to perform causal computation at every token position and maintain a KV cache that grows with the sequence length. We introduce TLDR, a patch-based autoregressive framework that accelerates codec-based AR-TTS by shifting the causal modeling from token-level speech sequences to patch-level sequences. TLDR groups consecutive codec tokens into compact latent patches using a lightweight compressor, models the resulting shorter patch sequence with a frozen pretrained AR-TTS backbone adapted by LoRA, and reconstructs fine-grained speech tokens within each patch using a speaker-conditioned extractor. With a patch size of 4, TLDR achieves a 1.8x inference speedup over the baseline AR-TTS model and reduces global KV-cache memory by up to 75%. Experimental results indicate that patch-level global causal modeling can be a practical way to reduce the inference cost of pretrained codec-based AR-TTS systems without replacing the existing modules.