Zero-Shot TTS

TTS: Text-to-Speech

Zero-shot TTS studies how to generate speech in a speaker’s voice from a short audio reference, often across languages or without a transcript. Common approaches include autoregressive models over discrete speech tokens, flow-matching and diffusion models, and masked non-autoregressive generation; some also model waveforms directly. Current work extends voice cloning with streaming, speech editing, and controls for pronunciation, accent, emotion, and prosody. A central challenge is balancing intelligibility, naturalness, and faithful speaker identity, while evaluation metrics do not always agree with human judgments or capture these goals separately.

Latest papers

61 papers11 papers in the last four weeks, up 38% on the four weeks before. 0.1% of all new papers.

ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis
Oct 26, 2026Hanchen Pei, Shujie Liu, Yayue Deng +4TTS Synthesis Zero-Shot TTS
Electronic Information School, Wuhan University, Wuhan, China · Microsoft Corporation, Redmond, US · The Chinese University of Hong Kong, Hong Kong, China
LeanTTS: Efficient Full-Stack Bilingual TTS for Zero-Shot Cloning and Voice Customization
Oct 25, 2026Yuheng Lu, Qiulin Li, Xi Wang +5TTS Synthesis Zero-Shot TTS
Tianjin University, Tianjin, China · Didichuxing Co. Ltd., Beijing, China · Tsinghua University, Beijing, China
Long-Prompt Zero-Shot TTS: Benchmarking Prosody Reproduction Beyond Voice Cloning
Oct 14, 2026Dekun Chen, Renjie Xu, Yingda Shen +2Zero-Shot TTS Speech Prosody
The Chinese University of Hong Kong, Shenzhen · Amphion Technology Co., Ltd.
Revise-TTS: A Non-Masking Refinement Framework for Masked Diffusion Models in Text-to-Speech
Oct 14, 2026Wanli Sun, Dan Su, Anton RagniTTS Synthesis Zero-Shot TTS
School of Computer Science, University of Sheffield, Sheffield, UK · HojoAI, Beijing, China
Training a Compact, Strong, and Realistic Text-to-Speech Model
Oct 10, 2026Runyuan Cai, Yu Lin, Yiming Wang +1TTS Synthesis Zero-Shot TTS
AutoArk
E3-System: Expressive EmoData, Explicitly Controllable, Easy-to-Use TTS
Oct 9, 2026Yiqiao Huang, Xiaolong Lin, Zhenpeng Zhan +1TTS Synthesis Expressive Speech Synthesis
Harvard University, Cambridge, MA, USA · Baidu Inc., Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China
Beyond Token Revision: Investigating Mask-and-Replace Diffusion for Zero-Shot Text-to-Speech
Oct 7, 2026Hounsu Kim, Joonyong Park, Yuki Saito +2Zero-Shot TTS Audio Diffusion Models
KAIST, Republic of Korea · The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan
BanglaBox: A Phonetically-Balanced Corpus and Data-Efficient Foundation-Model Adaptation for Bangla Text-to-Speech with Zero-Shot Voice Cloning
Oct 6, 2026Emtiaz Uddin Ahmed, Araf Mahmud, Sajib Hossain +3Text-to-Speech Low-Resource TTS Synthesis
A Comprehensive Objective Evaluation of Modern Text-to-Speech for Turkish Using Speech Quality Assessment Models
Oct 5, 2026Yunus Emre Ozkose, Alperen Kahraman, Ali HaznedarogluTTS Synthesis TTS Evaluation
Sestek, Ankara, Türkiye
Prompt-Consistency Inference for Zero-Shot Flow-Matching Text-to-Speech Models
Oct 3, 2026Vasily Zadorozhnyy, Can Goksen, Kazuhito Koishida +1TTS Synthesis Zero-Shot TTS
Applied Sciences Group, Microsoft Corporation, Redmond, USA
Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
Sep 29, 2026Jian Chen, You Zhang, Mark VintonTTS Synthesis Zero-Shot TTS
Dolby Laboratories
RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning
Sep 29, 2026Maxim Maslov, Kirill Borodin, Vasilii Kudryavtsev +2Audio Diffusion Models Zero-Shot TTS
lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia
DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS
Sep 26, 2026Ambuj Mehrish, Abhinaba Roy, Alex Ivanov +2TTS Synthesis Zero-Shot TTS
Ca’ Foscari University of Venice · Singapore University of Technology and Design · Kandinsky Lab +1
EditVoice: Variable-Length Non-Autoregressive Zero-Shot TTS and Speech Editing with Edit Flows
Sep 24, 2026Hongyao Deng, Wenhao Guan, Xuetao Lin +4TTS Synthesis Speech Editing
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China
Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning
Sep 24, 2026Yoomee Cho, Jisun LeeTTS Synthesis Multilingual TTS Synthesis
ESTsoft, Seoul, Republic of Korea
Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech
Sep 23, 2026Hyoeun Kim, Yujun Lee, Kyuhong ShimZero-Shot TTS Machine Unlearning
Sungkyunkwan University, Republic of Korea
TTS-Guard: Black-Box Ownership Verification of Text-to-Speech Models via Adaptive Adversarial Speaker-Pair Fingerprints
Sep 20, 2026Xubin Yue, Zhenhua Xu, Zhebo Wang +5Zero-Shot TTS
Zhejiang University · Binjiang Institute of Zhejiang University · Hangzhou International Innovation Institute, Beihang University +1
Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning
Sep 14, 2026Qingyu Liu, Rixi Xu, Yushen Chen +11TTS Synthesis Zero-Shot TTS
Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China +3
StepAudio 3 Gen Technical Report
Sep 11, 2026Bin Lin, Bo Zhao, Boyang Wang +68TTS Synthesis Zero-Shot TTS
Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language
Sep 10, 2026Lianru Gao, Yujie Guo, Yong QinTTS Synthesis Zero-Shot TTS
TMCC, College of Computer Science, Nankai University, Tianjin, China
FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
Aug 25, 2026Feiyu Shen, Fenglong Xie, Junjie Li +13Audio-Language Models Audio Understanding
Xiaohongshu
FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
Aug 18, 2026Feiyu Shen, Kun Xie, Yichen Wu +8TTS Synthesis Controllable Speech Generation
Xiaohongshu
Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization
Aug 12, 2026Peijie Chen, Zhuanling Zha, Zhipeng Nie +8Zero-Shot Voice Conversion Speech Processing
Didichuxing Co. Ltd
Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder
Aug 12, 2026Huaxuan Wang, Huimin Wang, Ruiyu Zhang +2TTS Synthesis Multilingual TTS Synthesis
NetEase Youdao, Beijing, China
Luna-TTS Family Technical Report
Aug 12, 2026Feng Yin, Shuai Shi, Junjie Zheng +19Non-Autoregressive Text Generation TTS Synthesis
VUI Labs Research
TTS-Q: Speaker-Aware Post-Training Quantization of Text-to-Speech Diffusion Transformers
Aug 11, 2026Xiaoqin Feng, Yuqi Li, Weilun Feng +9Diffusion Model Quantization Post-Training Quantization
Wyze Inc., USA · The City College of New York, City University of New York, USA · Institute of Computing Technology, Chinese Academy of Sciences, China +2
CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents
Aug 9, 2026Yuqian Zhang, Yao Shi, Kexin Huang +6TTS Synthesis Streaming TTS Synthesis
Work done during internship at OPPO. · OPPO AI Center · Fudan University +1
CtrlSpeech: Coarse-to-Fine Control for Expressive Speech Synthesis
Aug 8, 2026Zhisheng Zheng, Xiaohang Sun, Zhu Liu +5TTS Synthesis Zero-Shot TTS
The University of Texas at Austin · Amazon
SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
Aug 7, 2026Hanke Xie, Haopeng Lin, Jiale Qian +13Zero-Shot TTS Autoregressive Generation
Soul AI Lab, China
SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks
Aug 3, 2026Yu Zhang, Ruiqi Li, Changhao Pan +3Text-to-Audio Generation TTS Synthesis
1ByteDance · 2Zhejiang University

Oct 26, 2026J/K move · Enter open · S save

ImageTTS: Repurposing Pretrained Image Diffusion Transformers for Speech Synthesis

Hanchen Pei, Shujie Liu, Yayue Deng, Fangyun Wei+3

Electronic Information School, Wuhan University, Wuhan, China · Microsoft Corporation, Redmond, US · The Chinese University of Hong Kong, Hong Kong, China

TTS Synthesis Zero-Shot TTS Audio Diffusion Models Diffusion Transformer

Open paper AskPDF arXiv

Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2% to 2.7% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.