Voice Cloning

Follow this topic Ask about this topic

Voice cloning studies how to generate new speech that retains a person’s vocal identity while changing the words, language, or speaking style. Common approaches condition text-to-speech models on reference audio or speaker embeddings, using autoregressive token generation, diffusion or flow matching, and multilingual phonetic representations. Evaluation compares intelligibility, speaker similarity, prosody, and human judgments, with long-form consistency and preservation of accents and expressive traits remaining difficult. The field also faces an open tension between useful personalization and risks such as unauthorized imitation, unreliable clone attribution, and weak safeguards against misuse.

Latest papers

42 papers8 papers in the last four weeks, up 167% on the four weeks before. 0.0% of all new papers.

From Speech Tokens to Stolen Voices: Attacker-Chosen-Text Speaker Impersonation via Representation Inversion
Oct 26, 2026Ye Lu, Shen Wang, Runze Liu +5Voice Cloning Privacy-Preserving Speech Processing
School of Cyberspace Science, Harbin Institute of Technology
HeyGen Voice: Scene-Aware Voice Cloning for Talking Videos
Oct 21, 2026Pavel Andreev, Andrei Gusev, Adithya Narayan +15Voice Cloning TTS Synthesis
HeyGen Research
Closed-Loop Dual-Reference Guidance for Accent Control in Cross-Lingual Voice Cloning
Oct 21, 2026Yuanzhong Zheng, Wei Huang, Ming Ma +2Voice Cloning Multilingual TTS Synthesis
VoxNexus, Singapore
Beyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS
Oct 16, 2026Séverin Baroudi, Yanis Labrak, Petr Motlicek +2TTS Synthesis Voice Cloning
Univ Toulon, Aix Marseille Univ, LIS, CNRS, France · Mila – Quebec AI Institute · Idiap Research Institute, Martigny, Switzerland +3
Known-Voice Cloning and the Limits of Personality Transfer for Character Design
Oct 13, 2026Johanna Magdalena Kuch, Maud van Lier, Daksitha Senel Withanage Don +3Voice Cloning Anthropomorphism in AI
Augsburg University, Human-Centered Artificial Intelligence, Augsburg, Germany · Institute of Intelligent Systems and Robotics, CNRS, Sorbonne Université, Paris, France · German Research Center for Artificial Intelligence (DFKI), Affective Computing Group, Germany
Fish Audio Drama 3 Technical Report
Oct 13, 2026Dawei Han, Shidong Li, Shijia Liao +13TTS Synthesis Controllable Speech Generation
Intonation Perception in Real and Synthetic Speech across Varying Familiarity Levels: A Pilot Study of Equivalence Assessment
Sep 30, 2026Hanrui Zhou, Gaoyuan Zhang, Yixiang Chen +4Speech Prosody Speech Generation Evaluation
Institute of Software, Chinese Academy of Sciences, Beijing, China · Capital Normal University
Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
Sep 29, 2026Jian Chen, You Zhang, Mark VintonTTS Synthesis Zero-Shot TTS
Dolby Laboratories
RAWD-TTS: Ratio-Free Reward Alignment for Discrete-Diffusion Voice Cloning
Sep 29, 2026Maxim Maslov, Kirill Borodin, Vasilii Kudryavtsev +2Audio Diffusion Models Zero-Shot TTS
lab260, Yerevan, Armenia · BitmanagerAI, Dubai, UAE · MTUCI, Moscow, Russia
From Reliable Text to Real Voices: Trust-Aware Progressive Adaptation for Low-Resource TTS
Sep 22, 2026Jiayi Lu, Yizhong Geng, Jinghan Yang +4Synthetic-to-Real Domain Adaptation Low-Resource TTS Synthesis
Beijing Logic Intelligence Technology · University of Washington · Beijing University of Posts and Telecommunications +2
Taming Long-form Text-to-Speech
Sep 15, 2026Rongxiang Wang, Berkin Durmus, Aysegul Orhon +2TTS Synthesis TTS Evaluation
Argmax, Inc. · University of Virginia · Bilkent University
Cross-Lingual F5-TTS 2: A Simplified Framework for Language-Agnostic Voice Cloning
Sep 14, 2026Qingyu Liu, Rixi Xu, Yushen Chen +11TTS Synthesis Zero-Shot TTS
Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA · MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China +3
HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence
Aug 27, 2026Fei Ma, Zebang Cheng, Minghui Li +11Multimodal Model Evaluation Multimodal Emotion Recognition
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, China. · Shenzhen University, Shenzhen, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, China. +6
FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
Aug 18, 2026Feiyu Shen, Kun Xie, Yichen Wu +8TTS Synthesis Controllable Speech Generation
Xiaohongshu
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Aug 16, 2026Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko +4Voice Cloning Audio-Video Generation
Kandinsky Lab, Moscow, Russia
A survey of AI-generated voices and their detection
Aug 15, 2026Chengzhe Sun, Tianle Yang, Siwei LyuAudio Deepfake Detection Voice Cloning
Department of Computer Science and Engineering, University at Buffalo, Buffalo, New York, USA · Department of Linguistics, University at Buffalo, Buffalo, New York, USA
VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
Aug 12, 2026Jiarui Hai, Karan Thakkar, Ke Chen +5TTS Synthesis Controllable Speech Generation
Johns Hopkins University · Adobe Research
CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation
Aug 12, 2026Haowei Lou, Hye-Young Paik, Dai Jia +2Singing Voice Synthesis TTS Synthesis
UNSW Sydney · Dolby Laboratories
Extracting Voice Styles from Frozen TTS Models via Gradient-Based Inverse Optimization
Jul 28, 2026Gyeongmin KimTTS Synthesis Voice Cloning
Department of Computer Science, Hanyang University, Seoul, Korea
Synthetic Speech, Real Signal: Paralinguistic Preservation and Cross-Lingual Augmentation via Voice Cloning
Jul 24, 2026Roseline Polle, Owen Parsons, George Fairs +5Synthetic Data Augmentation Speech Processing
thymia, UK · The University of Edinburgh, UK · University of Southampton, UK
A Geometry-Limited Identification Floor and Its Consequences for Voice-Clone Attribution in Professional Voice Actors
Jul 17, 2026Shuhei KatoAudio Deepfake Detection Speaker Verification
The author is an independent researcher in Tokyo, Japan.
ZONOS2 Technical Report
Jun 23, 2026Gabriel Clark, Sofian Mejjoute, Mohamed Osman +2TTS Synthesis Streaming TTS Synthesis
Zyphra · San Francisco, CA
Dynamic Prosody Prediction in LLM-based TTS for Improving Speaker Similarity
Jun 13, 2026Zhenwei Mou, Liping Chen, Yajun Hu +3TTS Synthesis Speech Prosody
University of Science and Technology of China, Anhui, China · iFLYTEK, Anhui, China
Vocal Identity Under Siege by AI Voice Cloning Technologies
Jun 11, 2026Jyh-An Lee, Xuan SunAudio Generation Voice Cloning
Professor and Executive Director, Centre for Legal Innovation and Digital Society (CLINDS), The Chinese University of Hong Kong Faculty of Law. · PhD Candidate, The Chinese University of Hong Kong Faculty of Law.
Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation
Jun 5, 2026Ziyu Zhang, Chunyu Qiang, Xiaopeng Wang +10Controllable Music Generation Music Generation
Kuaishou Technology, China · Beijing Institute of Technology, China · Institute of Automation, Chinese Academy of Sciences, China
VoxCPM2 Technical Report
Jun 5, 2026Yixuan Zhou, Guoyang Zeng, Xin Liu +15Speech Foundation Models Multilingual TTS Synthesis
VoxCPM Team
Generative AI and Copyright Infringement: A Legal-Technical Analysis of AI Music Generation Systems Under 17 U.S.C. Title 17
May 20, 2026Zuhaib Hussain ButtText-to-Music Generation Music Generation
Gujranwala, Punjab, Pakistan
Voice "Cloning" is Style Transfer
May 15, 2026Kaitlyn Zhou, Federico Bianchi, Martijn Bartelds +3Controllable Speech Generation Trust in AI
Cornell University · TogetherAI · Stanford University
X-Voice: Enabling Everyone to Speak 30 Languages via Zero-Shot Cross-Lingual Voice Cloning
May 7, 2026Rixi Xu, Qingyu Liu, Haitao Li +10TTS Synthesis Zero-Shot TTS
1MoE Key Lab of Artificial Intelligence, X-LANCE Lab, Shanghai Jiao Tong University · Center for Language and Speech Processing, Johns Hopkins University · 2Shanghai Innovation Institute +4
MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model
May 5, 2026Jingyao GongSpeech Foundation Models Unified Multimodal Models
Independent Researcher

Oct 26, 2026J/K move · Enter open · S save

From Speech Tokens to Stolen Voices: Attacker-Chosen-Text Speaker Impersonation via Representation Inversion

Ye Lu, Shen Wang, Runze Liu, Zhitao Ou+4

School of Cyberspace Science, Harbin Institute of Technology

Voice Cloning Privacy-Preserving Speech Processing

Open paper AskPDF arXiv

Speech interaction systems have gradually evolved from traditional cascaded systems that use text as intermediate information to end-to-end speech language models that directly process speech representations. In some applications, the speech frontend runs on the user's device and sends speech representations to a remote server for inference, while the original audio remains local. However, these representations may still contain speaker identity information. If an attacker obtains them, it may use this information to imitate the user's voice, threatening their voice privacy. To investigate this risk, we propose \stia, a Speech Token Inversion Attack that generates speech with attacker-specified text in a victim's voice using a single short leaked representation, without access to the original audio or its transcript. Specifically, \stia converts discrete, continuous, or mixed speech representations into conditioning sequences and trains a conditional flow-matching speech generation model using pairs of different utterances from the same speaker in public speech datasets. The model combines speaker characteristics contained in the leaked representation with the specified text to generate speech in the victim's voice. Experiments show that, with a Moshi representation leak corresponding to 3 seconds of speech, \stia achieves an average CAM++ speaker cosine similarity of 0.78 and a target-text word error rate of 3.80%. These results show that speech frontend representations reveal reusable speaker identity information and highlight the need to consider privacy risks when designing speech frontend representation methods.