Speech Editing

Follow this topic Ask about this topic

Speech editing changes words, speaker traits, emotion, or delivery in recorded audio while aiming to leave unedited regions natural and consistent. Common approaches use autoregressive text-to-speech models, diffusion or flow-based generation, and codec-token infilling, often guided by transcripts or natural-language instructions. Benchmarks such as RealEdit and SpeechEditBench assess edit accuracy alongside preservation of speaker identity, timing, and non-target attributes. Models increasingly combine synthesis and editing, but compositional instructions and the tradeoff between successful changes and faithful preservation remain difficult.

Latest papers

23 papers5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Fish Audio Drama 3 Technical Report
Oct 13, 2026Dawei Han, Shidong Li, Shijia Liao +13TTS Synthesis Controllable Speech Generation
AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
Oct 11, 2026Yunjia Zhang, Junan Zhang, Jing Yang +3Audio Editing Speech Editing
The Chinese University of Hong Kong, Shenzhen · Institute of Media Technology and Experience Design, Huawei · City University of Macau +1
AegisVox: Factorized Privacy and Clinical Preservation for Long-Form Speech
Oct 10, 2026Praphul Singh, Akshat Agarwal, Ratul Ghosh +5Speaker Anonymization Speech Editing
Oracle Health AI
Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS
Oct 8, 2026Junchuan Zhao, Chenglin Xu, Wei Zeng +3TTS Synthesis Controllable Speech Generation
National University of Singapore · LIGHTSPEED · Nanyang Technological University +1
SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows
Sep 28, 2026Tianxin Xie, Pengfei Zhang, Kai Jiang +2TTS Synthesis Speech Editing
The Hong Kong University of Science and Technology (Guangzhou)
From Script to Drama: An Agentic Framework for Controllable Multi-Speaker Dialogue TTS
Sep 27, 2026Kangxiang Xia, Xinfa Zhu, HangRui Hu +8TTS Synthesis Speech Editing
Independent Researcher
EditVoice: Variable-Length Non-Autoregressive Zero-Shot TTS and Speech Editing with Edit Flows
Sep 24, 2026Hongyao Deng, Wenhao Guan, Xuetao Lin +4TTS Synthesis Speech Editing
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
Sep 8, 2026Ziyang Ma, Zhikang Niu, Wenming Tu +30Speech Foundation Models Speech Editing
FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation
Aug 25, 2026Feiyu Shen, Fenglong Xie, Junjie Li +13Audio-Language Models Audio Understanding
Xiaohongshu
FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
Aug 18, 2026Feiyu Shen, Kun Xie, Yichen Wu +8TTS Synthesis Controllable Speech Generation
Xiaohongshu
VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation
Aug 12, 2026Jiarui Hai, Karan Thakkar, Ke Chen +5TTS Synthesis Controllable Speech Generation
Johns Hopkins University · Adobe Research
Luna-TTS Family Technical Report
Aug 12, 2026Feng Yin, Shuai Shi, Junjie Zheng +19Non-Autoregressive Text Generation TTS Synthesis
VUI Labs Research
CookVoice: Unified Framework for Style Controllable Multi-Modal Human Voice Generation
Aug 12, 2026Haowei Lou, Hye-Young Paik, Dai Jia +2Singing Voice Synthesis TTS Synthesis
UNSW Sydney · Dolby Laboratories
Multi Codec Discrete Diffusion Model for Text Guided Speech Inpainting and Editing
Aug 5, 2026Iftach Shoham, Tali Dror, Oren Gal +3Speech Editing Audio Diffusion Models
Ben-Gurion University of the Negev · University of Haifa
dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model
Aug 2, 2026Hankun Wang, Bohan Li, Shi Lian +7Speech Foundation Models Speech Editing
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · Xiaohongshu Inc.
UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling
Jun 30, 2026Chuanbo Zhu, Wuyou Zhou, Rongxiu Zhong +4Audio Editing Speech Editing
The Hong Kong University of Science and Technology · China Mobile · Beijing Institute of Technology
RIVET: Robust Idempotent Voice Attribute Editing
Jun 17, 2026Dareen Alharthi, Bhuvan Koduru, Rita Singh +1Audio Editing Speech Editing
Carnegie Mellon University, Pittsburgh, PA, USA
SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing
Jun 1, 2026Hanlin Zhang, Daxin Tan, Dehua Tao +3Audio-Language Model Evaluation Audio Editing
Department of Computer Science, City University of Hong Kong · AI Lab, Leibniz Research Center, Huawei
UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion
May 29, 2026Zhaoqing Li, Haoning Xu, Jingran Su +9Text-to-Audio Generation TTS Synthesis
The Chinese University of Hong Kong · The Hong Kong Polytechnic University · City University of Hong Kong +3
CosyEdit2: Speech-Editing-Oriented Reinforcement Learning Unlocks Better Zero-Shot TTS
May 25, 2026Junyang Chen, Yuhang Jia, Hui Wang +3Speech Editing Zero-Shot TTS
College of Computer Science, Nankai University · College of Artificial Intelligence, Nankai University
AST: Adaptive, Seamless, and Training-Free Precise Speech Editing
Apr 17, 2026Sihan Lv, Yechen Jin, Zhen Li +5TTS Synthesis Audio Editing
School of Software Technology, Zhejiang University, Hangzhou, China · Institute of Remote Sensing Satellite, China Academy of Space Technology, Beijing, China · Innovation and Management Center of the School of Software (Ningbo), Zhejiang University, Ningbo, China +1
LLaDA-TTS: Unifying Speech Synthesis and Zero-Shot Editing via Masked Diffusion Modeling
Mar 27, 2026Xiaoyu Fan, Huizhi Xie, Wei Zou +1TTS Synthesis Zero-Shot TTS
BRVoice Team, Bairong, Inc., China
CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models
Jan 8, 2026Junyang Chen, Yuhang Jia, Hui Wang +2Audio Editing Speech Editing
College of Computer Science, Nankai University, Tianjin, China

Oct 13, 2026J/K move · Enter open · S save

Fish Audio Drama 3 Technical Report

Dawei Han, Shidong Li, Shijia Liao, Jiahua Liu+12

TTS Synthesis Controllable Speech Generation Speech Foundation Models Streaming TTS Synthesis Voice Cloning Speech Editing

Open paper AskPDF arXiv

We introduce Fish Audio Drama 3, an open-source speech generation model that turns text into directed performances, built on a 35B-parameter mixture-of-experts backbone that activates 3B parameters per token. It clones voices from multiple references, follows natural-language direction, revises a take when the next turn gives a note, voices multi-character scenes, edits existing speech by infilling, and synthesizes long-form and streaming audio. Multi-turn revision, scene dubbing, editing by infilling, and streaming text input are new in Drama 3, all in one model. Drama 3 pairs the hybrid linear-attention backbone with a new codebook multi-token-prediction audio decoder that predicts the acoustic codebooks of each frame in dedicated layers that attend over the entire sequence. The model is trained with a three-stage pre-training recipe, instruction fine-tuning, and large-batch reinforcement learning whose only reward is a single learned model of human preference. Its inference engine decodes a single stream at 271 frames per second on one H200, against 177 for Fish Audio S2, although Drama 3 has more than seven times as many parameters. The model weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/drama-3), and inference is available through SGLang-Omni (https://github.com/sgl-project/sglang-omni). We highly encourage readers to visit https://fish.audio to direct their own performances.