VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing and Its Learning Dynamics

Zhikang Niu; Wenming Tu; Yalin Li; Yanru Huo; Junxi Liu; Tianrui Wang; Ruiqi Yan; Junxuan Liang

cs.SDNov 5, 2026

VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing and Its Learning Dynamics

Authors: Zhikang Niu, Wenming Tu, Yalin Li, Yanru Huo, Junxi Liu, Tianrui Wang, Ruiqi Yan, Junxuan Liang,

Organizations: X-LANCE Lab, Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · Beijing Institute for General Artificial Intelligence, China · Tianjin University, China

Abstract

Recent advances in speech generation have spurred the development of speech editing techniques. However, achieving high-quality speech editing remains challenging due to two key limitations. First, the major hurdle is the lack of large-scale, high-quality speech editing datasets. Constructing such datasets is costly, while open-source, scalable data pipelines remain limited. Second, the learning dynamics of speech editing models remain poorly understood. In particular, how models acquire editing capabilities and the relationship between generation and editing remain unclear. To address these challenges, we introduce VoxCanvas-41M, a large-scale bilingual dataset comprising approximately 41 million speech editing pairs. VoxCanvas-41M covers two major categories: content editing, including single-span, multi-span, and location-reference editing, and paralinguistic editing, covering the insertion and deletion of 39 classes of nonverbal events such as laughter, sighs, and coughs. Building on VoxCanvas-41M, we train VoxCanvas-Edit, an instruction-based speech editing model, demonstrating the effectiveness of our dataset in improving editing performance. Furthermore, we systematically investigate the learning dynamics of speech editing, examining how models learn different editing operations and how data scale, task interactions, and training strategies affect editing performance. The dataset, pipeline, and demos are publicly available.

Figures & tables

Figure 1: Overview of the speech editing data construction pipeline. ASR: automatic speech recognition; LLM: large language model; TTS: text-to-speech.

Task	English instruction	Chinese instruction	#Pairs
Single-span editing
Insertion	Add ‘all’ after ‘you’.	在‘你们’后面加上‘都’。	10M
Deletion	Delete ‘very’ before ‘good’.	删除‘好’前面的‘很’。	10M
Substitution	Replace ‘red’ with ‘blue’.	把‘红色’改成‘蓝色’。	10M
Multi-span editing (2–6 edits per instruction)
Two edits	Add ‘all’ after ‘you’; delete ‘very’ before ‘good’.		10M

Table 1: Editing tasks in VoxCanvas-41M, with example instructions in English and Chinese and the total number of editing pairs across both languages.

Figure 2: Dataset statistics of VoxCanvas-41M. From left to right: source utterance duration distributions, multi-span complexity, edit-span length distributions, and nonverbal event category distributions.

Benchmark	Model	Metrics	Results
SpeechEditBench Content \| Paralinguistic			Content-ZH	Content-EN	Avg.	Paralinguistic
	AuK	ACC $\uparrow$	88.32	93.78	91.05	53.25
	AuK + GigaEdit-L		–	94.22	–	–
	VoxCanvas-Edit		93.66	96.78	95.22	54.50
Ming-Freeform-Audio-Edit Semantic Editing			Basic-ZH	Full-ZH	Basic-EN	Full-EN
	AuK	WER $\downarrow$	4.84	3.09	4.05	4.11

Table 2: Speech editing performance of AuK fine-tuned on different datasets. Avg. denotes the average content editing accuracy across Chinese and English on SpeechEditBench. GigaEdit-L contains only English training data, so its Chinese results and bilingual averages are omitted. Best results are highlighted in bold.

Figure 3: Effect of training data scale on the convergence of insertion, deletion, and substitution.

Figure 4: Convergence curves of individual editing tasks under single-operation training.

Figure 5: Task interactions under pairwise and joint training. (a)–(c) compare pairwise training with single-operation baselines, while (d) shows joint training of all three operations. Solid and dashed lines represent mixed and single-operation training, respectively.

Figure 6: Effect of data mixture ratios on speech editing convergence. The left and right panels show Exact Match (EM) and Edit Success (ES), respectively. The ratios represent generation-intensive operations to deletion, with the corresponding insertion:substitution:deletion ratios shown in parentheses.

Figure 7: Effect of joint editing and generation training on editing convergence. The left and right panels show Exact Match (EM) and Edit Success (ES), respectively.

Figure 8: Effect of editing supervision on generation performance on Seed-TTS test-en, measured against training steps (left) and cumulative generation-task audio exposure (right).

Figure 9: Effect of TTS initialization on speech editing convergence. Overall Exact Match (EM) and Edit Success (ES) are shown on the left and right, respectively.

Figure 10: Effect of curriculum learning on multi-span editing convergence. Exact Match (EM) and Edit Success (ES) are shown on the left and right, respectively.

Figure 11: Effect of data reversal on speech quality. Speaker similarity (SIM) and UTMOS are shown on the left and right, respectively.

Figure 12: Learning dynamics of content and paralinguistic editing for insertion (left) and deletion (right).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Transcripts	WER (%) $\downarrow$	SIM $\uparrow$	UTMOS $\uparrow$
Seed-TTS zh	Original	1.462	0.756	3.01
Seed-TTS zh	Qwen3-ASR	1.411	0.751	2.91
Seed-TTS en	Original	1.600	0.669	3.76
Seed-TTS en	Qwen3-ASR	1.469	0.662	3.66
LibriSpeech-PC	Original	2.146	0.664	3.93
LibriSpeech-PC	Qwen3-ASR	2.126	0.657	3.88

Appendix

Table 3: Effect of re-transcription on TTS performance. Both models are trained from scratch for 1.1M updates on Emilia, differing only in the training transcripts. Qwen3-ASR denotes Qwen3-ASR-1.7B.

Figure 13: Joint distributions of original span length ( $A$ ) and replacement span length ( $B$ ) for substitution editing in Chinese and English. The color scale indicates the percentage of samples on a logarithmic scale.

Transformer Backbone
Configuration	AuK	Ours
MMDiT blocks	10	4
DiT blocks	20	8
Hidden dimension	1536	1024
Attention heads	24	16
FFN dimension	3072	2048

Appendix

Table 4: Model architecture and training configurations for the learning dynamics experiments. Shared architectural settings are merged. Training hyperparameters are reported for our smaller model only.

Figure 14: Detailed convergence curves of individual editing tasks in English (top) and Chinese (bottom). The three columns correspond to insertion, deletion, and substitution. Each panel reports Exact Match (EM), Edit Success (ES), and ASR error rate (WER for English and CER for Chinese) over 200K training steps.

Figure 15: Detailed convergence curves of speech editing models with and without TTS initialization. Columns correspond to overall performance, insertion, deletion, and substitution. Rows report Exact Match (EM), Edit Success (ES), and Word Error Rate (WER), respectively.

Figure 16: Detailed learning curves under different data mixture ratios. Columns correspond to overall performance, insertion, substitution, and deletion. Rows report Exact Match (EM), Edit Success (ES), and Word Error Rate (WER), respectively. The three generation-intensive-to-deletion ratios are 1:1 (1:1:2), 2:1 (1:1:1), and 10:1 (5:5:1), with the ratios in parentheses denoting insertion, substitution, and deletion.

Figure 17: Detailed convergence curves of multi-span editing across different numbers of edits. The five columns correspond to 2, 3, 4, 5, and 6 edits, respectively. The top and bottom rows show Exact Match (EM) and Edit Success (ES).

Speech Editing

Speech editing modifies spoken content or vocal attributes while preserving the surrounding audio, speaker identity, and acoustic context.

See all 24 papers

SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing
Jun 1, 2026Hanlin Zhang, Daxin Tan, Dehua Tao +3Audio-Language Model Evaluation Audio Editing
Department of Computer Science, City University of Hong Kong · AI Lab, Leibniz Research Center, Huawei
CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models
Jan 8, 2026Junyang Chen, Yuhang Jia, Hui Wang +2Audio Editing Speech Editing
College of Computer Science, Nankai University, Tianjin, China
dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model
Aug 2, 2026Hankun Wang, Bohan Li, Shi Lian +7Speech Foundation Models Speech Editing
Xiaohongshu Inc. · X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University · School of Intelligence Science and Technology, Nanjing University

Jun 1, 2026J/K move · Enter open · S save

SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing

Hanlin Zhang, Daxin Tan, Dehua Tao, Xiao Chen+2

Department of Computer Science, City University of Hong Kong · AI Lab, Leibniz Research Center, Huawei

Audio-Language Model Evaluation Audio Editing Speech Editing Speech Processing Speech Language Models

Open paper AskPDF arXiv

Instruction-guided speech editing requires a model to modify specified speech attributes while preserving non-target characteristics. Despite rapid progress in Speech Large Language Models (Speech LLMs), systematic evaluation of this capability remains challenging, as existing benchmarks are fragmented across isolated editing tasks. To bridge this gap, we introduce SpeechEditBench, a bilingual multi-attribute benchmark for instruction-guided speech editing. SpeechEditBench encompasses seven atomic editing tasks, as well as compositional editing tasks that integrate multiple operations within a single instruction. We propose an anchor-based evaluation protocol that separately assesses the edit success of target attributes and the preservation of non-target linguistic content, leading to three metrics: target success, preservation success, and joint success. Using this benchmark, we evaluate mainstream Speech LLMs and specialized speech editing systems. The results reveal three key findings: (1) no single model performs well across all editing dimensions; (2) closed-source Speech LLMs generally outperform open-source models; (3) compositional editing poses a significant challenge, with even the most advanced models struggling to achieve high joint success. SpeechEditBench provides a rigorous diagnostic framework to identify bottlenecks in Speech LLMs, thereby facilitating the development of next-generation models with more precise instruction-guided editing capabilities. Data and code are available at https://github.com/daxintan-cuhk/SpeechEditBench.

Abstract

Speech Editing

Explore similar work

SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing

CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models

dots.tts.edit: Precisely Controlled Speech Editing with a Continuous Autoregressive Model