AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency

Yunjia Zhang; Junan Zhang; Jing Yang; Xueyao Zhang; Fan Fan; Zhizheng Wu

cs.SDOct 11, 2026

AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency

Authors: Yunjia Zhang, Junan Zhang, Jing Yang, Xueyao Zhang, Fan Fan, Zhizheng Wu

Organizations: The Chinese University of Hong Kong, Shenzhen · Institute of Media Technology and Experience Design, Huawei · City University of Macau · Amphion Technology Co., Ltd.

Abstract

We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: https://any-edit.github.io/. AnyEditBench: https://huggingface.co/datasets/Ririiiii/anyeditbench.

Figures & tables

Figure 1: Overall architecture of the AnyEdit framework. NFE indicates the Number of Function Evaluations (ODE integration steps) used during flow matching inference.

Figure 2: Training pipeline of the Noise-Robust Prosody Tokenizer via Teacher-Student Distillation.

Figure 3: AR model pretraining data construction and training process

Method	Construction Protocol	Target Realness	Melodic Match	Scalability	Primary Limitation
Web Crawled Hand-crafted SVS	Content creators manually tune synthesis software parameters to align specific timbres with different lyrics.	✗	✓	✗	Extremely labor-intensive; dependent on human tuning skills; limited data volume.
SVS Model Generation	Synthesizes target audio from lyrics using a trained SVS model (Synthetic Target).	✗	✓	✓	Model learns synthetic artifacts and robotic prosody; upper bound limited by the SVS teacher.
Random Truncation	Randomly masks/cuts audio segments for reconstruction (pre-training style).	✓	✗	✓	Lacks specific lyrical control; cannot guarantee melodic preservation when lyrics change.
Mined Real-to-Real Pairing (Ours)	Mines intra-song repetitions (Segment A/B) via Chromagram DTW alignment.	✓	✓	✓	Requires songs with repetitive structures (common in pop music).

Table 1: Comparison of data construction pipelines for singing voice editing. Target Realness : Whether the ground-truth target audio is a natural human recording. Melodic Match : Whether the target strictly follows the source melody contour.

Panel of Figure 4 — Figure 4: Example of a Real-to-Real mined singing pair (left) and the DPO training process (right). Different lyrics (“I don’t like your perfect crime” vs. “I don’t like your kingdom keys”)in the same song of Look what you made me do by Taylor Swift. The song pair is only used for visualization for research purpose.

	Speech Editing						Singing Voice Editing
Metric	SSR- Speech	Voice- Craft	Step- EditX	Vevo2	Ming- Uni	AnyEdit	Style- Singer	TCSinger	SoulX- Singer	Vevo2	Ming- Uni	AnyEdit
WER (en) $\downarrow$	0.0827	0.1081	0.1706	0.0778	0.5385	0.0616	–	0.9971	–	0.2809	1.0000	0.1749
CER (zh) $\downarrow$	0.1816	–	0.1296	0.1660	0.3000	0.1433	5.4133	0.3171	0.8919	0.1780	0.1538	0.0587
SIM-spk (en) $\uparrow$	0.9652	0.9604	0.9369	0.9650	0.9632	0.9675	–	0.7082	–	0.8785	0.3009	0.9167
SIM-spk (zh) $\uparrow$	0.8958	–	0.9054	0.9027	0.8085	0.9108	0.6745	0.7437	0.8942	0.8970	0.9502	0.9198
F0 Corr (en) $\uparrow$	–	–	–	–	–	–	–	0.0217	–	0.2030	-0.1780	0.2080

Table 2: Objective results on the AnyEditBench real-world subset English uses WER, Chinese uses CER; F0 Corr is singing-only. Bold indicates best per domain; underlined entries indicate that AnyEdit outperforms the unified baseline Vevo2 [ 1 ] .

	Speech Editing						Singing Voice Editing
Metric	SSR- Speech	Voice- Craft	Step- EditX	Vevo2	Ming- Uni	AnyEdit	Style- Singer	TCSinger	SoulX- Singer	Vevo2	Ming- Uni	AnyEdit
WER (en) $\downarrow$	0.1085	0.2173	0.1618	0.0686	0.0588	0.0770	–	1.2480	–	0.2579	0.5500	0.2368
CER (zh) $\downarrow$	1.0317	–	0.0875	0.1031	0.1163	0.1184	0.7595	0.6019	0.9028	0.3065	0.6250	0.2975
SIM-spk (en) $\uparrow$	0.9660	0.9383	0.9363	0.9296	0.9625	0.9535	–	0.5530	–	0.9015	0.8395	0.9223
SIM-spk (zh) $\uparrow$	0.9056	–	0.8840	0.9164	0.9136	0.9328	0.7093	0.7394	0.8723	0.8838	0.7996	0.9014
F0 Corr (en) $\uparrow$	–	–	–	–	–	–	–	-0.0221	–	0.2932	0.2436	0.3014

Table 3: Objective results on the AnyEditBench simulated subset (averaged over 15 conditions) Layout same as Table 2 .

Task	Condition	Model	SNRErr $\downarrow$	RT60Err $\downarrow$	WER/CER $\downarrow$	SIM-spk $\uparrow$	SIM-bg $\uparrow$	N-MOS $\uparrow$	S-MOS $\uparrow$
Speech	SNR=0 dB	Vevo2	13.321	0.011	0.196	0.806	0.471	4.02	3.96
	SNR=0 dB	AnyEdit	3.368	0.165	0.386	0.897	0.756	4.31	4.27
	RT60=1.5 s	Vevo2	10.383	0.031	0.093	0.929	0.504	4.08	4.01
	RT60=1.5 s	AnyEdit	1.480	0.0003	0.067	0.940	0.820	4.36	4.33
	Both (SNR0+RT60)	Vevo2	9.421	0.0005	0.156	0.884	0.458	3.95	3.90
	Both (SNR0+RT60)	AnyEdit	2.017	0.0006	0.105	0.911	0.848	4.24	4.18

Table 4: Comparison of AnyEdit and Vevo2 [ 1 ] under extreme conditions Bold indicates the better result in each pairwise comparison. Underlined entries highlight where AnyEdit outperforms Vevo2.

Table 9

Figure 5: Case study on real-world speech and singing voice editing.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Representation	Encoded Attributes	Excluded Attributes	System Role
Prosody Tokens ( $z^{\text{prosody}}$ )	Pitch contour, melody, rhythm	Phonetics, vocal style, timbre, acoustics	Condition for AR editor
Content-Style Tokens ( $z^{\text{CS}}$ )	Phonemes, articulation, vibrato, emotion	Speaker timbre, room/noise acoustics	Output target of AR editor
Acoustic Context (Mel)	Speaker timbre, background noise, RT60	Semantic text content	In-context prompt for FMT

Appendix

Table 7: Taxonomy and orthogonal properties of core representations in AnyEdit .

subset	Domain	Language	Source	Acoustic condition	Items
Clean	Speech	EN	LibriTTS, VCTK	Clean recordings; single-span and multi-span edits	50
	Speech	ZH	AISHELL-1/2/3	Clean recordings; single-span and multi-span edits	50
	Singing	EN	Clean SingVERSE test subset [ 8 ]	Clean recordings; single-span and multi-span edits	50
	Singing	ZH	Clean SingVERSE test subset [ 8 ]	Clean recordings; single-span and multi-span edits	50
Simulated	Speech	EN	Clean-source speech subsets	15 noise/reverberation conditions (SNR $\in\{0,10,20,\infty\}$ , RT60 $\in\{0,0.5,1.0,1.5\}$ )	50
	Speech	ZH	Clean-source speech subsets	15 noise/reverberation conditions (excluding clean case)	50

Appendix

Table 8: Summary of AnyEditBench . Each subset contains 50 evaluation items. The clean subset includes both single-span and multi-span edits.

Model	Supported Domain	Trained for Editing	Total for Inference
AnyEdit (Ours)	Speech + Singing (EN + ZH)	872M	1.1B
Vevo2 [ 1 ]	Speech + Singing (EN + ZH)	872M	1.1B
Ming-UniAudio [ 6 ]	Speech (EN + ZH)	$\sim$ 2.8B active	18.0B ( $\sim$ 2.8B active)
MiMo-Audio [ 7 ]	Speech (EN + ZH)	8.0B	9.3B
Step-Audio-EditX [ 3 ]	Speech (EN + ZH)	3.5B	3.7B
SSR-Speech [ 11 ]	Speech (EN + ZH) ∗	840M + 840M	1.8B

Appendix

Table 9: Comparison of model parameter counts and capability coverage.

	Speech Editing		Singing Voice Editing
Metric	Vevo2	AnyEdit	Vevo2	AnyEdit
WER (en) $\downarrow$	0.1717	0.1658	0.2795	0.2754
CER (zh) $\downarrow$	0.1950	0.1841	0.1986	0.1883
SIM-spk (en) $\uparrow$	0.9582	0.9583	0.9496	0.9519
SIM-spk (zh) $\uparrow$	0.9582	0.9597	0.9574	0.9553
F0 Corr (en) $\uparrow$	0.1801	0.1803	0.3341	0.3406

Appendix

Table 10: Comparison between Vevo2 and AnyEdit on Vevo 2’s speech and singing voice editing test sets.

	Speech Editing							Singing Voice Editing
Metric	SSR- Speech	Voice- Craft	Step- EditX	Vevo2	Ming- Uni	Mimo- Audio	AnyEdit	Style- Singer	TCSinger	SoulX- Singer	Vevo2	Ming- Uni	Mimo- Audio	AnyEdit
WER (en) $\downarrow$	0.0246	0.0428	0.0823	0.0600	0.1291	0.0423	0.0408	–	1.6463	–	0.1482	0.4123	0.1861	0.1425
CER (zh) $\downarrow$	0.0877	–	0.0419	0.0561	0.1091	0.0274	0.0487	0.6124	0.3135	0.8572	0.2203	0.4444	0.1957	0.0613
SIM-spk (en) $\uparrow$	0.9876	0.9714	0.9723	0.9739	0.9692	0.9448	0.9763	–	0.6569	–	0.9410	0.8314	0.8604	0.9551
SIM-spk (zh) $\uparrow$	0.9655	–	0.9430	0.9582	0.9274	0.9344	0.9540	0.7865	0.8309	0.9182	0.9458	0.8444	0.8836	0.9486
F0 Corr (en) $\uparrow$	–	–	–	–	–	–	–	–	-0.0538	–	0.1338	0.0120	0.0330	0.1589

Appendix

Table 11: Objective results on the AnyEditBench clean split

Metric	SSR- Speech	Voice- Craft	Step- EditX	Vevo2	Ming- Uni	Mimo- Audio	AnyEdit
WER $\downarrow$	0.0389	0.0531	0.1518	0.0266	0.1118	0.1029	0.0255
SIM-spk $\uparrow$	0.9685	0.9657	0.9598	0.9657	0.9532	0.9465	0.9689
F0 Corr $\uparrow$	0.1638	0.0076	0.2155	0.1966	0.2137	0.0248	0.2330

Appendix

Table 12: Objective results on speech editing

	VC		SVC		Spk2Sing
Metric	Vevo2	AnyEdit	Vevo2	AnyEdit	Vevo2	AnyEdit
WER (en) $\downarrow$	0.0639	0.0586	0.1884	0.2262	0.2038	0.1895
CER (zh) $\downarrow$	0.1028	0.0998	0.3552	0.3719	0.4302	0.3380
SIM-spk (en) $\uparrow$	0.9187	0.9084	–	–	–	–
SIM-spk (zh) $\uparrow$	0.9291	0.9134	–	–	–	–
F0 Corr (en) $\uparrow$	–	–	0.1678	0.1772	0.1528	0.1592

Appendix

Table 13: Zero-shot performance comparison between Vevo2 and AnyEdit across multiple conversion tasks.

Method	-5 dB	0 dB	5 dB	10 dB	15 dB	20 dB
Direct Clean Supervision	0.0896	0.2029	0.3367	0.4496	0.5497	0.6238
Teacher-Student Distillation (Ours)	0.3360	0.4587	0.5586	0.6306	0.6968	0.7410

Appendix

Table 14: Prosody token match rate between direct clean supervision and teacher-student distillation across varying SNR conditions.

Domain	Method	N-MOS (95% CI) $\uparrow$	S-MOS (95% CI) $\uparrow$	Fooling Rate (%)
Speech	Vevo2	4.15 (4.06, 4.24) ∗∗	4.02 (3.92, 4.12) ∗∗	31.5
	SSR-Speech	4.48 (4.40, 4.56) ${}^{\text{n.s.}}$	3.99 (3.90, 4.08) ∗∗	43.2 ${}^{\text{n.s.}}$
	VoiceCraft	4.04 (3.95, 4.13) ∗∗	4.32 (4.23, 4.41) ∗	35.7 ∗
	Step-Audio-EditX	3.95 (3.86, 4.04) ∗∗	4.02 (3.93, 4.11) ∗∗	33.8 ∗∗
	AnyEdit (Ours)	4.48 (4.41, 4.55)	4.45 (4.37, 4.53)	46.5
Singing	Vevo2	4.22 (4.13, 4.31) ∗∗	4.15 (4.05, 4.25) ∗∗	28.6 ∗∗

Appendix

Table 15: Subjective evaluation scores with 95% confidence intervals (CI). Superscripts denote paired Wilcoxon signed-rank tests against AnyEdit : ∗ (

p<0.05

), ∗∗ (

p<0.01

{}^{\text{n.s.}}

not significant (

p\geq 0.05

). Bold denotes optimal scores.

Method	WER/CER $\downarrow$	F0 Corr $\uparrow$		SIM-spk $\uparrow$
Method	WER/CER $\downarrow$	Speech	Singing	SIM-spk $\uparrow$
SSR-Speech	0.012	0.468	–	0.982
Vevo2	0.051	0.732	0.785	0.945
AnyEdit (w/o unmodified-area-aware sup.)	0.038	0.815	0.862	0.958
AnyEdit (Ours)	0.009	0.963	0.975	0.988

Appendix

Table 16: Unmodified Area Fidelity (UAF) on the simulated subset. “–” denotes unavailable/inapplicable entries.

Pipeline Strategy	WER $\downarrow$	SIM-bg $\uparrow$	SNRErr (dB) $\downarrow$	N-MOS $\uparrow$
MaskGCT + Sep (Direct Mix)	0.158	0.348	19.12	3.61 $\pm$ 0.10
Vevo2 + Sep (Direct Mix)	0.165	0.352	18.54	3.55 $\pm$ 0.11
Vevo2 + Sep (SNR-aware Mix)	0.165	0.415	6.85	3.72 $\pm$ 0.08
AnyEdit (Separation-Free, Ours)	0.142	0.819	3.13	4.38 $\pm$ 0.06

Appendix

Table 17: Quantitative comparison of explicit separation-based pipelines versus our separation-free framework on sampled real-world subset

Configuration	Circular Correlation $\uparrow$	Mean Pitch-Class Error (semitones) $\downarrow$	Onset Correlation $\uparrow$
12 bins	0.718	1.090	0.309
24 bins (Ours)	0.730	1.040	0.318

Appendix

Table 18: Agreement between chromagram representations and F0 pitch contours on singing vocals.

Model	WER/CER $\downarrow$	SIM-spk $\uparrow$	SIM-bg $\uparrow$	F0 Corr $\uparrow$
SSR-Speech	0.0842	0.9521	0.7240	0.1824
VoiceCraft	0.1265	0.9418	0.6512	0.0150
Vevo2	0.0894	0.9510	0.6128	0.2014
AnyEdit (Ours)	0.0521	0.9654	0.7850	0.2480

Appendix

Table 19: Quantitative results on the varying-length editing subset of AnyEditBench .

Audio Editing

Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.

See all 55 papers

CosyEdit2: Speech-Editing-Oriented Reinforcement Learning Unlocks Better Zero-Shot TTS
May 25, 2026Junyang Chen, Yuhang Jia, Hui Wang +3Speech Editing Zero-Shot TTS
College of Computer Science, Nankai University · College of Artificial Intelligence, Nankai University
CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models
Jan 8, 2026Junyang Chen, Yuhang Jia, Hui Wang +2Audio Editing Speech Editing
College of Computer Science, Nankai University, Tianjin, China
SongCraft: Unified Song Generation and Editing with Reconstructive Learning
Sep 14, 2026Haohe Liu, Varun Nagaraja, Gael Le Lan +5Text-to-Music Generation Audio Editing
Meta AI

May 25, 2026J/K move · Enter open · S save

CosyEdit2: Speech-Editing-Oriented Reinforcement Learning Unlocks Better Zero-Shot TTS

Junyang Chen, Yuhang Jia, Hui Wang, Jiaming Zhou+2

College of Computer Science, Nankai University · College of Artificial Intelligence, Nankai University

Speech Editing Zero-Shot TTS Text-to-Speech

Open paper AskPDF arXiv

Speech editing and zero-shot Text-to-Speech (TTS) share a similar generative foundation conditioned on speech prompts, yet speech editing demands far stricter local acoustic consistency with surrounding unedited content. While prior work has shown that Supervised Fine-Tuning (SFT) enables TTS models to acquire functional editing capability, this approach remains fundamentally bottlenecked by imperfect paired editing data and coarse-grained optimization signals. To address these limitations, we propose CosyEdit2, a speech editing model built on a two-stage post-training framework that progresses from supervised editing initialization to editing-oriented Group Relative Policy Optimization (GRPO) over target-speech-free data. Extensive experiments demonstrate that CosyEdit2 not only substantially advances speech editing performance, but also unlocks better zero-shot TTS capability, revealing a deeper mutual relationship between the two tasks. Audio samples are available at https://cjy1018.github.io/CosyEdit2.

Abstract

Audio Editing

Explore similar work

CosyEdit2: Speech-Editing-Oriented Reinforcement Learning Unlocks Better Zero-Shot TTS

CosyEdit: Unlocking End-to-End Speech Editing Capability from Zero-Shot Text-to-Speech Models

SongCraft: Unified Song Generation and Editing with Reconstructive Learning