AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency
Authors: Yunjia Zhang, Junan Zhang, Jing Yang, Xueyao Zhang, Fan Fan, Zhizheng Wu
Organizations: The Chinese University of Hong Kong, Shenzhen · Institute of Media Technology and Experience Design, Huawei · City University of Macau · Amphion Technology Co., Ltd.
We present AnyEdit, a unified, separation-free framework for speech and singing voice editing that preserves real-world acoustic environments, including noise, reverberation, and background music. Recent unified generative models already achieve strong speech and singing generation in clean conditions, but real-world vocal editing remains challenging because the model must revise content while simultaneously preserving speaker identity, prosody, and the surrounding acoustic scene. Directly modeling noisy audio end-to-end entangles semantic editing with environmental acoustics, weakening the core vocal generation capability. We therefore adopt a two-stage design that preserves clean-domain editing ability while deferring acoustic rendering to a separate stage: an instruction-guided autoregressive editor first predicts clean prosody and content-style tokens, supported by a teacher-student noise-robust prosody tokenizer that distills clean prosodic tokens from degraded recordings; a flow-matching acoustic model then performs in-context rendering conditioned on the source acoustic context, so the edited segment inherits surrounding noise, reverberation, and background sound while maintaining consistent speaker timbre. To stabilize unedited regions, we introduce unmodified-area-aware supervision; after pre-training, we further apply direct preference optimization on mined real-to-real singing pairs. We also release AnyEditBench, a benchmark covering both speech and singing voice editing under clean, simulated, and in-the-wild acoustic conditions. Experiments on Chinese and English speech and singing datasets showed that AnyEdit consistently outperforms strong baselines in content accuracy, subjective quality, and background consistency under noisy and reverberant conditions. Demo: https://any-edit.github.io/. AnyEditBench: https://huggingface.co/datasets/Ririiiii/anyeditbench.
Figures & tables
Figure 1: Overall architecture of the AnyEdit framework. NFE indicates the Number of Function Evaluations (ODE integration steps) used during flow matching inference.
Figure 2: Training pipeline of the Noise-Robust Prosody Tokenizer via Teacher-Student Distillation.
Figure 3: AR model pretraining data construction and training process
Method
Construction Protocol
Target Realness
Melodic Match
Scalability
Primary Limitation
Web Crawled Hand-crafted SVS
Content creators manually tune synthesis software parameters to align specific timbres with different lyrics.
✗
✓
✗
Extremely labor-intensive; dependent on human tuning skills; limited data volume.
SVS Model Generation
Synthesizes target audio from lyrics using a trained SVS model (Synthetic Target).
✗
✓
✓
Model learns synthetic artifacts and robotic prosody; upper bound limited by the SVS teacher.
Random Truncation
Randomly masks/cuts audio segments for reconstruction (pre-training style).
✓
✗
✓
Lacks specific lyrical control; cannot guarantee melodic preservation when lyrics change.
Mined Real-to-Real Pairing (Ours)
Mines intra-song repetitions (Segment A/B) via Chromagram DTW alignment.
✓
✓
✓
Requires songs with repetitive structures (common in pop music).
Table 1: Comparison of data construction pipelines for singing voice editing. Target Realness : Whether the ground-truth target audio is a natural human recording. Melodic Match : Whether the target strictly follows the source melody contour.
Figure 4: Example of a Real-to-Real mined singing pair (left) and the DPO training process (right). Different lyrics (“I don’t like your perfect crime” vs. “I don’t like your kingdom keys”)in the same song of Look what you made me do by Taylor Swift. The song pair is only used for visualization for research purpose.
Speech Editing
Singing Voice Editing
Metric
SSR- Speech
Voice- Craft
Step- EditX
Vevo2
Ming- Uni
AnyEdit
Style- Singer
TCSinger
SoulX- Singer
Vevo2
Ming- Uni
AnyEdit
WER (en) ↓
0.0827
0.1081
0.1706
0.0778
0.5385
0.0616
–
0.9971
–
0.2809
1.0000
0.1749
CER (zh) ↓
0.1816
–
0.1296
0.1660
0.3000
0.1433
5.4133
0.3171
0.8919
0.1780
0.1538
0.0587
SIM-spk (en) ↑
0.9652
0.9604
0.9369
0.9650
0.9632
0.9675
–
0.7082
–
0.8785
0.3009
0.9167
SIM-spk (zh) ↑
0.8958
–
0.9054
0.9027
0.8085
0.9108
0.6745
0.7437
0.8942
0.8970
0.9502
0.9198
F0 Corr (en) ↑
–
–
–
–
–
–
–
0.0217
–
0.2030
-0.1780
0.2080
Table 2: Objective results on the AnyEditBench real-world subset English uses WER, Chinese uses CER; F0 Corr is singing-only. Bold indicates best per domain; underlined entries indicate that AnyEdit outperforms the unified baseline Vevo2 [ 1 ] .
Speech Editing
Singing Voice Editing
Metric
SSR- Speech
Voice- Craft
Step- EditX
Vevo2
Ming- Uni
AnyEdit
Style- Singer
TCSinger
SoulX- Singer
Vevo2
Ming- Uni
AnyEdit
WER (en) ↓
0.1085
0.2173
0.1618
0.0686
0.0588
0.0770
–
1.2480
–
0.2579
0.5500
0.2368
CER (zh) ↓
1.0317
–
0.0875
0.1031
0.1163
0.1184
0.7595
0.6019
0.9028
0.3065
0.6250
0.2975
SIM-spk (en) ↑
0.9660
0.9383
0.9363
0.9296
0.9625
0.9535
–
0.5530
–
0.9015
0.8395
0.9223
SIM-spk (zh) ↑
0.9056
–
0.8840
0.9164
0.9136
0.9328
0.7093
0.7394
0.8723
0.8838
0.7996
0.9014
F0 Corr (en) ↑
–
–
–
–
–
–
–
-0.0221
–
0.2932
0.2436
0.3014
Table 3: Objective results on the AnyEditBench simulated subset (averaged over 15 conditions) Layout same as Table 2 .
Task
Condition
Model
SNRErr ↓
RT60Err ↓
WER/CER ↓
SIM-spk ↑
SIM-bg ↑
N-MOS ↑
S-MOS ↑
Speech
SNR=0 dB
Vevo2
13.321
0.011
0.196
0.806
0.471
4.02
3.96
AnyEdit
3.368
0.165
0.386
0.897
0.756
4.31
4.27
RT60=1.5 s
Vevo2
10.383
0.031
0.093
0.929
0.504
4.08
4.01
AnyEdit
1.480
0.0003
0.067
0.940
0.820
4.36
4.33
Both (SNR0+RT60)
Vevo2
9.421
0.0005
0.156
0.884
0.458
3.95
3.90
AnyEdit
2.017
0.0006
0.105
0.911
0.848
4.24
4.18
Table 4: Comparison of AnyEdit and Vevo2 [ 1 ] under extreme conditions Bold indicates the better result in each pairwise comparison. Underlined entries highlight where AnyEdit outperforms Vevo2.
Table 9
Figure 5: Case study on real-world speech and singing voice editing.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Representation
Encoded Attributes
Excluded Attributes
System Role
Prosody Tokens ( zprosody )
Pitch contour, melody, rhythm
Phonetics, vocal style, timbre, acoustics
Condition for AR editor
Content-Style Tokens ( zCS )
Phonemes, articulation, vibrato, emotion
Speaker timbre, room/noise acoustics
Output target of AR editor
Acoustic Context (Mel)
Speaker timbre, background noise, RT60
Semantic text content
In-context prompt for FMT
Appendix
Table 7: Taxonomy and orthogonal properties of core representations in AnyEdit .
subset
Domain
Language
Source
Acoustic condition
Items
Clean
Speech
EN
LibriTTS, VCTK
Clean recordings; single-span and multi-span edits
50
Speech
ZH
AISHELL-1/2/3
Clean recordings; single-span and multi-span edits
50
Singing
EN
Clean SingVERSE test subset [ 8 ]
Clean recordings; single-span and multi-span edits
50
Singing
ZH
Clean SingVERSE test subset [ 8 ]
Clean recordings; single-span and multi-span edits
Speech editing and zero-shot Text-to-Speech (TTS) share a similar generative foundation conditioned on speech prompts, yet speech editing demands far stricter local acoustic consistency with surrounding unedited content. While prior work has shown that Supervised Fine-Tuning (SFT) enables TTS models to acquire functional editing capability, this approach remains fundamentally bottlenecked by imperfect paired editing data and coarse-grained optimization signals. To address these limitations, we propose CosyEdit2, a speech editing model built on a two-stage post-training framework that progresses from supervised editing initialization to editing-oriented Group Relative Policy Optimization (GRPO) over target-speech-free data. Extensive experiments demonstrate that CosyEdit2 not only substantially advances speech editing performance, but also unlocks better zero-shot TTS capability, revealing a deeper mutual relationship between the two tasks. Audio samples are available at https://cjy1018.github.io/CosyEdit2.
Junyang Chen, Yuhang Jia, Hui Wang +3
College of Computer Science, Nankai University · College of Artificial Intelligence, Nankai University
Automatic speech editing aims to modify spoken content based on textual instructions, yet traditional cascade systems rely on explicit temporal alignment and complex preprocessing. To address these limitations, we propose CosyEdit, an end-to-end speech editing model adapted from CosyVoice through task-specific post-training and a complementary training paradigm, which internalizes text--speech alignment while ensuring high consistency between the speech before and after editing. Trained on only 250 hours of supervised data from our curated GigaEdit dataset, our 400M-parameter model achieves reliable speech editing performance. Extensive evaluations show that CosyEdit not only outperforms several billion-parameter language model baselines but also approaches state-of-the-art cascade systems. These results show that robust and efficient speech editing can be unlocked from a zero-shot TTS model through post-training, offering a cost-effective end-to-end solution for high-quality speech editing. Code and audio samples are available at https://cjy1018.github.io/CosyEditDemoPage/.
Junyang Chen, Yuhang Jia, Hui Wang +2
College of Computer Science, Nankai University, Tianjin, China
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.