Unified audio editors support diverse operations, yet often struggle to modify the intended source while preserving non-target content. We show that source selection in audio editing is shaped by early denoising states, while incorrect early selections can hardly be corrected during subsequent denoising. Motivated by these findings, we propose Source-selection Training via Early-trajectory Emphasis and Recovery (STEER), a two-stage post-training algorithm for reliable selective audio editing. STEER-Emphasis prioritizes training on task-critical early states to reduce source-selection errors, while STEER-Recovery learns corrective denoising from self-generated rollout states to recover from early deviations. To systematically assess source-selection reliability, we introduce the Selective Audio Editing Benchmark, enabling large-scale automatic evaluation of target manipulation and non-target preservation. Applying STEER to our pretrained multitask base editor gives SteerSound, a unified audio editor with leading selective editing performance among open-source audio editors. Experiments on our base editor, UNISON, and SAM Audio demonstrate that STEER consistently improves both selective editing capability and overall editing performance. Early-trajectory emphasis further brings substantial improvements in transcript-conditioned target speaker removal and separation, highlighting the broader applicability of this training principle.
Figures & tables
Figure 1: Strengthening selective audio editing with STEER. Left : Existing editors may leave the target unchanged or modify non-target content. Middle : Early states shape final source selection, while later denoising tends to follow the established editing direction rather than revise it. Right : STEER addresses these failures through early-trajectory emphasis (Emphasis) and recovery training (Recovery), yielding SteerSound with more reliable selective editing.
Figure 2: Erroneous source selection is formed and visible within the first few denoising steps. Each panel shows a failed edit: the reference, the target, and the zero-noise estimate decoded at step kobs , the earliest step at which the erroneous tendency becomes perceptible. A and B denote the two sources in the reference audio.
Figure 3: Bidirectional early-trajectory interventions on source selection. Corrective (top) and corruptive (bottom) prefix interventions are evaluated on 100 validation examples per task. Bars show per-step corrective or corruptive counts, while curves show the corresponding cumulative effects. For each task, k80 denotes the shortest intervention prefix length at which the cumulative effect reaches 80%. Removal consistently exhibits a shorter k80 than replacement.
Figure 4Figure 5
Params.
Generate
Add
Replace
Remove
Pitch
Volume
Model
Total
Train.
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
Audio-Omni
7.907B
3.047B
0.231
2.057
0.196
1.955
0.154
2.561
0.001
1.695
2.803
–
–
–
–
–
–
MMEdit
9.057B
0.616B
–
–
0.159
1.882
0.085
2.706
0.012
2.060
2.155
–
–
–
14.043
1.042
0.427
AUDIT
1.067B
0.860B
–
–
0.151
1.417
0.082
1.897
0.038
1.603
2.691
–
–
–
–
–
–
PPAE
1.318B
–
–
–
0.182
1.566
0.094
1.501
0.044
1.539
1.384
–
–
–
–
–
–
SDEdit
1.118B
–
–
–
0.111
2.347
0.129
2.222
0.129
1.778
1.969
0.923
0.942
1.963
5.047
0.895
1.992
Table 1: Unified audio editing performance across models on the Complex Audio Editing Test Set. Our Base Editor denotes the pretrained model before STEER; SteerSound denotes our final model after both STEER stages. “–” in the Train. column denotes training-free methods built upon existing pretrained audio generation models, while “–” in task columns indicates that the model or method does not support the corresponding task.
Model
STEER- Emphasis
STEER- Recovery
Generate
Add
Replace
Remove
Pitch
Volume
Overall
Succ. ↑
Succ. ↑
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Our Base Editor
–
–
92.80
93.20
67.90
24.10
72.50
22.80
51.20
48.40
39.10
24.70
70.42
28.48
+Emph. only
✓
–
94.00
94.60
78.90
14.00
88.10
7.90
87.90
11.70
61.60
6.80
84.02
9.99
+Full STEER
✓
✓
94.80
93.50
79.10
12.30
88.20
7.90
92.80
6.80
68.00
6.40
85.78
8.45
UNISON
–
–
96.40
90.10
33.40
50.90
25.30
26.90
–
–
–
–
61.30
38.90
+Emph. only
✓
–
94.80
85.60
78.40
13.60
38.30
8.50
–
–
–
–
74.28
11.05
Table 3: Effectiveness of STEER across models on the Selective Audio Editing Benchmark. Succ. and SErr. denote success and selective error rates, respectively. “–” indicates that the corresponding training stage is not applied or the editing task is not supported.
Figure 8
Backbone
Strategy
Complex
Selective
TBGall↑
TBGsel↑
Succ. ↑
SErr. ↓
Standard FM
0.92
0.66
81.05
13.58
Our Base Editor
STEER-Emphasis
1.67
2.97
84.02
9.99
Standard FM
6.97
12.89
68.18
18.95
UNISON
STEER-Emphasis
10.27
17.85
74.28
11.05
Standard FM
65.69
65.69
50.50
49.40
Table 9
Figure 10
Backbone
Strategy
Complex
Selective
TBGall↑
TBGsel↑
Succ. ↑
SErr. ↓
Emph. elb.
1.36
2.82
83.98
9.68
Our Base Editor
Emphasis
1.67
2.97
84.02
9.99
Table 11
Figure 12: Early emergence of source-selection tendencies on our base editor. Intermediate denoising states are extrapolated to estimated zero-noise outputs. Representative failed edits from the four selective editing tasks are shown, with kobs denoting the earliest step at which the erroneous source-selection tendency becomes perceptually distinguishable.
Figure 13: Bidirectional early-trajectory interventions on our base editor. Corrective (top) and corruptive (bottom) interventions are evaluated on 100 validation examples per selective editing task. Bars denote per-step intervention counts and curves denote cumulative effects. Dashed lines indicate the resulting critical ranges used by STEER-Emphasis: k∗=2 for selective suppression and k∗=15 for selective transformation.
Module
Params. (M)
DiT backbone
673.35
T5-Base
109.63
Audio VAE
509.34
HiFi-GAN
55.26
Total
1,347.58
Trainable
673.35
Table 8: Parameter counts by module for our base editor.
Figure 14: Training loss and learning rate.
Task
Conditional prompt
CFG reference prompt
Generate
Generate the sound of rain.
Empty string
Add
Add the sound of rain.
Empty string
Remove
Remove the sound of rain.
Empty string
Replace
Replace the sound of rain by the sound of wind.
Empty string
Volume ↑
Increase the volume of the sound of rain by 3.0.
Increase the volume by 3.0.
Volume ↓
Reduce the volume of the sound of rain by 3.0.
Reduce the volume by 3.0.
Table 9: Example conditional and CFG reference prompts.
Dataset
# Chunks
VGGSound
192,794
AudioCaps
101,369
FSD50K
51,182
Clotho V2
17,545
MACS
17,275
Total
380,165
Table 10: Data sources for P .
Figure 15: Overview of the online Complex Audio Editing Data constructor. Audio sources from the Class-Label Pool L and Fine-Grained Description Pool P are dynamically composed into reference–target–instruction pairs for six audio editing operations. ∣Δp∣ is capped at 4 semitones, and ∣Δv∣ is capped at 5 dB in the process.
Figure 16: Overview of the Selective Audio Editing Benchmark constructor. Audio sources from the Class-Label Pool L and Fine-Grained Description Pool P are organized into task-specific pools and composed into reference–target–instruction pairs for six audio editing operations with controlled source layouts. ∣Δp∣ is fixed at 4 semitones, and ∣Δv∣ is fixed at 5 dB in this process.
Figure 17: Validation dynamics of STEER on our base editor using the Complex Audio Editing validation set. (a) Validation TBGall across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Validation TBGall across recovery depths k . Markers denote checkpoints evaluated every 0.5 epoch at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 18: Validation dynamics of STEER on UNISON using the Complex Audio Editing validation set. (a) Validation TBGall across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Validation TBGall across recovery depths k . Markers denote checkpoints evaluated every 0.5 epoch at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 19: Validation dynamics of STEER on SAM Audio using the Complex Audio Editing validation set. (a) Validation TBGall across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Validation TBGall across recovery depths k . Markers denote checkpoints evaluated every 0.5 epoch at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 20: Validation dynamics of STEER on our base editor using the Selective Audio Editing validation set. (a) Overall validation success rate across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Overall validation success rate across recovery depths k . Markers denote checkpoints from the five training epochs at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 21: Validation dynamics of STEER on UNISON using the Selective Audio Editing validation set. (a) Overall validation success rate across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Overall validation success rate across recovery depths k . Markers denote checkpoints from the five training epochs at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 22: Validation dynamics of STEER on SAM Audio using the Selective Audio Editing validation set. (a) Overall validation success rate across STEER-Emphasis training epochs, with the selected checkpoint highlighted. (b) Overall validation success rate across recovery depths k . Markers denote checkpoints from the five training epochs at each depth, while the purple curve connects the validation-best checkpoint for each k .
Figure 23: Task-wise recovery from early-trajectory corruption across model variants on the Selective Audio Editing Benchmark. Larger k50 indicates stronger robustness to early-trajectory errors.
Figure 24: Human evaluation of our base editor, its STEER-Emphasis-only variant, and full STEER variant on the Complex Audio Editing Test Set, using 100 randomly sampled examples per task. Left: overall success rates and selective error counts. Right: task-wise distributions of successful edits and different failure types.
Table 12: Detailed comparison of STEER-Emphasis and standard Flow Matching post-training under matched updates on the Complex Audio Editing Test Set. TBG scores are computed relative to the corresponding pretrained model.
Model
Strategy
Generate
Add
Replace
Remove
Pitch
Volume
Overall
Succ. ↑
Succ. ↑
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Standard FM
93.70
92.90
76.20
16.40
83.00
13.40
82.44
16.91
58.40
8.40
81.05
13.58
Our Base Editor
STEER-Emphasis
94.00
94.60
78.90
14.00
88.10
7.90
87.90
11.70
61.60
6.80
84.02
9.99
Standard FM
95.20
82.40
67.40
23.30
27.70
14.60
–
–
–
–
68.18
18.95
UNISON
STEER-Emphasis
94.80
85.60
78.40
13.60
38.30
8.50
–
–
–
–
74.28
11.05
Standard FM
–
–
–
–
50.50
49.40
–
–
–
–
50.50
49.40
Table 13: Detailed comparison of STEER-Emphasis and standard Flow Matching post-training under matched updates on the Selective Audio Editing Benchmark. Succ. and SErr. denote success and selective error rates, respectively.
Model
Strategy
Generate
Add
Replace
Remove
Pitch
Volume
Overall
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
TBGall↑
TBGsel↑
Emphasis (continue)
0.22
2.08
0.25
1.32
0.24
1.04
0.14
1.25
0.93
0.05
0.49
0.28
0.68
0.43
0.20
3.61
5.50
Our Base Editor
STEER-Recovery
0.22
2.13
0.25
1.35
0.24
1.03
0.14
1.16
0.79
0.05
0.49
0.28
0.63
0.43
0.20
5.36
8.25
Emphasis (continue)
0.25
1.84
0.26
1.37
0.24
1.16
0.15
1.39
0.89
–
–
–
–
–
–
10.27
17.85
UNISON
STEER-Recovery
0.25
1.83
0.27
1.32
0.23
1.16
0.15
1.36
0.82
–
–
–
–
–
–
10.81
18.42
Emphasis (continue)
–
–
–
–
–
–
0.12
1.36
1.06
–
–
–
–
–
–
105.08
105.08
Table 14: Detailed comparison of STEER-Recovery and continued STEER-Emphasis under matched updates on the Complex Audio Editing Test Set. TBG scores are computed relative to the corresponding pretrained model.
Model
Strategy
Generate
Add
Replace
Remove
Pitch
Volume
Overall
Succ. ↑
Succ. ↑
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Emphasis (continue)
93.90
93.20
80.70
12.60
88.30
8.00
91.05
8.55
61.50
5.50
84.51
8.67
Our Base Editor
STEER-Recovery
94.80
93.50
79.10
12.30
88.20
7.90
92.83
6.77
68.00
6.40
85.78
8.45
Emphasis (continue)
94.80
85.60
78.40
13.60
38.30
8.50
–
–
–
–
74.28
11.05
UNISON
STEER-Recovery
94.90
85.40
79.60
13.10
41.10
8.00
–
–
–
–
75.25
10.55
Emphasis (continue)
–
–
–
–
82.10
17.50
–
–
–
–
82.10
17.50
Table 15: Detailed comparison of STEER-Recovery and continued STEER-Emphasis under matched updates on the Selective Audio Editing Benchmark. Succ. and SErr. denote success and selective error rates, respectively.
Dataset
Strategy
Generate
Add
Replace
Remove
Pitch
Volume
Overall
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
KL ↓
CLAP ↑
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
MAE ↓
LSD ↓
KL ↓
TBGall↑
TBGsel↑
Emph. (elbow)
0.22
2.14
0.25
1.35
0.23
1.06
0.14
1.25
0.95
0.05
0.49
0.27
0.74
0.43
0.20
1.36
2.82
Complex
STEER-Emphasis
0.22
2.12
0.25
1.32
0.23
1.05
0.14
1.22
0.90
0.05
0.49
0.28
0.71
0.43
0.20
1.67
2.97
Table 16: Task-wise comparison of task-group-specific and elbow-based shared early-trajectory ranges for STEER-Emphasis on the Complex Audio Editing Test Set. TBG scores are computed relative to our base editor.
Dataset
Strategy
Generate
Add
Replace
Remove
Pitch
Volume
Overall
Succ. ↑
Succ. ↑
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Succ. ↑
SErr. ↓
Emph. (elbow)
93.80
94.10
79.80
13.80
87.80
8.40
89.33
10.13
60.40
6.50
83.98
9.68
Selective
STEER-Emphasis
94.00
94.60
78.90
14.00
88.10
7.90
87.90
11.70
61.60
6.80
84.02
9.99
Table 17: Task-wise comparison of task-group-specific and elbow-based shared early-trajectory ranges for STEER-Emphasis on the Selective Audio Editing Benchmark. Succ. and SErr. denote success and selective error rates, respectively.
Figure 37: Architecture of our base editor adapted for the TSR task.
Module
Params.
DiT LoRA
15.73M
Queries
0.13M
Connector
52.46M
Projector
8.53M
Total
76.84M
Table 18: Trainable parameters.
Figure 38: Validation CLAP-AATSR during training on the TSR task with our adapted base editor. The horizontal line denotes the CLAP-AATSR of the mixture audio.
Module
Params.
SAM Audio LoRA
127.40M
Queries
0.13M
Connector
65.54M
Projector
18.09M
Total
211.17M
Table 19: Trainable parameters.
Figure 39: Validation CLAP-AATSS during training on the TSS task with SAM Audio. The horizontal line denotes the CLAP-AATSS of the mixture audio.
Figure 40: Human evaluation interface for the Complex Audio Editing Test Set.
Figure 41: Human evaluation interface for the TSR and TSS tasks.
Text-guided audio editing aims to modify the language-specified acoustic content while preserving edit-irrelevant source components. Existing training-free methods typically rely on inversion-based editing. While inversion-free editing is appealing as it decreases computational overhead and reconstruction errors, it remains largely unexplored for audio editing. The key challenge is to construct a source-to-target editing path through diffusion denoising dynamics. In this paper, we introduce DirectAudioEdit, the first attempt to develop a training-free and inversion-free method for audio editing. Experiments on music and event-level benchmarks across two backbones show that DirectAudioEdit reduces macro-averaged FAD and KL by 15.9% and 15.8% compared with DDPM inversion, while achieving up to 64.5% editing speedup.
Zhengkun Ge, Xiaoqian Liu, Haoran Zhang +5
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Kunming University of Science and Technology · NiuTrans Research, Shenyang, China
Text-to-audio (TTA) generation has made significant strides, yet achieving precise and consistent audio editing remains a major challenge. However, existing methods struggle to balance temporal consistency with background preservation. In this paper, we propose FreeSonic, a training-free framework leveraging the state-of-the-art Rectified Flow-based TangoFlux model. FreeSonic utilizes an optimized inversion-reverse process and joint text-audio attention maps for precise target segment extraction. For content editing, a novel scheduled attention decoupling confines modifications to target regions while preserving original acoustic context. Furthermore, task-oriented noise injection enhances versatility for tasks such as audio removal and non-rigid replacement. Extensive experimental results demonstrate that FreeSonic achieves a superior balance by providing a high-fidelity and efficient solution for precise and consistent audio editing. Project and demos: https://free-sonic.github.io/
Yuxuan Jiang, Mingyang Han, Yusheng Dai +12
Tsinghua University, China · Alibaba Group, China · Monash University, Australia +2
We introduce AudEdit, an inversion-free method for text-guided editing of real audio with a pretrained rectified-flow audio generator. Text-to-audio systems such as Stable Audio 3 already expose audio-to-audio editing by noising an input recording and denoising it under a new prompt, but this inversion-style route must trade prompt adherence against preservation of rhythm, transients, timbre, and long-range musical structure. Motivated by recent inversion-free flow editing in computer vision, we develop an audio-specific direct source-to-target ordinary differential equation for one-dimensional Stable Audio 3 latents: at each flow step, we compare the target- and source-conditioned velocity fields under a shared stochastic source marginal, and update the edited latent by their difference. The resulting editor requires no training, no paired edit data, no optimization, and no access to internal attention maps. Across sound-effect and music editing sets built from FSD50K and the Song Describer Dataset, AudEdit improves CLAP text alignment and audio preservation over SDEdit, ODE inversion, and FireFlow; for example, on sound effects it raises target-text CLAP similarity from 0.42 to 0.52 over the strongest baseline while reducing FAD from 65.70 to 50.37.