Harmony: Bridging Audio Understanding and Editing through Agentic Collaboration
Organizations: Worcester Polytechnic Institute · The University of Texas at Dallas
Abstract
Natural-language audio editing requires connecting a user's intent to specific sounds and time regions while preserving surrounding content. Complex requests make this connection demanding: an edit may select one occurrence, depend on a temporal relation, or modify a source that overlaps with other sounds. We introduce Harmony, an agentic framework that bridges audio understanding and editing through collaboration among an LLM planner, an audio-language model, and external tools. Harmony combines two mechanisms. Edit-driven collaborative planning uses targeted questions and tool measurements to determine the target, timing, and operation parameters. Evidence-guided execution uses observations of the first separated or generated audio to trigger bounded corrections before final mixing. Together, these mechanisms connect semantic decisions to waveform operations throughout the editing process. Harmony supports addition, removal, replacement, and volume adjustment. On 420 MMAE examples spanning these operations, Harmony achieves an exact match rate of 22.62%, compared with 13.57% for the strongest baseline, and improves aggregate instruction following and context preservation.
Figures & tables
| Correction | Evidence for selection | Action |
|---|---|---|
| Separation-query correction | Observations of the original audio and residual support a more specific description of the same requested target. | Repeat separation using the more specific target description. |
| Retained-source extraction | The instruction removes a complete background category, and the source inventory identifies exactly one complementary source to preserve. | Extract the retained source directly. |
| Residual clearing | Initial observations identify only content designated for replacement, with no protected or uncertain sources. | Silence the residual only within the selected replacement windows. |
| Generated-asset cleanup | Observation of the generated asset identifies an unwanted old instrument or voice. | Remove the identified content from the generated asset before placement. |
| Timbre correction | A generated instrumental asset has an incompatible timbre and satisfies the conditions for single-source pitched rendering. | Transcribe and re-render the generated notes with a supported target timbre. |
| At most 10 s ( ) | Longer than 10 s ( ) | |||||
|---|---|---|---|---|---|---|
| Method | IFR | CR | EMR | IFR | CR | EMR |
| SmartDJ | 50.74 | 65.58 | 7.39 | 54.70 | 63.84 | 5.33 |
| MMEDIT | 53.35 | 66.57 | 6.25 | 19.97 | 98.24 | 2.05 |
| WavCraft (original) | 59.23 | 81.38 | 17.05 | 57.65 | 75.75 | 11.07 |
| WavCraft (matched backbone) | 54.96 | 83.28 | 12.50 | 50.15 | 85.25 | 12.30 |
| AEC Agent | 44.68 | 75.60 | 7.95 | 44.13 | 74.12 | 8.61 |
| Addition ( ) | Removal ( ) | |||||
|---|---|---|---|---|---|---|
| Method | IFR | CR | EMR | IFR | CR | EMR |
| SmartDJ | 57.42 | 69.67 | 9.57 | 53.55 | 56.95 | 6.71 |
| MMEDIT | 39.13 | 82.56 | 6.09 | 31.36 | 93.94 | 3.36 |
| WavCraft (original) | 70.25 | 78.31 | 21.74 | 58.19 | 80.58 | 12.75 |
| WavCraft (matched backbone) | 62.80 | 83.70 | 17.39 | 51.01 | 89.45 | 13.42 |
| AEC Agent | 30.68 | 71.01 | 4.35 | 57.60 | 87.75 | 18.79 |
| Addition ( ) | Removal ( ) | |||||
|---|---|---|---|---|---|---|
| Configuration | IFR | CR | EMR | IFR | CR | EMR |
| 77.09 | 83.68 | 28.70 | 55.53 | 92.53 | 21.48 | |
| 79.10 | 81.52 | 27.83 | 60.95 | 90.86 | 22.15 | |
| 73.28 | 81.51 | 18.26 | 61.73 | 94.00 | 26.17 | |
| 79.14 | 82.00 | 31.30 | 57.29 | 93.12 | 22.82 | |
| Replacement ( ) | Volume adjustment ( ) | |||||
| Macro | Pooled | |||||
|---|---|---|---|---|---|---|
| Configuration | IFR | CR | EMR | IFR | CR | EMR |
| 57.19 | 88.23 | 17.63 | 58.75 | 88.52 | 19.29 | |
| 60.29 | 87.14 | 19.18 | 62.15 | 87.24 | 20.48 | |
| 58.38 | 86.46 | 15.57 | 60.35 | 87.25 | 17.62 | |
| 61.16 | 87.21 | 21.50 | 62.18 | 87.71 | 22.62 | |
| Operation | Time (s) | LLM calls | |
|---|---|---|---|
| Addition | 115 | 52.35 | 3.71 |
| Removal | 149 | 115.36 | 3.69 |
| Replacement | 79 | 173.08 | 7.04 |
| Volume adjustment | 77 | 95.36 | 3.29 |
| Overall | 420 | 105.30 | 4.25 |
| Method | Overall editing ( ) | Preservation ( ) |
|---|---|---|
| WavCraft (original) | 9.38 | 9.17 |
| AEC Agent | 8.59 | 16.67 |
| Harmony (ours) | 69.53 | 56.67 |
Audio Editing
Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.
See all 62 papers