cs.SDNov 10, 2026

Harmony: Bridging Audio Understanding and Editing through Agentic Collaboration

Authors: Zhongyi Tong, Yapeng Tian

Organizations: Worcester Polytechnic Institute · The University of Texas at Dallas

Abstract

Natural-language audio editing requires connecting a user's intent to specific sounds and time regions while preserving surrounding content. Complex requests make this connection demanding: an edit may select one occurrence, depend on a temporal relation, or modify a source that overlaps with other sounds. We introduce Harmony, an agentic framework that bridges audio understanding and editing through collaboration among an LLM planner, an audio-language model, and external tools. Harmony combines two mechanisms. Edit-driven collaborative planning uses targeted questions and tool measurements to determine the target, timing, and operation parameters. Evidence-guided execution uses observations of the first separated or generated audio to trigger bounded corrections before final mixing. Together, these mechanisms connect semantic decisions to waveform operations throughout the editing process. Harmony supports addition, removal, replacement, and volume adjustment. On 420 MMAE examples spanning these operations, Harmony achieves an exact match rate of 22.62%, compared with 13.57% for the strongest baseline, and improves aggregate instruction following and context preservation.

Figures & tables

Explore similar work

CardsList
  1. AURA: Unified Multimodal Framework for Conversational Music Editing

    Sep 13, 2026Quoc-Huy Trinh, Minh-Van Nguyen, Debesh JhaAudio EditingControllable Music Generation

  2. Bagpiper-Edit: Zero-Shot Open-Ended Audio Editing via Rich-Caption

    Jun 19, 2026Xun Gong, Jinchuan Tian, Haoran Wang +3Audio EditingAudio Generation

  3. Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing

    Apr 12, 2026Zeyue Tian, Binxin Yang, Zhaoyang Liu +8Audio GenerationAudio Editing