AURA: Unified Multimodal Framework for Conversational Music Editing

Quoc-Huy Trinh; Minh-Van Nguyen; Debesh Jha

cs.SDSep 13, 2026

AURA: Unified Multimodal Framework for Conversational Music Editing

Authors: Quoc-Huy Trinh, Minh-Van Nguyen, Debesh Jha

Organizations: Aalto University · OpenRB Lab · Technical University of Denmark · University of South Dakota

Abstract

Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.

Figures & tables

Figure 1: Overall architecture of AURA. Given an image $I$ , a multi-turn conversation $T$ , and a reference audio $X_{\mathrm{ref}}$ , the multimodal LLM $g_{\mathrm{mllm}}$ produces a natural-language response together with $\langle\mathrm{EDIT\_CONCEPT}\rangle$ tokens that summarize the editing intent inferred from all modalities and the dialogue history. The projector $g_{\mathrm{proj}}$ maps the hidden states of these tokens into the music decoder’s conditioning space. The concept-to-audio module $g_{\mathrm{c2a}}$ fuses this concept embedding, together with the reference audio stream, into the hidden states of the frozen music decoder, which then generates the edited audio $X_{\mathrm{target}}$ .

		Slakh2100-test (in-domain)							MoisesDB (out-of-domain)
Task	Model	FAD $\downarrow$	CLAP $\uparrow$	KL $\downarrow$	SSIM $\uparrow$	P-Dem. $\uparrow$	SI-SDR $\uparrow$	SDRi $\uparrow$	FAD $\downarrow$	CLAP $\uparrow$	KL $\downarrow$	SSIM $\uparrow$	P-Dem. $\uparrow$	SI-SDR $\uparrow$	SDRi $\uparrow$
Add	M2UGen	4.25	0.29	1.19	0.08	0.38	–	–	4.64	0.28	1.04	0.08	0.45	–	–
	Instruct-MG	2.01	0.28	0.65	0.37	0.48	–	–	3.84	0.18	0.84	0.32	0.51	–	–
	LeVo	2.30	0.28	0.82	0.11	0.35	–	–	3.92	0.25	0.87	0.10	0.44	–	–
	AURA	0.52	0.29	0.23	0.78	0.63	–	–	0.84	0.34	0.22	0.71	0.57	–	–
Remove	M2UGen	3.00	0.27	1.28	0.09	0.36	$-46.01$	$-52.43$	4.49	0.28	1.37	0.09	0.27	$-43.21$	$-49.87$

Table 1: Comparison with SOTA music-editing methods, in-domain (Slakh2100-test) and out-of-domain (MoisesDB). SI-SDR is undefined for add . AURA substantially improves edit correctness and content preservation across both benchmarks.

Metric	Model	T1	T2	T3	Ctrl
SSIM $\uparrow$	M2UGen	0.10	0.08	0.10	0.11
	Instruct-MG	0.42	0.32	0.22	0.32
	LeVo	0.08	0.07	0.08	0.08
	AURA	0.84	0.79	0.54	0.49
SI-SDR $\uparrow$	M2UGen	$-44.8$	$-45.9$	$-46.1$	$-47.1$
	Instruct-MG	$-0.9$	$-8.7$	$-19.3$	$-13.7$

Table 2: Multi-turn editing benchmark MoisesDB (out-of-domain). Turns apply

-

drums,

+

bass,

+

guitar in sequence; each is scored against the exact intermediate target. The control (Ctrl) applies only the final edit to the original mix, without the intermediate turns.

	FAD $\downarrow$		CLAP $\uparrow$		SSIM $\uparrow$
Task	Text	Concept	Text	Concept	Text	Concept
Add	13.32	0.84	0.10	0.34	0.08	0.71
Remove	9.49	0.72	0.15	0.33	0.068	0.70
Extract	9.97	4.24	0.14	0.26	0.099	0.48

Table 3: Comparison of the effects of

\langle\text{EDIT\_CONCEPT}\rangle

tokens with fully LLM-generated token hidden states on the MoiseDB dataset. Best results are in bold .

Audio Editing

Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.

See all 55 papers

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Oct 7, 2026William Chen, Prem Seetharaman, Ke Chen +10Instruction-Based Video Editing Unified Multimodal Models
Adobe Research · Carnegie Mellon University
Bagpiper-Edit: Zero-Shot Open-Ended Audio Editing via Rich-Caption
Jun 19, 2026Xun Gong, Jinchuan Tian, Haoran Wang +3Audio Editing Audio Generation
Auditory Cognition and Computational Acoustics Lab, Shanghai Jiao Tong University, Shanghai, China · Language Technologies Institute, Carnegie Mellon University
Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing
Apr 12, 2026Zeyue Tian, Binxin Yang, Zhaoyang Liu +8Audio Generation Audio Editing
Hong Kong University of Science and Technology, Hong Kong SAR, China · WeChat Vision, Tencent Inc, China · Peking University, Beijing, China

Oct 7, 2026J/K move · Enter open · S save

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto+9

Adobe Research · Carnegie Mellon University

Instruction-Based Video Editing Unified Multimodal Models Cross-Modal Learning Multimodal Instruction Tuning Video Editing Benchmarks

Open paper AskPDF arXiv

Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.

Abstract

Audio Editing

Explore similar work

CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing

Bagpiper-Edit: Zero-Shot Open-Ended Audio Editing via Rich-Caption

Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editing