SMEBENCH: Benchmarking Neural Music Editing with Synthetized MIDI Edits

Matthieu Cervera; Francesco Paissan; Micael Antunes; Mirco Ravanelli; Ricard Marxer; Cem Subakan

cs.SDOct 27, 2026

SMEBENCH: Benchmarking Neural Music Editing with Synthetized MIDI Edits

Authors: Matthieu Cervera, Francesco Paissan, Micael Antunes, Mirco Ravanelli, Ricard Marxer, Cem Subakan

Organizations: Mila – Quebec Artificial Intelligence Institute · Concordia University · Université Laval · National University of Ireland Maynooth ·

Abstract

Neural music editing requires a system to perform a requested change while preserving the musical content that should remain unchanged. Existing evaluations often lack a paired target edit, making these requirements difficult to measure separately. We introduce SMEBench, a large-scale benchmark of controlled music edits constructed by modifying symbolic MIDI performances and rendering matched source and target audio. Each example contains a source, an instruction, a paired target, edit metadata, and complementary descriptions. The benchmark covers twelve instrumentation, articulation, dynamics, and pitch operations and includes a balanced 3,600-example test set together with a smaller real-audio complement derived from multitrack recordings. The paired targets allow source preservation, target agreement, and textual compatibility to be evaluated independently. We use these references to diagnose common audio-editing metrics and benchmark six representative editing systems, showing that the metric used to rank a system can materially change the conclusion. The dataset is available at our repository.

Figures & tables

Figure 1: Overview of the SMEBench construction recipe. A 10-second MIDI excerpt is analyzed to determine which edits and targets are applicable. The selected intervention is applied to the symbolic performance, after which the source and target are rendered with the same SoundFont and filtered for audibility and quality. Structured facts extracted from the MIDI, audio, and edit metadata ground four complementary textual descriptions. Examples that pass the audio and caption checks become benchmark entries containing a known intervention and a paired reference target.

Panel of Figure 2 — Figure 2: SMEBench edit taxonomy. Each of the twelve operations is sampled only when the excerpt contains an eligible target. Instrument-level dynamics and pitch operations may target one instrument or the complete mixture without changing the operation category.

Criterion	Mean	Responses	95% CI (participant bootstrap, n=17)
Target quality	4.23	236	[4.06, 4.39]
Alignment with source	4.65	236	[4.57, 4.73]
Alignment with instruction	3.99	236	[3.72, 4.23]
Pipeline-caption quality	3.70	174	[3.46, 3.98]

Table 1: Human validation of the benchmark targets and captions. Ratings use a five-point scale. We report means over all available responses; the edit-wise breakdown appears in Appendix E .

Figure 3: Metric distributions by edit group on the SMEBench test split. The twelve operations are displayed in eight groups: crescendo and decrescendo are grouped as Dynamics, while instrument and drum variants are grouped under Swap, Remove, and Extract. Boxes show the median and interquartile range; whiskers extend to 1.5 times the interquartile range.

	Test set		Real-audio complement
Metric	Source	Target	Source	Target
CLAP-Text (caption) $\uparrow$	0.249	0.278	0.134	0.123
CLAP-Text (instruction) $\uparrow$	–	0.160	–	0.125
CLAP-Text (normalized) $\uparrow$	0.291	0.308	–	–
Audiobox Aesthetics $\uparrow$	6.903	6.734	7.247	6.952
LPAPS (Src $\leftrightarrow$ GT) $\downarrow$	1.513		2.167

Table 2: Descriptive characterization of the source and paired target audio. These aggregate statistics quantify source–target separation but do not by themselves establish edit correctness.

Figure 4: System performance on SMEBench and SME-Real under edit-caption conditioning (top) and edit-instruction conditioning (bottom). Paired points represent the same editing pipeline on SMEBench (blue) and SME-Real (orange), with connecting lines showing the synthetic-to-real change. All panels report raw metric values; arrows indicate the preferred direction. Target and source metrics separately measure agreement with the intended edit and preservation of the source.

Conditioning	System	CLAP-T $\uparrow$	CLAP-A GT $\uparrow$	LPAPS GT $\downarrow$
Edit caption	SDEdit	0.296 [0.293, 0.299]	0.625 [0.621, 0.628]	4.969 [4.951, 4.986]
	DDIM	0.283 [0.279, 0.286]	0.652 [0.648, 0.655]	4.981 [4.964, 4.998]
	VCI	0.299 [0.296, 0.303]	0.614 [0.610, 0.618]	4.908 [4.887, 4.929]
	ZETA	0.287 [0.284, 0.290]	0.654 [0.651, 0.657]	4.937 [4.918, 4.956]
	SteerMusic	0.269 [0.266, 0.272]	0.691 [0.688, 0.694]	4.482 [4.458, 4.506]
	MusicGen	0.350 [0.347, 0.353]	0.492 [0.489, 0.497]	5.926 [5.914, 5.939]

Table 3: System rankings on the SMEBench test split. The system preferred by CLAP-Text is not the system preferred by either paired-target audio metric. Values are means with 95% bootstrap confidence intervals. Because some edits admit multiple valid realizations, target similarity measures agreement with the benchmark’s known controlled realization rather than exhaustive edit correctness. Bold indicates the best observed mean within each conditioning mode and metric.

Source	Paired target	SDEdit	DDIM

VCI	ZETA	SteerMusic	MusicGen

Figure 5: Log-frequency spectrograms for one held-out SMEBench example. The paired target is one controlled valid realization; the remaining panels show the six system outputs. All panels use the same rendering configuration and show the same 2-second interval (4–6 s).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Edit operation	$n_{\text{trials}}$	$n_{\text{part.}}$	Instr.-following	Consistency	Overall
Remove drums	11	11	4.27 [3.64, 4.82]	4.82 [4.55, 5.00]	4.91 [4.73, 5.00]
Crescendo	12	12	4.50 [4.00, 4.92]	4.75 [4.50, 5.00]	4.67 [4.42, 4.92]
Decrescendo	26	15	4.50 [4.11, 4.82]	4.88 [4.70, 5.00]	4.54 [4.23, 4.81]
Staccato	25	14	4.44 [4.04, 4.78]	4.68 [4.46, 4.88]	4.56 [4.28, 4.81]
Double instrument	24	14	4.42 [4.15, 4.70]	4.83 [4.68, 4.96]	4.54 [4.26, 4.79]
Extract drums	20	11	4.35 [4.00, 4.64]	4.70 [4.53, 4.86]	4.50 [4.19, 4.77]

Appendix

Table 4: Instruction-following, source consistency, and overall quality by edit operation (mean and 95% participant-clustered bootstrap confidence interval). Operations marked

\dagger

do not reliably clear the midpoint of the instruction-following scale and are excluded only from the headline perceptual aggregate.

Figure 6: Source–target LPAPS distance and CLAP-Audio similarity distributions on the SMEBench test split (

n=11{,}160

Figure 7: Global CLAP-Text distributions for the edit caption, instruction, and normalized caption on the SMEBench test split (

n=11{,}160

Panel of Figure 8 — Figure 8: Audio-similarity metrics by the within-excerpt perceived-usage rank of the edited instrument. Lower ranks correspond to targets with greater perceived usage.

Figure 9: CLAP-Text distributions by the within-excerpt perceived-usage rank of the edited instrument. The analysis includes operations that target an individual instrument.

Figure 10: System-level comparison on SMEBench and SME-Real. Color distinguishes benchmarks, while line style and marker distinguish edit-caption from edit-instruction conditioning. Values are globally min–max normalized across the displayed systems, benchmarks, and conditions. LPAPS is reversed after normalization so that farther from the center consistently indicates better performance.

Real-audio complement, edit-caption conditioning
Method	Aesth. $\uparrow$	CLAP-A GT $\uparrow$	CLAP-A Src $\uparrow$	LPAPS GT $\downarrow$	LPAPS Src $\downarrow$
Identity	7.247 [7.2010, 7.2926]	1.000	0.804 [0.7830, 0.8247]	2.167 [1.9977, 2.3384]	0.000
Ground Truth	6.947 [6.877, 7.016]	0.804 [0.783, 0.824]	1.000	0.000	2.167 [1.998, 2.338]
SDEdit	6.737 [6.675, 6.797]	0.490 [0.474, 0.507]	0.503 [0.488, 0.519]	5.093 [4.990, 5.192]	4.947 [4.842, 5.054]
DDIM	6.589 [6.497, 6.679]	0.418 [0.404, 0.434]	0.414 [0.399, 0.429]	5.195 [5.091, 5.298]	5.058 [4.951, 5.163]
VCI	6.036 [5.912, 6.146]	0.447 [0.425, 0.468]	0.426 [0.405, 0.448]	4.806 [4.690, 4.918]	4.638 [4.519, 4.756]

Appendix

Table 5: System-level audio fidelity and quality with 95% bootstrap confidence intervals. GT and Src denote comparisons against the paired target and source, respectively.

Real-audio complement, edit-caption conditioning
Method	CLAP-T Edit $\uparrow$	CLAP-T Inst. $\uparrow$	CLAP-T Norm. $\uparrow$
Ground Truth	0.123 [0.114, 0.132]	0.125 [0.110, 0.140]	–
SDEdit	0.235 [0.226, 0.245]	0.150 [0.140, 0.161]	–
DDIM	0.222 [0.212, 0.233]	0.178 [0.167, 0.189]	–
VCI	0.198 [0.185, 0.211]	0.171 [0.159, 0.182]	–
ZETA	0.227 [0.216, 0.239]	0.165 [0.155, 0.175]	–

Appendix

Table 6: System-level text fidelity with 95% bootstrap confidence intervals. Missing normalized captions for SME-Real are shown as dashes.

	Mistral accepts	Mistral rejects
Ref accepts	3,228 (38.2%)	492 (5.8%)
Ref rejects	2,066 (24.4%)	2,675 (31.6%)

Appendix

Table 7: Agreement between the original Qwen ( ref ) judge and Mistral ( mistral ) judge on 8,461 generated captions. Entries in the agreement matrix show the number of captions, with percentages in parentheses.

Audio Editing

Audio editing studies how to modify speech, music, and sound while controlling the change and preserving the rest.

See all 55 papers

MMAE: A Massive Multitask Audio Editing Benchmark
Jun 5, 2026Ziyang Ma, Ruiqi Yan, Ruiyang Xu +35Audio Editing
Shanghai Jiao Tong University · Shanghai Innovation Institute · Nanyang Technological University +5
BeatEdit: Symbolic Music Generation as Explicit Editing
Jul 13, 2026Haoyu Gu, Lekai Qian, Haowu Zhou +2Symbolic Music Generation Music Generation
School of Future Technology, South China University of Technology, Guangzhou, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China
AURA: Unified Multimodal Framework for Conversational Music Editing
Sep 13, 2026Quoc-Huy Trinh, Minh-Van Nguyen, Debesh JhaAudio Editing Controllable Music Generation
Aalto University · OpenRB Lab · Technical University of Denmark +1

Jun 5, 2026J/K move · Enter open · S save

MMAE: A Massive Multitask Audio Editing Benchmark

Ziyang Ma, Ruiqi Yan, Ruiyang Xu, Jie Fang+34

Shanghai Jiao Tong University · Shanghai Innovation Institute · Nanyang Technological University+5

Audio Editing

Open paper AskPDF arXiv

We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.

Abstract

Audio Editing

Explore similar work

MMAE: A Massive Multitask Audio Editing Benchmark

BeatEdit: Symbolic Music Generation as Explicit Editing

AURA: Unified Multimodal Framework for Conversational Music Editing