What Selects, What Reconstructs: Repairing Exemplar-Based Complex-Spectrum Separation

Maxime Baelde

eess.SPSep 4, 2026

What Selects, What Reconstructs: Repairing Exemplar-Based Complex-Spectrum Separation

Authors: Maxime Baelde

Abstract

Exemplar methods separate a mixture by picking one learned spectrum per source and deforming it until it explains the observation, making one deformation class both reconstructor and selector. We show that the second role is empty as soon as the class can interpolate: the rule then ranks candidates on its regulariser, a choice made before the data, and the estimates sum back to the mixture whichever candidate wins. The condition is a parameter count, so the diagnosis runs before any experiment. On free per-bin deformation of complex spectra it explains the observed pathologies at once: a criterion that ranks candidates by their loudness, and half an output that is a mask on the mixture rather than an exemplar. The same theorem prescribes the repair, a selection class poorer than the reconstruction class: one complex gain and one pure delay rank the candidates, and a local combination of the best-aligned atoms, fitted jointly in closed form, rebuilds them. On MUSDB18 against the exact ceiling of the masking class, the distance between the criterion and an oracle inside its own candidate pool falls under the rigid selector from 6.2-7.7 to 0.5-2.8 dB, though only 0.9-1.2 dB of that reaches the output, and the per-frame latency of the deployed rule by a factor of 47 to 806. One lock remains, quantified: atoms are scored against the mixture, so the score carries a term for the other source that absorbs the capacity the reconstruction class gains, leaving the output 10.0 dB under the ceiling. Ranking hypotheses by the residual of a fit free enough to interpolate ranks them on the regulariser alone.

Figures & tables

Fig. 1: The alignment step, on a synthetic atom of known delay and gain. Left, the phase of the atom against that of the mixture; middle, the same once the ramp ( 8 ) is removed; right, the residual left after gain and delay are removed exactly. That residual is what the rigid class cannot explain, and what makes the criterion informative.

Fig. 2: The diagnosis. Gain over the mixture against dictionary size, both sources, for the original criterion and for an oracle picking the pair that minimises the true error in the same candidate pool. The criterion degrades as the dictionary grows while its own oracle improves, which no rule short of capacity does, and the gap is the same on both sources, as Corollary 1 predicts. Dashed lines are the derived ceilings,

m^{\star}

and the ideal ratio mask, on the tracks each rule was measured on.

Fig. 3: The selection gap before and after the phase constraint. Left, the oracle-to-criterion gap against dictionary size, under the free per-bin deformation and under the rigid class ( 8 ), both sources; right, the quality each criterion delivers. The gap collapses and its symmetry between sources breaks, the signature Corollary 1 predicts, while the criterion’s own quality now grows with dictionary size.

Fig. 4: Quality against capacity as

k

grows, both sources, 300 atoms per source. Capacity is the projection of the true source onto the span of

k

atoms aligned and chosen against the truth; quality is the same class populated by the deployed rule. The hatched selection lock absorbs the entire 4.4 dB that capacity gains between

k=1

and

k=16

		vocals		accompaniment
rule	class	$\Delta_{\mathrm{mix}}$	$\Delta_{\star}$	$\Delta_{\mathrm{mix}}$	$\Delta_{\star}$	latency
Def-MAP [ 7 ]	free per bin	$+5.71$	$-11.26$	$+0.38$	$-11.26$	1037.2
oracle pair, free deformation	free per bin	$+13.41$	$-3.56$	$+8.07$	$-3.56$	n/a
rigid pair, $k=1$	gain and delay	$+5.41$	$-11.56$	$-0.93$	$-12.57$	12.2
local combination, $k=4$	local, $\alpha=0$	$+6.11$	$-10.86$	$+0.20$	$-11.43$	10.7
local combination, $k=4$ , $\alpha>0$	local, $\alpha=0.75$	$+6.93$	$-10.04$	$+1.47$	$-10.16$	10.7

TABLE I: Main results on the MUSDB18 test split, fifty tracks on five-second excerpts, 300 atoms per source where a dictionary applies. No excerpt was dropped by the silence guard, no track by a missing reference.

\Delta_{\mathrm{mix}}

is the gain over the mixture,

\Delta_{\star}

the distance to the best real mask, both paired track by track. Latency is the per-frame time of the deployed rule alone. Rules whose estimates sum to the mixture read the same

\Delta_{\star}

on both sources: their per-source SDR difference is then the constant of Section V-B , 5.33 dB here, also the gap between the two

m^{\star}

columns.

atoms/src	Def-MAP	ramp	local $k{=}4$	local $k{=}16$	align + score
50	243.4	5.4	5.2	5.9	4.3 + 1.2
100	1037.2	12.2	10.7	11.6	9.1 + 2.2
300	9513.4	65.7	44.7	45.3	39.0 + 26.3
1000	103798.0	316.4	128.8	128.2	116.6 + 180.3

TABLE II: Per-frame latency in milliseconds, single core,

F=513

; one frame of 1024 samples at 44.1 kHz lasts 23 ms. The last column splits the repaired rule between alignment and pair scoring.

Audio Source Separation

Audio source separation isolates individual speakers, instruments, or sound events from mixtures for listening, analysis, and audio editing.

See all 65 papers

Sep 3, 2026

Geometric Ceilings on Time-Frequency Masking for Single-Channel Separation

Most single-channel separators estimate a source by applying a real gain to the mixture in each time-frequency bin. The optimum of that format, which the oracle masks used as bounds do not attain, is the orthogonal projection of the source onto the line spanned by the mixture, its residual set by the angle between them. Locating an estimator reduces to the block structure of a real-linear operator on stacked spectra, giving a chain of four nested classes whose three larger terms match three assumptions on the prior: zero means, circularity and absence of inter-frequency coupling. Held fixed the chain is a cascade of four orthogonal projections; refitted per frame it collapses onto its first term, attributing the whole residual to one missing real parameter per bin, the phase. When the phase posterior is symmetric about the mixture direction, the minimum mean-square estimate falls back onto the line, with gain the posterior mean of the oracle gain and excess error its variance. On MUSDB18 a posterior mean under a non-circular Gaussian-mixture prior leaves the class yet stays 11.44 dB under the per-frame ceiling, which four times as many components and 7.5x the data do not close; a closed-form gate attributes some 70% of it, in decibels, to the predicted variance. The widest fixed class stays 6.70 dB under the same ceiling. Leaving the class and minimising squared error are conflicting requests: the barrier lies in the criterion rather than in the prior.

Maxime Baelde

Independent researcher, Lille, France

Music Information Retrieval Audio Source Separation Speech Separation

Ask PDF

Oct 6, 2026

Restore, Separate, Restore: A Modular Framework for Music Source Restoration

Music Source Restoration (MSR) seeks to recover original, unprocessed instrument stems from mixed, mastered, and possibly degraded recordings. Unlike conventional source separation, which treats the mixture as a linear sum of clean sources, MSR must additionally invert nonlinear production effects, such as equalization and compression, and transmission-related degradations, such as codec artifacts. We propose a three-stage framework built around this distinction: (1) a mixture restoration model that addresses degradation before separation, (2) a single model that separates the restored mixture into eight target stems (vocals, guitars, keyboards, synthesizers, bass, drums, percussion, and orchestra), and (3) stem-specific restoration experts fine-tuned on the separator's own residual artifacts. Each stage improves restoration quality over the previous one on the MSR Challenge test set. We release code and models to support future research in MSR at https://github.com/theMoro/music_source_restoration.

Tobias Morocutti, Emmanouil Karystinaios, Gerhard Widmer

Institute of Computational Perception, Johannes Kepler University Linz, Austria · LIT Artificial Intelligence Lab, Linz, Austria

Audio Source Separation Music Source Separation

Ask PDF

Oct 5, 2026

SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation

Compact time-frequency separators that mask the mixture and refine through a shared cell face two limits. First, a bounded multiplicative mask only scales a mixture bin, so where overlapping components cancel, the estimate stays small. Second, a shared cell applies the same weights to every time-frequency token at every step, so enlarging it adds compute everywhere. We present SEAL (Sparse Expert routing with Additive Latent reconstruction) to address both. For reconstruction, a zero-sum additive residual bounded by the local mixture amplitude lets estimates be nonzero where components cancel yet still sum to the mixture. For routing, a query built from acoustic and inter-step evidence sends each token to one of six residual experts, and a norm cap keeps the step cue from overriding clear acoustic evidence. On EchoSet, SEAL (small) surpasses TIGER (small) by 0.31 dB SI-SDRi with 28% fewer parameters and 2.9 times fewer MACs, and SEAL (large) is within 0.07 dB SI-SDRi of TIGER (large) at 3.1 times fewer MACs.

Shao-Chun Hu, Zi-Xiang Lin, Jeih-Weih Hung +1

Department of Electrical Engineering, National Chi Nan University · Graduate Institute of AI Interdisciplinary Applied Technology, National Taiwan Normal University

Speech Separation Sparse Mixture-of-Experts

Ask PDF

Abstract

Audio Source Separation

Explore similar work

Geometric Ceilings on Time-Frequency Masking for Single-Channel Separation

Restore, Separate, Restore: A Modular Framework for Music Source Restoration

SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation