A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine

Amirhossein Hajavi; Hanhee Lee; Pushya Jain; Sky Qiao; Emmanuel Ko; Yuanhao Yu; Irina Kezele

cs.SDOct 7, 2026

A Study on Improving Multi-class Audio Source Separation Via Decoupled CLAP Query Optimization and an Automated Data Engine

Authors: Amirhossein Hajavi, Hanhee Lee, Pushya Jain, Sky Qiao, Emmanuel Ko, Yuanhao Yu, Irina Kezele

Organizations: Noah’s Ark Lab, Huawei Technologies Canada co.

Abstract

Language-queried audio source separation (LASS) enables extracting any sound source using natural language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.

Figures & tables

Figure 1: An overview of the proposed framework. (Top) The pipeline for the automated data engine which consists of an audio captioning and LLM-as-a-Judge component. (Bottom) The optimization process for control signals using an iterative procedure to select the candidate CLAP embeddings followed by a SGD optimizer.

	SDRi $\uparrow$			SI-SDRi $\uparrow$			CLAPScore $\uparrow$			CLAPScore A $\uparrow$
Target class	Baseline	Refined	Optimized	Baseline	Refined	Optimized	Baseline	Refined	Optimized	Baseline	Refined	Optimized
Ambient	-1.623	4.116	4.301	-0.004	10.349	10.677	0.043	0.028	0.034	0.559	0.708	0.721
Animal	6.502	7.749	7.911	12.148	13.730	13.863	0.210	0.218	0.215	0.763	0.744	0.767
Crowd	2.507	4.060	4.026	8.449	10.596	10.561	0.145	0.120	0.123	0.794	0.830	0.843
Music	5.378	6.142	6.232	11.475	12.316	12.435	0.207	0.197	0.198	0.780	0.753	0.765
Nature	0.071	4.285	4.083	5.050	10.547	10.366	0.146	0.142	0.144	0.766	0.866	0.869

Table 1: Performance comparison of the baseline, refined, and optimized control signals across target sound classes. Best results for each target class and metric are shown in bold.

\uparrow

indicates that higher values are better.

Figure 2: Subjective preference evaluation of the separation quality for the model trained using optimized control signal.

Explore similar work

Music-Source-Separation-Training (MSST): A Unified Framework for Training and Evaluating Music Demixing Models
Jul 25, 2026Roman Solovyev, Ilya Kiselev, Alexander Stempkovskiy +1Speech Separation Unified Framework
National Research University Higher School of Economics (HSE University), MIEM HSE · AlphaChip LLC. · Independent Researcher
CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
Sep 15, 2025Adhiraj Banerjee, Vipul AroraNeural Audio Codecs Speech Separation
Department of Electrical Engineering · Indian Institute of Technology, Kanpur
Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models
Jun 8, 2026Yuxuan Chen, Haoyuan Yu, Peize HeSpeech Separation Audio Understanding
Jilin University · Hunan University · University of Electronic Science and Technology of China

Jul 25, 2026, cs.SDJ/K move · Enter open · S save

Music-Source-Separation-Training (MSST): A Unified Framework for Training and Evaluating Music Demixing Models

Roman Solovyev, Ilya Kiselev, Alexander Stempkovskiy, Tatiana Gabruseva

National Research University Higher School of Economics (HSE University), MIEM HSE · AlphaChip LLC. · Independent Researcher

Speech Separation Unified Framework

Open paper AskPDF arXiv

Music Source Separation (MSS), the task of recovering individual sound components (stems) from a polyphonic mixture, is central to applications ranging from karaoke and remixing to audio restoration and content production. The separation quality depends on engineering decisions across the entire pipeline: model choice, training data preparation and augmentation, loss function and metrics choice, training configuration, validation, and post-processing. This paper presents MSST (Music-Source-Separation-Training) - a universal open-source framework for MSS tasks, which unifies training, validation, and inference for a broad range of modern demixing model families under a single, configuration-driven interface. The framework supports various model architectures, data preprocessing and augmentations, multiple loss functions and evaluation metrics, which helps with fast iterations and ablation studies. Additionally, the framework supports a range of practical techniques that improve separation quality, such as sliding-window inference with cross-fading, test-time augmentation, model ensembling, and fine-tuning via Low-Rank Adaptation (LORA). Our ablation studies demonstrate improvements of MSS using the above techniques. By consolidating these components into a reproducible, YAML-configurable framework, MSST lowers the barrier to systematic experimentation and enables rapid iteration from idea to verifiable result.

Abstract

Explore similar work

Music-Source-Separation-Training (MSST): A Unified Framework for Training and Evaluating Music Demixing Models

CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models