SelectSep: Target-Selective Video Queries for Audio Source Separation

Suyeon Lee; Doyeop Kwak; Joon Son Chung

eess.ASOct 28, 2026

SelectSep: Target-Selective Video Queries for Audio Source Separation

Authors: Suyeon Lee, Doyeop Kwak, Joon Son Chung

Organizations: Korea Advanced Institute of Science and Technology, South Korea

Abstract

Video-queried sound separation (VQSS) aims to isolate a target sound from an audio mixture using its corresponding video as a query. However, scene-level video queries can become ambiguous when multiple sounding objects are visible, particularly when the target and interferer are semantically similar. We propose SelectSep, a target-selective VQSS framework that jointly uses text and a single click coordinate to specify the target source and instance without requiring segmentation at inference time. We adapt the visual encoder through target-feature distillation with text and click conditioning, and then fine-tune the separator using the adapted visual features. We also introduce an on-screen interference setting, where both the target and interferer are visible, to evaluate target selectivity. Experiments on standard benchmarks and the on-screen interference setting demonstrate consistent improvements in separation quality and target selectivity, particularly for mixtures containing semantically similar sources.

Figures & tables

Figure 1: Training pipeline of SelectSep. In stage 1, we fine-tune the visual encoder with text and click conditions for target selectivity at the video feature level. In stage 2, we freeze the student video encoder and fine-tune the separator backbone.

	Query			VGGSound-Clean						MUSIC-Clean
Method	T	V	I	FAD $\downarrow$	IS $\uparrow$	CLAP $\uparrow$	CLAP-A $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$	FAD $\downarrow$	IS $\uparrow$	CLAP $\uparrow$	CLAP-A $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
GT (reconstructed)				2.59	10.00	25.49	88.98	25.89	0.718	4.92	2.78	32.71	92.20	38.74	0.163
FlowSep [ 22 ]	✓	-	-	1.71	9.86	24.42	81.48	24.86	0.844	5.42	2.73	29.08	79.00	29.94	0.461
OmniSep [ 4 ]	✓	-	✓	1.74	9.30	22.67	81.07	26.31	0.835	4.73	2.45	32.29	84.17	35.59	0.303
DAVIS-Flow [ 8 ]	-	✓	-	4.59	8.40	17.06	62.62	21.68	0.950	5.72	2.64	31.83	48.61	35.31	0.374
MMAudioSep [ 18 ]	✓	✓	-	3.54	8.59	21.71	76.88	22.51	0.769	5.50	2.70	32.10	86.54	37.68	0.232

Table 1: Source separation results. T, V, and I denote text, video, and image.

\dagger

denotes the target-selective video query conditioned on click and text inputs. Best and second-best scores are bolded and underlined.

Inter-class
Method	FAD $\downarrow$	IS $\uparrow$	CLAP $\uparrow$	CLAP-A $\uparrow$	IB-Score $\uparrow$	DeSync $\downarrow$
MMAudioSep [ 18 ]	15.92	7.10	16.49	63.62	20.56	0.677
+ Mixture-centered prior	16.35	7.24	16.76	63.46	22.96	0.647
+ Text	10.60	9.25	22.85	74.17	27.30	0.588
+ Click (Ours)	7.98	9.42	23.76	77.59	28.33	0.537
Intra-class

Table 2: Ablation study on VGG-MonoAudio. Components are added cumulatively.

Figure 2: Attention maps of the visual encoder under each conditioning mode. The marker indicates the clicked position.

Audio Source Separation

Audio source separation isolates individual speakers, instruments, or sound events from mixtures for listening, analysis, and audio editing.

See all 67 papers

FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation
Aug 22, 2026Yi Yuan, Xubo Liu, Haohe Liu +3Audio Source Separation Flow Matching
School of Computer Science and Electronic Engineering, University of Surrey, Guildford, UK · Department of Informatics, King’s College London, London, UK
CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents
Sep 15, 2025Adhiraj Banerjee, Vipul AroraAudio Source Separation
Department of Electrical Engineering · Indian Institute of Technology, Kanpur
Cinematic Audio Source Separation Using Visual Cues
Mar 27, 2026Kang Zhang, Suyeon Lee, Arda Senocak +1Audio Source Separation
School of Electrical Engineering, KAIST · Graduate School of Artificial Intelligence, UNIST

Aug 22, 2026J/K move · Enter open · S save

FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation

Yi Yuan, Xubo Liu, Haohe Liu, Xiyuan Kang+2

School of Computer Science and Electronic Engineering, University of Surrey, Guildford, UK · Department of Informatics, King’s College London, London, UK

Audio Source Separation Flow Matching

Open paper AskPDF arXiv

Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.

Abstract

Audio Source Separation

Explore similar work

FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation

CodecSep: Prompt-Driven Universal Sound Separation on Neural Audio Codec Latents

Cinematic Audio Source Separation Using Visual Cues