FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation
Authors: Yi Yuan, Xubo Liu, Haohe Liu, Xiyuan Kang, Mark D. Plumbley, Wenwu Wang
Organizations: School of Computer Science and Electronic Engineering, University of Surrey, Guildford, UK · Department of Informatics, King’s College London, London, UK
Language-queried audio source separation (LASS) aims to extract target sources from audio mixtures according to natural language descriptions, offering a flexible and scalable interface for audio source separation. However, most existing LASS methods rely on discriminative, mask-based models, which estimate masks from the input mixture. These methods often over-suppress target sounds or fail to fully separate them, especially when multiple sound events strongly overlap in complex acoustic scenes. In this work, we propose FlowSep2, a text-conditioned flow-matching generative model for LASS. Instead of directly predicting a separation mask, FlowSep2 learns to generate the target source representation from Gaussian noise in a latent space, conditioned on both the mixture representation and the text query. Specifically, we employ rectified flow matching with a Diffusion Transformer backbone. We further incorporate Self-Flow, a self-supervised flow-matching paradigm, into our LASS framework. By encouraging semantically structured latent representations under the generative objective, Self-Flow improves the model's ability to separate target sources according to text queries. Experiments on multiple LASS benchmarks show that FlowSep2 achieves state-of-the-art performance and demonstrates enhanced sound separation results in challenging scenarios with overlapping sound events.
Figures & tables
Fig. 1: The overall architecture of FlowSep2. Here, zm denotes the mixture latent, z0 denotes the Gaussian noise, z1 denotes the target latent, and zλ represents the intermediate latent along the flow trajectory. During training, the mixture latent zm is concatenated with the intermediate latent before being fed into the DiT flow module and the teacher receives a cleaner intermediate latent zλc to provide representation-level supervision. The teacher network is updated using EMA and the student network is optimized using both the flow matching loss LRFM and the representation loss Lrep .
Split
Dataset
Num.
Hours
Target Type
Train
VGGSound
183,727
510
Label
AudioCaps
49,837
140
Caption
AudioSetCaps
1,400,000
3,900
Caption
WavCaps
400,000
1,100
Caption
Test
VGGSound
2,000
5.6
Label
ESC-50
2,000
5.6
Label
TABLE I: Summary of datasets used for training and evaluation in FlowSep2.
Model
Training-Objectives
Encoder
Backbone
Params. (M)
AudioSep
Masking-Based
STFT
UNet
238.6
SAM-Audio
RFM
VAE
DiT
2,934
FlowSep
RFM
VAE
UNet
265.5
FlowSep2-UNet
RFM
VAE
UNet
352.1
FlowSep2-DDPM
DDPM
DiT
559.5
FlowSep2-RFM
RFM
VAE
DiT
558.2
TABLE II: Experimental setups and model sizes, SF denotes Self-Flow and RFM is short for Rectified Flow-matching
Model
FAD ↓
CLAP Score ↑
CLAP A Score ↑
AC
DE-S
VGG
ESC
AC
DE-S
DE-R
VGG
ESC
AC
DE-S
VGG
ESC
MUC
Unprocessed
−
−
−
−
31.7
42.7
41.7
33.6
38.6
54.4
66.3
62.7
66.8
50.2
LASS-Net [ 14 ]
5.09
1.83
3.09
3.28
33.9
44.4
44.8
36.4
40.5
65.2
72.1
64.5
75.6
−
AudioSep [ 20 ]
4.38
1.21
2.30
1.93
33.6
45.1
49.7
38.5
40.2
65.1
74.4
67.4
76.0
50.4
FlowSep [ 32 ]
2.86
0.90
2.06
1.49
41.9
46.4
50.3
39.5
42.2
76.7
75.6
69.2
75.7
54.1
SAM-Audio [ 48 ]
1.58
0.98
1.86
1.20
37.8
43.4
45.8
39.1
40.9
64.1
71.3
70.5
78.6
62.8
TABLE III: Objective evaluation on LASS, where AC, VGG, ESC, MUC are short for AudioCaps [ 63 ] , VGGSound [ 62 ] , ESC-50 [ 65 ] , and MUSDB18 [ 66 ] respectively. FAD cannot be applied to unprocessed data as it calculates the difference between two groups of audio, while the target and mixed audio share the same audio events.
Model
SAJ ↑
AA ↑
REL ↑
OVL ↑
AC
MUC
DE-S
AC
MUC
DE-S
DE-R
AC
MUC
DE-S
DE-R
AC
MUC
DE-S
DE-R
LASS-Net [ 14 ]
2.57
2.03
2.68
4.86
5.27
5.41
4.89
3.12
2.87
2.96
3.59
2.16
2.45
2.84
3.88
AudioSep [ 20 ]
2.84
2.97
2.93
5.18
5.94
6.03
5.57
3.66
3.99
3.24
3.93
2.69
3.85
3.53
4.02
SAM-Audio [ 48 ]
2.58
3.76
2.91
5.56
6.48
6.14
5.79
3.54
4.35
3.24
3.85
2.69
4.44
3.75
4.26
FlowSep [ 32 ]
2.73
3.08
3.02
5.36
6.07
5.83
5.28
4.08
3.71
3.62
4.11
3.98
3.84
3.72
4.26
FlowSep2-M
2.96
3.39
3.47
5.49
6.67
6.08
5.86
4.13
3.91
3.62
4.16
4.09
4.02
3.88
4.28
TABLE IV: Subjective and human-aligned evaluation results on LASS, where AC, DE-S and DE-R denote AudioCaps [ 63 ] , DCASE-Synth and DCASE-Real respectively. SAJ (SAM Audio Judge), AA (AudioBox Aesthetics), REL and OVL are evaluated using a 1 – 5 Likert scale. ↑ indicates that higher scores are better.
Backbone
Dif.
Sem.
Depth
FAD ↓
CLAP Score ↑
CLAP A Score ↑
AC
DE-S
VGG
ESC
AC
DE-S
DE-R
VGG
ESC
AC
DE-S
VGG
ESC
MUC
DiT
RFM
Self-Flow
16
0.88
0.72
1.32
1.15
44.9
47.5
51.0
42.0
44.5
80.5
78.5
72.0
80.6
58.5
UNet
RFM
-
18
1.68
1.03
1.85
1.58
40.5
45.4
49.4
39.1
42.1
74.5
75.8
70.1
77.1
54.8
DiT
DDPM
-
16
1.92
1.02
1.99
1.61
42.8
45.8
49.9
39.8
42.0
72.6
75.0
68.4
77.5
53.5
DiT
RFM
-
16
1.38
0.81
1.81
1.47
42.3
46.9
50.5
40.8
43.0
78.6
76.3
70.9
79.1
56.9
DiT
RFM
RAE
16
0.86
0.79
1.32
1.19
44.5
47.6
50.8
42.1
44.6
80.8
78.0
72.5
80.6
57.3
TABLE V: Ablation study on different design choices in FlowSep2, comparing backbone architectures (Backbone), diffusion training objectives (Dif.), semantic representation alignment strategies (Sem.), and model depths. Within each ablation group, only the modified design choices are highlighted in black, while unchanged settings are shown in gray. Performance values that improve upon the corresponding baseline (the first row of each ablation group) are underlined, and the best results are highlighted in bold.
Fig. 2: The comparison between different semantic-enhancement structures on different training steps. SRFM is short for Self-Flow
Fig. 3: A case study comparing separation results from different methods with the ground truth on the AudioCaps test set.
Language-queried audio source separation (LASS) enables extracting any sound source using natural language. However, adapting LASS models to application-specific sound classes is challenging due to noisy training data and limited semantic coverage of the CLAP-based control signals. We propose a framework comprised of an automated data engine for training-data curation and a two-stage optimization process for class-specific CLAP control signals. Our objective evaluations across seven sound classes show that data refinement and control signal optimization consistently improve source separation performance. Subjective evaluation with 17 participants further demonstrates perceptual improvements of the model trained with optimized control signals over baseline and similar commercial models.
The goal of single-channel source separation is to reconstruct K sources given their mixture. In supervised settings where vast amounts of clean source data are available, this challenging, ill-posed problem has been addressed successfully by generative diffusion and flow-based prior models. However, access to such clean source samples is often limited, and even when available, supervised models are vulnerable to domain shifts. To bridge this gap, we present Separation via Unsupervised Remixing Flow (SURF), an unsupervised flow matching approach for source separation that learns directly from observed mixtures. This method relies on a novel combination of state-of-the-art supervised flow matching and regression-based self-supervised techniques. At a high level, starting from a teacher model, we utilize a "remixing" step to bootstrap the learning of a student flow model from the teacher's estimates. We provide insights into the objectives optimized by this approach and draw a novel connection to the Wake-Sleep algorithm. Empirical evaluations on image and audio benchmarks demonstrate that SURF establishes a new state-of-the-art, significantly outperforming existing unsupervised methods. See our demo page for examples. https://google.github.io/df-conformer/surf/
Flow-matching transformers achieve strong audio separation, yet their attention dynamics are opaque. We adapt established causal-intervention principles into a deterministic, inference-time probing protocol for SAM Audio. Orthogonal probing uncovers a dual-pathway text-conditioning mechanism: additive injections control semantic identity, while cross-attention refines acoustic structure. We observe an asynchronous layerwise convergence: stable layers build temporal scaffolds early, whereas fast layers continue resolving artifacts during sampling. The model also attenuates temporal segmentation cues to maintain continuous-flow stability. Using these insights, we propose Layer-Selective Attention Caching (LSAC), a training-free acceleration method that caches attention in stable layers. Across acoustic complexities, LSAC cuts self-attention computation by about ~25% with negligible quality loss and yields up to 6.7x higher quality retention than naive step reduction.
Yuxuan Chen, Haoyuan Yu, Peize He
Jilin University · Hunan University · University of Electronic Science and Technology of China