MuseFlow-TSE: Music-Aware Target Speech Extraction via Cross-Source Band-Splitting and Data-to-Data Flow Matching

Lu Ma; Yongliang Wang; Zhewen Su; Xin Tian

cs.SDOct 21, 2026

MuseFlow-TSE: Music-Aware Target Speech Extraction via Cross-Source Band-Splitting and Data-to-Data Flow Matching

Authors: Lu Ma, Yongliang Wang, Zhewen Su, Xin Tian

Organizations: JD.com, Inc., Beijing, China

Abstract

Target speech extraction (TSE) isolates a speaker's voice given an enrollment utterance, but existing work is validated almost exclusively without music. Discriminative methods are content-faithful yet masking-bounded; generative methods improve naturalness but may hallucinate, drift in identity, and require many steps. We present MuseFlow-TSE, a discriminative deletion, generative insertion framework for music-bearing scenes. Stage-1 is a music-aware cross-source band-split extractor that estimates target, interfering speakers, music, and noise under conservation and music-leakage constraints. Stage-2 is a data-to-data flow refiner initialized by the discriminative estimate, negatively conditioned on interference stems, and run for 1--4 difficulty-adaptive Euler steps with an end-to-end fidelity objective.

Figures & tables

Figure 1: MuseFlow-TSE architecture. All spectra use the actual 512-point STFT with bins

0

–

255

retained (

F{=}256

). Stage-1 : band-split (

K{=}37

N_{b}{=}6

dual-path blocks with RoPE, FiLM-gated early split into

R{=}6

slots,

N_{c}{=}4

cross-source blocks, per-slot cIRMs. Stage-2 : 8 AdaLN-Zero DiT dual-path blocks with U-ViT skips over

[x_{t};\,X]

, conditioned on the flow time

t

\mathbf{e}_{\mathrm{tgt}}

, and the negative interference embedding

\mathbf{z}_{I}

; the difficulty head sets the step count

N

(dashed arrows: conditioning).

Method	SI-SDRi $\uparrow$	PESQ $\uparrow$	DNSMOS $\uparrow$	SIM $\uparrow$	WER $\downarrow$	MusicLeak $\downarrow$	NFE $\downarrow$
SpEx+ [ 11 ]	8.2	2.02	2.55	0.76	22.8	9.7	1
USEF-TSE [ 12 ]	9.1	2.18	2.68	0.79	20.5	8.9	1
TF-GridNet [ 13 ]	11.9	2.58	2.95	0.85	14.6	7.2	1
DPM-TSE [ 14 ]	9.6	2.35	3.05	0.77	19.8	8.4	50
SoloSpeech [ 2 ]	10.4	2.72	3.28	0.84	15.9	6.8	50 $\times$ 8
AD-FlowTSE [ 3 ]	10.8	2.66	3.18	0.83	16.4	7.0	${\sim}$ 8

Table 1: Main comparison (Complex scenario; WER/MusicLeak in %). SoloSpeech: 50 steps, 8 candidates.

#	Modification	Validates	Observed
1	Config-D $\to$ DG	drift-free refinement	SI-SDRi $+0.2$ dB; PESQ $+0.17$ ; SIM $+0.01$ ; DNSMOS preserved
2	Gaussian $\to$ $\hat{S}_{\mathrm{c}}$ start	start point	NFE 16 $\to$ 1.6; WER 16.8 $\to$ 12.2 %
3	w/o cross-source TF	decoupling	SI-SDRi 13.0 $\to$ 11.2 dB; cons. err. 0.021 $\to$ 0.048
4	w/o negative condition $\mathbf{z}_{I}$	anti-hallucination	MusicLeak 3.4 $\to$ 6.9 %; singing 2.1 $\to$ 5.4 %
5	w/o band-split	music modeling	MusicLeak 2.9 $\to$ 6.2 %; music SDR 9.4 $\to$ 6.1 dB
6	w/o conservation / grouped PIT	joint estimation	SI-SDRi $-1.2$ dB / $-0.9$ dB

Table 2: Ablation results on the Complex scenario (primary metrics; values are before

\to

after).

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 38 papers

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching
Mar 13, 2026Junwon Moon, Seungbeom Kim, Hansol Park +4Target Speaker Extraction Flow Matching
Sungkyunkwan University · University of Seoul
AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow
Mar 11, 2026Duojia Li, Shuhan Zhang, Zihan Qian +5Target Speaker Extraction One-Step Generative Modeling
School of Electronic Science and Engineering, Xiamen University, Xiamen, China · Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China +3
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Mar 13, 2026J/K move · Enter open · S save

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

Junwon Moon, Seungbeom Kim, Hansol Park, Hyunjin Choi+3

Sungkyunkwan University · University of Seoul

Target Speaker Extraction Flow Matching Speech Processing Speech Enhancement

Open paper AskPDF arXiv

Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech given a reference utterance. Existing masking-based approaches are lightweight and effective but suffer from an inability to synthesize missing content, leading to degraded perceptual quality. On the other hand, recent generative TSE models typically synthesize high-quality speech with diffusion, but require numerous iterative steps resulting in high computational costs and latency. We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms. We introduce the deletion/insertion (D/I) proportion, an analytical tool that reveals early flow steps predominantly remove signal components rather than synthesize them. Based on this finding, we decouple deletion from insertion: a masking-based module handles the deletion-dominant early steps, while a single flow-matching step performs the remaining insertion for high-quality reconstruction. Specifically, the first stage uses lightweight convolution for the masking module, while the second stage employs a Diffusion Transformer (DiT) adapted for TSE with speaker conditioning. Unlike prior approaches that start from Gaussian noise, our method starts from the masked spectrogram, enabling high-quality reconstruction in a single inference step. Experiments show that Mask2Flow-TSE produces high-quality extractions with only 85M parameters and one-step inference, while preserving clean single-speaker inputs with minimal degradation.

Abstract

Target Speaker Extraction

Explore similar work

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Training-Free Multi-Step Inference for Target Speaker Extraction