AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Duojia Li; Shuhan Zhang; Zihan Qian; Wenxuan Wu; Shuai Wang; Qingyang Hong; Lin Li; Haizhou Li

cs.SDMar 11, 2026

AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Authors: Duojia Li, Shuhan Zhang, Zihan Qian, Wenxuan Wu, Shuai Wang, Qingyang Hong, Lin Li, Haizhou Li

Organizations: School of Electronic Science and Engineering, Xiamen University, Xiamen, China · Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China · School of Informatics, Xiamen University, Xiamen, China

Abstract

In target speaker extraction (TSE), we aim to recover target speech from a multi-talker mixture using a short enrollment utterance as reference. Recent studies on diffusion and flow-matching generators have improved target-speech fidelity. However, multi-step sampling increases latency, and one-step solutions often rely on a mixture-dependent time coordinate that can be unreliable for real-world conversations. We present AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective. AlphaFlowTSE learns mean-velocity transport along a mixture-to-target trajectory starting from the observed mixture, eliminating auxiliary mixing-ratio prediction, and stabilizes training by combining flow matching with an interval-consistency teacher-student target. Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).

Figures & tables

Figure 1: Overall architecture of AlphaFlowTSE. Given a mixture waveform $y$ and an enrollment utterance $e$ , we compute complex STFT features and form the mixture feature $Y$ and enrollment feature $E$ (real/imaginary concatenation). During training, the backbone takes the current state feature $z_{t}$ ; during inference we initialize $z_{0}=Y$ . The enrollment feature is concatenated as a temporal prefix, yielding $[E\!\parallel\!z_{t}]$ (or $[E\!\parallel\!z_{0}]$ at inference), which is fed to the UDiT backbone. The backbone is conditioned via AdaLN on the absolute time $t$ and the interval length $\Delta=r-t$ (with $r=1$ at inference), and predicts the mean velocity for finite-interval transport, denoted $u_{\theta}(t,r,[E\!\parallel\!z_{t}])$ . One-step inference ( NFE $=1$ ) produces an estimated complex STFT $\hat{S}=(\hat{S}_{\mathrm{Re}},\hat{S}_{\mathrm{Im}})$ , which is converted to the target waveform $\hat{s}$ by iSTFT. The dashed module is an optional mixing-ratio predictor used only in the background-to-target ablation to predict the start coordinate $\hat{\tau}$ .

Method	Libri2Mix Clean					Libri2Mix Noisy
	PESQ $\uparrow$	ESTOI $\uparrow$	SI-SDR $\uparrow$	OVRL $\uparrow$	SpkSim $\uparrow$	PESQ $\uparrow$	ESTOI $\uparrow$	SI-SDR $\uparrow$	OVRL $\uparrow$	SpkSim $\uparrow$
Mixture	1.15	0.54	0.00	2.65	0.54	1.08	0.40	-1.93	1.63	0.46
DiffSep+SV [ 11 ]	1.85	0.79	–	3.14	0.83	1.32	0.60	–	2.78	0.62
DDTSE [ 11 ]	1.79	0.78	–	3.30	0.73	1.60	0.71	–	3.28	0.71
DiffTSE [ 10 ]	3.08	0.80	11.28	–	–	–	–	–	–	–
FlowTSE [ 16 ]	2.58	0.84	–	3.27	0.90	1.86	0.75	–	3.30	0.83

Table 1: Libri2Mix benchmark results ( min , 16 kHz).

\uparrow

indicates higher is better. DNSMOS OVRL is the DNSMOS-P.835 overall score and SpkSim denotes speaker similarity. Results of prior systems are taken from the literature [ 16 , 10 , 11 , 27 , 28 , 17 , 18 ] . For MR-indexed one-step systems, the default setting uses an MR predictor at inference.

Split	Method	MR pred.	DNSMOS				PESQ $\uparrow$	ESTOI $\uparrow$	SI-SDR $\uparrow$	SpkSim $\uparrow$	Relative decline ( w/o vs. w/ )
		MR pred.	SIG $\uparrow$	BAK $\uparrow$	OVRL $\uparrow$	P808 $\uparrow$	PESQ $\uparrow$	ESTOI $\uparrow$	SI-SDR $\uparrow$	SpkSim $\uparrow$	$\Delta$ OVRL(%)	$\Delta$ PESQ(%)	$\Delta$ SI-SDR (dB)
clean	AD-FlowTSE [ 17 ]	w/o	–	–	3.02	3.44	2.33	0.82	12.54	0.92	-4.1%	-19.4%	-4.95
		w/	3.47	3.90	3.15	3.59	2.89	0.90	17.49	0.95	-4.1%	-19.4%	-4.95
	MeanFlowTSE [ 18 ]	w/o	3.31	3.37	2.77	3.31	1.53	0.53	-6.00	0.60	-13.7%	-53.1%	-24.80
		w/	3.51	3.95	3.21	3.69	3.26	0.93	18.80	0.92	-13.7%	-53.1%	-24.80
	AlphaFlowTSE	w/o	3.52	3.94	3.16	3.61	3.04	0.92	18.50	0.92	-2.5%	-7.0%	-0.67

Table 2: MR-predictor ablation on Libri2Mix (NFE

=1

\uparrow

indicates higher is better. “w/” and “w/o” denote inference with and without MR prediction, respectively; relative-decline columns are computed as in the table header. Best values are in bold.

Method	NFE	Params (M)	Aux Params (M)
DiffSep+SV [ 11 ]	60	66	6.63
DDTSE [ 11 ]	10	71	6.63
SR-SSL [ 27 ]	5	431	–
SoloSpeech [ 28 ]	50	589	–
AD-FlowTSE [ 17 ]	1 or 5	342	15.57 (MR predictor)
MeanFlowTSE [ 18 ]	1	343	15.57 (MR predictor)

Table 3: Inference cost and model size comparison. NFE denotes the number of separator network function evaluations at test time. “Params” reports the separator (backbone) parameters, while “Aux Params” reports additional parameters required at inference (e.g., an MR predictor).

English subsets
Dataset	$Samples(N)$	DNSMOS OVRL $\uparrow$ (w/o MR predictor)			DNSMOS OVRL $\uparrow$ (w/ MR predictor)
		AD-FlowTSE	MeanFlowTSE	AlphaFlowTSE	AD-FlowTSE	MeanFlowTSE	AlphaFlowTSE
AMI	592	1.837	2.178	1.820	1.799	2.120	2.169
CHiME-6	545	1.460	1.174	1.843	1.597	1.896	1.858
DipCo	133	1.346	1.193	1.560	1.252	1.475	1.515
Avg.	1270	1.624	1.644	1.803	1.655	1.956	1.967

Table 4: DNSMOS OVRL on REAL-T [ 22 ] (higher is better). Models are trained on Libri2Mix noisy and evaluated zero-shot on REAL-T. We report inference without an MR predictor (w/o MR predictor) and with an MR predictor imported from synthetic training (w/ MR predictor), since REAL-T provides no MR labels.

Figure 2: ASR error rates on REAL-T under two inference settings: (a) w/o MR predictor and (b) w/ MR predictor. Left panels report English WER (average and subsets: AMI, CHiME-6, DipCo), and right panels report Chinese CER (average and subsets: AISHELL-4, AliMeeting) for AD-FlowTSE, MeanFlowTSE, and AlphaFlowTSE. Lower is better.

Figure 3: Speaker similarity (SIM / SpkSim) on REAL-T under two inference settings: (a) MR-free and (b) w/ MR predictor. Left panels report English SpkSim (average and subsets: AMI, CHiME-6, DipCo), and right panels report Chinese SpkSim (average and subsets: AISHELL-4, AliMeeting) for AD-FlowTSE, MeanFlowTSE, and AlphaFlowTSE. Higher is better.

Explore similar work

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching
Mar 13, 2026Junwon Moon, Seungbeom Kim, Hansol Park +4Target Speaker Extraction Flow Matching
Sungkyunkwan University · University of Seoul
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China
StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model
Apr 21, 2026Shuhai Peng, Hui Lu, Jinjiang Liu +8Target Speaker Extraction Speech Processing
Shenzhen International Graduate School, Tsinghua University · The Chinese University of Hong Kong · SenseTime Research

Mar 13, 2026J/K move · Enter open · S save

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

Junwon Moon, Seungbeom Kim, Hansol Park, Hyunjin Choi+3

Sungkyunkwan University · University of Seoul

Target Speaker Extraction Flow Matching Speech Processing Speech Enhancement

Open paper AskPDF arXiv

Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech given a reference utterance. Existing masking-based approaches are lightweight and effective but suffer from an inability to synthesize missing content, leading to degraded perceptual quality. On the other hand, recent generative TSE models typically synthesize high-quality speech with diffusion, but require numerous iterative steps resulting in high computational costs and latency. We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms. We introduce the deletion/insertion (D/I) proportion, an analytical tool that reveals early flow steps predominantly remove signal components rather than synthesize them. Based on this finding, we decouple deletion from insertion: a masking-based module handles the deletion-dominant early steps, while a single flow-matching step performs the remaining insertion for high-quality reconstruction. Specifically, the first stage uses lightweight convolution for the masking module, while the second stage employs a Diffusion Transformer (DiT) adapted for TSE with speaker conditioning. Unlike prior approaches that start from Gaussian noise, our method starts from the masked spectrogram, enabling high-quality reconstruction in a single inference step. Experiments show that Mask2Flow-TSE produces high-quality extractions with only 85M parameters and one-step inference, while preserving clean single-speaker inputs with minimal degradation.

Abstract

Explore similar work

Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching

Training-Free Multi-Step Inference for Target Speaker Extraction

StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model