AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Target speaker extraction (TSE) extracts the target speaker's voice from overlapping speech given a reference utterance. Existing masking-based approaches are lightweight and effective but suffer from an inability to synthesize missing content, leading to degraded perceptual quality. On the other hand, recent generative TSE models typically synthesize high-quality speech with diffusion, but require numerous iterative steps resulting in high computational costs and latency. We propose Mask2Flow-TSE, a two-stage framework combining the strengths of both paradigms. We introduce the deletion/insertion (D/I) proportion, an analytical tool that reveals early flow steps predominantly remove signal components rather than synthesize them. Based on this finding, we decouple deletion from insertion: a masking-based module handles the deletion-dominant early steps, while a single flow-matching step performs the remaining insertion for high-quality reconstruction. Specifically, the first stage uses lightweight convolution for the masking module, while the second stage employs a Diffusion Transformer (DiT) adapted for TSE with speaker conditioning. Unlike prior approaches that start from Gaussian noise, our method starts from the masked spectrogram, enabling high-quality reconstruction in a single inference step. Experiments show that Mask2Flow-TSE produces high-quality extractions with only 85M parameters and one-step inference, while preserving clean single-speaker inputs with minimal degradation.

Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.

While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications. Direct adaptation to streaming scenarios often leads to catastrophic inference performance degradation due to the severe mismatch between training and streaming inference. To bridge this gap, we present the first autoregressive (AR) models tailored for streaming TSE. Our approach introduces a Chunk-wise Interleaved Splicing Paradigm that ensures highly efficient and stable streaming inference. To ensure the coherence between the extracted speech segments, we design a historical context refinement mechanism that mitigates boundary discontinuities by leveraging historical information. Experiments on Libri2Mix show that while AR generative baseline exhibits performance degradation at low latencies, our approach maintains 100% stability and superior intelligibility. Furthermore, our streaming results are comparable to or even surpass offline baselines. Additionally, our model achieves a Real-Time-Factor (RTF) of 0.248 on consumer-level GPUs. This work provides empirical evidence that AR generative backbones are viable for latency-sensitive applications through the Chunk-wise Interleaved Splicing Paradigm.