We present Textraction, a text-guided speaker extraction framework that enables the selective extraction of desired speakers or noise sources from an audio mixture using only natural-language descriptions. The proposed method operates within a single unified model that supports both single-channel and multi-channel inputs and it allows conditioning on either semantic textual descriptions of the target source or on its spatial description. Dedicated single- and multi-channel encoders feed a shared stack of time--frequency transformer blocks, which is conditioned on the prompt and on the recording configuration in every block through cross-attention and estimates the target directly from the mixture in one forward pass, without a separation stage or a source-selection stage. The system can extract an arbitrary subset of entities from a mixture, assuming up to two simultaneously active speakers and two noise sources, and interprets prompts specifying which entities to keep, which to remove, or both. In addition, we show that the framework naturally applies to a speech-enhancement setting, in which a single target speaker is present alongside background noise. To circumvent the application of computationally costly large language models used in prior work, we introduce a lightweight, task-adapted text encoder obtained by contrastively fine-tuning a compact text-retrieval model, which learns to be sensitive to instruction polarity (e.g., keep vs. remove) while remaining robust to paraphrasing. Experiments across attribute-based extraction, spatially guided extraction, and speech enhancement show that the proposed approach effectively isolates the desired entities under various acoustic and spatial conditions, and matches or surpasses dedicated spatial-extraction and enhancement baselines while offering substantially greater flexibility.
Figures & tables
#
Base Spatial Prompts
1
The speaker positioned on the right
2
The speaker positioned on the left
3
The speaker located farther away
4
The speaker located closer
5
The speaker who is closer and oriented more to the left
6
The speaker who is closer and oriented more to the right
TABLE I: Base spatial prompts describing speaker position and orientation. Each base prompt is expanded into 1,000 free-form paraphrases.
Fig. 1: Overview of the proposed Textraction model. The mixture y is represented in the STFT real–imaginary domain and routed, according to the recording configuration, to the SC encoder (reference microphone) or to the MC encoder (both microphones); each encoder is a convolutional layer followed by a TF-Locoformer block. Five shared TF-Locoformer blocks follow. After the encoder and after every shared block, a cross-attention layer conditions the features on the pooled prompt embedding and the token-level states of the fine-tuned lightweight text encoder, together with a learned SC/MC embedding. A decoder outputs the target estimate s^ .
Positive
Negative
E5 (pre-trained)
0.9565
0.9889
E5 (fine-tuned)
0.9911
−0.5405
TABLE II: Cosine similarity between anchor prompts and their positive and negative counterparts, before and after fine-tuning the E5 model. After fine-tuning, opposite-polarity prompts are pushed apart while paraphrases remain close.
Input
Textraction
Task
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
MC Extraction
-0.03
0.575
1.54
76.1
11.11
0.797
1.61
33.3
SC Extraction
-0.03
0.575
1.53
75.8
10.83
0.789
1.61
34.4
Spatial Extraction
-3.83
0.474
1.41
100.8
10.53
0.857
2.36
29.2
MC Enhancement
5.70
0.792
1.71
22.1
18.66
0.960
2.30
8.2
SC Enhancement
5.55
0.790
1.70
21.1
18.15
0.955
2.33
9.7
TABLE III: Overall results across the five settings on the test items. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better. In the attribute-based extraction tasks, WER is computed over the items whose target contains a single speaker.
Input
Textraction
Action Category
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
Both Noises
-3.32
0.464
1.45
–
9.40
0.730
1.20
–
Both Targets
3.28
0.668
1.45
–
12.93
0.874
1.92
–
Both Targets + Noise
7.93
0.813
1.49
–
13.27
0.877
1.43
–
Single Noise
-7.95
0.282
1.49
–
3.99
0.491
1.15
–
Single Target
-3.58
0.503
1.61
81.7
11.70
0.856
2.30
33.4
TABLE IV: Per-configuration MC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
Input
Textraction
Action Category
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
Both Noises
-3.33
0.462
1.42
–
9.01
0.723
1.18
–
Both Targets
3.29
0.670
1.42
–
12.14
0.855
1.88
–
Both Targets + Noise
7.92
0.813
1.47
–
13.13
0.877
1.45
–
Single Noise
-7.94
0.281
1.47
–
3.61
0.480
1.17
–
Single Target
-3.88
0.497
1.61
81.5
11.23
0.843
2.27
36.3
TABLE V: Per-configuration SC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
Single Noise
Both Noises
Signal
Ref.
Cap.
Ref.
Cap.
Input mixture
0.771
0.629
0.786
0.632
AudioSep [ 44 ]
0.898
0.677
0.908
0.671
CLAPSep [ 45 ]
0.854
0.660
0.861
0.652
Textraction
0.950
0.681
0.967
0.672
Reference noise
1.000
0.688
1.000
0.676
TABLE VI: Noise extraction on the SC and MC test items (640 single-noise and 244 both-noise items): cosine similarity in the EmbeddingGemma 2 space between the output and the reference noise audio (Ref.) and the requested noise captions (Cap.). Higher is better. AudioSep and CLAPSep use their released weights and receive the requested noise caption as the query.
Fig. 2: The three spatial configurations of the spatially guided extraction test set. The target speaker (T) and the interfering speaker (I) differ in both DOA and distance (“Both”), only in DOA (“DOA gap”), or only in distance from the two-microphone array (“Radius gap”). Two directional noise sources are present in every scene.
Method
Action Category
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
Both
-3.99
0.466
1.42
101.3
DOA gap
-3.75
0.475
1.41
104.3
Radius gap
-3.75
0.480
1.41
96.7
SSF
Both
5.00
0.689
1.75
66.2
DOA gap
3.87
0.650
1.79
57.4
SpatialNet
Both
5.70
0.71
1.98
62.4
TABLE VII: Spatially guided extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better. SSF is conditioned on the oracle DOA of the target; it has no “Radius gap” row because a DOA cue cannot distinguish speakers that differ only in distance. SpatialNet separates both speakers without a cue, and its outputs are assigned to the prompts by the oracle permutation.
Fig. 3: Spatially guided extraction SI-SDR as a function of the spatial separation between the two speakers. Left: DOA gap, over the scenes in which the speakers differ in DOA (“Both” and “DOA gap”); right: radius gap, over the scenes in which they differ in distance (“Both” and “Radius gap”). Each point is the mean over the test items (prompt and complementary prompt) in the bin.
Method
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
5.70
0.792
1.71
22.1
TF-GridNet
18.15
0.959
2.24
7.9
Textraction
18.66
0.960
2.30
8.2
TABLE VIII: MC enhancement results on the same test items. WER in %.
Method
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
5.55
0.790
1.70
21.1
HDDEMUCS-TF
13.72
0.890
2.24
25.9
Textraction
18.15
0.955
2.33
9.7
TABLE IX: SC enhancement results. WER in %.
Task
Prompt
Items
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
MC Extraction
All
620
10.83
0.792
1.63
35.6
Keep
708
11.11
0.800
1.61
32.3
Remove
670
11.37
0.799
1.58
32.1
SC Extraction
All
667
10.63
0.780
1.61
36.7
Keep
642
11.11
0.796
1.61
33.2
Remove
689
10.77
0.792
1.60
33.4
TABLE X: Attribute-based extraction by prompt formulation. WER is computed over the items whose target contains a single speaker.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 4: SI-SDR as a function of the share of misspelled words in the prompt, per typo type. Left: mean over SC, MC, and spatially guided extraction; right: mean over SC and MC enhancement. 0% is the clean dataset prompt.
Extraction
Enhancement
Prompt
SC
MC
Spatial
SC
MC
Avg.
SI-SDR (dB)
Clean prompt
9.93
9.84
8.67
16.90
17.26
12.52
Swap
6.70
6.08
6.28
16.25
16.62
10.38
Delete
5.94
6.47
6.21
16.49
16.92
10.41
Insert
5.86
5.72
6.10
16.18
16.54
10.08
Appendix
TABLE XI: SI-SDR (dB) and STOI with 25% of the words misspelled, per typo type and task, and with every instruction word or every description word misspelled. Avg.: mean over the five tasks.
Condition
Prompt
Clean prompt
Maintain the loud female speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running motor vehicle engine.
Swap
Maitnain the loud female spaeker, include the bell rniging sound, keep the sotf male speaker, adn exclude the rnuning motor vehicle engine.
Delete
Mintain te loud fmale speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running mtor vehcle engne.
Insert
Maintain thue loud femalpe speakoer, incluide the bell ringing sokund, keep the soft male speaker, and exzclude the running motor vehicle engine.
Keyboard
Maintain the loud female speaker, incluce the bell ribging sound, keep the soft male speqker, amd exclude tge running motor vdhicle engine.
Merge words
Maintainthe loud femalespeaker, includethe bell ringingsound, keep the soft male speaker, and exclude the running motor vehicle engine.
Appendix
TABLE XII: One MC-extraction test prompt under every typo condition (25% of the words for the six typo types).
Target speech extraction (TSE) aims to recover a target speaker's voice from a mixture. While recent text-prompted approaches have shown promise, most approaches assume fully overlapped mixtures, limiting insight into behavior across realistic overlap ratios. We introduce VorTEX (Various overlap ratio for Target speech EXtraction), a text-prompted TSE architecture with a Decoupled Adaptive Multi-branch (DAM) Fusion block that separates primary extraction from auxiliary regularization pathways. To enable controlled analysis, we construct PORTE, a two-speaker dataset spanning overlap ratios from 0% to 100%. We further propose Suppression Ratio on Energy (SuRE), a diagnostic metric that detects suppression behavior not captured by conventional measures. Experiments show that existing models exhibit suppression or residual interference under overlap, whereas VorTEX achieves the highest separation fidelity across 20-100% overlap (e.g., 5.50 dB at 20% and 2.04 dB at 100%) while maintaining zero SuRE, indicating robust extraction without suppression-driven artifacts.
Ro-hoon Oh, Jihwan Seol, Bugeun Kim
Department of Artificial Intelligence, Chung-Ang University, Republic of Korea
Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.
Yutong Wen, Ke Chen, Prem Seetharaman +5
Adobe Research, USA · Massachusetts Institute of Technology, USA
Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.
Zhenghai You, Ying Shi, Lantian Li +1
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China