We present Textraction, a text-guided speaker extraction framework that enables the selective extraction of desired speakers or noise sources from an audio mixture using only natural-language descriptions. The proposed method operates within a single unified model that supports both single-channel and multi-channel inputs and it allows conditioning on either semantic textual descriptions of the target source or on its spatial description. Dedicated single- and multi-channel encoders feed a shared stack of time--frequency transformer blocks, which is conditioned on the prompt and on the recording configuration in every block through cross-attention and estimates the target directly from the mixture in one forward pass, without a separation stage or a source-selection stage. The system can extract an arbitrary subset of entities from a mixture, assuming up to two simultaneously active speakers and two noise sources, and interprets prompts specifying which entities to keep, which to remove, or both. In addition, we show that the framework naturally applies to a speech-enhancement setting, in which a single target speaker is present alongside background noise. To circumvent the application of computationally costly large language models used in prior work, we introduce a lightweight, task-adapted text encoder obtained by contrastively fine-tuning a compact text-retrieval model, which learns to be sensitive to instruction polarity (e.g., keep vs. remove) while remaining robust to paraphrasing. Experiments across attribute-based extraction, spatially guided extraction, and speech enhancement show that the proposed approach effectively isolates the desired entities under various acoustic and spatial conditions, and matches or surpasses dedicated spatial-extraction and enhancement baselines while offering substantially greater flexibility.
Figures & tables
#
Base Spatial Prompts
1
The speaker positioned on the right
2
The speaker positioned on the left
3
The speaker located farther away
4
The speaker located closer
5
The speaker who is closer and oriented more to the left
6
The speaker who is closer and oriented more to the right
TABLE I: Base spatial prompts describing speaker position and orientation. Each base prompt is expanded into 1,000 free-form paraphrases.
Fig. 1: Overview of the proposed Textraction model. The mixture y is represented in the STFT real–imaginary domain and routed, according to the recording configuration, to the SC encoder (reference microphone) or to the MC encoder (both microphones); each encoder is a convolutional layer followed by a TF-Locoformer block. Five shared TF-Locoformer blocks follow. After the encoder and after every shared block, a cross-attention layer conditions the features on the pooled prompt embedding and the token-level states of the fine-tuned lightweight text encoder, together with a learned SC/MC embedding. A decoder outputs the target estimate s^ .
Positive
Negative
E5 (pre-trained)
0.9565
0.9889
E5 (fine-tuned)
0.9911
−0.5405
TABLE II: Cosine similarity between anchor prompts and their positive and negative counterparts, before and after fine-tuning the E5 model. After fine-tuning, opposite-polarity prompts are pushed apart while paraphrases remain close.
Input
Textraction
Task
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
MC Extraction
-0.03
0.575
1.54
76.1
11.11
0.797
1.61
33.3
SC Extraction
-0.03
0.575
1.53
75.8
10.83
0.789
1.61
34.4
Spatial Extraction
-3.83
0.474
1.41
100.8
10.53
0.857
2.36
29.2
MC Enhancement
5.70
0.792
1.71
22.1
18.66
0.960
2.30
8.2
SC Enhancement
5.55
0.790
1.70
21.1
18.15
0.955
2.33
9.7
TABLE III: Overall results across the five settings on the test items. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better. In the attribute-based extraction tasks, WER is computed over the items whose target contains a single speaker.
Input
Textraction
Action Category
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
Both Noises
-3.32
0.464
1.45
–
9.40
0.730
1.20
–
Both Targets
3.28
0.668
1.45
–
12.93
0.874
1.92
–
Both Targets + Noise
7.93
0.813
1.49
–
13.27
0.877
1.43
–
Single Noise
-7.95
0.282
1.49
–
3.99
0.491
1.15
–
Single Target
-3.58
0.503
1.61
81.7
11.70
0.856
2.30
33.4
TABLE IV: Per-configuration MC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
Input
Textraction
Action Category
SI-SDR
STOI
DNSMOS
WER
SI-SDR
STOI
DNSMOS
WER
Both Noises
-3.33
0.462
1.42
–
9.01
0.723
1.18
–
Both Targets
3.29
0.670
1.42
–
12.14
0.855
1.88
–
Both Targets + Noise
7.92
0.813
1.47
–
13.13
0.877
1.45
–
Single Noise
-7.94
0.281
1.47
–
3.61
0.480
1.17
–
Single Target
-3.88
0.497
1.61
81.5
11.23
0.843
2.27
36.3
TABLE V: Per-configuration SC extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better, reported only when the target contains a single speaker.
Single Noise
Both Noises
Signal
Ref.
Cap.
Ref.
Cap.
Input mixture
0.771
0.629
0.786
0.632
AudioSep [ 44 ]
0.898
0.677
0.908
0.671
CLAPSep [ 45 ]
0.854
0.660
0.861
0.652
Textraction
0.950
0.681
0.967
0.672
Reference noise
1.000
0.688
1.000
0.676
TABLE VI: Noise extraction on the SC and MC test items (640 single-noise and 244 both-noise items): cosine similarity in the EmbeddingGemma 2 space between the output and the reference noise audio (Ref.) and the requested noise captions (Cap.). Higher is better. AudioSep and CLAPSep use their released weights and receive the requested noise caption as the query.
Fig. 2: The three spatial configurations of the spatially guided extraction test set. The target speaker (T) and the interfering speaker (I) differ in both DOA and distance (“Both”), only in DOA (“DOA gap”), or only in distance from the two-microphone array (“Radius gap”). Two directional noise sources are present in every scene.
Method
Action Category
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
Both
-3.99
0.466
1.42
101.3
DOA gap
-3.75
0.475
1.41
104.3
Radius gap
-3.75
0.480
1.41
96.7
SSF
Both
5.00
0.689
1.75
66.2
DOA gap
3.87
0.650
1.79
57.4
SpatialNet
Both
5.70
0.71
1.98
62.4
TABLE VII: Spatially guided extraction results. SI-SDR (dB), STOI, and DNSMOS OVRL: higher is better; WER (%): lower is better. SSF is conditioned on the oracle DOA of the target; it has no “Radius gap” row because a DOA cue cannot distinguish speakers that differ only in distance. SpatialNet separates both speakers without a cue, and its outputs are assigned to the prompts by the oracle permutation.
Fig. 3: Spatially guided extraction SI-SDR as a function of the spatial separation between the two speakers. Left: DOA gap, over the scenes in which the speakers differ in DOA (“Both” and “DOA gap”); right: radius gap, over the scenes in which they differ in distance (“Both” and “Radius gap”). Each point is the mean over the test items (prompt and complementary prompt) in the bin.
Method
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
5.70
0.792
1.71
22.1
TF-GridNet
18.15
0.959
2.24
7.9
Textraction
18.66
0.960
2.30
8.2
TABLE VIII: MC enhancement results on the same test items. WER in %.
Method
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
Input
5.55
0.790
1.70
21.1
HDDEMUCS-TF
13.72
0.890
2.24
25.9
Textraction
18.15
0.955
2.33
9.7
TABLE IX: SC enhancement results. WER in %.
Task
Prompt
Items
SI-SDR ↑
STOI ↑
DNSMOS ↑
WER ↓
MC Extraction
All
620
10.83
0.792
1.63
35.6
Keep
708
11.11
0.800
1.61
32.3
Remove
670
11.37
0.799
1.58
32.1
SC Extraction
All
667
10.63
0.780
1.61
36.7
Keep
642
11.11
0.796
1.61
33.2
Remove
689
10.77
0.792
1.60
33.4
TABLE X: Attribute-based extraction by prompt formulation. WER is computed over the items whose target contains a single speaker.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 4: SI-SDR as a function of the share of misspelled words in the prompt, per typo type. Left: mean over SC, MC, and spatially guided extraction; right: mean over SC and MC enhancement. 0% is the clean dataset prompt.
Extraction
Enhancement
Prompt
SC
MC
Spatial
SC
MC
Avg.
SI-SDR (dB)
Clean prompt
9.93
9.84
8.67
16.90
17.26
12.52
Swap
6.70
6.08
6.28
16.25
16.62
10.38
Delete
5.94
6.47
6.21
16.49
16.92
10.41
Insert
5.86
5.72
6.10
16.18
16.54
10.08
Appendix
TABLE XI: SI-SDR (dB) and STOI with 25% of the words misspelled, per typo type and task, and with every instruction word or every description word misspelled. Avg.: mean over the five tasks.
Condition
Prompt
Clean prompt
Maintain the loud female speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running motor vehicle engine.
Swap
Maitnain the loud female spaeker, include the bell rniging sound, keep the sotf male speaker, adn exclude the rnuning motor vehicle engine.
Delete
Mintain te loud fmale speaker, include the bell ringing sound, keep the soft male speaker, and exclude the running mtor vehcle engne.
Insert
Maintain thue loud femalpe speakoer, incluide the bell ringing sokund, keep the soft male speaker, and exzclude the running motor vehicle engine.
Keyboard
Maintain the loud female speaker, incluce the bell ribging sound, keep the soft male speqker, amd exclude tge running motor vdhicle engine.
Merge words
Maintainthe loud femalespeaker, includethe bell ringingsound, keep the soft male speaker, and exclude the running motor vehicle engine.
Appendix
TABLE XII: One MC-extraction test prompt under every typo condition (25% of the words for the six typo types).