PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

Yutong Wen; Ke Chen; Prem Seetharaman; Haonan Chen; Oriol Nieto; Paris Smaragdis; Justin Salamon; Zeyu Jin

cs.SDOct 16, 2026

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

Authors: Yutong Wen, Ke Chen, Prem Seetharaman, Haonan Chen, Oriol Nieto, Paris Smaragdis, Justin Salamon, Zeyu Jin

Organizations: Adobe Research, USA · Massachusetts Institute of Technology, USA

Abstract

Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.

Figures & tables

Figure 1: PromptSep-Speaker architecture.

		SI-SDRi $\uparrow$				mel- $L_{2}$ $\downarrow$				DNSMOS $\uparrow$				F1 Decision Error $\uparrow$
System	Cue	CapSp.	L2Mix	LTTS-R	Internal	CapSp.	L2Mix	LTTS-R	Internal	CapSp.	L2Mix	LTTS-R	Internal	CapSp.	L2Mix	LTTS-R	Internal
DAE-TSE	transcript	2.13	11.13	2.41	1.21	41.70	18.50	39.30	41.90	2.37	2.61	2.67	2.68	0.69	0.82	0.68	0.62
Ours	transcript	8.73	6.15	11.41	12.47	3.73	5.41	1.90	1.88	2.74	3.21	2.93	2.89	0.93	0.93	0.94	0.95
SoloSpeech	enroll	1.70	11.59	6.59	8.47	24.93	6.89	19.15	16.85	2.80	3.26	2.86	2.82	0.76	0.93	0.80	0.84
WeSep	enroll	7.07	11.14	11.19	13.18	31.20	20.00	26.80	26.60	2.27	2.59	2.49	2.50	0.76	0.82	0.74	0.75
Ours	enroll	4.01	5.04	11.79	13.30	7.48	6.00	2.31	1.84	2.69	3.21	2.86	2.89	0.87	0.92	0.95	0.96

Table 1: Evaluation results. CapSp.: CapSpeech-Mix; L2Mix: Libri2Mix, averaged over its clean and noisy conditions; LTTS-R: LibriTTS-R-Mix; Internal: a separate internal corpus, unused in training. A dash marks a cue the corpus does not support.

System	Cue	SI-SDRi $\uparrow$	mel- $L_{2}$ $\downarrow$	DNSMOS $\uparrow$	F1 Dec. Err. $\uparrow$
Ours	transcript	10.87	2.50	2.85	0.94
Ours	transcript (partial)	6.57	5.85	2.87	0.89
Ours	transcript (full ov.)	6.10	2.07	2.85	0.94
Ours	spk id (2 spk)	10.10	3.06	2.80	0.94
Ours	spk id (3 spk)	4.72	7.47	2.85	0.83
Ours (na)	transcript	-21.43	20.18	2.79	0.56

Table 2: Ablation studies across tasks with results averaged over CapSpeech, LTTS-R and internal sets. Task variants include using partial / full-overlap transcripts and three speakers. Model ablations include na (without alignment loss) and td (reverses the alignment cross-attention from frame-to-cue to cue-to-frame).

System	Cue	Speaker Accuracy	Speech Quality	Content Consistency
DAE-TSE	transcript	$3.28\pm 0.10$	$3.55\pm 0.11$	$3.96\pm 0.11$
Ours	transcript	$\mathbf{4.39\pm 0.12}$	$\mathbf{4.16\pm 0.11}$	$\mathbf{4.44\pm 0.12}$
DAE-TSE	trans. full-ov.	$3.09\pm 0.12$	$3.14\pm 0.12$	$3.67\pm 0.14$
Ours	trans. full-ov.	$\mathbf{4.19\pm 0.15}$	$\mathbf{4.01\pm 0.12}$	$\mathbf{4.18\pm 0.15}$
DAE-TSE	trans. partial	$3.09\pm 0.10$	$3.41\pm 0.11$	$3.94\pm 0.11$
Ours	trans. partial	$\mathbf{4.20\pm 0.15}$	$\mathbf{4.04\pm 0.12}$	$\mathbf{4.17\pm 0.15}$

Table 3: Listening test on our internal set; MOS

\pm

95\%

CI reported.

Figure 2: Alignment-module attention (layer 2) between transcript tokens and mixture frames for one two-speaker mixture, shared color scale, with (a) and without (b) the auxiliary loss.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 32 papers

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities
Sep 28, 2026Xinyuan Qian, Yanghao Zhou, Ziyang Jiang +10Target Speaker Extraction Multimodal Fusion
School of Computer and Communication Engineering, University of Science and Technology Beijing, 100083, China · Department of Computer Science and Technology, Beijing Institute of Technology, China · The Chinese University of Hong Kong, Shenzhen, China +5
What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective
Oct 12, 2026Wang Xiang, Bang Zeng, Ming LiTarget Speaker Extraction Disentangled Representation Learning
Wuhan University · The Chinese University of Hong Kong, Shenzhen
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Sep 28, 2026J/K move · Enter open · S save

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

Xinyuan Qian, Yanghao Zhou, Ziyang Jiang, Yu Chen+9

School of Computer and Communication Engineering, University of Science and Technology Beijing, 100083, China · Department of Computer Science and Technology, Beijing Institute of Technology, China · The Chinese University of Hong Kong, Shenzhen, China+5

Target Speaker Extraction Multimodal Fusion

Open paper AskPDF arXiv

Target Speaker Extraction (TSE) is pivotal in speech communication and human-computer interaction, enabling the isolation of a specific speaker's voice from complex acoustic environments, i.e., the cocktail party scenario. Although traditional TSE systems conditioned on enrollment speech have progressed substantially, enrollment speech as a cue has inherent limitations. Its reliability degrades when the target and interfering speakers have similar voice characteristics, when intra-speaker variability (e.g. changes in emotion or speaking style) creates a mismatch between the enrollment and target speech, or when the enrollment itself is contaminated by noise or competing speakers. This review surveys deep-learning-based TSE from the perspective of auxiliary target cues drawn from multiple modalities. We organize existing methods according to five types of information used to isolate the target speaker: audio enrollment, visual, spatial, textual/semantic, and neural cues. We also trace the evolution from discriminative estimators to variational, diffusion, flow, codec, and foundation-model-based systems and summarize representative datasets and evaluation metrics. We review the benefits and limitations of different cues and discuss challenges involving synchronization, missing or unreliable observations, data scarcity, privacy, computational cost, and real-time operation. Finally, we summarize future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment. By jointly reviewing cue design, model architecture, training objectives, datasets, and evaluation metrics, this article provides an overview of the current landscape and open problems in multimodal TSE.

Abstract

Target Speaker Extraction

Explore similar work

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective

Training-Free Multi-Step Inference for Target Speaker Extraction