What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective

Wang Xiang; Bang Zeng; Ming Li

cs.SDOct 12, 2026

What Makes Codec Tokens Effective for Target Speaker Extraction? A Semantic-Acoustic Disentanglement Perspective

Authors: Wang Xiang, Bang Zeng, Ming Li

Organizations: Wuhan University · The Chinese University of Hong Kong, Shenzhen

Abstract

Target speaker extraction (TSE) aims to recover a target speaker's speech from a speech mixture using an enrollment utterance. Recent codec-based generative approaches formulate TSE as conditional speech token generation and achieve promising perceptual quality. However, most existing systems rely on unified codec tokens that entangle linguistic content, speaker characteristics, and acoustic details, forcing a single model to jointly handle target selection, content semantic, and acoustic reconstruction. In this paper, we investigate whether disentangled semantic-acoustic codec representations are better suited to TSE. We first show that simply replacing unified codec tokens with semantic-acoustic tokens is insufficient and can even degrade performance when both remain handled by a unified generative model. Motivated by this observation, we propose a task-aware discriminative-generative framework, where a discriminative branch extracts target-related semantic information, while a generative branch predicts coarse acoustic tokens and refines them under semantic guidance. This specialization separates what is said from how it sounds while coordinating both factors in a factorized codec space. Experiments on Libri2Mix show that our method reduces dWER from 0.388 to 0.118 compared with a unified-codec baseline, while maintaining competitive perceptual quality and speaker consistency. Ablation studies confirm that the gains arise from task-aware semantic-acoustic modeling rather than representation disentanglement alone.

Figures & tables

Figure 1: Motivation for task-aware semantic–acoustic codec-token modeling. (a) Unified codec-token modeling entangles linguistic content, speaker characteristics, and acoustic details within a single generative model. (b) Representation-level disentanglement alone remains insufficient when semantic and acoustic token streams are still handled by a unified modeling paradigm. (c) Our task-aware framework assigns discriminative modeling to target semantics and generative modeling to acoustic reconstruction, aligning each token stream with its distinct role in TSE.

Figure 2: Overview of the proposed framework. A discriminative branch provides target-related semantic representations, while an autoregressive model predicts coarse acoustic tokens. The semantic-conditioned refiner recovers the complete acoustic embedding, which is combined with the semantic representation for waveform reconstruction.

Model	Category	Codecs	DNSMOS $\uparrow$			NISQA $\uparrow$	SBERT $\uparrow$	dWER $\downarrow$	WavLM $\uparrow$	WeSpeaker $\uparrow$
Model	Category	Codecs	SIG	BAK	OVRL	NISQA $\uparrow$	SBERT $\uparrow$	dWER $\downarrow$	WavLM $\uparrow$	WeSpeaker $\uparrow$
Mixture	–	–	3.383	3.098	2.653	2.453	0.572	0.792	0.847	0.759
X-TF-GridNet	D	–	3.328	3.607	2.895	3.216	0.784	0.389	–	–
USEF-SepFormer	D	–	3.324	3.698	2.927	2.880	0.830	0.156	–	–
SpEx+	D	–	3.472	4.027	3.186	3.349	0.878	0.148	0.973	0.935
WeSep	D	–	3.486	3.838	3.118	3.892	0.895	0.123	0.980	0.945

Table 1: Results on Libri2Mix clean. “D” and “G” denote discriminative and generative models, respectively. “Codecs” denotes the neural audio codecs used by each generative system.

(a) Oracle reconstruction of clean target speech
Methods	Codec	Modeling	Role	DNSMOS $\uparrow$			NISQA $\uparrow$	SBERT $\uparrow$	dWER $\downarrow$	WavLM $\uparrow$	WeSpk $\uparrow$
Methods	Codec	Modeling	Role	SIG	BAK	OVRL	NISQA $\uparrow$	SBERT $\uparrow$	dWER $\downarrow$	WavLM $\uparrow$	WeSpk $\uparrow$
Oracle	FunCodec	–	–	3.534	3.894	3.185	4.164	0.981	0.080	0.991	0.930
Oracle	DualCodec	–	–	3.559	4.084	3.284	4.058	0.947	0.105	0.972	0.829
(b) Controlled comparison of downstream TSE modeling
LauraTSE	FunCodec	G	–	3.609	4.084	3.336	4.333	0.908	0.159	0.974	0.876

Table 2: What Makes Codec Tokens Effective for TSE: Disentanglement or Specialization? “G” and “D–G” denote generative and discriminative–generative modeling. “Role” indicates whether semantic and acoustic representations are assigned task–aware modeling specialization roles.

Variant	NISQA $\uparrow$	SBERT $\uparrow$	dWER $\downarrow$	WeSpk $\uparrow$
Full model	4.052	0.902	0.118	0.837
w/o sem condition	3.992	0.881	0.122	0.821
w/o aco refiner	3.612	0.767	0.495	0.772

Table 3: Ablation of semantic guidance and acoustic refinement.

Training	Decoding	OVRL $\uparrow$	NISQA $\uparrow$	dWER $\downarrow$	WeSpk $\uparrow$
Split	AR	3.225	4.055	0.119	0.834
Split	NAR	3.256	4.091	0.113	0.848
Joint	AR	3.260	4.052	0.118	0.837
Joint	NAR	3.258	4.092	0.097	0.844

Table 4: Analysis of training and decoding strategies.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 32 papers

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment
Oct 16, 2026Yutong Wen, Ke Chen, Prem Seetharaman +5Target Speaker Extraction
Adobe Research, USA · Massachusetts Institute of Technology, USA
AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow
Mar 11, 2026Duojia Li, Shuhan Zhang, Zihan Qian +5Target Speaker Extraction One-Step Generative Modeling
School of Electronic Science and Engineering, Xiamen University, Xiamen, China · Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, Shenzhen, China +3
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Oct 16, 2026J/K move · Enter open · S save

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

Yutong Wen, Ke Chen, Prem Seetharaman, Haonan Chen+4

Adobe Research, USA · Massachusetts Institute of Technology, USA

Target Speaker Extraction

Open paper AskPDF arXiv

Target speaker extraction (TSE) isolates a desired speaker from a multi-speaker mixture using cues that identify the target. Prior work has explored acoustic and textual cues, such as enrollment audio and different types of speaker descriptors, but typically relies on a single cue family. Yet no single cue is both broadly available and reliably discriminative. Moreover, most TSE systems adopt mask-prediction models, which are prone to audible artifacts and generalize poorly across acoustic conditions. To address these limitations, we propose PromptSep-Speaker, a generative TSE model based on latent diffusion that unifies four interchangeable cues: speaker temporal order, transcript, enrollment audio, and speaker caption. The generative formulation delivers higher separation quality and multi-modal prompting improves applicability across scenarios. To fully exploit each cue, we further introduce a frozen WavLM representation with an auxiliary alignment loss that supervises query-mixture cross-attention without extra labeling. Across four evaluation sets and a listening study, PromptSep-Speaker achieves state-of-the-art performance over TSE and universal source separation baselines, improving separation accuracy and quality across cue modalities.

Abstract

Target Speaker Extraction

Explore similar work

PromptSep-Speaker: Generative Target Speaker Extraction via Multimodal Prompting and Semantic Alignment

AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow

Training-Free Multi-Step Inference for Target Speaker Extraction