Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues

Wenxuan Wu; Shuhan Zhang; Shuai Wang; Haizhou Li

cs.SDSep 24, 2026

Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues

Authors: Wenxuan Wu, Shuhan Zhang, Shuai Wang, Haizhou Li

Abstract

Target speech extraction (TSE) typically trains a separate extractor per cue, and visual-cue systems often need corruption-matched training to remain robust under visual frame corruption. We show that one autoregressive LLM backbone, TSE-Omni, can serve both temporally synchronous cues (lip movements, co-speech gestures) and asynchronous cues (enrollment audio, text). TSE-Omni is driven by next-token prediction: each step predicts target speech semantic tokens from its own past outputs, which we term self-enrollment, forming a continuous target-speech context initialized by the enrollment cue (asynchronous audio or text, or a short visual prefix). This enables audio-visual compensation: the model uses synchronized visuals when intact and its token history when visual frames are missing. Under clean visuals, TSE-Omni matches strong discriminative and generative baselines (SpeechBERTScore 0.81 on VoxCeleb2 and 0.89 on LRS3 zero-shot) with higher DNSMOS. On the same VoxCeleb2 test set, after a 2 s clean visual start, removing the remaining visual frames leaves SpeechBERTScore at 0.81. It remains usable under sparse overlap and multi-speaker interference, and supports streaming inference. Project page: https://alexwxwu.github.io/tseomni-main/.

Figures & tables

Fig. 1: Cue types. Cues are either temporally synchronous with the mixture (lip movements, co-speech gestures) or temporally asynchronous (pre-recorded audio, text).

Fig. 2: Self-enrollment compensation: historical predicted target-speech semantic tokens are fed back as a running enrollment cue. This self-loop is inherent to AR decoding. TSE-Omni is the first to exploit it to compensate for corrupted visual cues without a corruption-matched training stage in AR LLM framework.

Fig. 3: Overview of the TSE-Omni training framework. (a) AR Stage : predicts target speech semantic tokens from the mixture, conditioned on a pre-recorded audio clip or temporally synchronized visual cues. A short initial visual segment (default 2 s) goes to the cue encoder for recognition. The visual integration module is used only for visual-cue TSE. After $a_{\mathrm{sos}}$ , visual tokens start at $v_{2}$ : each $\hat{a}_{t}$ ( $t\geq 2$ ) is conditioned on $e_{t-1}=f_{av}(\hat{a}_{t-1},v_{t})$ . (b) NAR Stage : reconstructs the waveform from the predicted semantic tokens, the mixture, and the cue.

Fig. 4: Visual frame corruption on the VoxCeleb2 evaluation protocol: full missing, partial occlusion, and low resolution. The first 2 s stay clean; later corrupted or missing frames are zeros.

Test set

Model

Param

Arch.

Obj.

Inp. rep.

Rate

Type

Cue

Semantic-Speaker-Quality

\uparrow

DNSMOS

\uparrow

SBS

SIM

NISQA

SIG

BAK

OVL

LRS3

USEV [ 34 ]

16.4M

RNN

Mask

Emb

800 Hz

0.83

0.96

2.95

3.24

3.61

2.80

AV-Sepf [ 37 ]

163M

Sepf.

Mask

Emb

2000 Hz

0.87

0.96

2.46

3.19

2.90

2.43

AV-Mamba [ 9 ]

126M

Mamba

Mask

Emb

800 Hz

0.89

0.95

2.82

3.23

3.34

2.66

AVDiffuSS [ 38 ]

241M

U-Net

Noise

STFT

125 Hz

D+G

0.65

0.87

3.18

3.35

3.45

2.78

FlowAVSE [ 39 ]

241M

U-Net

Vel.

STFT

125 Hz

D+G

0.74

0.93

2.66

3.31

2.89

2.51

TABLE I: Comprehensive performance benchmarking across LRS3, VoxCeleb2, and Libri2mix. LRS3 supports both visual and audio cues. Visual-cue baselines on LRS3 are trained in-domain; TSE-Omni (V) and TSE-Omni (A) LRS3 rows are zero-shot from the VoxCeleb2 core checkpoint. VoxCeleb2 rows are in-domain. On Libri2mix, audio baselines are trained in-domain; TSE-Omni (A) is zero-shot from the VoxCeleb2 core checkpoint; TSE-Omni (A+Pretrain) is initialized from LauraTSE and fine-tuned on VoxCeleb2. On LRS3 and VoxCeleb2, the best visual-cue value per metric is in bold (audio-cue TSE-Omni is listed for reference). On Libri2mix, the best value per metric is in bold. Higher is better. Param.: speech extractor backbone parameters; Arch.: speech extractor backbone architecture; Obj.: prediction objective; Inp. rep.: intermediate modeling representation; Rate: frame rate; Type: D = discriminative, G = generative; Cue: A = audio, V = visual.

Fig. 5: Speech semantic token compensation. Only the first 2 s of visual cues are available; later visual frames are zeros. TSE-Omni continues extraction from the historical predicted speech semantic tokens (self-enrollment), with the visual module still enabled.

Model	Cue	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
Model	Cue	SBS	SIM	NISQA	SIG	BAK	OVL
USEV [ 34 ]	V	$0.71$	$0.86$	$2.56$	$3.17$	$2.94$	$2.43$
AV-Sepf [ 37 ]	V	$0.70$	$0.84$	$1.96$	$3.04$	$2.54$	$2.21$
AV-Mamba [ 9 ]	V	$0.79$	$0.86$	$2.42$	$3.14$	$2.89$	$2.39$
AVDiffuSS [ 38 ]	V	$0.60$	$0.74$	$2.89$	$3.21$	$2.89$	$2.46$
FlowAVSE [ 39 ]	V	$0.62$	$0.76$	$2.69$	$3.11$	$2.96$	$2.46$

TABLE II: VoxCeleb2 core test set (

2{,}000

utterances), full-missing visual cues: first 2 s clean, later visual frames zeros. Same mixtures as the clean VoxCeleb2 test in Table I .

Fig. 7: Target-speaker switch on a visual cue: a switch token (SWS) is added to the vocabulary. The LLM detects a conflict between the new visual cue and the self-enrolled speech history, resets the AR context, and follows the new target.

Method	Visual	SWS	Sem.-Spk.-Qual $\uparrow$			DNSMOS $\uparrow$
Method	Visual	SWS	SBS	SIM	NISQA	OVL
USEV	vclean	–	$0.808$	0.949	$2.635$	$2.744$
TSE-Omni (V)		w/o	$0.774$	$0.928$	$3.495$	$3.082$
TSE-Omni (V)		w/	0.812	$0.936$	3.560	3.120
USEV	vocc	–	$0.736$	$0.926$	$2.477$	$2.650$
TSE-Omni (V)		w/o	$0.766$	$0.925$	$3.474$	$3.074$

TABLE III: Target speaker switch on LRS3 (fine-tuned from the VoxCeleb2 core checkpoint). “SWS” is the switch token. We report DNSMOS-OVL only; SIG/BAK follow the same trend. Best values within each visual condition are bold.

Model	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
Model	SBS	SIM	NISQA	SIG	BAK	OVL
USEV [ 34 ]	$0.626$	$0.879$	$1.774$	$3.037$	$2.327$	$2.087$
TSE-Omni (V)	$0.629$	$0.867$	$2.820$	$3.393$	$2.819$	$2.513$
TSE-Omni (A)	0.658	0.898	2.827	3.395	2.843	2.535

TABLE IV: Three-speaker mixtures on the VoxCeleb2 test set (zero-shot from the core checkpoint). Best values within each metric are bold.

Model	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
Model	SBS	SIM	NISQA	SIG	BAK	OVL
USEV [ 34 ]	0.774	0.872	$1.300$	$2.620$	$2.600$	$1.980$
TSE-Omni (V)	$0.750$	$0.821$	1.520	2.850	2.910	2.260
TSE-Omni (A)	$0.743$	$0.817$	$1.510$	$2.840$	$2.870$	$2.240$

TABLE V: Target speaker extraction under sparse overlap on IEMOCAP. TSE-Omni (V) uses clean visual cues; TSE-Omni (A) uses audio enrollment. Fine-tuned from the VoxCeleb2 core checkpoint. Best values are bold.

Model	Backbone	Cue	SBS	SIM	NISQA	SIG	BAK	OVL	RTF $\downarrow$
MeMo [ 7 ]	–	V	0.76	0.95	$2.02$	$3.15$	$2.71$	$2.35$	0.03
TSE-Omni	LauraGPT	A	$0.75$	$0.90$	2.73	$3.36$	3.21	2.73	$0.64$
	LauraGPT	V	$0.72$	$0.89$	$2.64$	$3.35$	$3.19$	$2.71$	$0.65$
	Qwen2.5	A	$0.64$	$0.73$	$2.53$	3.40	$3.09$	$2.68$	$0.23$
	Qwen2.5	V	$0.65$	$0.72$	$2.56$	$3.38$	$3.14$	$2.69$	$0.24$

TABLE VI: Streaming on the VoxCeleb2 test set, 1 s chunks. Qwen2.5 is the 77M from-scratch backbone (same size as LauraGPT), not Qwen2.5-0.5B. Best values are bold.

Fig. 8: Multi-cue fusion: each encoder is mapped by a linear layer into the shared $D$ -dimensional space; a missing cue is a zero tensor. Used for Table VII .

Model	Cue	Vox2			YGD
Model	SBS	SIM	OVL	SBS	SIM	OVL
USEV / SEG †	V	$0.800$	$0.940$	$2.600$	$0.650$	$0.840$	$2.580$
TSE-Omni	A	$0.790$	$0.921$	$2.851$	0.713	$0.835$	$2.893$
	T	$0.602$	$0.746$	2.874	$0.442$	$0.519$	$2.643$
	V	$0.803$	$0.937$	$2.831$	$0.602$	$0.732$	$2.776$
	A+T	$0.771$	$0.925$	$2.848$	$0.704$	0.844	2.964

TABLE VII: Ablation of cue combinations. VoxCeleb2 columns: trained from scratch on VoxCeleb2 (V = lip movements). YGD columns: trained from scratch on YGD-2mix (V = co-speech gestures). Not the same checkpoint. Bold = best among TSE-Omni rows.

Backbone	Param	Mode	Cue	SBS	SIM	NISQA	OVL
Q1: LLM architecture (from scratch)
LauraGPT	77M	Scr	A	$0.783$	$0.931$	$3.230$	$2.840$
LauraGPT	77M	Scr	V	0.806	0.941	3.240	2.850
Qwen2.5	77M	Scr	A	$0.706$	$0.900$	$3.180$	$2.740$
Qwen2.5	77M	Scr	V	$0.710$	$0.893$	$3.220$	$2.770$
Q2: Pretrained textual weights (Qwen2.5-0.5B)

TABLE VIII: LLM architecture, textual initialization, and LauraTSE initialization. All rows are trained on VoxCeleb2 (in-domain).

Fig. 9: Token integration. Dual-Path: the visual module is used only with a visual cue. Uni-Path: one module for both cues, with zeros in place of visual tokens for audio-cue TSE.

Cue	Strategy	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
Cue	Strategy	SBS	SIM	NISQA	SIG	BAK	OVL
Audio	Dual-Path	0.91	0.97	3.89	3.50	3.79	3.08
Audio	Uni-Path	$0.89$	0.97	$3.72$	$3.46$	$3.73$	$3.05$
Visual	Dual-Path	0.89	0.96	3.84	3.51	3.76	3.08
Visual	Uni-Path	$0.88$	$0.95$	$3.79$	$3.48$	$3.75$	$3.05$

TABLE IX: Dual-Path vs. Uni-Path token integration on the LRS3 two-speaker test set (zero-shot from the VoxCeleb2 core checkpoint). Best values within each cue are bold.

$\alpha$	$\beta$	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
$\alpha$	$\beta$	SBS	SIM	NISQA	SIG	BAK	OVL
0	1	$0.79$	$0.92$	$3.74$	$3.33$	$3.74$	$2.94$
1	1	0.89	0.96	3.84	3.51	3.76	3.08
1	10	$0.86$	$0.96$	$3.81$	$3.46$	$3.72$	$3.01$
10	1	$0.88$	$0.96$	$3.64$	$3.43$	$3.61$	$2.95$

TABLE X: Effect of the residual scales on the LRS3 two-speaker test set (zero-shot from the VoxCeleb2 core checkpoint).

\alpha

: scale of the speech semantic tokens;

\beta

: scale of the fused audio-visual tokens. Best values are bold.

In-V Rep.	V-Loss	SBS	$\Delta$ SBS
VSR (ResNet)	$\times$	$0.894$	$-0.001$
VSR (ResNet)	$\checkmark$	$0.887$	$-0.008$
AV-HuBERT (L)	$\times$	0.895	$0.000$
AV-HuBERT (L)	$\checkmark$	$0.884$	$-0.011$

TABLE XI: Visual modeling strategies on the LRS3 two-speaker test set (zero-shot from the VoxCeleb2 core checkpoint). In-V Rep. : input visual representation. V-Loss : whether the visual CE loss (and hence the visual head) is enabled.

\Delta

SBS is the change in SBS relative to the best value in the table (

0.895

). All configurations are within

\pm 0.011

of this best, i.e., differences are statistically negligible. The default model uses VSR (ResNet) without V-Loss.

Cue	Enr. Stage	Semantic-Speaker-Quality $\uparrow$			DNSMOS $\uparrow$
Cue	Enr. Stage	SBS	SIM	NISQA	SIG	BAK	OVL
Audio	AR	$0.894$	$0.967$	$3.680$	$3.460$	$3.740$	$3.020$
Audio	AR+NAR	0.914	0.973	3.890	3.500	3.790	3.080
Visual	AR	$0.886$	$0.960$	$3.820$	$3.490$	$3.740$	$3.050$
Visual	AR+NAR	0.894	0.960	3.840	3.510	3.760	3.080

TABLE XII: Effect of the cue enrollment stage on the LRS3 two-speaker test set (zero-shot from the VoxCeleb2 core checkpoint). Best values within each cue are bold.

Explore similar work

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities
Sep 28, 2026Xinyuan Qian, Yanghao Zhou, Ziyang Jiang +10Target Speaker Extraction Speaker
School of Computer and Communication Engineering, University of Science and Technology Beijing, 100083, China · Department of Computer Science and Technology, Beijing Institute of Technology, China · The Chinese University of Hong Kong, Shenzhen, China +5
StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model
Apr 21, 2026Shuhai Peng, Hui Lu, Jinjiang Liu +8Target Speaker Extraction Autoregressive Text-To-Speech
Shenzhen International Graduate School, Tsinghua University · The Chinese University of Hong Kong · SenseTime Research
UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
Oct 23, 2025Haoyin Yan, Chengwei Liu, Shaofei Xue +4Speech Enhancement Autoregressive Model
Qwen Business Unit of Alibaba, China · Tongyi AI Lab of Alibaba, China

Sep 28, 2026, cs.SDJ/K move · Enter open · S save

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

Xinyuan Qian, Yanghao Zhou, Ziyang Jiang, Yu Chen+9

School of Computer and Communication Engineering, University of Science and Technology Beijing, 100083, China · Department of Computer Science and Technology, Beijing Institute of Technology, China · The Chinese University of Hong Kong, Shenzhen, China+5

Target Speaker Extraction Speaker Modalities

Open paper AskPDF arXiv

Target Speaker Extraction (TSE) is pivotal in speech communication and human-computer interaction, enabling the isolation of a specific speaker's voice from complex acoustic environments, i.e., the cocktail party scenario. Although traditional TSE systems conditioned on enrollment speech have progressed substantially, enrollment speech as a cue has inherent limitations. Its reliability degrades when the target and interfering speakers have similar voice characteristics, when intra-speaker variability (e.g. changes in emotion or speaking style) creates a mismatch between the enrollment and target speech, or when the enrollment itself is contaminated by noise or competing speakers. This review surveys deep-learning-based TSE from the perspective of auxiliary target cues drawn from multiple modalities. We organize existing methods according to five types of information used to isolate the target speaker: audio enrollment, visual, spatial, textual/semantic, and neural cues. We also trace the evolution from discriminative estimators to variational, diffusion, flow, codec, and foundation-model-based systems and summarize representative datasets and evaluation metrics. We review the benefits and limitations of different cues and discuss challenges involving synchronization, missing or unreliable observations, data scarcity, privacy, computational cost, and real-time operation. Finally, we summarize future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment. By jointly reviewing cue design, model architecture, training objectives, datasets, and evaluation metrics, this article provides an overview of the current landscape and open problems in multimodal TSE.

Abstract

Explore similar work

Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities

StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model

UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement