Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

Doyeop Kwak; Suyeon Lee; Joon Son Chung

eess.ASMar 20, 2026

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

Authors: Doyeop Kwak, Suyeon Lee, Joon Son Chung

Organizations: Korea Advanced Institute of Science and Technology, South Korea

Abstract

The goal of this paper is to provide a new perspective on audio-visual target speaker extraction (AV-TSE) by decoupling separation and target selection. Conventional AV-TSE systems typically integrate audio and visual features deeply to re-learn the entire separation process, which can act as a fidelity ceiling due to the noisy nature of in-the-wild audio-visual datasets. To address this, we propose Plug-and-Steer, which assigns high-fidelity separation to a frozen audio-only backbone and limits the role of the visual modality strictly to target selection. We introduce the Latent Steering Matrix (LSM), a minimalist linear transformation that re-routes latent features within the backbone to anchor the target speaker to a designated channel. Experiments across four representative architectures show that our method effectively preserves the acoustic priors of diverse backbones, achieving perceptual quality comparable to that of the original backbones. Audio samples are available at: https://plugandsteer.github.io

Figures & tables

Figure 1: Training process of the latent steering matrix

W

for the

i

-th separator block.

\hat{s}_{1}

and

\hat{s}_{2}

are the outputs from the first and second channels of the pre-trained audio-only backbone.

Figure 2: Training process of the AV-TSE module by learning the frame-wise gate value

g_{t}

. The visual steering module predicts the value

g_{t}

to control the degree of steering.

Figure 3: Layer-wise performance preservation rate (%) across different AOSS backbones.

Method	AO	# params.	SI-SDRi (dB)	DNSMOS	NISQA
Libri2Mix GT		-	-	3.16	3.93
LRS2-2mix GT		-	-	2.38	3.19
AO-oriented
Conv-TasNet [ 21 ]		5.1M	7.12	2.35	2.31
+ Residual	✗	6.6M	11.72	2.31	2.57
+ Residual	✓	1.5M	9.37	2.13	2.27

Table 1: Performance of AV-TSE methods on LRS2-2mix grouped by audio-only (AO) backbone, where the AO backbones are pre-trained on Libri2Mix. AO indicates whether the pre-trained AO backbone is frozen or not. # params. denotes the number of trainable parameters. Bold and underlined values represent global and backbone-specific best scores.

Method	AO	SI-SDRi (dB)	DNSMOS	NISQA
MossFormer2 [ 2 ]		15.42	2.88	3.88
+ Residual	✗	17.11	2.53	3.28
+ Residual	✓	16.24	2.51	3.21
+ LSM (Ours)	✓	15.40	2.88	3.87
AV-MossFormer2 [ 11 ]		15.52	2.64	3.06

Table 2: AV-TSE performance on LRS2-2mix using a powerful backbone. The baseline (AV-MossFormer2) is trained solely on LRS2-2mix.

Method	AO	SI-SDRi (dB)	DNSMOS	NISQA
Conv-TasNet [ 21 ]		11.50	2.30	2.32
+ Residual	✗	11.33	2.29	2.33
+ Residual	✓	11.28	2.28	2.31
+ LSM (Ours)	✓	11.21	2.26	2.25
DPRNN [ 23 ]		12.60	2.38	2.55
+ Residual	✗	12.59	2.34	2.51

Table 3: In-domain AV-TSE performance on LRS2-2mix. Both AO and AV models are trained on the same data.

Routing	Method	Accuracy (%)		FLOPs	RTF
		1 epoch	Best epoch
Internal	$\mathcal{L}_{\text{BCE}}$	50.00	99.73	256.82 G	0.147
	$\mathcal{L}_{\text{SI-SNR}}$ + LSM	97.03	99.60
	$\mathcal{L}_{\text{BCE}}$ + $\mathcal{L}_{\text{SI-SNR}}$ + LSM	99.01	99.93
Post-hoc	LSE-C [ 32 , 33 ]	99.80		308.56 G	0.209
	LSE-D [ 32 , 33 ]	99.83

Table 4: Internal routing vs. post-hoc selection. ‘Internal’ denotes our routing module with various loss combinations; ‘Post-hoc’ refers to SyncNet-score-based selection.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 38 papers

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information
Sep 25, 2026Shuhan Zhang, Wenxuan Wu, Haizhou LiTarget Speaker Extraction Speech Processing
Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China
Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes
Oct 23, 2026Wendi Sang, Kai Li, Yifan Li +2Target Speaker Extraction Streaming Inference
School of Computer Technology and Application, Qinghai University, Xining 810016, China · Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua Laboratory of Brain and Intelligence (THBI), IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China · Intelligent Computing and Application Laboratory of Qinghai Province, Qinghai University, Xining 810016, China +1
Training-Free Multi-Step Inference for Target Speaker Extraction
Mar 11, 2026Zhenghai You, Ying Shi, Lantian Li +1Target Speaker Extraction
Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Sep 25, 2026J/K move · Enter open · S save

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

Shuhan Zhang, Wenxuan Wu, Haizhou Li

Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China

Target Speaker Extraction Speech Processing

Open paper AskPDF arXiv

In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.

Abstract

Target Speaker Extraction

Explore similar work

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Training-Free Multi-Step Inference for Target Speaker Extraction