Phonology-Aware Speaker Extraction Frontend for Audio LLMs

Yanxiang Chen; Haoyuan Deng; Jinyu Wang; Hongyi Chen; Dingshan Li; Runlin Huang; Weifeng Su

cs.SDOct 9, 2026

Phonology-Aware Speaker Extraction Frontend for Audio LLMs

Authors: Yanxiang Chen, Haoyuan Deng, Jinyu Wang, Hongyi Chen, Dingshan Li, Runlin Huang, Weifeng Su

Organizations: Sun Yat-Sen University, Zhuhai, China · Beijing Normal-Hong Kong Baptist University, Zhuhai, China · Fudan University, Shanghai, China

Abstract

Target speaker extraction (TSE) is a key frontend for enabling audio large language models (LLMs) to recognize and understand a designated speaker in overlapping speech. However, multilingual mixtures exhibit heterogeneous extraction demands, causing fixed-depth systems to either under-process challenging inputs or waste computation on inputs that have already converged. Existing TSE frontends generally apply uniform computation, while Audio-LLM adaptation methods neither explicitly model language-conditioned extraction demands nor preserve plug-and-play compatibility with frozen downstream models. We therefore propose LATTE, a lightweight 5.2M-parameter frontend that combines an acoustic Difficulty Estimator, a task-adaptive phonology-informed prior learned from multi-depth TSE gains, and an Adaptive Hourglass Extractor to route each mixture without requiring language identity at inference. On PolyExtract, LATTE achieves a macro SI-SNRi of 15.07 dB at 0.30 GFLOPs, retaining performance within 0.09 dB of a 0.53-GFLOP fixed-depth model while reducing computation by 43.4%, and lowers the average WER from 16.8% to 14.6% relative to TF-GridNet. By jointly modeling sample-level extraction demands and phonological structure, LATTE provides a practical framework for quality-aware, compute-efficient multilingual TSE and offers a scalable path toward robust Audio-LLM deployment.

Figures & tables

Figure 1: Motivation of the proposed LATTE framework.

Figure 2: LATTE combines semantic and spectral evidence, estimates a low/medium/high compute budget, and routes the mixture through shared multi-exit hourglass blocks before target-speech reconstruction.

Language	Key Phonological Features	PDI
English	non-tonal, moderate clusters	0.15
Turkish	agglutinative, 8-way vowel harmony	0.30
Mandarin	4 tones, simple onsets	0.35
Polish	complex clusters ( szcz- , prz- )	0.42
Kinyarwanda	2 Bantu tones, prenasalized stops	0.48
Cantonese	6–9 tones, checked codas	0.55

Table 1: PolyExtract language overview.

	SpEx+	TD- SB	AD-Flow TSE	TCE	TF-GridNet	LATTE
Model Complexity
Trainable Params $\downarrow$	11.1M	14.4M	85.0M	8.7M	14.5M	5.2M
Avg. GFLOPs $\downarrow$	0.62	0.74	5.80	0.55	1.42	0.30
All Directed Language Pairs
Macro SI-SNRi $\uparrow$	12.8	13.2	14.6	13.5	13.8	15.07
Macro PESQ $\uparrow$	2.92	2.98	3.14	3.01	3.06	3.22

Table 2: Signal-level extraction quality and computational cost across directed target–interferer language pairs. Results are macro-averaged over language pairs.

Language	Metric	LoRA			Plugin
Language	Metric	Qwen3 Omni	Whisper v3	DeSTA 2.5-A	Voice Filter	LATTE
PolyExtract	WER $\downarrow$	15.4	18.8	15.6	20.6	14.6
PolyExtract	FLOPs $\downarrow$	18.2T	4.8T	12.6T	0.6T	0.3T
Chinese	CER $\downarrow$	11.8	13.5	12.9	17.8	12.6
Chinese	FLOPs $\downarrow$	18.2T	4.8T	12.6T	0.6T	0.3T
English	WER $\downarrow$	7.9	8.9	8.1	12.3	8.2

Table 3: Multilingual ASR performance across speaker extraction approaches.

			PolyExtract (Avg.)			Mandarin			English
Category	Method	Trainable Params $\downarrow$	Mix WER	Ext. WER $\downarrow$	RelRed	Mix CER	Ext. CER $\downarrow$	RelRed	Mix WER	Ext. WER $\downarrow$	RelRed
LoRA	Qwen3-Omni [ 22 ]	7.0B	38.5	15.4	60.0%	32.4	11.8	63.6%	24.6	7.9	67.9%
	Whisper-v3 [ 16 ]	1.5B	38.5	18.8	51.2%	32.4	13.5	58.3%	24.6	8.9	63.8%
	DeSTA2.5-Audio [ 11 ]	7.2B	38.5	15.6	59.5%	32.4	12.9	60.2%	24.6	8.1	67.1%
Plug-in	VoiceFilter [ 14 ]	8.2M	38.5	20.6	46.5%	32.4	17.8	45.1%	24.6	12.3	50.0%
	TD-SpeakerBeam [ 4 ]	14.4M	38.5	19.8	48.6%	32.4	16.9	47.8%	24.6	11.4	53.7%

Table 4: Multi-speaker ASR results on PolyExtract.

	Task A:Slot Filling				Task B:Yes/No				Task C:Attribution
System	en	zh	yue	Poly.	en	zh	yue	Poly.	en	zh	yue	Poly.
Mix $\to$ Qwen	24	21	15	20	61	58	51	57	43	40	33	39
LoRA $\to$ Qwen	70	57	47	58	88	77	67	77	85	70	59	71
LATTE $\to$ Qwen	58	55	49	54	78	75	69	74	72	68	61	67
Clean $\to$ Qwen	74	69	61	68	91	89	83	88	89	85	79	84

Table 5: LLM semantic understanding accuracy (%).

Variant	Avg	Hard	GFLOPs
LATTE-Full	15.1	14.6	0.30g
Adaptive compute
W/o DE (Fixed-6)	13.5	11.8	0.53g
W/o refinement	14.3	12.5	0.28g
W/o $\mathcal{L}_{\textsc{bgt}}$	14.9	13.0	0.39g
DE design

Table 6: Ablation study on PolyExtract. Hard : average over yue/vi/zu. GFLOPs are per-utterance averages over the test set.

Figure 3: Evaluation of difficulty estimation and adaptive computation. (a) Calibration of the predicted compute demand against the task-observed oracle target. Large markers denote demand-level means, and the dashed line denotes perfect calibration. (b) Quality–computation comparison among fixed-depth and adaptive routing strategies. The upper-left region represents higher extraction quality with lower average computation.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 32 papers

Mar 11, 2026

Training-Free Multi-Step Inference for Target Speaker Extraction

Target speaker extraction (TSE) aims to recover a target speaker's speech from a mixture using a reference utterance as a cue. Most TSE systems adopt conditional auto-encoder architectures with one-step inference. Inspired by test-time scaling, we propose a training-free multi-step inference method that enables iterative refinement with a frozen pretrained model. At each step, new candidates are generated by interpolating the original mixture and the previous estimate, and the best candidate is selected for further refinement until convergence. Experiments show that, when ground-truth target speech is available, optimizing an intrusive metric (SI-SDRi) yields consistent gains across multiple evaluation metrics. Without ground truth, optimizing non-intrusive metrics (UTMOS or SpkSim) improves the corresponding metric but may hurt others. We therefore introduce joint metric optimization to balance these objectives, enabling controllable extraction preferences for practical deployment.

Zhenghai You, Ying Shi, Lantian Li +1

Beijing University of Posts and Telecommunications, China · Center for Speech and Language Technologies, BNRist, Tsinghua University, China

Target Speaker Extraction

Ask PDF

Jul 9, 2026

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable. We present PS4, a proxy-supervised training framework for TSE in real conversational mixtures, with two main contributions. First, we construct a large-scale corpus of 71,771 training samples derived from four public datasets, covering both Chinese and English scenarios. Each sample contains an overlapping speech mixture, per-speaker enrollment audio, a ground-truth transcript, and frame-level voice activity labels. Second, we propose a proxy-supervised joint training strategy that fine-tunes a BSRNN-based TSE model using four complementary differentiable objectives: ASR cross-entropy, speaker similarity, frame-level voice activity detection, and perceptual audio quality. Starting from a publicly available pre-trained checkpoint, only the BSRNN separator is updated during fine-tuning. On the REAL-T challenge leaderboard, PS4 ranks 2nd overall, achieving the best speaker similarity and timing F1 among all submitted systems.

Wanyi Ning, Wei Zhou, Yingpeng Li +3

Yijiahe AI, Nanjing, China · Tianjin University, Tianjin, China · Nanjing University, Nanjing, China

Target Speaker Extraction Speech Processing Weakly Supervised Learning Audio Source Separation Speech Separation

Ask PDF

Sep 25, 2026

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.

Shuhan Zhang, Wenxuan Wu, Haizhou Li

Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China

Target Speaker Extraction Speech Processing

Ask PDF

Abstract

Target Speaker Extraction

Explore similar work

Training-Free Multi-Step Inference for Target Speaker Extraction

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information