Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Wendi Sang; Kai Li; Yifan Li; Jianqiang Huang; Xiaolin Hu

cs.SDOct 23, 2026

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Authors: Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu

Organizations: School of Computer Technology and Application, Qinghai University, Xining 810016, China · Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua Laboratory of Brain and Intelligence (THBI), IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China · Intelligent Computing and Application Laboratory of Qinghai Province, Qinghai University, Xining 810016, China · Chinese Institute for Brain Research (CIBR), Beijing 100010, China

Abstract

Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.

Figures & tables

Panel of Figure 1 — Figure 1: Real-world AVTSE scenarios motivating RealSSA. The four scene types, ranging from small rooms to outdoor streets, cover varying number of speakers, near-/far-field interference, reverberation, and sudden environmental noise beyond standard fixed-speaker benchmarks.

Benchmark	NF speakers	FF speakers	Reverb control	Env. noise	Sudden noise	Real-recorded eval
NTCD-TIMIT+noise [ 2017 ]	2	✗	✗	✓	✗	✗
LRS2-2Mix [ 2018a ]	2	✗	✗	✗	✗	✗
LRS3-2Mix [ 2018b ]	2	✗	✗	✗	✗	✗
VoxCeleb2-2Mix [ 2018 ]	2	✗	✗	✗	✗	✗
LRS3+WHAM! [ 2023 ]	2	✗	✗	✓	✗	✗
RealSSA (ours)	2–6	✓	✓	✓	✓	✓

Table 1: AVTSE benchmarks organized along three axes: speaker configuration, acoustic conditions, and evaluation realism. “NF” and “FF” denote near- and far-field sources; “Env. noise” means an external noise corpus is mixed in.

Scene	Space size	RT60	Near speakers	Far speakers	Env. noise SIR	Sudden noise
Small meeting	$4\text{-}6\times 3\text{-}5\times 2.5\text{-}3\,\mathrm{m}$	0.3–0.6 s	1–5, 0.5–1.5 m	none	5–15 dB	30%, 1 event
Medium indoor	$8\text{-}15\times 8\text{-}12\times 3\text{-}4\,\mathrm{m}$	0.5–0.9 s	1–3, 1.0–2.5 m	10–15, 3–15 m	5–10 dB	40%, 1–2 events
Large public	$20\text{-}40\times 15\text{-}30\times 4\text{-}8\,\mathrm{m}$	1.0–2.0 s	1–4, 1.0–3.0 m	10–15, 5–20 m	2–8 dB	50%, 1–3 events
Outdoor	$50\times 50\times 20\,\mathrm{m}$	0.05–0.1 s	1–4, 1.0–4.0 m	10–15, 5–25 m	$-2$ – $5$ dB	40%, 1–3 events

Table 2: Acoustic scene configurations in RealSSA-Sim, covering four room types with controlled reverberation, speaker distance, and noise conditions. Near speakers and Far speakers denote interfering speakers and exclude the target.

Figure 2: Overall pipeline of Falcon for audio-visual target speaker extraction.

Figure 3: Single-Pass Multi-Scale Separator.

\phi

denotes the upsampling operation.

Figure 4: Time-Frequency Refinement Block.

Method	RealSSA-Real	RealSSA-Sim		LRS2-2Mix		LRS3-2Mix		VoxCeleb2-2Mix
Method	DNSMOS $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$
AV-ConvTasNet [ 2021 ]	2.580	1.2	2.6	9.8	10.3	12.5	12.8	6.9	8.2
AV-Sepformer [ 2023 ]	2.813	5.5	6.3	10.6	11.1	13.4	13.7	6.9	8.3
AV-TF-GridNet [ 2023 ]	3.030	7.6	8.1	12.7	13.1	14.3	14.8	5.8	7.7
RTFSNet-12 [ 2024 ]	3.066	4.9	5.7	12.4	12.8	14.0	14.4	8.4	10.7
CTCNet [ 2024b ]	2.840	5.4	6.2	8.0	8.8	9.4	10.2	2.5	4.7

Table 3: Performance of causal AVTSE methods. LRS2-2Mix and LRS3-2Mix baseline results are from Swift-Net; other baseline results use our retrained models. Bold and underline mark the best and second-best results.

Method	RealSSA-Sim		RealSSA-Real
Method	WER (%) $\downarrow$	Similarity $\uparrow$	WER (%) $\downarrow$	Similarity $\uparrow$
AV-Sepformer [ 2023 ]	32.64	38.69	34.61	38.25
AV-TF-GridNet [ 2023 ]	27.31	40.44	34.95	39.88
RTFSNet-12 [ 2024 ]	31.91	36.51	35.56	35.78
CTCNet [ 2024b ]	34.44	37.08	38.81	35.33
AV-Mossformer2 [ 2025 ]	32.44	37.34	32.00	36.42

Table 4: WER (%,

\downarrow

) and target-speaker similarity (cosine similarity

\times 100

\uparrow

). Ground truth evaluates the target waveform using the same protocol; “–” denotes unavailable isolated target waveforms. Bold and underline mark the best and second-best extraction systems, excluding ground truth.

Method	Params (M) $\downarrow$	MACs (G) $\downarrow$	Latency (ms) $\downarrow$		Memory (MB) $\downarrow$
Method	Params (M) $\downarrow$	MACs (G) $\downarrow$	CPU	GPU	RAM	GPU Inf.
AV-TF-GridNet [ 2023 ]	6.2	213.8	5646.3	567.8	136.3	1015.8
AV-Mossformer2 [ 2025 ]	56.7	224.2	16670.7	114.9	330.9	539.8
Swift-Net-12 [ 2025 ]	0.5	45.3	4163.8	172.1	482.3	411.7
Dolphin [ 2026a ]	5.6	18.1	2411.0	87.9	560.2	390.3
Falcon (Ours)	0.5	10.6	1361.2	51.3	125.3	364.7

Table 5: Efficiency comparison among Falcon and the four best-performing causal AVTSE baselines from Table 3 . Lower is better for all metrics.

Method	VoxCeleb2		RealSSA-Sim (Ours)
Method	NISQA	DNSMOS	NISQA	DNSMOS
AV-TF-GridNet	2.495	2.757	2.413	3.030
AV-Mossformer2	1.995	2.715	2.256	2.912
Swift-Net-12	2.481	3.064	3.484	3.276
Dolphin	1.593	2.608	2.590	2.960
Falcon (Ours)	2.431	2.878	3.615	3.276

Table 6: Evaluation on RealSSA-Real when models were trained on VoxCeleb2 versus RealSSA.

Stages	SI-SNRi $\uparrow$	SDRi $\uparrow$
1	7.1	7.7
2	7.4	7.9
3 (Ours)	7.7	8.2

Table 7: Effect of downsampling depth.

Model	Loss	SI-SNRi $\uparrow$	SDRi $\uparrow$
Falcon	Without	7.3	7.8
Falcon	+ Ours	7.7	8.2
SwiftNet-12	Baseline	7.3	7.8
SwiftNet-12	+ Ours	7.5	8.0
Dolphin	Baseline	6.7	7.3
Dolphin	+ Ours	7.0	7.6

Table 8: Encoder-space speaker contrastive loss. Gains are consistent across three architectures with different separator designs.

$\tau$ Strategy	SI-SNRi $\uparrow$	SDRi $\uparrow$
Fixed $\tau{=}0$	7.5	8.1
Fixed $\tau{=}1$	7.4	7.9
Anneal $0\!\to\!2$	7.5	8.0
Anneal $0\!\to\!1$ (Ours)	7.7	8.2

Table 9: Sensitivity to the

\tau

schedule. Moderate annealing (

0\!\to\!1

) balances early exploration and late hard-negative focus.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Role	Hours	Speakers	Native SR	Used SR
LRS3 [ 10 ]	Target (audio–visual)	$\sim$ 438	$\sim$ 9,545	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
LibriSpeech [ 29 ]	Near-/far-field interferers	$\sim$ 1000	$\sim$ 2,484	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
WHAM! [ 12 ]	Continuous environmental noise	$\sim$ 80	–	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
Hive [ 30 ]	Semantic environmental and sudden events	$\sim$ 544	–	$44.1\,\mathrm{kHz}$	$16\,\mathrm{kHz}$

Appendix

Table 10: Source corpora used to construct RealSSA-Sim. “Role” summarizes the function of each corpus in the mixture. “Hours” denotes the approximate total speech duration. “Speakers” indicates the number of distinct speakers (marked “–” if not applicable). “Native SR” is the original sampling rate, and “Used SR” is the resampled rate used in RealSSA-Sim.

Panel of Figure 5 — Figure 5: Spatial layouts of the four RealSSA-Sim acoustic scene types. The blue human icons indicate the positions of near-field speakers, while the green human icons indicate the positions of far-field speakers. The green spheres denote environmental sound sources, the orange spheres denote sudden noise sources, and the black microphone denotes the microphone position. Since different sound sources may have different spatial locations, the visualization icons can appear at different heights.

Scene	Space size	RT60	Near speakers	Far speakers	Near SIR	Env. noise SIR	Applied Far SIR	Sudden noise
Small meeting	$4\text{-}6\times 3\text{-}5\times 2.5\text{-}3\,\mathrm{m}$	0.3–0.6 s	1–5, 0.5–1.5 m	none	2–5 dB	5–15 dB	–	30%, 1 event
Medium indoor	$8\text{-}15\times 8\text{-}12\times 3\text{-}4\,\mathrm{m}$	0.5–0.9 s	1–3, 1.0–2.5 m	10–15, 3–15 m	2–8 dB	5–10 dB	20–35 dB	40%, 1–2 events
Large public	$20\text{-}40\times 15\text{-}30\times 4\text{-}8\,\mathrm{m}$	1.0–2.0 s	1–4, 1.0–3.0 m	10–15, 5–20 m	2–8 dB	2–8 dB	20–35 dB	50%, 1–3 events
Outdoor	$50\times 50\times 20\,\mathrm{m}$	0.05–0.1 s	1–4, 1.0–4.0 m	10–15, 5–25 m	2–15 dB	$-2$ to $5$ dB	20–35 dB	40%, 1–3 events

Appendix

Table 11: Extended acoustic-scene configuration for RealSSA-Sim. Near speakers and Far speakers denote interfering speakers and exclude the target. Near SIR specifies the scene-dependent range for near-field interferers. Applied Far SIR is

20

–

35\,\mathrm{dB}

in all scenes with far-field interferers; small-meeting scenes contain none. The mixing procedure is detailed in Appendix A.2 .

Scene	Environmental noise labels	Sudden-noise labels
Small meeting	air conditioning, fan, clock, typing, printer, keyboard, tick, hiss	writing, clicking, crumple, rustle, knock, telephone, door, drawer, squeak, paper
Medium indoor	microwave, blender, dishes, pots/pans, refrigerator	cutlery, glass, cupboard, frying, cash register, gurgling, steam, kettle, sneeze, cough, laughter
Large public	footsteps, shuffle, speech, babble, crowd	sliding door, siren, child speech, crying, roll, alarm, buzzer, announcement
Outdoor	wind, rain, traffic, stream, city ambience	car, bus, truck, motorcycle, car passing, skidding, reversing beep, horn, police/ambulance/fire siren, dog, bird, cricket, jackhammer, hammer, power tool
All scenes (shared)	–	key jangling, coin dropping, slam, bang, thump, crash, squeal

Appendix

Table 12: Hive label assignment per RealSSA-Sim scene. “All scenes (shared)” lists generic transient events that are added to the sudden-noise pool of every scene.

Split	Samples	Total hours	Avg. duration	Avg. speakers	Avg. Near SIR	Avg. Applied Far SIR
Train	$100{,}826$	$129.2$	$4.61$ s	$13.21$	$8.51$ dB	$27.49$ dB
Val	$9{,}133$	$11.7$	$4.61$ s	$13.29$	$8.51$ dB	$27.47$ dB
Test	$9{,}169$	$11.7$	$4.61$ s	$13.23$	$8.49$ dB	$27.50$ dB

Appendix

Table 13: Per-split summary statistics of RealSSA-Sim. Total number of speakers includes the target and all near-/far-field interferers. Number of speakers, near-/far-field SIR, and RT60 are sampled per mixture from the ranges of Table 11 ; the values reported here are realized averages over the released metadata.

Figure 6: RealSSA-Sim distribution statistics. (a) Total number of speakers per mixture, including the target and all interferers. (b) Scene-type proportion. (c) Realized near-field and far-field SIR distributions. (d) Realized RT60 distribution. Histograms are computed from the released training-split metadata.

Simulated scene	Real-world environment	Sessions	Total duration (m:s)
Small meeting	Study room	$12$	$10{:}36$
Small meeting	Classroom	$8$	$10{:}16$
Medium indoor	Internet cafe	$5$	$10{:}22$
Medium indoor	Billiard hall	$5$	$10{:}39$
Large public	Shopping mall	$5$	$9{:}48$
Large public	Supermarket	$5$	$10{:}46$

Appendix

Table 14: Per-environment composition of RealSSA-Real. Total durations are reported as “minutes:seconds”. The environments are grouped by the simulated scene category they correspond to in Table 11 .

Figure 7: Eight environments covered by RealSSA-Real. All recordings were captured with a single Apple iPhone 17 Pro; detected faces have been pixelated for privacy before release.

Symbol	Module	Value
$C_{\text{e}}$	STFT encoder output channels	$112$
$C$	Audio bottleneck output channels	$112$
$C_{\text{h}}$	Separator hidden channels	$64$
$C_{\text{v}}$	Visual hidden channels	$32$
$\dim(\mathbf{q})$	Speaker embedding dimension	$64$
$h_{\mathcal{F}}$	TFRB frequency-LSTM hidden size	$24$

Appendix

Table 15: Channel-width settings of Falcon. Symbols follow the definitions in Sections 4.1 and 4.2 .

Item	Value
Optimizer	AdamW, learning rate $1\times 10^{-3}$ , weight decay $0.01$
LR scheduler	ReduceLROnPlateau, factor $0.5$ , patience $15$
Early stopping	Patience $30$ on validation loss
Max epochs	$1000$
Batch size per GPU	$4$
Hardware	$8\times$ NVIDIA RTX 3090

Appendix

Table 16: Training protocol of Falcon.

Method	Small meeting					Medium indoor			Large public				Outdoor
Method	$1$	$2$	$3$	$4$	$5$	$1$	$2$	$3$	$1$	$2$	$3$	$4$	$1$	$2$	$3$	$4$
AV-ConvTasNet [ 39 ]	$-0.6$	0.1	0.4	0.7	0.7	1.6	1.5	1.5	1.3	1.4	1.6	1.5	0.9	1.5	2.0	1.9
AV-DPRNN [ 26 ]	4.0	4.4	4.3	4.3	4.1	5.1	4.9	4.8	5.4	5.4	5.5	5.6	6.6	6.8	7.0	7.1
AV-Sepformer [ 27 ]	4.0	4.4	4.3	4.4	4.2	4.9	4.8	4.7	5.4	5.4	5.6	5.6	7.1	7.3	7.5	7.4
CTCNet [ 6 ]	4.0	4.4	4.4	4.5	4.3	5.0	4.9	4.8	5.3	5.3	5.5	5.7	6.6	6.9	7.1	7.3
RTFSNet-12 [ 16 ]	2.5	3.4	3.5	3.9	3.7	4.5	4.6	4.5	4.7	4.9	4.9	5.3	4.8	5.4	5.8	6.0

Appendix

Table 17: Per-scene and per-speaker-count SI-SNRi (dB,

\uparrow

) on the RealSSA-Sim test split.

N_{\text{near}}

denotes the number of near-field interfering speakers in the mixture, excluding the target, so that

N_{\text{near}}=1

is the one-interferer case. Columns group the four RealSSA-Sim scene types of Table 11 . For each column, bold marks the best result and underline marks the second-best.

Panel of Figure 8 — Figure 8: Qualitative spectrogram comparison of target speech extraction results on four representative test mixtures. Each example compares the ground-truth target speech with the outputs of Falcon, Dolphin, and SwiftNet. Red boxes highlight regions where the methods differ in preserving target speech components or suppressing residual interference.

Recurrent Unit	SI-SNRi $\uparrow$	SDRi $\uparrow$	Params (M) $\downarrow$	MACs (G) $\downarrow$
RNN	6.2	7.1	0.08	1.90
SRU	7.4	7.9	0.15	2.71
GRU	7.3	7.9	0.20	4.42
LSTM (Ours)	7.7	8.2	0.25	5.69

Appendix

Table 18: Recurrent unit comparison in TFRB. LSTM achieved the highest SI-SNRi with moderate parameter and computational overhead.

Figure 9: PCA visualization of encoder-space speaker embeddings for the top-15 target speakers in the RealSSA-Sim training set. Left: without the proposed contrastive loss, embeddings from different speakers are scattered and overlap. Right: with the contrastive loss, same-speaker embeddings become more compact and different speakers are more clearly separated.

Output formulation	SI-SNRi $\uparrow$	SDRi $\uparrow$
Complex-valued mask (Falcon)	7.7	8.2
Real-valued mask	7.3	7.9
Direct real/imaginary prediction	7.1	7.2

Appendix

Table 19: Comparison of output formulations on RealSSA-Sim. SI-SNRi and SDRi are reported in dB; higher is better. Bold indicates the best result.

Overlap range	Samples	Mean overlap	SI-SNRi $\uparrow$	SDRi $\uparrow$
$0.11$ – $0.72$	$1{,}834$	$0.635$	$7.95$	$8.39$
$0.72$ – $0.79$	$1{,}834$	$0.756$	$7.71$	$8.30$
$0.79$ – $0.84$	$1{,}834$	$0.811$	$7.80$	$8.33$
$0.84$ – $0.89$	$1{,}834$	$0.863$	$7.73$	$8.28$
$0.89$ – $1.00$	$1{,}833$	$0.946$	$7.73$	$8.22$

Appendix

Table 20: Falcon’s performance across speech overlap ratios on RealSSA-Sim. SI-SNRi and SDRi are reported in dB.

Method	Missing pattern	$p=0$	$p=0.25$	$p=0.50$	$p=0.75$	$p=1.00$
AV-Sepformer [ 27 ]	Random Drop	5.26 / 5.88	4.98 / 5.68	4.57 / 5.38	2.35 / 4.02	$-44.09$ / $-4.66$
AV-Sepformer [ 27 ]	Block Drop	5.26 / 5.88	2.42 / 3.65	$-4.17$ / $-0.58$	$-12.42$ / $-3.46$	$-44.09$ / $-4.66$
RTFSNet [ 16 ]	Random Drop	4.01 / 4.61	3.87 / 4.50	3.42 / 4.15	1.69 / 3.02	$-10.80$ / $-1.73$
RTFSNet [ 16 ]	Block Drop	4.01 / 4.61	2.77 / 3.66	0.51 / 2.20	$-3.38$ / 0.31	$-10.80$ / $-1.73$
CTCNet [ 6 ]	Random Drop	5.07 / 5.72	4.83 / 5.53	4.30 / 5.09	2.90 / 4.03	$-14.85$ / $-3.72$
CTCNet [ 6 ]	Block Drop	5.07 / 5.72	3.04 / 4.10	$-0.28$ / 1.76	$-4.01$ / $-0.39$	$-14.85$ / $-3.72$

Appendix

Table 21: Performance under missing visual information on

500

RealSSA-Sim test samples. Each entry reports SI-SNRi / SDRi in dB. The

p=0

column is the uncorrupted control for this subset. At

p=1

, the two missing patterns are equivalent. Bold indicates the best result for each metric under the same pattern and missing ratio.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 38 papers

Sep 25, 2026

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.

Shuhan Zhang, Wenxuan Wu, Haizhou Li

Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China

Target Speaker Extraction Speech Processing

Ask PDF

Mar 20, 2026

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

The goal of this paper is to provide a new perspective on audio-visual target speaker extraction (AV-TSE) by decoupling separation and target selection. Conventional AV-TSE systems typically integrate audio and visual features deeply to re-learn the entire separation process, which can act as a fidelity ceiling due to the noisy nature of in-the-wild audio-visual datasets. To address this, we propose Plug-and-Steer, which assigns high-fidelity separation to a frozen audio-only backbone and limits the role of the visual modality strictly to target selection. We introduce the Latent Steering Matrix (LSM), a minimalist linear transformation that re-routes latent features within the backbone to anchor the target speaker to a designated channel. Experiments across four representative architectures show that our method effectively preserves the acoustic priors of diverse backbones, achieving perceptual quality comparable to that of the original backbones. Audio samples are available at: https://plugandsteer.github.io

Doyeop Kwak, Suyeon Lee, Joon Son Chung

Korea Advanced Institute of Science and Technology, South Korea

Target Speaker Extraction Audio Source Separation Activation Steering

Ask PDF

Jul 16, 2026

SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordings

We introduce the REAL-TSE Challenge, an IEEE SLT 2026 satellite challenge on target speaker extraction~(TSE) from real conversational recordings. Given a multi-speaker mixture and one or more enrollment utterances from a target speaker, participating systems must recover only the target speech. Unlike simulated read-speech benchmarks, REAL-TSE evaluates Mandarin and English recordings that contain natural overlap, reverberation, noise, channel mismatch, and conversational dynamics. The challenge defines two complementary tracks: an Online track for low-latency streaming extraction and an Offline track for full-context processing. Systems are evaluated with Token Error Rate (TER), Speaker Similarity (SpkSim), DNSMOS, and target-speaker activity F1. This overview paper describes the task definition, datasets, baselines, evaluation protocol, submitted systems, condition-wise findings, and lessons for future real-world TSE benchmarks.

Shuai Wang, Zihan Qian, Ke Zhang +9

Nanjing University, China · Chinese University of Hong Kong (Shenzhen), China · Brno University of Technology, Czechia +3

Target Speaker Extraction Speech Processing Speech Separation

Ask PDF

Abstract

Target Speaker Extraction

Explore similar work

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

SLT 2026 REAL-TSE Challenge: Real-world Target Speaker Extraction from Conversational Recordings