Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Wendi Sang; Kai Li; Yifan Li; Jianqiang Huang; Xiaolin Hu

cs.SDOct 23, 2026

Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes

Authors: Wendi Sang, Kai Li, Yifan Li, Jianqiang Huang, Xiaolin Hu

Organizations: School of Computer Technology and Application, Qinghai University, Xining 810016, China · Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua Laboratory of Brain and Intelligence (THBI), IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China · Intelligent Computing and Application Laboratory of Qinghai Province, Qinghai University, Xining 810016, China · Chinese Institute for Brain Research (CIBR), Beijing 100010, China

Abstract

Streaming audio-visual target speaker extraction (AVTSE) is essential for latency-sensitive applications. However, current causal systems face two coupled gaps. First, standard benchmarks fix the number of speakers at two and ignore the near-/far-field interference, reverberation, and sudden noise of real scenes. Second, high-performance causal architectures are too heavy for edge deployment. Lightweight alternatives try to reclaim capacity by repeatedly invoking a small separator, which offsets the savings. To address the first gap, we release RealSSA, a realistic AVTSE benchmark consisting of RealSSA-Sim and RealSSA-Real. RealSSA-Sim provides controllable simulated scenes with 1--5 dynamic near-field interfering speakers; scenes with far-field interference additionally include 10--15 far-field interfering speakers. RealSSA-Real provides a held-out real-recorded evaluation set. To address the second gap, we propose Falcon, a causal AVTSE method. Its Separator performs multi-scale time-frequency separation in a single encoder--decoder pass. We train Falcon with an encoder-space speaker contrastive loss. This loss suppresses near-field leakage at zero inference cost and transfers across backbones. Falcon achieves state-of-the-art extraction quality on RealSSA-Sim, LRS2-2Mix, LRS3-2Mix, and VoxCeleb2-2Mix. Compared with the causal AV-TFGridNet baseline, it cuts parameters by 91.9%, computation by 20.2 times, and GPU inference latency by 11.1 times. Code and demo are available at https://cslikai.cn/Falcon/.

Figures & tables

Panel of Figure 1 — Figure 1: Real-world AVTSE scenarios motivating RealSSA. The four scene types, ranging from small rooms to outdoor streets, cover varying number of speakers, near-/far-field interference, reverberation, and sudden environmental noise beyond standard fixed-speaker benchmarks.

Benchmark	NF speakers	FF speakers	Reverb control	Env. noise	Sudden noise	Real-recorded eval
NTCD-TIMIT+noise [ 2017 ]	2	✗	✗	✓	✗	✗
LRS2-2Mix [ 2018a ]	2	✗	✗	✗	✗	✗
LRS3-2Mix [ 2018b ]	2	✗	✗	✗	✗	✗
VoxCeleb2-2Mix [ 2018 ]	2	✗	✗	✗	✗	✗
LRS3+WHAM! [ 2023 ]	2	✗	✗	✓	✗	✗
RealSSA (ours)	2–6	✓	✓	✓	✓	✓

Table 1: AVTSE benchmarks organized along three axes: speaker configuration, acoustic conditions, and evaluation realism. “NF” and “FF” denote near- and far-field sources; “Env. noise” means an external noise corpus is mixed in.

Scene	Space size	RT60	Near speakers	Far speakers	Env. noise SIR	Sudden noise
Small meeting	$4\text{-}6\times 3\text{-}5\times 2.5\text{-}3\,\mathrm{m}$	0.3–0.6 s	1–5, 0.5–1.5 m	none	5–15 dB	30%, 1 event
Medium indoor	$8\text{-}15\times 8\text{-}12\times 3\text{-}4\,\mathrm{m}$	0.5–0.9 s	1–3, 1.0–2.5 m	10–15, 3–15 m	5–10 dB	40%, 1–2 events
Large public	$20\text{-}40\times 15\text{-}30\times 4\text{-}8\,\mathrm{m}$	1.0–2.0 s	1–4, 1.0–3.0 m	10–15, 5–20 m	2–8 dB	50%, 1–3 events
Outdoor	$50\times 50\times 20\,\mathrm{m}$	0.05–0.1 s	1–4, 1.0–4.0 m	10–15, 5–25 m	$-2$ – $5$ dB	40%, 1–3 events

Table 2: Acoustic scene configurations in RealSSA-Sim, covering four room types with controlled reverberation, speaker distance, and noise conditions. Near speakers and Far speakers denote interfering speakers and exclude the target.

Figure 2: Overall pipeline of Falcon for audio-visual target speaker extraction.

Figure 3: Single-Pass Multi-Scale Separator.

\phi

denotes the upsampling operation.

Figure 4: Time-Frequency Refinement Block.

Method	RealSSA-Real	RealSSA-Sim		LRS2-2Mix		LRS3-2Mix		VoxCeleb2-2Mix
Method	DNSMOS $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$	SI-SNRi $\uparrow$	SDRi $\uparrow$
AV-ConvTasNet [ 2021 ]	2.580	1.2	2.6	9.8	10.3	12.5	12.8	6.9	8.2
AV-Sepformer [ 2023 ]	2.813	5.5	6.3	10.6	11.1	13.4	13.7	6.9	8.3
AV-TF-GridNet [ 2023 ]	3.030	7.6	8.1	12.7	13.1	14.3	14.8	5.8	7.7
RTFSNet-12 [ 2024 ]	3.066	4.9	5.7	12.4	12.8	14.0	14.4	8.4	10.7
CTCNet [ 2024b ]	2.840	5.4	6.2	8.0	8.8	9.4	10.2	2.5	4.7

Table 3: Performance of causal AVTSE methods. LRS2-2Mix and LRS3-2Mix baseline results are from Swift-Net; other baseline results use our retrained models. Bold and underline mark the best and second-best results.

Method	RealSSA-Sim		RealSSA-Real
Method	WER (%) $\downarrow$	Similarity $\uparrow$	WER (%) $\downarrow$	Similarity $\uparrow$
AV-Sepformer [ 2023 ]	32.64	38.69	34.61	38.25
AV-TF-GridNet [ 2023 ]	27.31	40.44	34.95	39.88
RTFSNet-12 [ 2024 ]	31.91	36.51	35.56	35.78
CTCNet [ 2024b ]	34.44	37.08	38.81	35.33
AV-Mossformer2 [ 2025 ]	32.44	37.34	32.00	36.42

Table 4: WER (%,

\downarrow

) and target-speaker similarity (cosine similarity

\times 100

\uparrow

). Ground truth evaluates the target waveform using the same protocol; “–” denotes unavailable isolated target waveforms. Bold and underline mark the best and second-best extraction systems, excluding ground truth.

Method	Params (M) $\downarrow$	MACs (G) $\downarrow$	Latency (ms) $\downarrow$		Memory (MB) $\downarrow$
Method	Params (M) $\downarrow$	MACs (G) $\downarrow$	CPU	GPU	RAM	GPU Inf.
AV-TF-GridNet [ 2023 ]	6.2	213.8	5646.3	567.8	136.3	1015.8
AV-Mossformer2 [ 2025 ]	56.7	224.2	16670.7	114.9	330.9	539.8
Swift-Net-12 [ 2025 ]	0.5	45.3	4163.8	172.1	482.3	411.7
Dolphin [ 2026a ]	5.6	18.1	2411.0	87.9	560.2	390.3
Falcon (Ours)	0.5	10.6	1361.2	51.3	125.3	364.7

Table 5: Efficiency comparison among Falcon and the four best-performing causal AVTSE baselines from Table 3 . Lower is better for all metrics.

Method	VoxCeleb2		RealSSA-Sim (Ours)
Method	NISQA	DNSMOS	NISQA	DNSMOS
AV-TF-GridNet	2.495	2.757	2.413	3.030
AV-Mossformer2	1.995	2.715	2.256	2.912
Swift-Net-12	2.481	3.064	3.484	3.276
Dolphin	1.593	2.608	2.590	2.960
Falcon (Ours)	2.431	2.878	3.615	3.276

Table 6: Evaluation on RealSSA-Real when models were trained on VoxCeleb2 versus RealSSA.

Stages	SI-SNRi $\uparrow$	SDRi $\uparrow$
1	7.1	7.7
2	7.4	7.9
3 (Ours)	7.7	8.2

Table 7: Effect of downsampling depth.

Model	Loss	SI-SNRi $\uparrow$	SDRi $\uparrow$
Falcon	Without	7.3	7.8
Falcon	+ Ours	7.7	8.2
SwiftNet-12	Baseline	7.3	7.8
SwiftNet-12	+ Ours	7.5	8.0
Dolphin	Baseline	6.7	7.3
Dolphin	+ Ours	7.0	7.6

Table 8: Encoder-space speaker contrastive loss. Gains are consistent across three architectures with different separator designs.

$\tau$ Strategy	SI-SNRi $\uparrow$	SDRi $\uparrow$
Fixed $\tau{=}0$	7.5	8.1
Fixed $\tau{=}1$	7.4	7.9
Anneal $0\!\to\!2$	7.5	8.0
Anneal $0\!\to\!1$ (Ours)	7.7	8.2

Table 9: Sensitivity to the

\tau

schedule. Moderate annealing (

0\!\to\!1

) balances early exploration and late hard-negative focus.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Role	Hours	Speakers	Native SR	Used SR
LRS3 [ 10 ]	Target (audio–visual)	$\sim$ 438	$\sim$ 9,545	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
LibriSpeech [ 29 ]	Near-/far-field interferers	$\sim$ 1000	$\sim$ 2,484	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
WHAM! [ 12 ]	Continuous environmental noise	$\sim$ 80	–	$16\,\mathrm{kHz}$	$16\,\mathrm{kHz}$
Hive [ 30 ]	Semantic environmental and sudden events	$\sim$ 544	–	$44.1\,\mathrm{kHz}$	$16\,\mathrm{kHz}$

Appendix

Table 10: Source corpora used to construct RealSSA-Sim. “Role” summarizes the function of each corpus in the mixture. “Hours” denotes the approximate total speech duration. “Speakers” indicates the number of distinct speakers (marked “–” if not applicable). “Native SR” is the original sampling rate, and “Used SR” is the resampled rate used in RealSSA-Sim.

Panel of Figure 5 — Figure 5: Spatial layouts of the four RealSSA-Sim acoustic scene types. The blue human icons indicate the positions of near-field speakers, while the green human icons indicate the positions of far-field speakers. The green spheres denote environmental sound sources, the orange spheres denote sudden noise sources, and the black microphone denotes the microphone position. Since different sound sources may have different spatial locations, the visualization icons can appear at different heights.

Scene	Space size	RT60	Near speakers	Far speakers	Near SIR	Env. noise SIR	Applied Far SIR	Sudden noise
Small meeting	$4\text{-}6\times 3\text{-}5\times 2.5\text{-}3\,\mathrm{m}$	0.3–0.6 s	1–5, 0.5–1.5 m	none	2–5 dB	5–15 dB	–	30%, 1 event
Medium indoor	$8\text{-}15\times 8\text{-}12\times 3\text{-}4\,\mathrm{m}$	0.5–0.9 s	1–3, 1.0–2.5 m	10–15, 3–15 m	2–8 dB	5–10 dB	20–35 dB	40%, 1–2 events
Large public	$20\text{-}40\times 15\text{-}30\times 4\text{-}8\,\mathrm{m}$	1.0–2.0 s	1–4, 1.0–3.0 m	10–15, 5–20 m	2–8 dB	2–8 dB	20–35 dB	50%, 1–3 events
Outdoor	$50\times 50\times 20\,\mathrm{m}$	0.05–0.1 s	1–4, 1.0–4.0 m	10–15, 5–25 m	2–15 dB	$-2$ to $5$ dB	20–35 dB	40%, 1–3 events

Appendix

Table 11: Extended acoustic-scene configuration for RealSSA-Sim. Near speakers and Far speakers denote interfering speakers and exclude the target. Near SIR specifies the scene-dependent range for near-field interferers. Applied Far SIR is

20

–

35\,\mathrm{dB}

in all scenes with far-field interferers; small-meeting scenes contain none. The mixing procedure is detailed in Appendix A.2 .

Scene	Environmental noise labels	Sudden-noise labels
Small meeting	air conditioning, fan, clock, typing, printer, keyboard, tick, hiss	writing, clicking, crumple, rustle, knock, telephone, door, drawer, squeak, paper
Medium indoor	microwave, blender, dishes, pots/pans, refrigerator	cutlery, glass, cupboard, frying, cash register, gurgling, steam, kettle, sneeze, cough, laughter
Large public	footsteps, shuffle, speech, babble, crowd	sliding door, siren, child speech, crying, roll, alarm, buzzer, announcement
Outdoor	wind, rain, traffic, stream, city ambience	car, bus, truck, motorcycle, car passing, skidding, reversing beep, horn, police/ambulance/fire siren, dog, bird, cricket, jackhammer, hammer, power tool
All scenes (shared)	–	key jangling, coin dropping, slam, bang, thump, crash, squeal

Appendix

Table 12: Hive label assignment per RealSSA-Sim scene. “All scenes (shared)” lists generic transient events that are added to the sudden-noise pool of every scene.

Split	Samples	Total hours	Avg. duration	Avg. speakers	Avg. Near SIR	Avg. Applied Far SIR
Train	$100{,}826$	$129.2$	$4.61$ s	$13.21$	$8.51$ dB	$27.49$ dB
Val	$9{,}133$	$11.7$	$4.61$ s	$13.29$	$8.51$ dB	$27.47$ dB
Test	$9{,}169$	$11.7$	$4.61$ s	$13.23$	$8.49$ dB	$27.50$ dB

Appendix

Table 13: Per-split summary statistics of RealSSA-Sim. Total number of speakers includes the target and all near-/far-field interferers. Number of speakers, near-/far-field SIR, and RT60 are sampled per mixture from the ranges of Table 11 ; the values reported here are realized averages over the released metadata.

Figure 6: RealSSA-Sim distribution statistics. (a) Total number of speakers per mixture, including the target and all interferers. (b) Scene-type proportion. (c) Realized near-field and far-field SIR distributions. (d) Realized RT60 distribution. Histograms are computed from the released training-split metadata.

Simulated scene	Real-world environment	Sessions	Total duration (m:s)
Small meeting	Study room	$12$	$10{:}36$
Small meeting	Classroom	$8$	$10{:}16$
Medium indoor	Internet cafe	$5$	$10{:}22$
Medium indoor	Billiard hall	$5$	$10{:}39$
Large public	Shopping mall	$5$	$9{:}48$
Large public	Supermarket	$5$	$10{:}46$

Appendix

Table 14: Per-environment composition of RealSSA-Real. Total durations are reported as “minutes:seconds”. The environments are grouped by the simulated scene category they correspond to in Table 11 .

Figure 7: Eight environments covered by RealSSA-Real. All recordings were captured with a single Apple iPhone 17 Pro; detected faces have been pixelated for privacy before release.

Symbol	Module	Value
$C_{\text{e}}$	STFT encoder output channels	$112$
$C$	Audio bottleneck output channels	$112$
$C_{\text{h}}$	Separator hidden channels	$64$
$C_{\text{v}}$	Visual hidden channels	$32$
$\dim(\mathbf{q})$	Speaker embedding dimension	$64$
$h_{\mathcal{F}}$	TFRB frequency-LSTM hidden size	$24$

Appendix

Table 15: Channel-width settings of Falcon. Symbols follow the definitions in Sections 4.1 and 4.2 .

Item	Value
Optimizer	AdamW, learning rate $1\times 10^{-3}$ , weight decay $0.01$
LR scheduler	ReduceLROnPlateau, factor $0.5$ , patience $15$
Early stopping	Patience $30$ on validation loss
Max epochs	$1000$
Batch size per GPU	$4$
Hardware	$8\times$ NVIDIA RTX 3090

Appendix

Table 16: Training protocol of Falcon.

Method	Small meeting					Medium indoor			Large public				Outdoor
Method	$1$	$2$	$3$	$4$	$5$	$1$	$2$	$3$	$1$	$2$	$3$	$4$	$1$	$2$	$3$	$4$
AV-ConvTasNet [ 39 ]	$-0.6$	0.1	0.4	0.7	0.7	1.6	1.5	1.5	1.3	1.4	1.6	1.5	0.9	1.5	2.0	1.9
AV-DPRNN [ 26 ]	4.0	4.4	4.3	4.3	4.1	5.1	4.9	4.8	5.4	5.4	5.5	5.6	6.6	6.8	7.0	7.1
AV-Sepformer [ 27 ]	4.0	4.4	4.3	4.4	4.2	4.9	4.8	4.7	5.4	5.4	5.6	5.6	7.1	7.3	7.5	7.4
CTCNet [ 6 ]	4.0	4.4	4.4	4.5	4.3	5.0	4.9	4.8	5.3	5.3	5.5	5.7	6.6	6.9	7.1	7.3
RTFSNet-12 [ 16 ]	2.5	3.4	3.5	3.9	3.7	4.5	4.6	4.5	4.7	4.9	4.9	5.3	4.8	5.4	5.8	6.0

Appendix

Table 17: Per-scene and per-speaker-count SI-SNRi (dB,

\uparrow

) on the RealSSA-Sim test split.

N_{\text{near}}

denotes the number of near-field interfering speakers in the mixture, excluding the target, so that

N_{\text{near}}=1

is the one-interferer case. Columns group the four RealSSA-Sim scene types of Table 11 . For each column, bold marks the best result and underline marks the second-best.

Panel of Figure 8 — Figure 8: Qualitative spectrogram comparison of target speech extraction results on four representative test mixtures. Each example compares the ground-truth target speech with the outputs of Falcon, Dolphin, and SwiftNet. Red boxes highlight regions where the methods differ in preserving target speech components or suppressing residual interference.

Recurrent Unit	SI-SNRi $\uparrow$	SDRi $\uparrow$	Params (M) $\downarrow$	MACs (G) $\downarrow$
RNN	6.2	7.1	0.08	1.90
SRU	7.4	7.9	0.15	2.71
GRU	7.3	7.9	0.20	4.42
LSTM (Ours)	7.7	8.2	0.25	5.69

Appendix

Table 18: Recurrent unit comparison in TFRB. LSTM achieved the highest SI-SNRi with moderate parameter and computational overhead.

Figure 9: PCA visualization of encoder-space speaker embeddings for the top-15 target speakers in the RealSSA-Sim training set. Left: without the proposed contrastive loss, embeddings from different speakers are scattered and overlap. Right: with the contrastive loss, same-speaker embeddings become more compact and different speakers are more clearly separated.

Output formulation	SI-SNRi $\uparrow$	SDRi $\uparrow$
Complex-valued mask (Falcon)	7.7	8.2
Real-valued mask	7.3	7.9
Direct real/imaginary prediction	7.1	7.2

Appendix

Table 19: Comparison of output formulations on RealSSA-Sim. SI-SNRi and SDRi are reported in dB; higher is better. Bold indicates the best result.

Overlap range	Samples	Mean overlap	SI-SNRi $\uparrow$	SDRi $\uparrow$
$0.11$ – $0.72$	$1{,}834$	$0.635$	$7.95$	$8.39$
$0.72$ – $0.79$	$1{,}834$	$0.756$	$7.71$	$8.30$
$0.79$ – $0.84$	$1{,}834$	$0.811$	$7.80$	$8.33$
$0.84$ – $0.89$	$1{,}834$	$0.863$	$7.73$	$8.28$
$0.89$ – $1.00$	$1{,}833$	$0.946$	$7.73$	$8.22$

Appendix

Table 20: Falcon’s performance across speech overlap ratios on RealSSA-Sim. SI-SNRi and SDRi are reported in dB.

Method	Missing pattern	$p=0$	$p=0.25$	$p=0.50$	$p=0.75$	$p=1.00$
AV-Sepformer [ 27 ]	Random Drop	5.26 / 5.88	4.98 / 5.68	4.57 / 5.38	2.35 / 4.02	$-44.09$ / $-4.66$
AV-Sepformer [ 27 ]	Block Drop	5.26 / 5.88	2.42 / 3.65	$-4.17$ / $-0.58$	$-12.42$ / $-3.46$	$-44.09$ / $-4.66$
RTFSNet [ 16 ]	Random Drop	4.01 / 4.61	3.87 / 4.50	3.42 / 4.15	1.69 / 3.02	$-10.80$ / $-1.73$
RTFSNet [ 16 ]	Block Drop	4.01 / 4.61	2.77 / 3.66	0.51 / 2.20	$-3.38$ / 0.31	$-10.80$ / $-1.73$
CTCNet [ 6 ]	Random Drop	5.07 / 5.72	4.83 / 5.53	4.30 / 5.09	2.90 / 4.03	$-14.85$ / $-3.72$
CTCNet [ 6 ]	Block Drop	5.07 / 5.72	3.04 / 4.10	$-0.28$ / 1.76	$-4.01$ / $-0.39$	$-14.85$ / $-3.72$

Appendix

Table 21: Performance under missing visual information on

500

RealSSA-Sim test samples. Each entry reports SI-SNRi / SDRi in dB. The

p=0

column is the uncorrupted control for this subset. At

p=1

, the two missing patterns are equivalent. Bold indicates the best result for each metric under the same pattern and missing ratio.

Target Speaker Extraction

Target speaker extraction isolates a chosen voice from overlapping speech using speaker identity, visual, spatial, textual, or neural cues.

See all 39 papers

Tap2Listen: Instant Target Switching for Streaming Audio-Visual Speaker Extraction
Oct 27, 2026Hansol Park, Minkyung Song, Zinu Kim +4Target Speaker Extraction Speech Processing
Sungkyunkwan University
Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information
Sep 25, 2026Shuhan Zhang, Wenxuan Wu, Haizhou LiTarget Speaker Extraction Speech Processing
Shenzhen Loop Area Institute, Shenzhen, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · The Chinese University of Hong Kong, Shatin, N.T., Hong Kong SAR, China
Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction
Mar 20, 2026Doyeop Kwak, Suyeon Lee, Joon Son ChungTarget Speaker Extraction Audio Source Separation
Korea Advanced Institute of Science and Technology, South Korea

Oct 27, 2026J/K move · Enter open · S save

Tap2Listen: Instant Target Switching for Streaming Audio-Visual Speaker Extraction

Hansol Park, Minkyung Song, Zinu Kim, Yujun Lee+3

Sungkyunkwan University

Target Speaker Extraction Speech Processing

Open paper AskPDF arXiv

Streaming audio-visual target speaker extraction (AV-TSE) uses a target speaker's face to extract the speaker's speech from an input audio mixture in real time. However, existing systems fix the target speaker for an entire clip, while conversation demands frequent and immediate target switches. To address this problem, we propose Tap2Listen, a streaming AV-TSE model designed for instant response to target switches. The model separates computation by modality: the target-agnostic audio encoder processes only the mixture, while the target-specific visual cues of the selected face condition only the decoder. A switch requires only a 0.12ms re-projection and takes effect immediately. We also design component-wise losses: a prediction loss for the encoder and a supervised extraction loss for the decoder. For evaluation, we measure how quickly output quality recovers after a switch. On the standard LRS3-2mix test set, Tap2Listen improves SI-SNRi from 8.85 to 9.68 dB at a comparable parameter count. After a switch to a known face, the model produces full-quality output from the first frame, whereas the baseline needs about 1 second to recover. Finally, we show that the proposed streaming AV-TSE model runs in real time on a single CPU core.

Abstract

Target Speaker Extraction

Explore similar work

Tap2Listen: Instant Target Switching for Streaming Audio-Visual Speaker Extraction

Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction