Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration

Pu Wang; Hugo Van hamme

cs.SDOct 23, 2026

Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration

Authors: Pu Wang, Hugo Van hamme

Organizations: KU Leuven, Department of Electrical Engineering, Leuven, Belgium

Abstract

Active speaker detection (ASD) relies on audiovisual correspondence between speech and visible faces, yet this correspondence becomes unreliable in crowded scenes, or when visual evidence is unavailable. We introduce ROAM-ASD-C and ROAM-ASD-CR to extend ROAM-ASD with contextual and identity-aware reasoning. ROAM-ASD-C jointly reasons across simultaneously visible candidates, allowing other faces to provide complementary evidence when the target observation is ambiguous or incomplete. ROAM-ASD-CR incorporates registered-speaker representations derived from enrollment speech, allowing detection of whether a speaker is active even when that speaker is not visually observed. ROAM-ASD-C achieves state-of-the-art performance across five ASD benchmarks: 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW. It also improves ROAM-ASD by up to 13.3 mAP under 75% target-track visual removal.

Figures & tables

Figure 1: Overview of ROAM-ASD [ 14 ] , ROAM-ASD-C, and ROAM-ASD-CR. ROAM-ASD-C adds inter-face contextual reasoning across temporally aligned face candidates, while ROAM-ASD-CR further incorporates enrollment speech as registration tokens for registered-speaker activity and face association. Original ROAM-ASD skip connections are omitted for clarity but retained unchanged.

Method	AVA	WASD	UniTalk	Talkies	ASW
TalkNCE [ 5 ]	–	–	$83.2$	–	–
Light-ASD [ 8 ]	$94.1$	$93.7$	–	–	–
MuSED [ 13 ]	$95.6$	–	–	–	$98.3$
LoCoNet [ 15 ]	$95.2$	–	$82.2$	$96.1$	–
ROAM-ASD [ 14 ]	$96.5$	$98.8$	$87.9$	$98.2$	$99.3$
ROAM-ASD-C	$\mathbf{96.8}$	$\mathbf{98.9}$	$\mathbf{88.8}$	$\mathbf{98.7}$	$\mathbf{99.5}$

Table 1: Overall mAP (

\uparrow

, %) across five benchmarks, compared with the best-performing prior methods and ROAM-ASD.

Figure 2: Inter-face context gain ( $\Delta\mathrm{mAP}=\mathrm{mAP}_{\text{ROAM-ASD-C}}-\mathrm{mAP}_{\text{ROAM-ASD}}$ ) across (a) WASD and (b) UniTalk categories.

Figure 3: Inter-face context gain ( $\Delta\mathrm{mAP}=\mathrm{mAP}_{\text{ROAM-ASD-C}}-\mathrm{mAP}_{\text{ROAM-ASD}}$ ) with (a) different numbers of visible faces and under continuous (b) mouth, (c) audio, and (d) all-visual missing segments.

Method	1 face	2 faces	3 faces
ASC [ 1 ]	$91.8$	$83.8$	$67.6$
MAAS [ 2 ]	$93.3$	$85.8$	$68.2$
TalkNet [ 12 ]	$95.4$	$89.6$	$80.3$
ASDNet [ 7 ]	$95.7$	$92.4$	$83.7$
EASEE [ 3 ]	$96.5$	$92.4$	$83.9$
Light-ASD [ 8 ]	$96.2$	$92.6$	$84.4$

Table 2: mAP (

\uparrow

, %) on AVA with different numbers of visible faces.

	Registered-speaker AP		Face ASD	Association	FA
Dataset	Full	Visual absent	Visual absent	Acc.	Out-of-scene $+20$
ASW	$93.3$	$80.4$	$52.8$	$98.4$	$0.0$
AVA	$89.0$	$78.0$	$58.9$	$95.5$	$0.04$
WASD	$87.4$	$81.8$	$42.4$	$92.7$	$1.3$
UniTalk	$72.4$	$60.3$	$36.4$	$79.9$	$0.0$

Table 3: Speaker-registration performance of ROAM-ASD-CR (%).

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion
Sep 22, 2026Pu Wang, Hugo Van hammeAudio-Visual Understanding Multimodal Robustness
KU Leuven, Department of Electrical Engineering, Leuven, Belgium
CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection
Oct 10, 2026Pratham Gupta, Haotian Qi, Tushar Sandhan +1Causal Attention
Department of Electrical Engineering, Indian Institute of Technology Kanpur · Department of Speech, Music and Hearing, KTH Royal Institute of Technology
$C^3$ ASD: Multi-Level Consistency-Driven Representation Learning
Jul 3, 2026Jin Hong, Jisoo Park, Junseok KwonMultimodal Robustness Audio-Visual Representation Learning
Chung-Ang University, Seoul, Republic of Korea

Sep 22, 2026J/K move · Enter open · S save

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

Pu Wang, Hugo Van hamme

KU Leuven, Department of Electrical Engineering, Leuven, Belgium

Audio-Visual Understanding Multimodal Robustness Multimodal Fusion

Open paper AskPDF arXiv

Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.

Abstract

Audio-Visual Understanding

Explore similar work

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection

C3C^3C3ASD: Multi-Level Consistency-Driven Representation Learning

$C^3$ ASD: Multi-Level Consistency-Driven Representation Learning