Active speaker detection (ASD) relies on audiovisual correspondence between speech and visible faces, yet this correspondence becomes unreliable in crowded scenes, or when visual evidence is unavailable. We introduce ROAM-ASD-C and ROAM-ASD-CR to extend ROAM-ASD with contextual and identity-aware reasoning. ROAM-ASD-C jointly reasons across simultaneously visible candidates, allowing other faces to provide complementary evidence when the target observation is ambiguous or incomplete. ROAM-ASD-CR incorporates registered-speaker representations derived from enrollment speech, allowing detection of whether a speaker is active even when that speaker is not visually observed. ROAM-ASD-C achieves state-of-the-art performance across five ASD benchmarks: 96.8% mAP on AVA, 98.9% on WASD, 88.8% on UniTalk, 98.7% on Talkies, and 99.5% on ASW. It also improves ROAM-ASD by up to 13.3 mAP under 75% target-track visual removal.
Figures & tables
Figure 1: Overview of ROAM-ASD [ 14 ] , ROAM-ASD-C, and ROAM-ASD-CR. ROAM-ASD-C adds inter-face contextual reasoning across temporally aligned face candidates, while ROAM-ASD-CR further incorporates enrollment speech as registration tokens for registered-speaker activity and face association. Original ROAM-ASD skip connections are omitted for clarity but retained unchanged.
Method
AVA
WASD
UniTalk
Talkies
ASW
TalkNCE [ 5 ]
–
–
83.2
–
–
Light-ASD [ 8 ]
94.1
93.7
–
–
–
MuSED [ 13 ]
95.6
–
–
–
98.3
LoCoNet [ 15 ]
95.2
–
82.2
96.1
–
ROAM-ASD [ 14 ]
96.5
98.8
87.9
98.2
99.3
ROAM-ASD-C
96.8
98.9
88.8
98.7
99.5
Table 1: Overall mAP ( ↑ , %) across five benchmarks, compared with the best-performing prior methods and ROAM-ASD.
Figure 2: Inter-face context gain ( ΔmAP=mAPROAM-ASD-C−mAPROAM-ASD ) across (a) WASD and (b) UniTalk categories.
Figure 3: Inter-face context gain ( ΔmAP=mAPROAM-ASD-C−mAPROAM-ASD ) with (a) different numbers of visible faces and under continuous (b) mouth, (c) audio, and (d) all-visual missing segments.
Method
1 face
2 faces
3 faces
ASC [ 1 ]
91.8
83.8
67.6
MAAS [ 2 ]
93.3
85.8
68.2
TalkNet [ 12 ]
95.4
89.6
80.3
ASDNet [ 7 ]
95.7
92.4
83.7
EASEE [ 3 ]
96.5
92.4
83.9
Light-ASD [ 8 ]
96.2
92.6
84.4
Table 2: mAP ( ↑ , %) on AVA with different numbers of visible faces.
Registered-speaker AP
Face ASD
Association
FA
Dataset
Full
Visual absent
Visual absent
Acc.
Out-of-scene +20
ASW
93.3
80.4
52.8
98.4
0.0
AVA
89.0
78.0
58.9
95.5
0.04
WASD
87.4
81.8
42.4
92.7
1.3
UniTalk
72.4
60.3
36.4
79.9
0.0
Table 3: Speaker-registration performance of ROAM-ASD-CR (%).
Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.
Pu Wang, Hugo Van hamme
KU Leuven, Department of Electrical Engineering, Leuven, Belgium
We present CAST-ASD (Causal Attention over Speakers and Time), a causal joint model for active speaker detection (ASD). Recent ASD models improve performance by jointly processing all faces in a scene, but operate offline, with the full clip available before any frame is labelled. Low-latency causal detectors instead process each face independently. CAST-ASD combines joint face reasoning with fully causal inference. It applies scene-level attention across faces within the current frame and causal temporal attention over each face's history, enabling information sharing across speakers with zero lookahead. Its parameter count does not grow with the number of faces. On MSDWild, where most negative face-frames contain a visible competing speaker, CAST-ASD reduces competing-speaker false alarms by 32% relative to the causal per-face detector it extends, with larger gains as scene occupancy increases. It outperforms all causal baselines on both validation splits while using only one-tenth the inter-speaker parameters of a three-block fixed-roster convolution. Code and models are available at https://github.com/prathamg007/CAST-ASD.
Pratham Gupta, Haotian Qi, Tushar Sandhan +1
Department of Electrical Engineering, Indian Institute of Technology Kanpur · Department of Speech, Music and Hearing, KTH Royal Institute of Technology
Active Speaker Detection determines whether a visible person in a video is speaking at each moment. While recent audio-visual fusion methods perform well on clean data, they degrade under real-world corruptions such as background noise, occlusion, or simultaneous modality degradation. We attribute this limitation to the absence of explicit consistency constraints that promote robust, semantically aligned representations across modalities. Without such guidance, models tend to learn fragile modality-specific shortcuts that fail under corrupted conditions. We propose C3ASD, a multi-level consistency-driven framework with three complementary constraints: embedding-level inter-modality consistency aligns audio-visual representations during speech; sequence-level intra-modality consistency separates speaking and non-speaking clusters via track-aware contrastive learning; and prediction-level consistency stabilizes fusion through knowledge distillation. Extensive experiments demonstrate significant improvements under diverse audio, visual and joint corruptions, while maintaining competitive performance on clean data.