CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection

Pratham Gupta; Haotian Qi; Tushar Sandhan; Gabriel Skantze

cs.CVOct 10, 2026

CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection

Authors: Pratham Gupta, Haotian Qi, Tushar Sandhan, Gabriel Skantze

Organizations: Department of Electrical Engineering, Indian Institute of Technology Kanpur · Department of Speech, Music and Hearing, KTH Royal Institute of Technology

Abstract

We present CAST-ASD (Causal Attention over Speakers and Time), a causal joint model for active speaker detection (ASD). Recent ASD models improve performance by jointly processing all faces in a scene, but operate offline, with the full clip available before any frame is labelled. Low-latency causal detectors instead process each face independently. CAST-ASD combines joint face reasoning with fully causal inference. It applies scene-level attention across faces within the current frame and causal temporal attention over each face's history, enabling information sharing across speakers with zero lookahead. Its parameter count does not grow with the number of faces. On MSDWild, where most negative face-frames contain a visible competing speaker, CAST-ASD reduces competing-speaker false alarms by 32% relative to the causal per-face detector it extends, with larger gains as scene occupancy increases. It outperforms all causal baselines on both validation splits while using only one-tenth the inter-speaker parameters of a three-block fixed-roster convolution. Code and models are available at https://github.com/prathamg007/CAST-ASD.

Figures & tables

		MSDWild
	AVA	few.val	many.val
Scenes with $\geq 5$ faces	0 6.3	0 0.1	69.6
Scenes with $\leq 3$ faces	88.8	80.0	15.4
Competing negatives	19.4	79.7	76.2
Overlapping speech	0 0.5	0 9.9	12.0

Table 1: Corpus statistics, computed with identical code over the validation annotations of AVA-ActiveSpeaker and MSDWild. All entries are percentages of face-frames. A negative face-frame is competing when another tracked face is genuinely speaking at that instant.

Figure 1: CAST-ASD. The visual branch runs independently for each of the $F$ faces and the audio branch once per scene, so every face receives the same mixed audio. The inset shows one CAST block: the speaker pass attends across the faces present at a frame, the time pass attends over a face’s own past under a causal mask, and neither reads a future frame. Face crops are grayscale, and are coloured here only to match each crop to its track. Right: an example scene, with the active speaker boxed in green.

Causal: zero lookahead, verified by perturbation
	many.val ( $5+$ speakers)				few.val
System	mAP	mAP c	FA c	DER ∗	mAP
CAST-ASD $\times 1$ (Mimi)	90.56 $\pm$ .10	71.51 $\pm$ .41	8.61 $\pm$ .18	27.59 $\pm$ .23	96.99 $\pm$ .02
LoCoNet $\times 3$ (VGGish)	89.15 $\pm$ .13	69.63 $\pm$ .81	11.14 $\pm$ .04	31.57 $\pm$ .37	96.71 $\pm$ .05
Per-face backbone (Mimi)	88.58 $\pm$ .16	69.95 $\pm$ .43	12.72 $\pm$ .34	32.17 $\pm$ .46	96.17 $\pm$ .05
TalkNet (MFCC)	87.15 $\pm$ .51	68.48 $\pm$ .96	13.84 $\pm$ .76	33.92 $\pm$ .98	95.27 $\pm$ .09

Table 2: Causal systems on the two MSDWild validation splits, with the offline configuration of each architecture shown for reference. The audio encoder is given in parentheses and

\times N

denotes the number of mixer blocks. Bold marks the best causal entry in each column.

Audio	Mixer	$s$	Blocks	Param.	mAP	FA c
Mimi	CAST	—	1	0 1.57M	90.56 $\pm$ .10	0 8.61 $\pm$ .18
	CAST	—	3	0 4.72M	90.65 $\pm$ .14	0 8.44 $\pm$ .17
	LoCoNet	3	1	0 5.18M	89.60 $\pm$ .05	10.77 $\pm$ .29
		3	3	15.55M	89.91 $\pm$ .24	10.26 $\pm$ .42
		4	3	25.18M	90.14 $\pm$ .17	0 9.68 $\pm$ .31
		5	3	37.57M	90.37 $\pm$ .15	0 9.42 $\pm$ .18

Table 3: Mixing operator against audio front end on many.val , three seeds per row. The two mixers are exchanged between the two front ends at matched depth, and the convolution is additionally given wider rosters (

s\in\{4,5\}

) on our own front end. Parameters are those of the scene mixer.

Competing false alarms FA c
Faces visible at the frame	2	3	4	$5+$
Proportion of frames (%)	5.4	9.6	14.5	69.6
Per-face backbone (Mimi)	8.25	9.97	17.17	12.45
LoCoNet $\times 3$ (VGGish)	6.94	7.02	13.33	11.39
CAST-ASD $\times 1$ (Mimi)	6.72	7.83	12.28	0 8.15
CAST-ASD $\times 3$ (Mimi)	6.30	7.15	11.56	0 8.14

Table 4: Competing-speaker false alarms and mAP on many.val , computed separately by the number of faces visible at the frame (when more than one). Bold marks the best causal entry in each column.

Architecture	Causal	Offline	$\Delta$
CAST-ASD $\times 1$ (Mimi)	90.56	91.70	1.14 $\pm$ .11
CAST-ASD $\times 3$ (Mimi)	90.65	92.08	1.42 $\pm$ .24
TalkNet (MFCC)	87.15	88.78	1.63 $\pm$ .89
LoCoNet $\times 3$ (VGGish)	89.15	91.82	2.67 $\pm$ .59
Light-ASD (MFCC)	85.10	88.64	3.54 $\pm$ .56

Table 5: The cost of causal inference on many.val , that is, the mAP an architecture gives up when its causal constraints are applied.

System	Causal	Offline
CAST-ASD $\times 1$ (Mimi)	91.07 $\pm$ .15	94.2 $\pm$ .17
LoCoNet $\times 3$ (VGGish)	90.19 $\pm$ .06	95.2 †
TalkNet (MFCC)	89.11 $\pm$ .18	92.3 †

Table 6: AVA-ActiveSpeaker validation mAP, computed with the official ActivityNet scorer. † Reported by the original authors under their own training protocol.

Causal Attention

See all 15 papers

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion
Sep 22, 2026Pu Wang, Hugo Van hammeAudio-Visual Understanding Multimodal Robustness
KU Leuven, Department of Electrical Engineering, Leuven, Belgium
$C^3$ ASD: Multi-Level Consistency-Driven Representation Learning
Jul 3, 2026Jin Hong, Jisoo Park, Junseok KwonMultimodal Robustness Audio-Visual Representation Learning
Chung-Ang University, Seoul, Republic of Korea
STAM-ASR: Speaker-Temporal Anchoring with Memory for Multi-Speaker ASR
Sep 24, 2026Victor Tolulope Olufemi, Syeda Faiza Ahmed Sara, Shammur Absar ChowdhuryMulti-Speaker ASR Audio-Language Models
Qatar Computing Research Institute (QCRI), Doha, Qatar

Sep 22, 2026J/K move · Enter open · S save

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

Pu Wang, Hugo Van hamme

KU Leuven, Department of Electrical Engineering, Leuven, Belgium

Audio-Visual Understanding Multimodal Robustness Multimodal Fusion

Open paper AskPDF arXiv

Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.

Abstract

Causal Attention

Explore similar work

ROAM-ASD: Robust Open-World Active Speaker Detection with Flexible Multimodal Fusion

C3C^3C3ASD: Multi-Level Consistency-Driven Representation Learning

STAM-ASR: Speaker-Temporal Anchoring with Memory for Multi-Speaker ASR

$C^3$ ASD: Multi-Level Consistency-Driven Representation Learning