SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

Arnav Sharma; Anand Mishra; Renu Rameshan

cs.CVOct 11, 2026

SlotAVS: Object-Centric Audio-Visual Segmentation via Bi-Modal Slot Attention

Authors: Arnav Sharma, Anand Mishra, Renu Rameshan

Organizations: Indian Institute of Technology Jodhpur · Vehant Technologies Pvt Ltd

Abstract

Audio-Visual Segmentation (AVS) requires predicting pixel-wise masks of sounding objects by grounding acoustic signals in visual space. In multi-source scenes, concurrent sound sources often remain entangled within shared visual representations, leading to merged masks or unstable specialization when using standard alignment-focused transformer architectures. To address this, we present SlotAVS, an object-centric framework driven by competitive bi-modal slot attention. While robust foundation models provide rich semantic priors, SlotAVS shifts the paradigm from global cross-modal fusion to explicit competitive binding. By initializing discrete object slots from temporal audio sequences and enforcing strict slot-wise competition over spatial visual tokens, our method structurally mitigates the blending of distinct acoustic entities. Further, to stabilize instance-level learning, we introduce a Hungarian matching strategy that optimally aligns slots with ground-truth objects using spatial and boundary overlap. Coupled with a numerically stable, confidence-aware mask fusion mechanism, SlotAVS robustly aggregates multi-source predictions. Extensive experiments on the AVSBench dataset demonstrate the efficacy of our approach, achieving 88.6% mIoU on the Single-Source (S4) split and 71.7% mIoU on the Multi-Source (MS3) split. These results show that explicit competitive binding is critical for complex audio-visual scene understanding and effectively leveraging foundation-model priors without relying solely on their capacity.

Figures & tables

Figure 1: Object-centric disentanglement via bi-modal slot attention. We visualize the inner workings of SlotAVS in (a) single-source and (b) multi-source acoustic scenes. Instead of relying on a global attention map that risks entangling concurrent sounds, our architecture routes visual features into $K=5$ explicitly competitive slots. As shown, distinct sounding entities (e.g., different instruments or individuals) are strictly isolated into separate slots with high confidence scores. Unmatched slots are naturally suppressed, allowing the fused predictions to cleanly match the instance-level ground truth (GT) without hallucination or mask merging. (Best viewed in color) .

Figure 2: Overall architecture of SlotAVS. The framework leverages frozen foundation models (DINOv3 and CLAP) to extract visual and audio representations. A bi-modal slot attention module performs competitive cross-modal binding to establish object-centric slots. These slots condition the spatial features via Multi-Slot ACSA and act as queries in a mask decoder, followed by instance aware Hungarian matching, probabilistic fusion, and tri modal Contrastive Feature Distillation (CFD).

Figure 3: Detailed schematic of the (a) Bi-modal slot attention and (b) Multi-slot audio-conditioned spatial attention (ACSA) modules. Note the slot wise softmax operation that encourages zero sum competitive binding among the

K

slots for visual tokens. (Best viewed in color) .

Models	Backbone	Size	S4		MS3		AVSS
			$\mathcal{M}_{J}$	$\mathcal{M}_{F}$	$\mathcal{M}_{J}$	$\mathcal{M}_{F}$	$\mathcal{M}^{1}_{J}$
AV-SAM [ 49 ]	ViT-H	1024	40.5	0.566	–	–	-
SAM [ 28 ]	ViT-H	1024	55.1	0.739	54.0	0.638	-
AVSBench [ 71 ]	PVT-v2	224	78.7	0.879	54.0	0.645	29.7
ECMVAE [ 46 ]	PVT-v2	224	81.7	0.901	57.8	0.708	-
DiffusionAVS [ 45 ]	PVT-v2	224	81.4	0.902	58.2	0.709	-

Table 1: Quantitative comparison on S4 and MS3.

\mathcal{M}_{J}

and

\mathcal{M}_{F}

denote region similarity and contour accuracy, respectively. Baseline methods are sorted by MS3

\mathcal{M}_{J}

. AVSS reports

\mathcal{M}^{1}_{J}

	(1) Single-sound		(2) Mixed-sound		(3) Multi-entity		(4) Off-screen
Method	CIoU(%)	AUC(%)	CIoU(%)	AUC(%)	CIoU(%)	AUC(%)	TN(%)
SLAVC(144k) [ 47 ]	9.07	10.60	6.31	7.88	6.41	7.96	96.46
EZ-VSL(10k) [ 48 ]	9.66	11.07	8.16	9.35	6.87	8.32	96.91
EZ-VSL(144k) [ 48 ]	10.92	12.22	6.97	8.34	5.80	7.42	96.47
EZ-VSL(full) [ 48 ]	12.17	13.38	7.67	8.91	6.96	8.40	95.43
SSL-TIE(144k) [ 38 ]	13.10	14.23	5.19	6.76	5.50	7.12	90.82

Table 2: Quantitative evaluation on the AVATAR audio-visual localization benchmark. We compare SlotAVS against localization baselines across four distinct acoustic scenarios: Single-sound, Mixed-sound, Multi-entity, and Off-screen. Performance is measured using Consensus Intersection over Union (CIoU), Area Under Curve (AUC), and True Negative (TN) rates. Higher values indicate better performance.

Architecture Variant	S4 Split		MS3 Split
Architecture Variant	$\mathcal{M}_{J}$	$\mathcal{M}_{F}$	$\mathcal{M}_{J}$	$\mathcal{M}_{F}$
Global Audio (w/o Slots)	86.9	0.928	69.2	0.769
Standard Cross Attention	86.8	0.932	70.3	0.831
SlotAVS (Ours)	87.1	0.923	71.4	0.811

Table 3: Ablation on object-centric Routing and Competitive Binding. Evaluation of core architectural components on the S4 and MS3 splits. Higher values indicate better performance.

Adaptation Strategy	S4 Split		MS3 Split
Adaptation Strategy	$\mathcal{M}_{J}$	$\mathcal{M}_{F}$	$\mathcal{M}_{J}$	$\mathcal{M}_{F}$
Audio LoRA Only	82.58	0.885	60.79	0.679
Both LoRA (Audio + Visual)	86.95	0.925	65.72	0.768
None (Completely Frozen)	85.11	0.912	67.07	0.790
Visual LoRA Only (Ours)	87.10	0.923	71.40	0.811

Table 4: Ablation on Asymmetric Foundation Adaptation: Impact of applying LoRA to different modalities. The visual only adaptation (our main setting) provides the optimal balance of routing adaptability and semantic stability.

Figure 4: Qualitative comparison in dynamic multi-source scenes. SlotAVS distinctly segregates concurrent sounding objects (e.g., voice, guitar, piano) into separated instance masks, successfully mitigating the superposition and mask-merging effects commonly observed in standard global fusion approaches. (Best viewed in color) .

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Binaural Audio-Visual Instance Segmentation
Sep 26, 2026Saijun Wang, Guanfeng Tang, Hongbo Zhao +4Audio-Visual Understanding Instance Segmentation
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China
PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangAudio-Visual Understanding Referring Expression Segmentation
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

Dec 23, 2025J/K move · Enter open · S save

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Abstract

Audio-Visual Understanding

Explore similar work

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Binaural Audio-Visual Instance Segmentation

PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition