SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation

Yingjian Zhu; Ying Wang; Yuyang Hong; Ruohao Guo; Kun Ding; Xin Gu; Bin Fan; Shiming Xiang

cs.CVMar 2, 2026

SeaVIS: Sound-Enhanced Association for Online Audio-Visual Instance Segmentation

Authors: Yingjian Zhu, Ying Wang, Yuyang Hong, Ruohao Guo, Kun Ding, Xin Gu, Bin Fan, Shiming Xiang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 101408, China. · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University, Beijing 100871, China. · Research and Development Department, China Academy of Launch Vehicle Technology, Beijing 100076, China. ·

Abstract

Recently, an audio-visual instance segmentation (AVIS) task has been introduced, aiming to identify, segment and track individual sounding instances in videos. However, prevailing methods primarily adopt the offline paradigm, that cannot associate detected instances across consecutive clips, making them unsuitable for real-world scenarios that involve continuous video streams. To address this limitation, we introduce SeaVIS, the first online framework designed for audio-visual instance segmentation. SeaVIS leverages the Causal Cross Attention Fusion (CCAF) module to enable efficient online processing, which integrates visual features from the current frame with the entire audio history under strict causal constraints. A major challenge for conventional VIS methods is that appearance-based instance association fails to distinguish between an object's sounding and silent states, resulting in the incorrect segmentation of silent objects. To tackle this, we employ an Audio-Guided Contrastive Learning (AGCL) strategy to generate instance prototypes that encode not only visual appearance but also sounding activity. In this way, instances preserved during per-frame prediction that do not emit sound can be effectively suppressed during instance association process, thereby significantly enhancing the audio-following capability of SeaVIS. Extensive experiments conducted on the AVISeg dataset demonstrate that SeaVIS surpasses existing state-of-the-art models across multiple evaluation metrics while maintaining a competitive inference speed suitable for real-time processing.

Figures & tables

Figure 1: Two key limitations of offline models. (A) Different inference paradigms between offline and online models. (A.1) Offline models process the entire sequence of video frames simultaneously and generate only video-level predictions. Consequently, the segmentation of each frame depends heavily on information from future frames. (A.2) In contrast, online models process input frames in a streaming, frame-by-frame manner, followed by post-processing to associate instances across frames. (B) Offline models face the continual inference gap problem: after completing inference on a fixed-length video segment, the model cannot incrementally process new frames. Consequently, newly predicted segments cannot be associated with previously identified instances (e.g., $p_{1}$ and $d_{1}$ ), leading to association failure such as $p_{1}\neq p_{3}$ and $d_{1}\neq d_{2}$ .

Figure 2: Overview of the proposed SeaVIS. SeaVIS operates through two sequential stages: per-frame instance segmentation prediction followed by cross-frame instance association. We propose two key components in our framework: (a) A causal cross attention fusion module enabling cross-frame integration between visual and audio modalities under the temporal constraints of online processing; (b) Dual-level audio-guided contrastive learning at both frame and instance levels to optimize sound-aware instance embedding for audio-visual instance segmentation.

Figure 3: Illustration of Causal Cross Attention Fusion(CCAF). This module receives both visual and audio signals, and output the visual feature which is enhanced by audio features. At each frame, the visual features can integrate audio features from all previous and current frames using cross-attention.

Figure 4: Illustration of the Audio-Guided Contrastive Learning (AGCL) strategy. (a) A tracking example contained the essential components of the proposed strategy. At each frame, an anchor is obtained from the corresponding audio clip. The detection results at each time step include sounding instances, non-sounding instances, and background. (b) Frame-level contrastive learning: At each frame, the audio anchor attracts sounding instances while repelling both non-sounding instances and background. (c) Instance-level contrastive learning: For each instance tracked across time, an average audio anchor is computed from frames where the instance is sounding. This anchor pulls the sounding embeddings of that instance while repelling its non-sounding embeddings. (d) The learned sound-aware embeddings assist in filtering out silent instances during the association process.

Task	Model	Venue	Audio	FSLA	HOTA	mAP	FSLAn	FSLAs	FSLAm	AssA	DetA
VIS	Mask2Former-VIS [ 15 ]	CVPR’ 22	✗	29.75	52.03	28.66	0.00	25.47	36.37	64.49	43.33
	TeViT [ 58 ]	CVPR’ 22	✗	32.28	53.67	31.52	0.00	28.07	39.18	65.27	45.10
	SeqFormer [ 20 ]	ECCV’ 22	✗	30.32	54.32	32.79	25.03	21.76	36.46	67.25	45.23
	VITA [ 5 ]	NeurIPS’ 22	✗	38.04	57.48	36.25	15.04	27.98	47.45	69.86	48.96
	DAVIS [ 59 ]	ICCV’ 23	✗	23.99	49.12	19.83	14.61	24.83	24.69	63.51	40.11
	LBVQ [ 60 ]	TCSVT’ 24	✗	34.73	56.97	36.58	27.71	29.52	38.96	68.34	48.83

Table 1: Quantitative evaluation of different models from related tasks on the AVISeg test set. The best results are highlighted in bold.

Method	Backbone	FSLA	HOTA	mAP	fps
DVIS-online [ 59 ]	Res-50	0.03	29.97	44.29	39.57
IDOL [ 10 ]	Res-50	7.66	46.42	45.18	23.71
GenVIS-online [ 63 ]	Res-50	12.31	24.97	25.16	25.6
MinVIS [ 64 ]	Res-50	2.45	35.13	44.66	50.38
AVISM [ 4 ]	Res-50	44.42	64.52	45.04	20.46
SeaVIS(ours)	Res-50	47.09	66.47	46.28	34.65

Table 2: Comprehensive performance comparison between SeaVIS and existing state-of-the-art methods. The best results are highlighted in bold. Inference speed (FPS) is measured on an NVIDIA A800-SXM4-40GB GPU.

CCAF	AGCL	FSLA	mAP	HOTA
		42.03 $\pm$ 0.14	43.93 $\pm$ 0.12	63.54 $\pm$ 0.17
✓		43.10 $\pm$ 0.09	45.74 $\pm$ 0.16	66.03 $\pm$ 0.15
✓	✓	47.03 $\pm$ 0.15	46.11 $\pm$ 0.12	66.39 $\pm$ 0.13

Table 3: Impact of the Causal Cross Attention Fusion Module (CCAF) and Audio-guided Contrastive Learning strategy (AGCL). Values represent mean

\pm

standard deviation.

Fusion Mode	FSLA	mAP	HOTA
audio $\rightarrow$ visual	47.09	46.28	66.47
visual $\rightarrow$ audio	45.91	45.06	64.28
bilateral fusion	46.17	44.89	66.91

Table 4: Comparisons between different fusion modes.

Figure 5: Visualization of attention scores from CCAF module during inference. The red boxes highlight that for a given frame, the module assigns high attention scores to critical preceding audio time steps.

$\lambda_{f}$	$\lambda_{i}$	FSLA	mAP	HOTA
0	0	43.12	45.92	66.03
1	0	45.96	45.06	65.64
0	1	44.58	46.02	66.97
1	1	47.09	46.28	66.47

Table 5: Ablation study on Audio-guided Contrastive Learning (AGCL) Strategy.

Queries	FSLA	mAP	HOTA
add	47.09	46.28	66.47
all	45.91	45.06	64.28

Table 6: Ablation study on query initialization.

Figure 6: Qualitative results of AVISM and SeaVIS on AVISeg test set. These examples show performance under no-sound scene (the left), single sound source (the left) and multi sound sources (the right). The red boxes highlight false negatives (FNs), the yellow boxes highlight false positives (FPs) and blue ones mark inferior segmentations.

$N_{f}$	FSLA	mAP	HOTA
2	43.85	42.69	62.91
5	45.26	44.96	64.87
10	47.09	46.28	66.47
15	46.64	45.76	66.12

Table 7: Ablation study on number of training frames.

\toprule Duration	CCAF	GPU Mem (MB)	Power (W)	FPS	mAP
\midrule 1 min	✗	5,950	224	24.23	44.19
\midrule 1 min	✓	6,054	228	24.12	46.28
\midrule 30 min	✗	20,516	228	26.58	52.18
\midrule 30 min	✓	21,228	225	25.49	52.96
\bottomrule

Table 8: Ablation analysis of the CCAF module’s impact on computational efficiency and inference performance over extended video durations.

\toprule Audio Encoder	AGCL	FSLA	HOTA	mAP
\midrule VGGish	✓	47.09	66.47	46.28
AudioMAE	✓	49.24	67.19	46.19
AudioMAE	✗	46.02	66.95	45.86
\bottomrule

Table 9: Impact of advanced audio backbones and the consistent effectiveness of the AGCL strategy.

\toprule Method	Audio	FSLA	HOTA	mAP
\midrule AVISM	Clean	44.42	64.52	45.04
AVISM	Noisy	43.19 ( $\downarrow$ 1.23)	63.19 ( $\downarrow$ 1.33)	44.89 ( $\downarrow$ 0.15)
\midrule SeaVIS (Ours)	Clean	47.09	66.47	46.28
SeaVIS (Ours)	Noisy	44.17 ( $\downarrow$ 2.92)	64.26 ( $\downarrow$ 2.21)	46.09 ( $\downarrow$ 0.19)
\bottomrule

Table 10: Robustness analysis under noisy audio conditions.

\toprule Method	IDSW ( $\downarrow$ )
\cmidrule (lr)2-3	Single-Source	Multi-Source
\midrule AVISM	17	71
SeaVIS (Ours)	10	63
\bottomrule

Table 11: Quantitative analysis of tracking stability (IDSW) under different acoustic complexities.

Figure 7: More visual results of our SeaVIS on AVISeg dataset from different scenarios. Each row have six sampled frames from a video sequence. Zoom in to see details.

Figure 8: Failure cases of our SeaVIS on AVISeg dataset from different scenarios. Each row have six sampled frames from a video sequence. Zoom in to see details.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation
Aug 4, 2026Leiye Liu, Miao Zhang, Jiahong Jiang +7Video Object Segmentation Instance Segmentation
Dalian University of Technology Dalian, Liaoning, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Yale University New Haven, Connecticut, USA
Binaural Audio-Visual Instance Segmentation
Sep 26, 2026Saijun Wang, Guanfeng Tang, Hongbo Zhao +4Audio-Visual Understanding Instance Segmentation
College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China · College of Electronic and Information Engineering, Shanghai Institute of Intelligent Science and Technology, State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University, Shanghai 201804, China
Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation
Mar 9, 2026Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn +1Audio-Visual Understanding Continual Learning
Purdue University, West Lafayette, IN 47906, USA

Aug 4, 2026J/K move · Enter open · S save

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

Leiye Liu, Miao Zhang, Jiahong Jiang, Jingjing Li+6

Dalian University of Technology Dalian, Liaoning, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Yale University New Haven, Connecticut, USA

Video Object Segmentation Instance Segmentation Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-visual instance segmentation (AVIS) requires accurately identifying and tracking individual sounding objects with pixel-level masks. Existing methods struggle to match overlapping acoustic events with visual instances and handle asynchronous audio-visual dynamics. Therefore, two critical questions arise: how can a model establish precise correspondence between overlapping sound sources and visual instances, and how can a model maintain robust tracking when audio and visual signals are temporally misaligned?This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms. The Acoustic-Semantic Projector (ASP) disentangles mixed audio and establishes hierarchical correspondence from semantic to spatial domains. The Asynchronous Dynamics Modulator (ADM) adaptively adjusts state transitions via audio-modulated Mamba, prioritizing current information during dynamic variations and maintaining continuity in stable periods.Experiments on AVISeg show H2S achieves SOTA performance, attaining 48.54 mAP with a COCO pretrained ResNet50 and surpassing the previous by 7.8%. The code will be open-sourced once the paper is accepted. The source code will be publicly available at https://github.com/leiyeliu/H2S.

Abstract

Instance Segmentation

Explore similar work

Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation

Binaural Audio-Visual Instance Segmentation

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation