Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Junbo Zhang; Hang Su; Zhaofan Li; Hang Dong; Chao Sun

cs.CVNov 6, 2026

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Authors: Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun

Organizations: School of Computer Science, Wuhan University · Academy of Advanced Interdisciplinary Studies, Wuhan University

Abstract

Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.

Figures & tables

Figure 1: llustration of challenges from intra-modal noise and fine-grained localization. (a) When the target is guitar only, performer–instrument interaction introduces visual interference, while environmental sounds corrupt audio signal, causing blurred boundaries and partial segmentation ( e.g . , missing guitar head). (b) When the target is to guitar & woman, visual confusion is alleviated, but audio noise and cross-modal misalignment persist, leading to inaccurate boundaries and incomplete segmentation.

Figure 2: Image denoising methods: (a) The pretrained SAM [ 25 ] model is employed to generate a prior mask, which is then fused with the original image information; (b) Offline generated prototype centers are introduced to guide pixel clustering within the image; (c) The prototype memory bank is dynamically updated following a momentum strategy based on the input data, and it constrains pixel-wise classification through prototype alignment.

Figure 3: Overall architecture of BYOAVP. After encoding, both image and audio inputs are transformed into feature representations. The first-layer visual features are fed into the DPC module for pixel classification under prototype bank constraints, which are dynamically updated. Meanwhile, the audio features and last-layer visual features undergo cross-modal self-supervised contrastive learning to obtain enhanced audio embeddings. These embeddings interact with the classified visual features to produce target-enhanced mask features and semantically aligned queries, which are then fed into the segmentation head for final prediction.

Method	Visual Backbone	Audio Backbone	AVS-Object-S4			AVS-Object-MS3			AVS-Semantic
			$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
AVSBench [ 58 ] [ECCV’22]	PVT-v2	VGGish	83.3	78.7	87.9	59.3	54.0	64.5	32.5	29.8	35.2
ECMVAE [ 42 ] [ICCV’23]	PVT-v2	VGGish	85.9	81.7	90.1	64.3	57.8	70.8	–	–	–
AVSC [ 33 ] [ACM MM’23]	Swin-B	VGGish	85.0	81.3	88.6	62.6	59.5	65.8	–	–	–
CATR [ 29 ] [ACM MM’23]	PVT-v2	VGGish	87.9	84.4	91.3	68.6	62.7	74.5	35.7	32.8	38.5
AQFormer [ 22 ] [IJCAI’23]	PVT-v2	VGGish	85.5	81.6	89.4	67.5	62.2	72.7	–	–	–

Table 1: Quantitative comparisons on AVSBench dataset [ 58 , 57 ] with Transformer backbone. Best results in Bold , while second best underlined . Methods marked with * leverage prior knowledge, such as SAM-generated masks or offline prototype centers.

Figure 4: Qualitative comparison of our BYOAVP with AVSBench [ 58 , 57 ] and VCT [ 23 ] on the AVSBench dataset. Representative examples are drawn from the AVS-Object-S4 (left), AVS-Object-MS3 (middle), and AVS-Semantic (right) test sets.

Method	Visual Backbone	Audio Backbone	Trainable Params	VPO-SS			VPO-MS			VPO-MSMI
				$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
AVSBench [ 58 ] [ECCV’22]	PVT-v2	VGGish	101.32M	44.6	41.6	47.6	45.7	42.3	49.1	43.2	40.0	46.3
AVSegFormer [ 13 ] [AAAI’24]	PVT-v2	VGGish	186.05M	46.0	43.8	48.1	43.7	47.3	40.1	50.0	47.2	52.7
CAVP [ 10 ] [CVPR’24]	ResNet50	VGGish	119.79M	67.0	58.8	75.2	61.3	53.2	69.4	56.5	48.2	64.8
CPM [ 11 ] [ECCV’24]	ResNet50	VGGish	–	73.5	67.1	79.9	72.9	65.9	79.9	68.1	60.6	75.6
DDESEG* [ 36 ] [CVPR’25]	MIT-B5	HTSAT	103.84M	74.4	67.6	81.2	74.3	67.6	81.0	68.4	62.1	74.7

Table 2: Quantitative comparisons on VPO dataset [ 10 ] . Best results in Bold , while second best underlined . The * indicates that the model utilizes additional prior information, such as offline-generated prototype centers.

Figure 5: Qualitative comparison of our proposed components. The predicted mask heatmaps show that, from top to bottom, the model progressively concentrates on sounding objects while effectively suppressing surrounding noise.

Module	AVS-Object-MS3			AVS-Semantic
	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
Baseline	72.48	67.58	77.38	61.89	58.24	65.53
SSAE	76.22	72.93	79.50	65.08	61.21	68.95
DPC	77.98	74.88	81.08	67.40	63.01	71.79
SSAE+DPC	80.50	77.43	83.57	69.61	64.68	74.54

Table 3: Component effectiveness study of BYOAVP.

Figure 6: t-SNE visualization of audio features. Easy indicates that the selected categories do not share the same superclass ( e.g . , animals, instruments), while hard denotes categories belonging to the same superclass or producing acoustically similar sounds.

Strategy			AVS-Object-MS3			AVS-Semantic
CA	BN	GN	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
$\times$	$\times$	$\times$	77.98	74.88	81.08	67.40	63.01	71.79
$\checkmark$	$\times$	$\times$	77.54	74.23	80.85	66.60	62.17	71.02
$\times$	$\checkmark$	$\times$	76.95	73.59	80.31	66.09	61.33	70.84
$\times$	$\times$	$\checkmark$	77.77	74.75	80.79	67.38	62.79	71.96
$\checkmark$	$\checkmark$	$\times$	78.85	75.41	82.29	68.76	63.72	73.80

Table 4: Ablation study of the SSAE module on the AVSBench dataset. “

\checkmark

” and “

\times

” denote whether a component is enabled. CA represents the Cross-Attention layer, while BN and GN denote BatchNorm and GroupNorm applied after the projection.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 159 papers

Dec 23, 2025

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Jingqi Tian, Yiheng Du, Haoji Zhang +6

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Ask PDF

Sep 23, 2025

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Audio-visual segmentation (AVS) plays a critical role in multimodal machine learning by effectively integrating audio and visual cues to precisely segment objects or regions within visual scenes. Recent AVS methods have demonstrated significant improvements. However, they overlook the inherent frequency-domain contradictions between audio and visual modalities--the substantial noise often present in audio high-frequency signals vs. the structurally rich details in visual high-frequency signals. Ignoring these differences can result in suboptimal performance. In this paper, we rethink the AVS task from a deeper perspective by reformulating AVS task as a frequency-domain decomposition and recomposition problem. To this end, we introduce a novel Frequency-Aware Audio-Visual Segmentation (FAVS) framework consisting of two key modules: Frequency-Domain Enhanced Decomposer (FDED) module and Synergistic Cross-Modal Consistency (SCMC) module. FDED module employs a residual-based iterative frequency decomposition to discriminate modality-specific semantics and structural features, and SCMC module leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing. Extensive experiments demonstrate that our FAVS framework achieves state-of-the-art performance on three benchmark datasets, and abundant qualitative visualizations further verify the effectiveness of the proposed FDED and SCMC modules. The project repository is available at https://github.com/happylife-pk/FAVS.

Kai Peng, Yunzhe Shen, Miao Zhang +6

Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Audio-Visual Understanding Multimodal Semantic Segmentation Multimodal Fusion Frequency-Domain Feature Learning

Ask PDF

Mar 15, 2026

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation

The ability to capture and segment sounding objects in dynamic visual scenes is crucial for the development of Audio-Visual Segmentation (AVS) tasks. While significant progress has been made in this area, the interaction between audio and visual modalities still requires further exploration. In this work, we aim to answer the following questions: How can a model effectively suppress audio noise while enhancing relevant audio information? How can we achieve discriminative interaction between the audio and visual modalities? To this end, we propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy. The proposed SNRP mitigates audio noise interference by selectively emphasizing relevant auditory cues, while DAMF ensures more consistent audio-visual representations. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets, especially in multi-source and complex scenes. The code and model are available at https://github.com/happylife-pk/SDAVS.

Kai Peng, Yunzhe Shen, Miao Zhang +6

Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Audio-Visual Understanding Multimodal Fusion

Ask PDF

Abstract

Audio-Visual Understanding

Explore similar work

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation