Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Junbo Zhang; Hang Su; Zhaofan Li; Hang Dong; Chao Sun

cs.CVNov 6, 2026

Bootstrap Your Own AV-Proxies: Adaptive Contrastive and Prototype Learning for Audio-Visual Segmentation

Authors: Junbo Zhang, Hang Su, Zhaofan Li, Hang Dong, Chao Sun

Organizations: School of Computer Science, Wuhan University · Academy of Advanced Interdisciplinary Studies, Wuhan University

Abstract

Audio-Visual Segmentation (AVS) aims to accurately segment sounding objects in video frames by leveraging audio-visual correspondence cues. However, it remains challenging due to the intrinsic semantic incompleteness within a single modality and the semantic gap between audio and visual representations. Traditional feature-fusion decoding approaches struggle to suppress fusion noise effectively, while recent methods that incorporate data-dependent priors often increase the complexity of modeling audio-visual correlations, leading to poor cross-domain generalization. To address these issues, we propose a novel adaptive contrastive and prototype learning framework, BYOAVP, for AVS. Specifically, we design a Self-Supervised Audio Enhancement (SSAE) module that introduces contrastive learning to adaptively align audio representations with gradient-blocked visual embeddings, thus narrowing the semantic gap between modalities. Furthermore, a Dynamic Prototype Constraint (DPC) module is developed to refine pixel-wise category perception via momentum-based prototype updating, while enhancing localization of sounding regions through cross-modal interaction. Extensive experiments demonstrate that our method achieves SOTA performance across two AVS benchmarks and six sub-tasks, exhibiting strong robustness and generalization ability.

Figures & tables

Figure 1: llustration of challenges from intra-modal noise and fine-grained localization. (a) When the target is guitar only, performer–instrument interaction introduces visual interference, while environmental sounds corrupt audio signal, causing blurred boundaries and partial segmentation ( e.g . , missing guitar head). (b) When the target is to guitar & woman, visual confusion is alleviated, but audio noise and cross-modal misalignment persist, leading to inaccurate boundaries and incomplete segmentation.

Figure 2: Image denoising methods: (a) The pretrained SAM [ 25 ] model is employed to generate a prior mask, which is then fused with the original image information; (b) Offline generated prototype centers are introduced to guide pixel clustering within the image; (c) The prototype memory bank is dynamically updated following a momentum strategy based on the input data, and it constrains pixel-wise classification through prototype alignment.

Figure 3: Overall architecture of BYOAVP. After encoding, both image and audio inputs are transformed into feature representations. The first-layer visual features are fed into the DPC module for pixel classification under prototype bank constraints, which are dynamically updated. Meanwhile, the audio features and last-layer visual features undergo cross-modal self-supervised contrastive learning to obtain enhanced audio embeddings. These embeddings interact with the classified visual features to produce target-enhanced mask features and semantically aligned queries, which are then fed into the segmentation head for final prediction.

Method	Visual Backbone	Audio Backbone	AVS-Object-S4			AVS-Object-MS3			AVS-Semantic
			$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
AVSBench [ 58 ] [ECCV’22]	PVT-v2	VGGish	83.3	78.7	87.9	59.3	54.0	64.5	32.5	29.8	35.2
ECMVAE [ 42 ] [ICCV’23]	PVT-v2	VGGish	85.9	81.7	90.1	64.3	57.8	70.8	–	–	–
AVSC [ 33 ] [ACM MM’23]	Swin-B	VGGish	85.0	81.3	88.6	62.6	59.5	65.8	–	–	–
CATR [ 29 ] [ACM MM’23]	PVT-v2	VGGish	87.9	84.4	91.3	68.6	62.7	74.5	35.7	32.8	38.5
AQFormer [ 22 ] [IJCAI’23]	PVT-v2	VGGish	85.5	81.6	89.4	67.5	62.2	72.7	–	–	–

Table 1: Quantitative comparisons on AVSBench dataset [ 58 , 57 ] with Transformer backbone. Best results in Bold , while second best underlined . Methods marked with * leverage prior knowledge, such as SAM-generated masks or offline prototype centers.

Figure 4: Qualitative comparison of our BYOAVP with AVSBench [ 58 , 57 ] and VCT [ 23 ] on the AVSBench dataset. Representative examples are drawn from the AVS-Object-S4 (left), AVS-Object-MS3 (middle), and AVS-Semantic (right) test sets.

Method	Visual Backbone	Audio Backbone	Trainable Params	VPO-SS			VPO-MS			VPO-MSMI
				$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
AVSBench [ 58 ] [ECCV’22]	PVT-v2	VGGish	101.32M	44.6	41.6	47.6	45.7	42.3	49.1	43.2	40.0	46.3
AVSegFormer [ 13 ] [AAAI’24]	PVT-v2	VGGish	186.05M	46.0	43.8	48.1	43.7	47.3	40.1	50.0	47.2	52.7
CAVP [ 10 ] [CVPR’24]	ResNet50	VGGish	119.79M	67.0	58.8	75.2	61.3	53.2	69.4	56.5	48.2	64.8
CPM [ 11 ] [ECCV’24]	ResNet50	VGGish	–	73.5	67.1	79.9	72.9	65.9	79.9	68.1	60.6	75.6
DDESEG* [ 36 ] [CVPR’25]	MIT-B5	HTSAT	103.84M	74.4	67.6	81.2	74.3	67.6	81.0	68.4	62.1	74.7

Table 2: Quantitative comparisons on VPO dataset [ 10 ] . Best results in Bold , while second best underlined . The * indicates that the model utilizes additional prior information, such as offline-generated prototype centers.

Figure 5: Qualitative comparison of our proposed components. The predicted mask heatmaps show that, from top to bottom, the model progressively concentrates on sounding objects while effectively suppressing surrounding noise.

Module	AVS-Object-MS3			AVS-Semantic
	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
Baseline	72.48	67.58	77.38	61.89	58.24	65.53
SSAE	76.22	72.93	79.50	65.08	61.21	68.95
DPC	77.98	74.88	81.08	67.40	63.01	71.79
SSAE+DPC	80.50	77.43	83.57	69.61	64.68	74.54

Table 3: Component effectiveness study of BYOAVP.

Figure 6: t-SNE visualization of audio features. Easy indicates that the selected categories do not share the same superclass ( e.g . , animals, instruments), while hard denotes categories belonging to the same superclass or producing acoustically similar sounds.

Strategy			AVS-Object-MS3			AVS-Semantic
CA	BN	GN	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}$ $\uparrow$	$\mathcal{F}$ $\uparrow$
$\times$	$\times$	$\times$	77.98	74.88	81.08	67.40	63.01	71.79
$\checkmark$	$\times$	$\times$	77.54	74.23	80.85	66.60	62.17	71.02
$\times$	$\checkmark$	$\times$	76.95	73.59	80.31	66.09	61.33	70.84
$\times$	$\times$	$\checkmark$	77.77	74.75	80.79	67.38	62.79	71.96
$\checkmark$	$\checkmark$	$\times$	78.85	75.41	82.29	68.76	63.72	73.80

Table 4: Ablation study of the SSAE module on the AVSBench dataset. “

\checkmark

” and “

\times

” denote whether a component is enabled. CA represents the Cross-Attention layer, while BN and GN denote BatchNorm and GroupNorm applied after the projection.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 159 papers

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
Sep 23, 2025Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Semantic Segmentation
Dalian University of Technology, Dalian 116024, China · the School of Medicine, Yale University, New Haven, CT 06520 USA · the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada
Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation
Mar 15, 2026Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Fusion
Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Dec 23, 2025J/K move · Enter open · S save

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Jingqi Tian, Yiheng Du, Haoji Zhang, Yuji Wang+5

Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China

Audio-Visual Understanding Audio-Visual Representation Learning Audio-Visual Alignment

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by integrating auditory and visual cues. However, existing methods often struggle with multi-source entanglement and audio-visual misalignment, leading to a dominance bias toward acoustically or visually salient objects (i.e., louder or larger ones) at the expense of subtler or co-occurring sources. To address these challenges, we propose DDAVS: Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This process is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS achieves state-of-the-art performance across single-source, multi-source, and multi-class multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Abstract

Audio-Visual Understanding

Explore similar work

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation