Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

Tianrui Hui; Shaofei Huang; Qisong Han; Yaxiong Wang; Lechao Cheng; Zhedong Zheng; Zhun Zhong; Richang Hong

cs.CVAug 29, 2026

Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

Authors: Tianrui Hui, Shaofei Huang, Qisong Han, Yaxiong Wang, Lechao Cheng, Zhedong Zheng, Zhun Zhong, Richang Hong,

Organizations: Hefei University of Technology Hefei, Anhui, China · Cardiff University Cardiff, Wales, United Kingdom · University of Macau Macau, China

Abstract

Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.

Figures & tables

Figure 1. Motivation of our method. (a) The previous method adopts a class-agnostic foreground definition, grouping semantically diverse objects into a highly heterogeneous positive set. (b) Our method reformulates the foreground definition of OV-AVSS to be class-specific. By explicitly grounding acoustic events to cost representations, we achieve accurate intra-category soundingness discovery and robust inter-category distractor discrimination.

Figure 2. Overall architecture of our method. In the Audio-Modulated Cost Generation module, category-wise cost maps are obtained by correlating visual features from CLIP-V with category text features from CLIP-T. Audio features from CLAP-A are injected into CLIP-V to modulate visual features and softly highlight sounding regions. The resulting cost maps are further refined in the Multimodal-Guided Cost Refinement module to produce sharper spatial boundaries and better temporal consistency. The refined cost maps are then fed into a mask decoder to generate category-wise sounding object segmentation masks. During training, we apply the Synergistic Distractor Mining (SDM) strategy to the generated cost maps to select acoustically and semantically confusing distractor categories for more discriminative learning.

		Vision	Text	mIoU
Method	Reference	Encoder	Encoder	Unseen	Seen	Harmonic	Overall
Closed-Set Methods
TPAVI ( Zhou et al., 2022 ; Zhou et al., 2025 )	[ECCV’22]	ResNet-50	-	0.00	28.45	0.00	18.08
CATR ( Li et al., 2023 )	[ACM MM’23]	ResNet-50	-	0.00	30.64	0.00	19.73
VCT ( Huang et al., 2025a ) *	[CVPR’25]	Swin-Base	-	0.00	52.09	0.00	32.00
Zero-Shot Method

Table 1. Comparison with state-of-the-art methods on the AVSBench-OV dataset. Our method obtains significant performance gains for the challenging unseen categories with the same CLIP model as previous methods. * denotes the results are reproduced by us using the authors’ released code. Best results are presented in bold .

Figure 3. A simplified example of Synergistic Distractor Mining. $k_{a}$ acoustic distractors are selected based on the feature similarities between the input audio and category texts. $k_{s}$ semantic distractors are selected based on text feature similarities between ground-truth and other categories. Here, $k_{a}$ and $k_{s}$ are set to $2$ for better illustration.

				mIoU
AMCG	AGTA	SDM-A	SDM-S	Unseen	Seen	Harmonic	Overall
				26.47	56.66	36.08	44.40
✓				29.11	58.66	38.91	46.91
	✓			28.44	58.29	38.23	46.67
✓	✓			30.45	60.65	40.54	48.87
✓	✓	✓		32.32	59.27	41.83	48.58

Table 2. Ablation study on different components of our AGCL framework. SDM-A and SDM-S denote mining acoustically and semantically confusing distractors, respectively.

	mIoU
Method	Unseen	Seen	Harmonic	Overall
-	26.47	56.66	36.08	44.40
SA	26.68	57.78	36.51	45.40
CA	27.43	58.92	37.44	46.10
AMCG	29.11	58.66	38.91	46.91

Table 3. Ablation study on different approaches of incorporating audio cues for cost generation in our AMCG module.

Figure 4. Qualitative comparison between OV-AVSS ( Guo et al., 2024 ) and our AGCL on both seen and unseen categories. Sounding categories correspond to masks of the same colors.

Figure 5. Visualization of cost maps. (a) Original video frame. (b) Ground-truth mask. (c) Our prediction. (d) Cost maps generated by the original CLIP model. (e) Cost maps generated in our AMCG module. (f) Cost maps after refinement.

	mIoU
Guidance	Unseen	Seen	Harmonic	Overall
w/o AGTA	29.11	58.66	38.91	46.91
No Guide	29.27	60.00	39.34	48.00
Visual	29.70	60.01	39.73	48.19
Audio	30.45	60.65	40.54	48.87

Table 4. Ablation study on different types of guidance features used in the AGTA module.

		VPO (MS)			VPO (MSMI)
Method	Type	MIoU $\uparrow$	$F_{\beta}\uparrow$	FDR $\downarrow$	MIoU $\uparrow$	$F_{\beta}\uparrow$	FDR $\downarrow$
TPAVI ( Zhou et al., 2022 )	ID	44.08	58.14	30.82	50.37	66.80	29.82
AVSegFormer ( Gao et al., 2024 )	ID	56.46	71.89	24.66	50.96	64.96	32.72
CAVP ( Chen et al., 2024 )	ID	61.85	74.60	20.24	57.22	72.26	24.04
OV-AVSS ( Guo et al., 2024 )	OOD	45.04	56.42	43.51	42.30	53.18	47.59
AGCL (Ours)	OOD	57.72	72.91	24.05	54.24	68.65	29.52

Table 5. Evaluation results of our AGCL w/o training on the VPO dataset. “OOD” and “ID” denotes out-of-distribution and in-distribution, respectively. Best and Second Best .

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation
Mar 9, 2026Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn +1Audio-Visual Understanding Continual Learning
Purdue University, West Lafayette, IN 47906, USA
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation
Mar 15, 2026Kai Peng, Yunzhe Shen, Miao Zhang +6Audio-Visual Understanding Multimodal Fusion
Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Yidong Han, Yongri Piao, and Huchuan Lu are with Dalian University of Technology, Dalian 116024, China · Wei Ji is with the School of Medicine, Yale University, New Haven, CT 06520 USA · Jingjing Li is with the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T5V 1A4, Canada

Mar 9, 2026J/K move · Enter open · S save

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu

Purdue University, West Lafayette, IN 47906, USA

Audio-Visual Understanding Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

Audio-Visual Segmentation (AVS) aims to produce pixel-level masks of sound producing objects in videos, by jointly learning from audio and visual signals. However, real-world environments are inherently dynamic, causing audio and visual distributions to evolve over time, which challenge existing AVS systems that assume static training settings. To address this gap, we introduce the first exemplar-free continual learning benchmark for Audio-Visual Segmentation, comprising four learning protocols across single-source and multi-source AVS datasets. We further propose a strong baseline, ATLAS, which uses audio-guided pre-fusion conditioning to modulate visual feature channels via projected audio context before cross-modal attention. Finally, we mitigate catastrophic forgetting by introducing Low-Rank Anchoring (LRA), which stabilizes adapted weights based on loss sensitivity. Extensive experiments demonstrate competitive performance across diverse continual scenarios, establishing a foundation for lifelong audio-visual perception. Code is available at

{}^{*}

\footnote{Paper under review} - \hyperlink{https://gitlab.com/viper-purdue/atlas}{https://gitlab.com/viper-purdue/atlas} \keywords{Continual Learning \and Audio-Visual Segmentation \and Multi-Modal Learning}

Abstract

Audio-Visual Understanding

Explore similar work

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation