Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

Siddeshwar Raghavan; Gautham Vinod; Bruce Coburn; Fengqing Zhu

cs.CVMar 9, 2026

Can You Hear, Localize, and Segment Continually? An Exemplar-Free Continual Learning Benchmark for Audio-Visual Segmentation

Authors: Siddeshwar Raghavan, Gautham Vinod, Bruce Coburn, Fengqing Zhu

Organizations: Purdue University, West Lafayette, IN 47906, USA

Abstract

Audio-Visual Segmentation (AVS) aims to produce pixel-level masks of sound producing objects in videos, by jointly learning from audio and visual signals. However, real-world environments are inherently dynamic, causing audio and visual distributions to evolve over time, which challenge existing AVS systems that assume static training settings. To address this gap, we introduce the first exemplar-free continual learning benchmark for Audio-Visual Segmentation, comprising four learning protocols across single-source and multi-source AVS datasets. We further propose a strong baseline, ATLAS, which uses audio-guided pre-fusion conditioning to modulate visual feature channels via projected audio context before cross-modal attention. Finally, we mitigate catastrophic forgetting by introducing Low-Rank Anchoring (LRA), which stabilizes adapted weights based on loss sensitivity. Extensive experiments demonstrate competitive performance across diverse continual scenarios, establishing a foundation for lifelong audio-visual perception. Code is available at ${}^{*}$ \footnote{Paper under review} - \hyperlink{https://gitlab.com/viper-purdue/atlas}{https://gitlab.com/viper-purdue/atlas} \keywords{Continual Learning \and Audio-Visual Segmentation \and Multi-Modal Learning}

Figures & tables

Figure 1: An overview of Exemplar Free Continual Learning Benchmark for Audio-Visual Segmentation

Figure 2: Overview of ATLAS: The framework performs exemplar-free continual audio-visual segmentation using frozen encoder backbones with parameter-efficient LoRA adapters in the visual pathway. Audio-guided pre-fusion conditioning is followed by cross-attention, where conditioned visual features act as queries and audio features serve as keys and values. To mitigate catastrophic forgetting across tasks, our Low-Rank Anchoring (LRA) module dynamically restricts LoRA parameter drift. The fused representation is decoded for segmentation with an optional classification branch.

	SS-AVS (11-2 split)						MS-AVS (31-5 split)
	TIL		CIL		DIL		TF-CL
	Avg mAP	Avg For	Avg mAP	Avg For	Avg mAP	Avg For	Avg mAP	Avg For
Finetune	27.91	7.85	28.04	10.89	25.38	9.30	16.61	11.58
LwF [ 28 ] [TPAMI ‘16]	26.70	10.34	26.89	7.52	24.79	12.52	21.59	13.43
EWC [ 22 ] [PNAS ‘17]	54.90	23.12	55.03	23.52	41.37	23.47	18.89	31.81
SI [ 48 ] [ICML ‘17]	59.10	13.74	56.52	15.57	42.44	22.77	19.51	33.25

Table 1: Performance on the Continual AVS Benchmark exemplar-free CL methods adapted to AVS; static AVS models extended to continual learning; continual segmentation adapted to AVS; the CL-AVS baseline with replay; and our method. Performance is measured by mAP across tasks and average backward forgetting.

Figure 3: Qualitative AVS results from the top four methods in our CL-AVS setting : Input frame, predicted binary mask for the middle frame is shown along with the ground truth mask. (Left) SS-AVS CIL and (Right) MS-AVS TF-CL

Figure 4: Forward Transfer vs. Forgetting: Trade-off between forward transfer and forgetting across models in the CL-AVS benchmark on the SS-AVS dataset under the TIL and DIL protocols.

Figure 5: Heatmap of MaxF scores on the SS-AVS test set under the CIL protocol with an 11–2 split, evaluated after each incremental task. Four best-performing methods are included for clarity.

			SS-AVS (11-2 split)				MS-AVS (31-5 split)
Method	Pre-cond.	LRA	Avg mAP	Avg For	Avg F1	Avg mIoU	Avg mAP	Avg For	Avg F1	Avg mIoU
ATLAS	✓	✓	74.67	9.42	62.76	50.85	47.53	23.71	42.27	39.44
ATLAS	✗	✓	71.17	12.11	59.97	47.78	43.12	25.07	39.38	38.2 7
ATLAS	✓	✗	67.18	18.03	50.38	39.16	34.41	26.61	36.72	24.12
ATLAS	✗	✗	65.33	21.00	47.31	33.58	29.19	27.01	31.16	20.12
AVSBench	-	-	63.84	22.05	50.63	37.28	31.15	21.09	34.39	20.15

Table 2: Component ablation of ATLAS. We analyze the contribution of audio-guided pre-conditioning and Low-Rank Anchoring (LRA) on the CL-AVS benchmark

Figure 6: Radar plot summarizing normalized performance across six dimensions: Accuracy, Retention, Plasticity, Stability, Speed, and Memory Efficiency for the top six performing methods on the SS-AVS dataset under the CIL protocol and the MS-AVS dataset under the TF-CL protocol.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Taming Modality Entanglement in Continual Audio-Visual Segmentation
Oct 20, 2025Yuyang Hong, Qi Yang, Tao Zhang +5Audio-Visual Understanding Replay-Based Continual Learning
School of Artificial Intelligence, UCAS · 2MAIS, Institute of Automation · School of Intelligent Science and Technology, University of Science and Technolog Beijing
Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation
Dec 23, 2025Jingqi Tian, Yiheng Du, Haoji Zhang +6Audio-Visual Understanding Audio-Visual Representation Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University, China · Peking University, China
LightAVSeg: Lightweight Audio-Visual Segmentation
May 9, 2026Qing Zhong, Guodong Ding, Lingqiao Liu +3Audio-Visual Understanding Audio-Visual Alignment
College of Informatics, Huazhong Agricultural University, Wuhan, China · School of Computing, National University of Singapore, Singapore · School of Computer Science, Adelaide University, Australia +2

Oct 20, 2025J/K move · Enter open · S save

Taming Modality Entanglement in Continual Audio-Visual Segmentation

Yuyang Hong, Qi Yang, Tao Zhang, Zili Wang+4

School of Artificial Intelligence, UCAS · 2MAIS, Institute of Automation · School of Intelligent Science and Technology, University of Science and Technolog Beijing

Audio-Visual Understanding Replay-Based Continual Learning

Open paper AskPDF arXiv

Recently, significant progress has been made in multi-modal continual learning, aiming to learn new tasks sequentially in multi-modal settings while preserving performance on previously learned ones. However, existing methods mainly focus on coarse-grained tasks, with limitations in addressing modality entanglement in fine-grained continual learning settings. To bridge this gap, we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to continuously segment new classes guided by audio. Through comprehensive analysis, two critical challenges are identified: 1) multi-modal semantic drift, where a sounding objects is labeled as background in sequential tasks; 2) co-occurrence confusion, where frequent co-occurring classes tend to be confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework is designed to address these challenges. Specifically, for multi-modal semantic drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select samples with high modal consistency for rehearsal. Meanwhile, for co-occurence confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed, allowing for the increase of rehearsal sample frequency of those confusable classes during training process. Moreover, we construct three audio-visual incremental scenarios to verify effectiveness of our method. Comprehensive experiments demonstrate that our method significantly outperforms single-modal continual learning methods. Code can be seen at https://github.com/cqu-student/CAVS-CMR.

Abstract

Audio-Visual Understanding

Explore similar work

Taming Modality Entanglement in Continual Audio-Visual Segmentation

Delayed Bidirectional Alignment via Disentangled Audio Semantics for Audio-Visual Segmentation

LightAVSeg: Lightweight Audio-Visual Segmentation