Generalizing Multimodal Foundational Models in Context-Driven Class Incremental Learning

Avi Gupta; Nilotpal Sinha; Vishnu Raj; Sambuddha Saha; Pratik Joshi; Koteswar Rao Jerripothula; Tammam Tillo

cs.CVOct 19, 2026

Generalizing Multimodal Foundational Models in Context-Driven Class Incremental Learning

Authors: Avi Gupta, Nilotpal Sinha, Vishnu Raj, Sambuddha Saha, Pratik Joshi, Koteswar Rao Jerripothula, Tammam Tillo

Organizations: Indraprastha Institute of Information Technology, Delhi · Dolby Laboratories · Indian Institute of Technology, Kanpur · TYUST, China

Abstract

Class-Incremental Learning (CIL) aims to learn new classes continuously without forgetting the previously learned ones. Recent works in CIL have significantly advanced across various modalities; however, at the intersection of audio-visual modality, this problem remains underexplored. While representations from multimodal foundational models (MFMs) encapsulate rich static priors, our preliminary analysis shows that these representations struggle in incremental settings. Motivated by this, we propose a framework that generalizes frozen MFMs to audio-visual CIL. Specifically, we leverage their dense audio and visual representations and employ an attention-based contextual guidance module, in which the audio representations serve as task-stable contextual guidance for the visual representations, enabling effective class recognition in the incremental setting. To further mitigate catastrophic forgetting and preserve stability across tasks, we employ feature- and logit-level distillation, which are primarily responsible for the reduced forgetting. Extensive evaluations across various audio-visual CIL benchmarks demonstrate that our approach consistently outperforms state-of-the-art methods. Code is available at https://github.com/avigupta2798/MFM-CIL.

Figures & tables

Figure 1: Comparison of standard audio-visual feature extractor approach with MFMs. Standard approach [ 57 ] : Traditional method processes incoming visual and audio data through stand-alone feature extractors. The t-SNE plots on the AVE dataset from incremental tasks (Task 0 to Task 3) show the overlap and cause clutter, leading to severe catastrophic forgetting, reflected in the decline in testing accuracy [Top-right] and high forgetting [Bottom-right] ( ). MFM-based Approach : MFMs as feature extractor projects both visual features and audio features into a shared latent space, thereby maintaining clear, distinctly separated class clusters as new tasks arrive and also reflecting in high test accuracy and less forgetting throughout the incremental learning tasks ( ). Linear-Probe Naive concatenation of MFM’s features with no distillation losses and contextual guidance for analyzing the impact of MFMs ( ).

Figure 2: [Left] The overall architecture. At current task $t$ , audio ( $z_{a}$ ) and visual ( $z_{v}$ ) representations are derived from audio signal ( $x_{a}$ ) and video frames ( $x_{v}$ ) using frozen audio ( $\psi^{model}_{a}$ ) and visual ( $\psi^{model}_{v}$ ) encoder, respectively of a Foundational Multimodal model ( $model$ ). The audio representations are utilized as contextual guidance for the corresponding visual representations in a transformer-based attention module. The final video representations ( $z^{\prime}_{t,v}$ ) are passed through the classifier for precise class prediction ( $\hat{y}_{t}$ ). The model is trained using $\mathcal{L}_{ce}$ (for class prediction), $\mathcal{L}_{fd}$ (distillation at feature level), and $\mathcal{L}_{kd}$ (distillation at logits level). [Top-Right] We leverage three different multimodal foundational models ( $model\in\{sam,ibind,qwen\}$ ) to support our hypothesis.

Models (model)	$d^{model}$	$T_{v}$	$T^{model}_{a}$
SAM-Audio	1024	16	59
ImageBind	1024	16	1
Qwen2.5-Omni-3B	2048	16	251

Table 1: Architecture details.

Figure 3: Comparative analysis of task-wise test accuracy (

\%

) between AV-CIL [ 57 ] and our proposed approach (on ImageBind). (a) Individual task performance for the AVE-CI dataset on 4 incremental tasks. (b) Individual task performance for the VS100-CI dataset on 10 incremental tasks. (c) Individual task performance for the K-S-CI dataset on 5 incremental tasks.

Methods	Exemplar	AVE-CI		VS100-CI		K-S-CI
Methods	Exemplar	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$
Fine-tuning	✗	42.40	70.99	26.21	89.37	41.18	89.62
LwF [ 46 ]	✗	58.07	26.90	59.34	23.01	65.54	16.55
iCaRL-NME [ 62 ]	✓	56.15	11.71	56.19	12.80	64.51	18.70
iCaRL-FC [ 62 ]	✓	65.88	26.08	64.22	29.94	65.54	40.57
SS-IL [ 4 ]	✓	61.94	22.49	69.20	9.75	69.71	10.53

Table 2: Comparison of our proposed approach with previous works on different datasets.

\dagger

represents results that are produced for fair comparison. Best results (

\%

) are marked in Bold .

Methods	AVE-CI		VS100-CI		K-S-CI
Methods	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$
AV-CIL † (SAM-Audio)	84.38	2.24	74.04	4.11	80.21	1.45
Ours (SAM-Audio)	88.68	0.42	79.64	0.34	84.01	0.43
AV-CIL † (Qwen2.5)	82.98	5.91	80.43	5.40	79.32	4.28
Ours (Qwen2.5)	86.14	4.18	82.40	1.20	81.85	3.84
AV-CIL † (ImageBind)	85.80	3.04	77.70	7.72	77.96	7.09

Table 3: Comparison of our proposed approach on different MFMs with AV-CIL [ 57 ] .

\dagger

represents results that are produced for fair settings. Best results (

\%

) are marked in Bold .

Figure 4: Analysis of AV-CIL & ours on different backbones (AVE-CI).

Methods	AVE-CI		VS100-CI		K-S-CI
Methods	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$
Linear Probe (SAM-Audio)	77.62	3.43	61.31	4.75	79.71	2.87
Ours (SAM-Audio)	88.68	0.42	79.64	0.34	84.01	0.43
Linear Probe (Qwen2.5)	72.30	9.13	73.36	6.15	72.84	7.31
Ours (Qwen2.5)	86.14	4.18	82.40	1.20	81.85	3.84
Linear Probe (ImageBind)	80.17	4.47	70.22	8.47	71.23	7.54

Table 4: Comparison of our proposed approach on different MFMs with the naive concatenation (Linear Probe). Best results (

\%

) are in Bold .

Figure 5: Comparative analysis among MFM-based approaches for different task sequences using mean test accuracy (

\%

). (a) Three task sequences for AVE-CI, 4 tasks (7 classes per task), 7 tasks (4 classes per task), and 14 tasks (2 classes per task); (b) Four task sequences for VS100-CI, 2 tasks (50 classes per task), 5 tasks (20 classes per task), 10 tasks (10 classes per task), 20 tasks (5 classes per task); (c) Four task sequences for K-S-CI, 2 tasks (15 classes per task), 5 tasks (6 classes per task), 6 tasks (5 classes per task), 15 tasks (2 classes per task).

$\mathcal{L}_{ce}$	$\mathcal{L}_{kd}$	$\mathcal{L}_{fd}$	SAM-Audio		ImageBind		Qwen2.5
$\mathcal{L}_{ce}$	$\mathcal{L}_{kd}$	$\mathcal{L}_{fd}$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$	$\mathcal{A}_{m}\uparrow$	$\mathcal{F}_{avg}\downarrow$
✓	✗	✗	87.26	4.29	86.55	3.18	83.20	7.32
✓	✓	✗	87.34	2.30	87.60	2.80	83.94	5.45
✓	✗	✓	87.79	1.32	87.78	1.02	84.49	5.01
✓	✓	✓	88.68	0.42	91.48	0.52	86.14	4.18

Table 5: Effectiveness of different loss components (

\mathcal{L}_{ce}

\mathcal{L}_{fd}

, and

\mathcal{L}_{kd}

) on AVE-CI dataset. Bold denotes the best results.

$\lambda_{fd}$	$\lambda_{kd}$	AVE-CI			K-S-CI
$\lambda_{fd}$	$\lambda_{kd}$	SAM-Audio	ImageBind	Qwen2.5	SAM-Audio	ImageBind	Qwen2.5
0.2	1.0	88.04	88.66	82.92	84.50	84.88	81.52
0.4	1.0	87.90	89.33	82.68	84.36	85.13	81.52
0.6	1.0	88.06	89.48	82.41	84.07	85.11	81.49
0.8	1.0	87.92	89.36	82.57	83.72	85.05	81.46
1.0	1.0	88.68	91.48	86.14	84.01	86.29	81.85

Table 6: Sensitivity analysis of distillation hyperparameters

\lambda_{fd}

and

\lambda_{kd}

by comparing the accuracy (

\mathcal{A}_{m}

) for different MFM-based backbones. Bold denotes the adopted configuration.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Audio	Video	SAM-Audio	Qwen2.5
✓	✗	30.75	67.75
✗	✓	38.98	69.84
✓	✓	88.68	86.14

Appendix

Table 7: Effectiveness of individual modality on the AVE dataset. Bold denotes the best accuracy (

\mathcal{A}_{m}

Methods	AVE-CI				VS100-CI				K-S-CI
	AV-CIL		Ours		AV-CIL		Ours		AV-CIL		Ours
	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$	$\mathcal{A}_{m}$	$\mathcal{F}_{avg}$
AudioMAE/VideoMAE	74.04	7.63	78.06 $\pm$ 1.17	1.12 $\pm$ 0.45	72.80	5.49	79.88 $\pm$ 1.37	4.65 $\pm$ 0.55	73.06	6.48	80.24 $\pm$ 1.32	5.45 $\pm$ 0.76
SAM-Audio	84.38	2.24	88.72 $\pm$ 0.67	0.43 $\pm$ 0.03	74.04	4.11	79.11 $\pm$ 0.76	0.37 $\pm$ 0.08	80.21	1.45	83.34 $\pm$ 1.75	0.54 $\pm$ 0.12
ImageBind	85.80	3.04	91.25 $\pm$ 0.51	0.55 $\pm$ 0.06	77.70	7.72	84.01 $\pm$ 1.21	0.10 $\pm$ 0.06	77.96	7.09	86.11 $\pm$ 0.55	0.76 $\pm$ 0.54
Qwen2.5	82.98	5.91	86.97 $\pm$ 1.12	4.66 $\pm$ 0.97	80.43	5.40	82.50 $\pm$ 0.24	1.34 $\pm$ 0.32	79.32	4.28	82.20 $\pm$ 0.44	3.54 $\pm$ 0.40

Appendix

Table 8: Analysis over three random seeds for three benchmark datasets.

Backbone	Frozen Params	Trainable Params	GFLOPs	Latency	VRAM
SAMAudio	$\approx$ 500M	27.31M	1.06G	4.11ms	1.58GB
ImageBind	$\approx$ 700M	27.31M	0.54G	5.84ms	1.41GB
Qwen2.5	$\approx$ 1.5B	109.15M	2.17G	6.12ms	5.61GB

Appendix

Table 9: Compute/Memory overhead for different backbones.

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
Step 0	86.15	90.39	91.34	95.20	90.38	92.30
Step 1	85.23	91.43	87.14	95.71	83.33	86.19
Step 2	83.01	86.35	84.75	86.67	81.58	83.80
Step 3	83.14	86.55	79.90	88.32	76.64	82.23

Appendix

Table 10: Comparison of our proposed approach on different MFMs with AV-CIL on AVE-CI dataset 4 Tasks (7 classes per task) . Best results (

\%

) are marked in Bold .

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
Step 0	83.14	91.33	92.33	93.34	82.54	92.94
Step 1	81.84	85.70	86.30	89.02	80.93	86.20
Step 2	78.06	83.03	84.10	87.46	80.47	87.19
Step 3	76.45	82.02	81.12	86.91	77.0	83.59
Step 4	76.36	80.61	78.80	86.24	75.31	83.35

Appendix

Table 11: Comparison of our proposed approach on different MFMs with AV-CIL on the VGGSound100-CI dataset 10 Tasks (10 classes per task) . Best results (

\%

) are marked in Bold .

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
Step 0	89.59	95.97	93.14	96.77	92.74	94.09
Step 1	80.05	85.97	77.97	85.71	77.77	81.95
Step 2	80.23	80.85	76.78	84.75	77.57	80.24
Step 3	79.27	79.42	73.98	83.17	76.46	78.64
Step 4	74.23	77.83	67.93	81.05	72.04	74.31

Appendix

Table 12: Comparison of our proposed approach on different MFMs with AV-CIL on Kinetics-CI dataset 5 Tasks (6 classes per task) . Best results (

\%

) are marked in Bold .

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
7 Tasks (4 classes per task)
Step 0	86.55	99.2	90.00	98.89	88.00	98.28
Step 1	86.36	70.0	77.00	80.0	55.45	63.63
Step 2	85.45	86.93	75.34	91.47	78.07	78.98
Step 3	83.83	88.07	74.36	92.59	62.30	68.72

Appendix

Table 13: Task-wise Comparison of our proposed approach on different MFMs with AV-CIL on AVE-CI dataset. Best results (

\%

) are marked in Bold .

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
5 Tasks (20 classes per task)
Step 0	84.36	91.74	91.74	93.66	82.35	88.78
Step 1	79.38	80.51	81.04	83.38	76.25	72.92
Step 2	73.20	75.65	76.93	79.28	71.62	85.89
Step 3	70.17	75.16	71.44	77.76	67.83	82.46

Appendix

Table 14: Task-wise Comparison of our proposed approach on different MFMs with AV-CIL on VGGSound100-CI dataset. Best results (

\%

) are marked in Bold .

Steps	SAMAudio		ImageBind		Qwen2.5
Steps	AV-CIL	Ours	AV-CIL	Ours	AV-CIL	Ours
6 Tasks (5 classes per task)
Step 0	89.19	95.82	92.74	98.12	84.35	95.18
Step 1	81.42	86.09	77.58	86.75	70.90	83.28
Step 2	70.77	76.04	76.39	85.62	69.68	76.04
Step 3	79.37	81.75	72.00	82.91	66.76	79.35

Appendix

Table 15: Task-wise Comparison of our proposed approach on different MFMs with AV-CIL on Kinetics-CI dataset. Best results (

\%

) are marked in Bold .

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio
Jun 9, 2026Avi Gupta, Nilotpal Sinha, Vishnu Raj +4Class-Incremental Learning Audio-Visual Representation Learning
IIIT Delhi · Dolby Laboratories · IIT Kanpur +1
BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
Nov 14, 2025Lan Li, Tao Hu, Da-Wei Zhou +3Class-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China
Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University

Jun 9, 2026J/K move · Enter open · S save

Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio

Avi Gupta, Nilotpal Sinha, Vishnu Raj, Sambuddha Saha+3

IIIT Delhi · Dolby Laboratories · IIT Kanpur+1

Class-Incremental Learning Audio-Visual Representation Learning Incremental Learning

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. While recent CIL advances have spurred significant interest across various modalities, the audio-visual setting remains underexplored. Furthermore, although foundational multimodal models like SAM-Audio encapsulate rich static priors, our empirical analysis reveals that these representations struggle in incremental settings. This work bridges this gap by integrating SAM-Audio's audio-visual priors into the CIL setting. Specifically, we leverage its dense audio and visual representations and employ a novel guided attention strategy where the audio features contextually guide the visual representations. To further mitigate catastrophic forgetting, we introduce dual-level distillation objectives at both the feature and logit levels. Extensive evaluations on audio-visual CIL benchmarks demonstrate that our approach consistently outperforms state-of-the-art methods.

Abstract

Class-Incremental Learning

Explore similar work

Listen, Look, and Learn: Learning Without Forgetting through SAM-Audio

BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning

Visual Branch is What You Need for CLIP-based Class-Incremental Learning