Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

He Li; Yuhui Zhang; Xiaohan Wang; Kaifeng Lyu; Serena Yeung-Levy

cs.CVMar 15, 2026

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

Authors: He Li, Yuhui Zhang, Xiaohan Wang, Kaifeng Lyu, Serena Yeung-Levy

Organizations: Stanford University · Tsinghua University

Abstract

The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.

Figures & tables

Figure 1: Evaluation matrix. A 2 $\times$ 2 design crossing text and image . In this work, for both text and images, we define in-distribution (ID) data as samples drawn from the same probability distribution as the training set. Conversely, out-of-distribution (OOD) data originates from a distribution not encountered during training; during evaluation, we report average accuracy within each quadrant. This setup allows us to systematically evaluate a comprehensive range of training and evaluation scenarios. Further details are provided in Appendix A.1 .

Figure 2: Single-task fine-tuning across the evaluation matrix. Each curve traces checkpoints during fine-tuning: x-axis = ID accuracy on ImageNet validation (the fine-tuned task), y-axis = accuracy on an ID/OOD evaluation. Layout and colors follow Figure 1 . Legends show trainable part (method, learning rate) . Performance is largely maintained in ID T – OOD I and OOD T – OOD I with simplest regularization on parameter updata, with a notable drop only in OOD T – ID I . Full hyperparameters are in Appendix B.1 .

		Final Acc.	$\Delta$ vs. zero-shot
Trainable Part	Settings	Validation (%)	OOD T –ID I (pp)	OOD T –OOD I (pp)
LLM Backbone	Full, 1e-5	91.56	-16.56	-33.64
LLM Backbone	LoRA, 1e-4	91.08 (-0.48)	0.46	-2.97
LLM Backbone	Full, 1e-6	91.37 (-0.19)	1.06	-1.51
Vision Encoder	Full, 1e-6	90.96 (-0.60)	-1.36	0.49
Vision Encoder	Full, 1e-5	91.08 (-0.48)	-9.90	-2.76

Table 1: ID accuracy and robustness deltas across recipes. “ Final Acc ” is ImageNet-VQA validation accuracy; in parentheses we show the difference to LLM Backbone, Full, 1e-5 . “

\Delta

vs. zero-shot ” reports percentage-point change relative to the pre-trained model on OOD T –ID I and OOD T –OOD I . To enhance visual clarity, we use red to highlight performance degradations

>

3pp and blue for changes within a

\pm

3pp margin. Rows corresponding to settings where all results fall within this margin are shaded gray . This suggests that most of regularization strategies mitigate catastrophic forgetting without compromising the model’s learning capacity.

Table 2: Generalization of the recipe. The default setting referenced in § 3.2 is shaded in gray . The results show that all findings in § 3.1 are consistent across: (a) different model sizes and families; (b) rare domains including surgery and microscopy; (c) different fine-tuning datasets size; Full training details are in Appendix B.2 . More experiments on the generalizability of our findings could be found in Appendix D .

Figure 3: ImageWikiQA with class-label distractors. Left: an example transformation where one distractor is replaced by the correct class name. Right: accuracy with/without a class-name distractor, before fine-tuning and after fine-tuning, using LLM Backbone, Full, 1e-6 . The substantial decrease in accuracy and the concurrent increase in “mischoice on class name” after fine-tuning indicate that the model ceases to follow prompt instructions, instead defaulting to outputting the choice with class label directly. Therefore, the primary issue is task-specific overfitting rather than catastrophic forgetting.

Panel of Figure 4 — Figure 4: Ablations for data-hybrid training. (a) Mixing ImageNet-VQA with Flowers102, OCR-VQA, or LLaVA-665K (each at 50% of training instances). (b) Varying the LLaVA-665K mix from 0% to 70%; larger, darker markers denote higher ratios. Augmenting the training data with diverse textual inputs helps to alleviate task-specific overfitting . Consequently, this data-hybrid method improves model robustness in the OOD T – ID I setting with minimal trade-offs for ID performance. Training details are in Appendix B.3 .

Method	RS (%)		Med (%)		AD (%)		Sci (%)		Fin (%)
	Last	Average	Last	Average	Last	Average	Last	Average	Last	Average
Zero-shot	32.29	-	28.28	-	15.59	-	35.55	-	62.56	-
w/ replay buffer
LoRA	29.57	80.87	29.19	58.60	7.09	38.95	19.55	36.41	63.60	36.78
MoELoRA	40.23	80.00	23.58	56.91	5.19	34.69	18.35	31.70	74.89	31.36
O-LoRA	76.21	80.13	51.34	70.23	36.50	61.35	42.64	53.34	90.20	59.38

Table 3: Continual learning on the MLLM-CL benchmark. We highlight best and second best separately for with and without replay. Our simple methods (IncLoRA, SeqFull) are competitive with specialized approaches under replay, and dominate most columns without replay.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Image	Question	Choices
	What is the identity of this cell captured using brightfield in LED array microscopy?	A. eosinophil B. neutrophil C. t_lymphocyte D. plasma_cell E. debris_or_artifact F. unclassified_cell
	What type of white blood cell is shown in this dark field (50 illumination) microscopy image?	A. plasma_cell B. nk_lymphocyte C. b_lymphocyte D. neutrophil E. unclassified_cell F. debris_or_artifact
	Based on the morphological features visible in this differential phase contrast (left illumination) image, what is the cell type?	A. basophil B. unclassified_cell C. debris_or_artifact D. t_lymphocyte E. blast_cell F. lymphocyte

Appendix

Table 4: VQA Dataset Curated from BSCCM

Image	Question	Choices
	What is the major surgical instrument being used in this frame?	A. tissue_glue B. retractable_knife C. haemostatic_foam D. no_secondary_instrument E. out_of_patient F. no_visible_instrument/occluded_image_inside_patient
	What is the major surgical instrument being used in this frame?	A. plasma_cell B. nk_lymphocyte C. b_lymphocyte D. neutrophil E. unclassified_cell F. debris_or_artifact
	What is the major surgical instrument being used in this frame?	A. out_of_patient B. ring_curette C. no_visible_instrument/occluded_image_inside_patient D. freer_elevator E. micro_doppler F. no_secondary_instrument

Appendix

Table 5: VQA Dataset Curated from BSCCM

Table 10

Table 11

Config	Value
Optimizer	AdamW
Batch Size	16
Learning Rate Schedule	cosine decay
Warmup Ratio	0.1
Learning Rate	$1\text{\times}{10}^{-6}$
Training Steps	80000

Appendix

(a) Configuration for ablation study across hybriding different datasets and different hybrid ratio.

Table 13

	MMMU-dev (%)	MMMU-val (%)	VMCBench (%)
Before Fine-tuning	48.00%	48.44%	75.20%
After Fine-tuning	48.67%	45.00%	74.20%

Appendix

Table 14

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 746 papers

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
Aug 12, 2026Juncheng Liao, Jinfan Lv, Guoming Wang +3Continual Learning for LLMs LLM Fine-Tuning
School of Software Technology, Zhejiang University, Hangzhou, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · School of Artificial Intelligence, Sun Yat-Sen University, Guangzhou, China +1
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
Jul 13, 2026Jing Liu, Chenxuanyin Zou, Jiayang Ren +5Multimodal Federated Learning Generative Replay
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · College of Future Information Technology, Fudan University, Shanghai 200433, China · Department of Chemical and Biological Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada +3
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Jun 22, 2026Chuangxin Zhao, Canran Xiao, Siyuan Ma +5Visual Question Answering Cross-Modal Attention
The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University · Nanyang Technological University +2

Aug 12, 2026J/K move · Enter open · S save

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Juncheng Liao, Jinfan Lv, Guoming Wang, Jupeng Zheng+2

School of Software Technology, Zhejiang University, Hangzhou, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · School of Artificial Intelligence, Sun Yat-Sen University, Guangzhou, China+1

Continual Learning for LLMs LLM Fine-Tuning Multimodal Large Language Models Catastrophic Forgetting

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.

Abstract

LLM Fine-Tuning

Explore similar work

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs