How to Teach Large Multimodal Models New Skills

Zhen Zhu; Yiming Gong; Yao Xiao; Yaoyao Liu; Derek Hoiem

cs.AIOct 9, 2025

How to Teach Large Multimodal Models New Skills

Authors: Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, Derek Hoiem

Organizations: University of Illinois Urbana-Champaign · Google DeepMind · Carnegie Mellon University

Abstract

How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. Surprisingly, we find that performance lost on held-out tasks after fine-tuning on one skill can partly recover when the model is subsequently tuned on a different skill. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that shows the shift co-varies with forgetting. Guided by this insight, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers (SA Proj., $Δ$ learning +24.9 / $Δ$ held-out forgetting -0.6), and (ii) updating only the MLP Gate&Up while freezing the Down projection (+30.5 / -2.1). Both substantially outperform full-LLM tuning (+31.8 / -23.3) in the learning-forgetting trade-off. We also compare against common forgetting mitigation methods: Learning without Forgetting (LwF), LoRA, Mixture-of-Experts, and weight-space interpolation (WiSE-FT), and find that our selective tuning recipes match or exceed their learning-stability balance while remaining simpler, requiring no replay, auxiliary parameters, or per-stage tuning. These results hold across LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL, confirming that the key to teaching LMMs new skills without forgetting lies in controlling output distribution shift by choosing which components to tune. Code will be made available.

Figures & tables

Figure 1: Surprising Forgetting Behavior in LMMs : Left : When fine-tuning most components on one target task, we see major improvement in that task (“Learning”) but a substantial drop in performance of other tasks (“Forgetting”, total across tasks shown here), as expected. But if we only tune self-attention projection layers (SA Proj.) in the language model, we still get substantial learning on the target task with minimal forgetting. Right : Even fine-tuning SA Proj. for multiple tasks sequentially, we see no forgetting. For others, we see large forgetting on the PixmoCount task, but the models somehow partly recover what they "forgot" in learning the next specialized task. Our paper documents and analyzes these and other interesting phenomena of learning and forgetting in LMMs, leading to simple and effective ways to teach LMMs new skills.

Figure 2: Architecture of our evaluated LMMs. The input contains visual inputs such as images or videos, which are converted to visual tokens by the vision encoder, and text input is processed by a tokenizer containing a visual placeholder token <image> . Visual tokens are converted by the projector and concatenated with text tokens as input for the language model. We visualize the architecture of the transformer decoder layer of the language model. "LN", "MHA", and "MLP" represent layer norm, multi-head attention, and multi-layer perceptron, respectively. $r^{(l)}$ is the final output of layer $l$ .

Method	Target Learning	Target Forgetting	Target Overall	Held-out Forgetting
Baseline	43.9	0.0	43.9	76.4
Full	$+29.9$	$-25.9$	$+9.2$	$-27.4$
\|-- Vision Encoder	$+9.6$	$-12.7$	$-0.5$	$-10.8$
\|-- Projector	$+2.3$	$-0.8$	$+1.7$	$-1.3$
\|-- LLM	$+31.8$	$-23.5$	$+13.0$	$-23.3$
\|-- SA Proj.	$+24.9$	$-2.3$	$+23.1$	$-0.6$

Table 1: Effect of tuning different components: learning, forgetting, and overall performance, averaged over three five-task sequences. Cells are colored using a blue-orange colormap to show performance changes. Blue indicates a positive change, where a darker shade is better. Orange indicates a negative change, where a lighter shade is better. We underline numbers that do not reflect a significantly different task-average distribution from the best, based on a two-sided paired sample t-test.

Figure 3: Learning–forgetting tracks output–distribution shift. On LLaVA - OneVision tuned for counting, we plot five curves over log - spaced steps for LLM, SA Proj., MLP, MLP (Gate&Up) and MLP (LwF). The dashed line represents the base model. Left: PixmoCount accuracy rises for all methods. Middle: mean held - out performance drops sharply for LLM and MLP, remains nearly unchanged for SA Proj., and is preserved by MLP (LwF); Right: the expected likelihood of number tokens on non - counting captions (LCS - 558K [ 34 ] ) surges for LLM and MLP, stays near baseline for SA Proj., and has little changes for MLP (LwF).

Figure 4: Visualizations on counting and captioning examples after tuning tuning MLP and SA Proj. on the counting task. The counting and examples are sampled from the PixmoCount dataset and the LCS - 558K [ 34 ] dataset, respectively.

Method	LLaVA-NeXT (LLaMA-3 8B)				Qwen2.5-VL (7B)
Method	T. Learn	T. Forget	T. Overall	H. Forget	T. Learn	T. Forget	T. Overall	H. Forget
Baseline	31.5	0.0	31.5	59.9	52.1	0.0	52.1	77.9
Full	$+31.7$	$-20.3$	$+15.4$	$-32.0$	$+17.3$	$-5.2$	$+13.1$	$-17.5$
\|-- Vision + Projector	$+0.1$	$-1.8$	$-1.3$	$-13.4$	$+12.1$	$-9.1$	$+4.9$	$-6.2$
\|-- LLM	$+36.2$	$-21.2$	$+19.3$	$-35.9$	$+16.8$	$-5.9$	$+12.1$	$-24.6$
\|-- SA Proj.	$+28.3$	$-7.9$	$+21.9$	$-7.7$	$+16.1$	$-1.6$	$+14.9$	$+0.6$

Table 2: Component-level tuning experiments with LLaVA-NeXT and Qwen2.5-VL . "T" represents "Target" and "H" is for "Held-out". Underlined text of each column denotes the best method.

Figure 5: Comparison of different continual learning techniques in the default sequential task curriculum. For LwF, WiSE-FT, only the MLP layers are tuned. LoRA adapters are wrapped only on the MLP layers. MoE is also applied to the MLP layers.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Components	#Params
$\Theta_{\mathrm{VE}}$	SigLIP vision encoder	$\approx 400\,$ M
$\Theta_{\mathrm{Proj}}$	Multimodal projector	$\approx 20\,$ M
$\Theta_{\mathrm{SA}}$	All blocks: $W_{Q},W_{K},W_{V},W_{O}$	$\approx 822\,$ M
$\Theta_{\mathrm{MLP}}$	All blocks: $W_{\mathrm{gate}},\,W_{\mathrm{up}},\,W_{\mathrm{down}}$	$\approx 5{,}703\,$ M
$\Theta_{\mathrm{Emb}}$	Input token embeddings	$\approx 545\,$ M
$\Theta_{\mathrm{LM}}$	LM head $U$	$\approx 545\,$ M

Appendix

Table 3: Parameter groups and counts for LLaVA OneVision Qwen2-7B

Index / Token		Index / Token		Index / Token		Index / Token
15	0	16	1	17	2	18	3
19	4	20	5	21	6	22	7
23	8	24	9	603	one	1960	ten
3966	One	5225	ONE	11613	Two	14154	zero
17999	Zero	19641	Three	19789	two	26972	Four
27856	three	32687	Ten	34024	four	37020	Five

Appendix

Table 4: Numeric token indices and their corresponding tokens.

Method	CUB200		PixmoCount		PathVQA		TextVQA		TimeClock		Average
Method	Target	Held-out	Target	Held-out	Target	Held-out	Target	Held-out	Target	Held-out	Target	Held-out
Baseline	53.7	76.4	52.4	76.4	36.3	76.4	76.0	76.4	1.1	76.4	43.9	76.4
Proj.	+5.7	-0.0	+4.2	-0.1	+0.6	-0.1	+0.5	+0.2	+0.4	-0.5	+2.3	-0.1
Vis. Enc.	+16.1	-0.8	+11.6	-4.7	+3.7	-2.8	+1.0	-0.7	+12.7	-11.9	+9.0	-4.2
SA Proj.	+31.8	+0.3	+15.2	-0.2	+14.4	-0.3	+3.5	+0.3	+56.0	-0.1	+24.2	+0.0
MLP	+36.4	+0.3	+17.8	-4.0	+26.5	-0.4	+3.8	+0.0	+73.3	-3.1	+31.6	-1.4

Appendix

Table 5: Single task fine-tuning by component. Each individual target task is fine-tuned from the original model, and the performance for that task (“Target”) and average held-out performance (“Held-out”) is measured. Each row is for tuning a different component or set of components: “Proj.”, “Vis. Enc.”, “SA Proj.”, “MLP”, “LM”, “Full” represent tuning the projector, vision encoder, self-attention projections in the LLM, MLP in LLM, full LLM, and all parameters, respectively. “+” is an increment over the baseline (original LLaVA-OneVision-7B checkpoint), and “-” is a decrease.

Figure 6: Sequential fine-tuning by component. The target tasks in the

x

-axis are trained sequentially, from left to right. After training each task, the average performance of all target tasks ( left ) and all held-out tasks ( right ) are measured. Each line shows the performance after tuning a different component or set of components: LLaVA-OneVision (Full, Vision Encoder, Projector, LLM, SA Proj., SA Proj. (QKV),, MLP, MLP (Gate&Up)). The dashed horizontal gray line marks the average performance of the original model.

Figure 7: Sequential fine-tuning by component. Tasks are arranged as TimeClock

\rightarrow

TextVQA

\rightarrow

PathVQA

\rightarrow

PixmoCount

\rightarrow

CUB200.

Figure 8: Sequential fine-tuning by component. Tasks are arranged as PathVQA

\rightarrow

CUB200

\rightarrow

TextVQA

\rightarrow

TimeClock

\rightarrow

PixmoCount.

Figure 9: Composing stable updates. We compare SA Proj. and MLP (Gate&Up) to two compositions: SA Proj. + Gate&Up and SA Proj. (QKV only) + Gate&Up using the default five-task sequential tuning.

Figure 10: Layer-wise residual-delta magnitude across training iterations. Each subplot shows the average logit-space

\lVert\Delta\text{residual}\rVert_{2}

attributable to MLP (solid reds) and self-attention projections (dashed blues) versus transformer layer index. The five method configurations are: SA Proj., MLP, LLM, MLP (Gate & Up), and MLP (LwF). Color shade encodes checkpoint iteration (darker = later). The top-left panel is a legend; other panels omit legends for clarity.

Figure 11: Sequential tuning of LLM layer subsets. We compare tuning the full LLM against tuning only the first 23 layers or only the last 5 layers of the 28-layer Qwen2-7B model.

Method	Total Params	Trainable Params	Train SPS $\uparrow$
	(B)	(B)	(#samples / sec.)
SA Proj.	8.03	0.82	1.46
MLP (Gate&Up)	8.03	3.80	1.45
MLP	8.03	5.70	1.44
LoRA	8.51	0.50	1.27
LwF	16.06	5.70	0.81

Appendix

Table 6: Training efficiency and parameter footprint of the evaluated model variants (4 × H100 96 GB, DeepSpeed, bfloat16). Total Params is the total number of model parameters in billions. Trainable Params is the subset that requires gradient updates in billions. Train SPS is the number of training samples processed per second, collected from training models on the same 384 samples.

Figure 12: Analysis of the forgetting and recovery cycle over three sequential tasks. We plot performance on (Left) average target tasks, (Middle) average held-out tasks, and (Right) the likelihood of numeric tokens on a general held-out dataset (LCS-558K). The plots track performance after training on CUB200 (Stage 1), PixmoCount (Stage 2), and PathVQA (Stage 3). The key finding is in the rightmost plot: for methods susceptible to forgetting (e.g., LLM, MLP), the numeric token bias (output shift) spikes at Stage 2 (correlating with the performance drop) and then recedes at Stage 3 (correlating with the performance recovery). This provides direct quantitative evidence that the "recovery" phenomenon is a re-correction of the output distribution shift.

Figure 13: Results of different models on captioning examples from the LCS - 558K [ 34 ] dataset after sequentially fine-tuning on CUB200, PixmoCount, and PathVQA.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	87.7	87.6	87.4	87.3	87.0
PixmoCount	52.4	53.9	65.5	68.2	63.5	59.9
PathVQA	36.3	36.2	35.4	61.3	57.4	57.8
TextVQA	76.0	76.0	75.3	75.3	78.5	76.3

Appendix

Table 7: Detailed performance of using MoE to mitigate forgetting by performing sequential fine-tuning on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	80.6	80.0	78.0	78.2	77.0
PixmoCount	52.4	52.6	67.8	64.2	62.5	63.7
PathVQA	36.3	36.2	35.1	58.1	51.6	53.3
TextVQA	76.0	76.0	75.2	75.2	79.1	76.2

Appendix

Table 8: Detailed performance of using LoRA to mitigate forgetting by performing sequential fine-tuning on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	90.0	89.8	89.6	89.5	89.3
PixmoCount	52.4	53.2	67.6	67.8	61.6	61.0
PathVQA	36.3	35.9	35.0	61.1	58.3	57.6
TextVQA	76.0	76.3	76.5	76.6	80.3	79.4

Appendix

Table 9: Detailed performance of using LwF to mitigate forgetting by performing sequential fine-tuning on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	89.9	89.2	89.5	89.0	89.0
PixmoCount	52.4	53.7	69.7	68.0	68.0	66.1
PathVQA	36.3	35.9	34.2	59.1	56.3	56.1
TextVQA	76.0	76.4	74.2	76.1	79.7	78.3

Appendix

Table 10: Detailed performance of using WiSE-FT using

\beta=0.3

to mitigate forgetting by performing sequential fine-tuning on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	90.7	89.2	88.1	88.0	86.9
PixmoCount	52.4	54.9	73.0	64.6	63.1	59.4
PathVQA	36.3	34.8	3.7	63.6	59.8	58.6
TextVQA	76.0	76.6	59.0	74.6	79.6	68.9

Appendix

Table 11: Detailed performance of sequentially fine-tuning the full model on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	69.9	57.5	58.8	61.5	55.3
PixmoCount	52.4	22.1	64.2	44.0	59.0	37.8
PathVQA	36.3	35.4	31.8	37.0	34.4	34.2
TextVQA	76.0	75.5	72.2	70.7	76.4	72.7

Appendix

Table 12: Detailed performance of sequentially fine-tuning the vision tower on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	59.4	57.7	57.5	58.2	57.4
PixmoCount	52.4	53.2	56.6	57.9	58.2	57.7
PathVQA	36.3	36.1	36.0	35.5	36.4	35.9
TextVQA	76.0	76.1	76.3	76.4	77.0	76.9

Appendix

Table 13: Detailed performance of sequentially fine-tuning the projector in the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	90.7	89.4	89.6	88.9	87.8
PixmoCount	52.4	54.3	70.2	62.2	63.3	56.6
PathVQA	36.3	35.2	4.4	63.2	58.6	56.7
TextVQA	76.0	76.6	60.5	74.7	79.6	71.2

Appendix

Table 14: Detailed performance of sequentially fine-tuning the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	85.5	85.1	84.8	84.4	84.0
PixmoCount	52.4	53.9	67.8	68.2	64.8	66.3
PathVQA	36.3	35.7	35.0	55.9	52.6	51.4
TextVQA	76.0	76.1	76.4	75.8	79.3	78.9

Appendix

Table 15: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	78.7	78.4	78.4	78.2	77.6
PixmoCount	52.4	53.2	65.9	67.2	62.4	65.4
PathVQA	36.3	36.1	36.4	44.2	42.6	43.0
TextVQA	76.0	76.2	76.9	76.2	78.6	78.3

Appendix

Table 16: Detailed performance of sequentially fine-tuning the SA Proj. (QKV) in the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	90.1	89.5	89.6	89.3	88.9
PixmoCount	52.4	54.1	71.5	67.6	68.0	62.0
PathVQA	36.3	35.6	17.0	64.1	60.9	60.9
TextVQA	76.0	76.6	66.2	75.3	79.8	74.0

Appendix

Table 17: Detailed performance of sequentially fine-tuning the MLP in the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	53.7	90.2	89.8	89.8	89.6	89.5
PixmoCount	52.4	53.4	71.5	67.8	68.4	67.2
PathVQA	36.3	36.1	35.0	61.9	58.5	58.9
TextVQA	76.0	76.4	75.5	76.1	80.0	79.3

Appendix

Table 18: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	84.8	76.8	77.2	76.6	69.6
PixmoCount	45.7	37.6	63.3	48.5	32.4	44.0
PathVQA	13.2	24.8	0.7	62.0	55.6	45.2
TextVQA	65.4	52.2	31.0	56.1	72.9	42.8

Appendix

Table 19: Detailed performance of sequentially fine-tuning the full model of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	4.2	4.2	12.8	22.1	13.4
PixmoCount	45.7	0.6	44.9	40.8	48.9	40.3
PathVQA	13.2	0.3	1.8	34.1	34.1	33.1
TextVQA	65.4	0.9	0.9	50.1	69.9	59.3

Appendix

Table 20: Detailed performance of sequentially fine-tuning the vision encoder and projector of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	85.2	68.6	72.0	72.1	68.8
PixmoCount	45.7	32.0	57.5	55.1	21.9	41.6
PathVQA	13.2	23.3	14.1	62.7	56.4	42.7
TextVQA	65.4	56.9	35.0	57.8	72.6	40.1

Appendix

Table 21: Detailed performance of sequentially fine-tuning the LLM of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	78.1	68.2	72.3	70.6	70.3
PixmoCount	45.7	38.8	60.5	50.6	9.9	54.7
PathVQA	13.2	28.7	28.4	53.9	43.9	44.3
TextVQA	65.4	64.3	62.9	62.0	73.5	64.9

Appendix

Table 22: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	84.3	78.7	76.9	76.9	72.0
PixmoCount	45.7	34.6	58.6	53.9	35.8	52.8
PathVQA	13.2	28.5	26.3	61.8	56.2	52.7
TextVQA	65.4	61.5	54.6	62.0	73.2	59.2

Appendix

Table 23: Detailed performance of sequentially fine-tuning the MLP in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	32.6	78.4	73.9	74.3	71.5	71.9
PixmoCount	45.7	32.4	58.4	57.9	29.8	44.8
PathVQA	13.2	27.9	27.9	59.5	51.6	54.0
TextVQA	65.4	63.5	63.7	64.7	74.0	63.9

Appendix

Table 24: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	93.5	0.2	12.1	92.6	92.4
PixmoCount	58.6	55.6	50.6	48.3	47.4	51.1
PathVQA	29.2	18.5	0.0	60.8	58.2	59.1
TextVQA	83.0	69.5	17.2	73.3	81.5	62.7

Appendix

Table 25: Detailed performance of sequentially fine-tuning the full model of Qwen2.5-VL on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	92.3	81.4	81.5	81.5	88.0
PixmoCount	58.6	56.4	59.0	59.2	58.2	33.1
PathVQA	29.2	30.3	29.1	29.3	30.2	35.4
TextVQA	83.0	82.5	83.2	83.1	83.1	71.0

Appendix

Table 26: Detailed performance of sequentially fine-tuning the vision encoder and projector of Qwen2.5-VL on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	93.8	0.0	64.4	67.6	91.5
PixmoCount	58.6	55.8	47.0	50.6	41.0	49.1
PathVQA	29.2	4.9	0.0	63.0	59.7	60.2
TextVQA	83.0	47.9	11.6	73.3	82.1	61.8

Appendix

Table 27: Detailed performance of sequentially fine-tuning the LLM of Qwen2.5-VL on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	93.7	93.6	93.4	93.4	93.2
PixmoCount	58.6	59.9	53.4	56.7	53.6	53.4
PathVQA	29.2	35.7	35.3	61.0	57.3	58.3
TextVQA	83.0	77.8	77.4	81.8	83.6	80.6

Appendix

Table 28: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM of Qwen2.5-VL on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	94.1	83.5	93.1	93.0	92.6
PixmoCount	58.6	56.6	50.0	53.4	31.8	52.2
PathVQA	29.2	32.4	2.8	61.5	60.6	60.4
TextVQA	83.0	76.7	8.2	78.9	83.4	64.4

Appendix

Table 29: Detailed performance of sequentially fine-tuning the MLP in the LLM of Qwen2.5-VL on each target task.

Target
Dataset	Baseline	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
Dataset	–	CUB200	PixmoCount	PathVQA	TextVQA	TimeClock
CUB200	81.4	94.1	94.2	93.8	94.0	94.0
PixmoCount	58.6	59.7	49.6	50.0	50.9	54.7
PathVQA	29.2	36.0	22.0	61.5	61.7	62.3
TextVQA	83.0	75.9	74.7	81.1	83.8	79.6

Appendix

Table 30: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM of Qwen2.5-VL on each target task.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,092 papers

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
Mar 15, 2026He Li, Yuhui Zhang, Xiaohan Wang +2LLM Fine-Tuning Multimodal Large Language Models
Stanford University · Tsinghua University
Towards Continual Learning: Mitigating Forgetting With Subspace Tuning
Oct 15, 2026Arnav Chavan, Kusha Sareen, Rishabh Tiwari +6Continual Learning for LLMs LLM Fine-Tuning
UC Berkeley · Mila · McGill University +5
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
Aug 12, 2026Juncheng Liao, Jinfan Lv, Guoming Wang +3Continual Learning for LLMs LLM Fine-Tuning
School of Software Technology, Zhejiang University, Hangzhou, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · School of Artificial Intelligence, Sun Yat-Sen University, Guangzhou, China +1

Mar 15, 2026J/K move · Enter open · S save

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

He Li, Yuhui Zhang, Xiaohan Wang, Kaifeng Lyu+1

Stanford University · Tsinghua University

LLM Fine-Tuning Multimodal Large Language Models Catastrophic Forgetting Continual Learning for VLMs

Open paper AskPDF arXiv

The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.

Abstract

Multimodal Large Language Models

Explore similar work

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

Towards Continual Learning: Mitigating Forgetting With Subspace Tuning

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention