How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. Surprisingly, we find that performance lost on held-out tasks after fine-tuning on one skill can partly recover when the model is subsequently tuned on a different skill. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that shows the shift co-varies with forgetting. Guided by this insight, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers (SA Proj., Δ learning +24.9 / Δ held-out forgetting -0.6), and (ii) updating only the MLP Gate&Up while freezing the Down projection (+30.5 / -2.1). Both substantially outperform full-LLM tuning (+31.8 / -23.3) in the learning-forgetting trade-off. We also compare against common forgetting mitigation methods: Learning without Forgetting (LwF), LoRA, Mixture-of-Experts, and weight-space interpolation (WiSE-FT), and find that our selective tuning recipes match or exceed their learning-stability balance while remaining simpler, requiring no replay, auxiliary parameters, or per-stage tuning. These results hold across LLaVA-OneVision, LLaVA-NeXT, and Qwen2.5-VL, confirming that the key to teaching LMMs new skills without forgetting lies in controlling output distribution shift by choosing which components to tune. Code will be made available.
Figures & tables
Figure 1: Surprising Forgetting Behavior in LMMs : Left : When fine-tuning most components on one target task, we see major improvement in that task (“Learning”) but a substantial drop in performance of other tasks (“Forgetting”, total across tasks shown here), as expected. But if we only tune self-attention projection layers (SA Proj.) in the language model, we still get substantial learning on the target task with minimal forgetting. Right : Even fine-tuning SA Proj. for multiple tasks sequentially, we see no forgetting. For others, we see large forgetting on the PixmoCount task, but the models somehow partly recover what they "forgot" in learning the next specialized task. Our paper documents and analyzes these and other interesting phenomena of learning and forgetting in LMMs, leading to simple and effective ways to teach LMMs new skills.
Figure 2: Architecture of our evaluated LMMs. The input contains visual inputs such as images or videos, which are converted to visual tokens by the vision encoder, and text input is processed by a tokenizer containing a visual placeholder token <image> . Visual tokens are converted by the projector and concatenated with text tokens as input for the language model. We visualize the architecture of the transformer decoder layer of the language model. "LN", "MHA", and "MLP" represent layer norm, multi-head attention, and multi-layer perceptron, respectively. r(l) is the final output of layer l .
Method
Target Learning
Target Forgetting
Target Overall
Held-out Forgetting
Baseline
43.9
0.0
43.9
76.4
Full
+29.9
−25.9
+9.2
−27.4
|-- Vision Encoder
+9.6
−12.7
−0.5
−10.8
|-- Projector
+2.3
−0.8
+1.7
−1.3
|-- LLM
+31.8
−23.5
+13.0
−23.3
|-- SA Proj.
+24.9
−2.3
+23.1
−0.6
Table 1: Effect of tuning different components: learning, forgetting, and overall performance, averaged over three five-task sequences. Cells are colored using a blue-orange colormap to show performance changes. Blue indicates a positive change, where a darker shade is better. Orange indicates a negative change, where a lighter shade is better. We underline numbers that do not reflect a significantly different task-average distribution from the best, based on a two-sided paired sample t-test.
Figure 3: Learning–forgetting tracks output–distribution shift. On LLaVA - OneVision tuned for counting, we plot five curves over log - spaced steps for LLM, SA Proj., MLP, MLP (Gate&Up) and MLP (LwF). The dashed line represents the base model. Left: PixmoCount accuracy rises for all methods. Middle: mean held - out performance drops sharply for LLM and MLP, remains nearly unchanged for SA Proj., and is preserved by MLP (LwF); Right: the expected likelihood of number tokens on non - counting captions (LCS - 558K [ 34 ] ) surges for LLM and MLP, stays near baseline for SA Proj., and has little changes for MLP (LwF).
Figure 4: Visualizations on counting and captioning examples after tuning tuning MLP and SA Proj. on the counting task. The counting and examples are sampled from the PixmoCount dataset and the LCS - 558K [ 34 ] dataset, respectively.
Method
LLaVA-NeXT (LLaMA-3 8B)
Qwen2.5-VL (7B)
T. Learn
T. Forget
T. Overall
H. Forget
T. Learn
T. Forget
T. Overall
H. Forget
Baseline
31.5
0.0
31.5
59.9
52.1
0.0
52.1
77.9
Full
+31.7
−20.3
+15.4
−32.0
+17.3
−5.2
+13.1
−17.5
|-- Vision + Projector
+0.1
−1.8
−1.3
−13.4
+12.1
−9.1
+4.9
−6.2
|-- LLM
+36.2
−21.2
+19.3
−35.9
+16.8
−5.9
+12.1
−24.6
|-- SA Proj.
+28.3
−7.9
+21.9
−7.7
+16.1
−1.6
+14.9
+0.6
Table 2: Component-level tuning experiments with LLaVA-NeXT and Qwen2.5-VL . "T" represents "Target" and "H" is for "Held-out". Underlined text of each column denotes the best method.
Figure 5: Comparison of different continual learning techniques in the default sequential task curriculum. For LwF, WiSE-FT, only the MLP layers are tuned. LoRA adapters are wrapped only on the MLP layers. MoE is also applied to the MLP layers.
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Group
Components
#Params
ΘVE
SigLIP vision encoder
≈400 M
ΘProj
Multimodal projector
≈20 M
ΘSA
All blocks: WQ,WK,WV,WO
≈822 M
ΘMLP
All blocks: Wgate,Wup,Wdown
≈5,703 M
ΘEmb
Input token embeddings
≈545 M
ΘLM
LM head U
≈545 M
Appendix
Table 3: Parameter groups and counts for LLaVA OneVision Qwen2-7B
Index / Token
Index / Token
Index / Token
Index / Token
15
0
16
1
17
2
18
3
19
4
20
5
21
6
22
7
23
8
24
9
603
one
1960
ten
3966
One
5225
ONE
11613
Two
14154
zero
17999
Zero
19641
Three
19789
two
26972
Four
27856
three
32687
Ten
34024
four
37020
Five
Appendix
Table 4: Numeric token indices and their corresponding tokens.
Method
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Average
Target
Held-out
Target
Held-out
Target
Held-out
Target
Held-out
Target
Held-out
Target
Held-out
Baseline
53.7
76.4
52.4
76.4
36.3
76.4
76.0
76.4
1.1
76.4
43.9
76.4
Proj.
+5.7
-0.0
+4.2
-0.1
+0.6
-0.1
+0.5
+0.2
+0.4
-0.5
+2.3
-0.1
Vis. Enc.
+16.1
-0.8
+11.6
-4.7
+3.7
-2.8
+1.0
-0.7
+12.7
-11.9
+9.0
-4.2
SA Proj.
+31.8
+0.3
+15.2
-0.2
+14.4
-0.3
+3.5
+0.3
+56.0
-0.1
+24.2
+0.0
MLP
+36.4
+0.3
+17.8
-4.0
+26.5
-0.4
+3.8
+0.0
+73.3
-3.1
+31.6
-1.4
Appendix
Table 5: Single task fine-tuning by component. Each individual target task is fine-tuned from the original model, and the performance for that task (“Target”) and average held-out performance (“Held-out”) is measured. Each row is for tuning a different component or set of components: “Proj.”, “Vis. Enc.”, “SA Proj.”, “MLP”, “LM”, “Full” represent tuning the projector, vision encoder, self-attention projections in the LLM, MLP in LLM, full LLM, and all parameters, respectively. “+” is an increment over the baseline (original LLaVA-OneVision-7B checkpoint), and “-” is a decrease.
Figure 6: Sequential fine-tuning by component. The target tasks in the x -axis are trained sequentially, from left to right. After training each task, the average performance of all target tasks ( left ) and all held-out tasks ( right ) are measured. Each line shows the performance after tuning a different component or set of components: LLaVA-OneVision (Full, Vision Encoder, Projector, LLM, SA Proj., SA Proj. (QKV),, MLP, MLP (Gate&Up)). The dashed horizontal gray line marks the average performance of the original model.
Figure 7: Sequential fine-tuning by component. Tasks are arranged as TimeClock → TextVQA → PathVQA → PixmoCount → CUB200.
Figure 8: Sequential fine-tuning by component. Tasks are arranged as PathVQA → CUB200 → TextVQA → TimeClock → PixmoCount.
Figure 9: Composing stable updates. We compare SA Proj. and MLP (Gate&Up) to two compositions: SA Proj. + Gate&Up and SA Proj. (QKV only) + Gate&Up using the default five-task sequential tuning.
Figure 10: Layer-wise residual-delta magnitude across training iterations. Each subplot shows the average logit-space ∥Δresidual∥2 attributable to MLP (solid reds) and self-attention projections (dashed blues) versus transformer layer index. The five method configurations are: SA Proj., MLP, LLM, MLP (Gate & Up), and MLP (LwF). Color shade encodes checkpoint iteration (darker = later). The top-left panel is a legend; other panels omit legends for clarity.
Figure 11: Sequential tuning of LLM layer subsets. We compare tuning the full LLM against tuning only the first 23 layers or only the last 5 layers of the 28-layer Qwen2-7B model.
Method
Total Params
Trainable Params
Train SPS ↑
(B)
(B)
(#samples / sec.)
SA Proj.
8.03
0.82
1.46
MLP (Gate&Up)
8.03
3.80
1.45
MLP
8.03
5.70
1.44
LoRA
8.51
0.50
1.27
LwF
16.06
5.70
0.81
Appendix
Table 6: Training efficiency and parameter footprint of the evaluated model variants (4 × H100 96 GB, DeepSpeed, bfloat16). Total Params is the total number of model parameters in billions. Trainable Params is the subset that requires gradient updates in billions. Train SPS is the number of training samples processed per second, collected from training models on the same 384 samples.
Figure 12: Analysis of the forgetting and recovery cycle over three sequential tasks. We plot performance on (Left) average target tasks, (Middle) average held-out tasks, and (Right) the likelihood of numeric tokens on a general held-out dataset (LCS-558K). The plots track performance after training on CUB200 (Stage 1), PixmoCount (Stage 2), and PathVQA (Stage 3). The key finding is in the rightmost plot: for methods susceptible to forgetting (e.g., LLM, MLP), the numeric token bias (output shift) spikes at Stage 2 (correlating with the performance drop) and then recedes at Stage 3 (correlating with the performance recovery). This provides direct quantitative evidence that the "recovery" phenomenon is a re-correction of the output distribution shift.
Figure 13: Results of different models on captioning examples from the LCS - 558K [ 34 ] dataset after sequentially fine-tuning on CUB200, PixmoCount, and PathVQA.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
87.7
87.6
87.4
87.3
87.0
PixmoCount
52.4
53.9
65.5
68.2
63.5
59.9
PathVQA
36.3
36.2
35.4
61.3
57.4
57.8
TextVQA
76.0
76.0
75.3
75.3
78.5
76.3
Appendix
Table 7: Detailed performance of using MoE to mitigate forgetting by performing sequential fine-tuning on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
80.6
80.0
78.0
78.2
77.0
PixmoCount
52.4
52.6
67.8
64.2
62.5
63.7
PathVQA
36.3
36.2
35.1
58.1
51.6
53.3
TextVQA
76.0
76.0
75.2
75.2
79.1
76.2
Appendix
Table 8: Detailed performance of using LoRA to mitigate forgetting by performing sequential fine-tuning on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
90.0
89.8
89.6
89.5
89.3
PixmoCount
52.4
53.2
67.6
67.8
61.6
61.0
PathVQA
36.3
35.9
35.0
61.1
58.3
57.6
TextVQA
76.0
76.3
76.5
76.6
80.3
79.4
Appendix
Table 9: Detailed performance of using LwF to mitigate forgetting by performing sequential fine-tuning on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
89.9
89.2
89.5
89.0
89.0
PixmoCount
52.4
53.7
69.7
68.0
68.0
66.1
PathVQA
36.3
35.9
34.2
59.1
56.3
56.1
TextVQA
76.0
76.4
74.2
76.1
79.7
78.3
Appendix
Table 10: Detailed performance of using WiSE-FT using β=0.3 to mitigate forgetting by performing sequential fine-tuning on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
90.7
89.2
88.1
88.0
86.9
PixmoCount
52.4
54.9
73.0
64.6
63.1
59.4
PathVQA
36.3
34.8
3.7
63.6
59.8
58.6
TextVQA
76.0
76.6
59.0
74.6
79.6
68.9
Appendix
Table 11: Detailed performance of sequentially fine-tuning the full model on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
69.9
57.5
58.8
61.5
55.3
PixmoCount
52.4
22.1
64.2
44.0
59.0
37.8
PathVQA
36.3
35.4
31.8
37.0
34.4
34.2
TextVQA
76.0
75.5
72.2
70.7
76.4
72.7
Appendix
Table 12: Detailed performance of sequentially fine-tuning the vision tower on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
59.4
57.7
57.5
58.2
57.4
PixmoCount
52.4
53.2
56.6
57.9
58.2
57.7
PathVQA
36.3
36.1
36.0
35.5
36.4
35.9
TextVQA
76.0
76.1
76.3
76.4
77.0
76.9
Appendix
Table 13: Detailed performance of sequentially fine-tuning the projector in the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
90.7
89.4
89.6
88.9
87.8
PixmoCount
52.4
54.3
70.2
62.2
63.3
56.6
PathVQA
36.3
35.2
4.4
63.2
58.6
56.7
TextVQA
76.0
76.6
60.5
74.7
79.6
71.2
Appendix
Table 14: Detailed performance of sequentially fine-tuning the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
85.5
85.1
84.8
84.4
84.0
PixmoCount
52.4
53.9
67.8
68.2
64.8
66.3
PathVQA
36.3
35.7
35.0
55.9
52.6
51.4
TextVQA
76.0
76.1
76.4
75.8
79.3
78.9
Appendix
Table 15: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
78.7
78.4
78.4
78.2
77.6
PixmoCount
52.4
53.2
65.9
67.2
62.4
65.4
PathVQA
36.3
36.1
36.4
44.2
42.6
43.0
TextVQA
76.0
76.2
76.9
76.2
78.6
78.3
Appendix
Table 16: Detailed performance of sequentially fine-tuning the SA Proj. (QKV) in the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
90.1
89.5
89.6
89.3
88.9
PixmoCount
52.4
54.1
71.5
67.6
68.0
62.0
PathVQA
36.3
35.6
17.0
64.1
60.9
60.9
TextVQA
76.0
76.6
66.2
75.3
79.8
74.0
Appendix
Table 17: Detailed performance of sequentially fine-tuning the MLP in the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
53.7
90.2
89.8
89.8
89.6
89.5
PixmoCount
52.4
53.4
71.5
67.8
68.4
67.2
PathVQA
36.3
36.1
35.0
61.9
58.5
58.9
TextVQA
76.0
76.4
75.5
76.1
80.0
79.3
Appendix
Table 18: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
84.8
76.8
77.2
76.6
69.6
PixmoCount
45.7
37.6
63.3
48.5
32.4
44.0
PathVQA
13.2
24.8
0.7
62.0
55.6
45.2
TextVQA
65.4
52.2
31.0
56.1
72.9
42.8
Appendix
Table 19: Detailed performance of sequentially fine-tuning the full model of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
4.2
4.2
12.8
22.1
13.4
PixmoCount
45.7
0.6
44.9
40.8
48.9
40.3
PathVQA
13.2
0.3
1.8
34.1
34.1
33.1
TextVQA
65.4
0.9
0.9
50.1
69.9
59.3
Appendix
Table 20: Detailed performance of sequentially fine-tuning the vision encoder and projector of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
85.2
68.6
72.0
72.1
68.8
PixmoCount
45.7
32.0
57.5
55.1
21.9
41.6
PathVQA
13.2
23.3
14.1
62.7
56.4
42.7
TextVQA
65.4
56.9
35.0
57.8
72.6
40.1
Appendix
Table 21: Detailed performance of sequentially fine-tuning the LLM of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
78.1
68.2
72.3
70.6
70.3
PixmoCount
45.7
38.8
60.5
50.6
9.9
54.7
PathVQA
13.2
28.7
28.4
53.9
43.9
44.3
TextVQA
65.4
64.3
62.9
62.0
73.5
64.9
Appendix
Table 22: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
84.3
78.7
76.9
76.9
72.0
PixmoCount
45.7
34.6
58.6
53.9
35.8
52.8
PathVQA
13.2
28.5
26.3
61.8
56.2
52.7
TextVQA
65.4
61.5
54.6
62.0
73.2
59.2
Appendix
Table 23: Detailed performance of sequentially fine-tuning the MLP in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
32.6
78.4
73.9
74.3
71.5
71.9
PixmoCount
45.7
32.4
58.4
57.9
29.8
44.8
PathVQA
13.2
27.9
27.9
59.5
51.6
54.0
TextVQA
65.4
63.5
63.7
64.7
74.0
63.9
Appendix
Table 24: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM of LLaVA-NeXT (LLaMA 3) on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
93.5
0.2
12.1
92.6
92.4
PixmoCount
58.6
55.6
50.6
48.3
47.4
51.1
PathVQA
29.2
18.5
0.0
60.8
58.2
59.1
TextVQA
83.0
69.5
17.2
73.3
81.5
62.7
Appendix
Table 25: Detailed performance of sequentially fine-tuning the full model of Qwen2.5-VL on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
92.3
81.4
81.5
81.5
88.0
PixmoCount
58.6
56.4
59.0
59.2
58.2
33.1
PathVQA
29.2
30.3
29.1
29.3
30.2
35.4
TextVQA
83.0
82.5
83.2
83.1
83.1
71.0
Appendix
Table 26: Detailed performance of sequentially fine-tuning the vision encoder and projector of Qwen2.5-VL on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
93.8
0.0
64.4
67.6
91.5
PixmoCount
58.6
55.8
47.0
50.6
41.0
49.1
PathVQA
29.2
4.9
0.0
63.0
59.7
60.2
TextVQA
83.0
47.9
11.6
73.3
82.1
61.8
Appendix
Table 27: Detailed performance of sequentially fine-tuning the LLM of Qwen2.5-VL on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
93.7
93.6
93.4
93.4
93.2
PixmoCount
58.6
59.9
53.4
56.7
53.6
53.4
PathVQA
29.2
35.7
35.3
61.0
57.3
58.3
TextVQA
83.0
77.8
77.4
81.8
83.6
80.6
Appendix
Table 28: Detailed performance of sequentially fine-tuning the SA projection layers in the LLM of Qwen2.5-VL on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
94.1
83.5
93.1
93.0
92.6
PixmoCount
58.6
56.6
50.0
53.4
31.8
52.2
PathVQA
29.2
32.4
2.8
61.5
60.6
60.4
TextVQA
83.0
76.7
8.2
78.9
83.4
64.4
Appendix
Table 29: Detailed performance of sequentially fine-tuning the MLP in the LLM of Qwen2.5-VL on each target task.
Dataset
Baseline
Stage 1
Stage 2
Stage 3
Stage 4
Stage 5
–
CUB200
PixmoCount
PathVQA
TextVQA
TimeClock
Target
CUB200
81.4
94.1
94.2
93.8
94.0
94.0
PixmoCount
58.6
59.7
49.6
50.0
50.9
54.7
PathVQA
29.2
36.0
22.0
61.5
61.7
62.3
TextVQA
83.0
75.9
74.7
81.1
83.8
79.6
Appendix
Table 30: Detailed performance of sequentially fine-tuning the MLP (Gate & Up) in the LLM of Qwen2.5-VL on each target task.
The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.
Large language models (LLMs) acquire new skills through fine-tuning, but it can destroy capabilities they already have: catastrophic forgetting remains a central obstacle to continual learning in LLMs. Recent works connect forgetting to how far, and along which directions, fine-tuning moves the weights. On-policy methods such as reinforcement learning and on-policy distillation avoid updating the principal singular directions of weight matrices and thus forget less. In contrast, SFT concentrates updates along these directions and is susceptible to catastrophic forgetting. In this work, we introduce Subspace Tuning (SST), which restricts each weight update to the bottom singular subspace, eliminating changes along the principal directions. SST is optimizer-agnostic and composes well with fine-tuning recipes including SFT and self-distillation. SST combines tight subspace protection, dense gradient estimation, and a simple refresh mechanism, enabling stronger learning than prior orthogonal fine-tuning methods. With these mechanisms, protecting as much as 99% of the weight subspace substantially improves retention while preserving strong target-task performance. This behavior strengthens with model and data size and nearly eliminates forgetting in the large-model regime. SST also keeps the model close to its base distribution, inducing 2--4× less KL drift than the baseline recipes. Overall, SST advances the learning--forgetting Pareto frontier, making it a practical default for fine-tuning pre-trained language models.
Multimodal Large Language Models (MLLMs) exhibit strong generalization and reasoning abilities due to large-scale multimodal pre-training. However, fine-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task-specific knowledge degrades previously acquired capabilities. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs. To address this challenge, we propose Activation-Weighted Adaptive REtention (AWARe), a fine-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns. AWARe assigns activation-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks. Importantly, AWARe operates without modifying model architectures, ensuring compatibility with existing inference engines. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods. Code is available at https://github.com/kaln27/AWARe.
Juncheng Liao, Jinfan Lv, Guoming Wang +3
School of Software Technology, Zhejiang University, Hangzhou, China · College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China · School of Artificial Intelligence, Sun Yat-Sen University, Guangzhou, China +1