MAny: Merge Anything for Multimodal Continual Instruction Tuning
Authors: Zijian Gao, Wangwang Jia, Xingxing Zhang, Pengfei Qian, Tao Sun, Bo Ding, Yong Dou, Huaimin Wang, +1 more
Organizations: College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology · School of Computer Science, Tsinghua University
Multimodal Continual Instruction Tuning (MCIT) is essential for sequential task adaptation of Multimodal Large Language Models (MLLMs) but is severely restricted by catastrophic forgetting. While existing literature focuses on the reasoning language backbone, in this work, we expose a critical yet neglected dual-forgetting phenomenon across both perception drift in Cross-modal Projection Space and reasoning collapse in Low-rank Parameter Space. To resolve this, we present MAny (Merge Anything), a framework that merges task-specific knowledge through Cross-modal Projection Merging (CPM) and Low-rank Parameter Merging (LPM). Specifically, CPM recovers perceptual alignment by adaptively merging cross-modal visual representations via visual-prototype guidance, ensuring accurate feature recovery during inference. Simultaneously, LPM eliminates mutual interference among task-specific low-rank modules by recursively merging low-rank weight matrices. By leveraging recursive least squares, LPM provides a closed-form solution that mathematically guarantees an optimal fusion trajectory for reasoning stability. Notably, MAny operates as a training-free paradigm that achieves knowledge merging via efficient CPU-based algebraic operations, eliminating additional gradient-based optimization beyond initial tuning. Our extensive evaluations confirm the superior performance and robustness of MAny across multiple MLLMs and benchmarks. Specifically, on the UCIT benchmark, MAny achieves significant leads of up to 8.57% and 2.85% in final average accuracy over state-of-the-art methods across two different MLLMs, respectively.
Figures & tables
Figure 1: Performance diagnosis on UCIT benchmark for LLaVA-1.5-7B and InternVL-Chat-7B during sequential LoRA fine-tuning. Tasks follow a clockwise sequence from Image-R.
Figure 2: Visualizing Perception Drift and Reasoning Collapse. (a) Cosine similarity of projector features across tasks. (b) Layer-wise feature similarity between oracle states and the final model. In both aspects, MAny effectively mitigates internal representation drift.
Figure 3: Overview of MAny. The dual-track design explicitly decouples perception from reasoning.
Method
Venue
Image-R
ArxivQA
Viz-cap
IconQA
CLEVR
Flickr30k
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
LLaVA-1.5-7B
Zero-shot
–
16.27
57.73
38.39
19.20
20.63
41.88
–
–
–
LoRA-FT Hu et al. (2022)
ICLR’22
65.50
76.23
44.41
66.37
54.87
57.80
60.86
67.61
17.44
O-LoRA Wang et al. (2023a)
EMNLP’23
83.40
94.20
41.51
58.50
56.80
53.37
64.63
75.98
8.03
MoELoRA Chen et al. (2024a)
NeurIPS’24
68.57
77.73
44.26
65.43
45.20
57.31
59.75
75.57
17.54
ModalPrompt Zeng et al. (2025)
EMNLP’25
51.07
87.27
48.11
39.23
46.57
42.93
52.53
57.67
0.15
CL-MoE Huai et al. (2025)
CVPR’25
67.80
73.47
44.58
69.37
47.80
57.40
60.07
74.89
18.25
Table 1: Results on the UCIT benchmark using LLaVA-1.5-7B and InternVL-Chat-7B models.
CPM
LPM
LLaVA-1.5-7B
InternVL-Chat-7B
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
60.86
67.61
17.44
65.91
77.89
10.12
✓
68.71
75.65
10.45
68.20
80.14
8.22
✓
70.87
81.93
5.85
69.99
81.85
3.71
✓
✓
74.50
83.42
0.37
72.35
83.35
1.77
Table 2: Ablation study of different components in MAny on the UCIT benchmark.
Method
w/o CPM
w/ CPM
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
Task-specific LoRA
73.99
82.56
4.40
76.98
84.01
0.23
LoRA-FT
63.09
73.27
18.64
68.71
75.65
10.45
O-LoRA
64.63
75.98
8.03
68.53
80.58
2.78
HiDE
65.93
79.04
5.58
68.93
80.23
1.96
SEFE
65.17
76.64
12.80
69.04
79.37
5.76
Table 3: Comparison of different methods w/ and w/o the CPM module on the UCIT benchmark.
Figure 4: Sensitivity of λ . Dashed lines denote the SOTA performance of all baseline.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Venue
RS
Med
AD
Sci
Fin
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
LLaVA-1.5-7B
Zero-shot
–
32.29
28.28
15.59
35.55
62.56
–
–
–
LoRA-FT Hu et al. (2022)
ICLR’22
74.13
47.63
33.53
41.42
89.66
57.27
63.23
11.43
O-LoRA Wang et al. (2023a)
EMNLP’23
74.79
39.78
30.68
39.11
83.34
53.54
59.34
7.03
MoELoRA Chen et al. (2024a)
NeurIPS’24
76.96
47.09
31.19
41.96
89.74
57.39
63.11
11.65
ModalPrompt Zeng et al. (2025)
EMNLP’25
47.92
29.10
13.35
32.77
89.03
42.44
49.75
30.52
CL-MoE Huai et al. (2025)
CVPR’25
73.05
48.00
30.27
41.48
90.11
56.58
62.81
12.74
Appendix
Table 4: Results on the MLLM-DCL benchmark using LLaVA-1.5-7B and InternVL-Chat-7B models.
Projector Setting
LoRA Setting
UCIT
Metrics
Img-R
Arxiv
VizCap
Icon
CLEVR
F30k
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
Task Projector
Task LoRA (Oracle)
91.10
91.77
60.45
76.17
75.10
57.80
75.40
-
-
Merged LoRA
88.83
89.43
56.29
75.43
58.57
56.77
70.89
82.05
5.92
Final Task LoRA
81.20
76.40
46.49
78.63
74.70
57.99
69.24
77.80
10.08
Merged Projector
Task LoRA (Oracle)
91.63
91.10
60.79
83.07
77.17
58.13
76.98
84.01
0.23
Merged LoRA
90.87
91.97
60.48
80.20
67.37
56.09
74.50
83.24
0.37
Appendix
Table 5: Under the two settings of the projector corresponding to the task and the merged projector of CPM, the impact of using the corresponding task’s LoRA and the LPM’s Merged LoRA and the last task LoRA using LLaVA-1.5-7B.
Modality
UCIT
Metrics
Img-R
Arxiv
VizCap
Icon
CLEVR
F30k
FAA ( ↑ )
CAA ( ↑ )
FFM ( ↓ )
Text
91.20
92.67
57.81
80.70
67.67
56.32
74.40
83.48
0.70
Text + Image
91.03
92.63
58.89
80.23
68.57
56.29
74.61
83.46
0.38
Image (Ours)
90.87
91.97
60.48
80.20
67.37
56.09
74.50
83.24
0.37
Appendix
Table 6: Comparison of modality choices for CPM prototype construction.
Figure 5: Impact of γ . Dashed lines denote the SOTA performance of all baseline.
Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.
Duzhen Zhang, Yahan Yu, Qiaoyi Su +2
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.
Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks. In such scenarios, Multimodal Continual Instruction Tuning (MCIT) aims to acquire new capabilities while limiting catastrophic forgetting. Existing methods mainly follow a module-composition paradigm: they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, and reasoning demands. This motivates instance-level adaptation to individual query-image pairs rather than only selecting or combining task-level modules. To this end, we propose DRAPE (Dynamic Cross-Modal Prompt Generation), a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. Instead of selecting prompts from a fixed pool, DRAPE derives prompt queries from the textual instruction and cross-attends to visual patch features, producing query-image conditioned prompts that are prepended to the frozen LLM. To mitigate forgetting during sequential updates, DRAPE applies null-space gradient projection to the shared projector and uses CLIP-based prototype routing for task-label-free generator selection at inference. Extensive experiments on MCIT benchmarks show that DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.
Tao Hu, Da-Wei Zhou
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University