MAny: Merge Anything for Multimodal Continual Instruction Tuning

Zijian Gao; Wangwang Jia; Xingxing Zhang; Pengfei Qian; Tao Sun; Bo Ding; Yong Dou; Huaimin Wang

cs.LGApr 15, 2026

MAny: Merge Anything for Multimodal Continual Instruction Tuning

Authors: Zijian Gao, Wangwang Jia, Xingxing Zhang, Pengfei Qian, Tao Sun, Bo Ding, Yong Dou, Huaimin Wang, +1 more

Organizations: College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology · School of Computer Science, Tsinghua University

Abstract

Multimodal Continual Instruction Tuning (MCIT) is essential for sequential task adaptation of Multimodal Large Language Models (MLLMs) but is severely restricted by catastrophic forgetting. While existing literature focuses on the reasoning language backbone, in this work, we expose a critical yet neglected dual-forgetting phenomenon across both perception drift in Cross-modal Projection Space and reasoning collapse in Low-rank Parameter Space. To resolve this, we present MAny (Merge Anything), a framework that merges task-specific knowledge through Cross-modal Projection Merging (CPM) and Low-rank Parameter Merging (LPM). Specifically, CPM recovers perceptual alignment by adaptively merging cross-modal visual representations via visual-prototype guidance, ensuring accurate feature recovery during inference. Simultaneously, LPM eliminates mutual interference among task-specific low-rank modules by recursively merging low-rank weight matrices. By leveraging recursive least squares, LPM provides a closed-form solution that mathematically guarantees an optimal fusion trajectory for reasoning stability. Notably, MAny operates as a training-free paradigm that achieves knowledge merging via efficient CPU-based algebraic operations, eliminating additional gradient-based optimization beyond initial tuning. Our extensive evaluations confirm the superior performance and robustness of MAny across multiple MLLMs and benchmarks. Specifically, on the UCIT benchmark, MAny achieves significant leads of up to 8.57% and 2.85% in final average accuracy over state-of-the-art methods across two different MLLMs, respectively.

Figures & tables

Figure 1: Performance diagnosis on UCIT benchmark for LLaVA-1.5-7B and InternVL-Chat-7B during sequential LoRA fine-tuning. Tasks follow a clockwise sequence from Image-R.

Figure 2: Visualizing Perception Drift and Reasoning Collapse. (a) Cosine similarity of projector features across tasks. (b) Layer-wise feature similarity between oracle states and the final model. In both aspects, MAny effectively mitigates internal representation drift.

Figure 3: Overview of MAny. The dual-track design explicitly decouples perception from reasoning.

	Method	Venue	Image-R	ArxivQA	Viz-cap	IconQA	CLEVR	Flickr30k	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
LLaVA-1.5-7B	Zero-shot	–	16.27	57.73	38.39	19.20	20.63	41.88	–	–	–
	LoRA-FT Hu et al. (2022)	ICLR’22	65.50	76.23	44.41	66.37	54.87	57.80	60.86	67.61	17.44
	O-LoRA Wang et al. (2023a)	EMNLP’23	83.40	94.20	41.51	58.50	56.80	53.37	64.63	75.98	8.03
	MoELoRA Chen et al. (2024a)	NeurIPS’24	68.57	77.73	44.26	65.43	45.20	57.31	59.75	75.57	17.54
	ModalPrompt Zeng et al. (2025)	EMNLP’25	51.07	87.27	48.11	39.23	46.57	42.93	52.53	57.67	0.15
	CL-MoE Huai et al. (2025)	CVPR’25	67.80	73.47	44.58	69.37	47.80	57.40	60.07	74.89	18.25

Table 1: Results on the UCIT benchmark using LLaVA-1.5-7B and InternVL-Chat-7B models.

CPM	LPM	LLaVA-1.5-7B			InternVL-Chat-7B
CPM	LPM	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
		60.86	67.61	17.44	65.91	77.89	10.12
✓		68.71	75.65	10.45	68.20	80.14	8.22
	✓	70.87	81.93	5.85	69.99	81.85	3.71
✓	✓	74.50	83.42	0.37	72.35	83.35	1.77

Table 2: Ablation study of different components in MAny on the UCIT benchmark.

Method	w/o CPM			w/ CPM
Method	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
Task-specific LoRA	73.99	82.56	4.40	76.98	84.01	0.23
LoRA-FT	63.09	73.27	18.64	68.71	75.65	10.45
O-LoRA	64.63	75.98	8.03	68.53	80.58	2.78
HiDE	65.93	79.04	5.58	68.93	80.23	1.96
SEFE	65.17	76.64	12.80	69.04	79.37	5.76

Table 3: Comparison of different methods w/ and w/o the CPM module on the UCIT benchmark.

Figure 4: Sensitivity of $\lambda$ . Dashed lines denote the SOTA performance of all baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Method	Venue	RS	Med	AD	Sci	Fin	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
LLaVA-1.5-7B	Zero-shot	–	32.29	28.28	15.59	35.55	62.56	–	–	–
	LoRA-FT Hu et al. (2022)	ICLR’22	74.13	47.63	33.53	41.42	89.66	57.27	63.23	11.43
	O-LoRA Wang et al. (2023a)	EMNLP’23	74.79	39.78	30.68	39.11	83.34	53.54	59.34	7.03
	MoELoRA Chen et al. (2024a)	NeurIPS’24	76.96	47.09	31.19	41.96	89.74	57.39	63.11	11.65
	ModalPrompt Zeng et al. (2025)	EMNLP’25	47.92	29.10	13.35	32.77	89.03	42.44	49.75	30.52
	CL-MoE Huai et al. (2025)	CVPR’25	73.05	48.00	30.27	41.48	90.11	56.58	62.81	12.74

Appendix

Table 4: Results on the MLLM-DCL benchmark using LLaVA-1.5-7B and InternVL-Chat-7B models.

Projector Setting	LoRA Setting	UCIT						Metrics
Projector Setting	LoRA Setting	Img-R	Arxiv	VizCap	Icon	CLEVR	F30k	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
Task Projector	Task LoRA (Oracle)	91.10	91.77	60.45	76.17	75.10	57.80	75.40	-	-
	Merged LoRA	88.83	89.43	56.29	75.43	58.57	56.77	70.89	82.05	5.92
	Final Task LoRA	81.20	76.40	46.49	78.63	74.70	57.99	69.24	77.80	10.08
Merged Projector	Task LoRA (Oracle)	91.63	91.10	60.79	83.07	77.17	58.13	76.98	84.01	0.23
	Merged LoRA	90.87	91.97	60.48	80.20	67.37	56.09	74.50	83.24	0.37

Appendix

Table 5: Under the two settings of the projector corresponding to the task and the merged projector of CPM, the impact of using the corresponding task’s LoRA and the LPM’s Merged LoRA and the last task LoRA using LLaVA-1.5-7B.

Modality	UCIT						Metrics
Modality	Img-R	Arxiv	VizCap	Icon	CLEVR	F30k	FAA ( $\uparrow$ )	CAA ( $\uparrow$ )	FFM ( $\downarrow$ )
Text	91.20	92.67	57.81	80.70	67.67	56.32	74.40	83.48	0.70
Text + Image	91.03	92.63	58.89	80.23	68.57	56.29	74.61	83.46	0.38
Image (Ours)	90.87	91.97	60.48	80.20	67.37	56.09	74.50	83.24	0.37

Appendix

Table 6: Comparison of modality choices for CPM prototype construction.

Figure 5: Impact of $\gamma$ . Dashed lines denote the SOTA performance of all baseline.

Explore similar work

Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Mixture-of-Experts Models Multimodal Instruction Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
May 11, 2026Tao Hu, Da-Wei ZhouCross-Modal Learning Continual Learning for LLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University

Jul 29, 2026J/K move · Enter open · S save

Progressive Multimodal Alignment for Continual Instruction Tuning

Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong+1

Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University+2

Cross-Modal Alignment Multimodal Large Language Models Continual Learning for VLMs Multimodal Instruction Tuning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.

Abstract

Explore similar work

Progressive Multimodal Alignment for Continual Instruction Tuning

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning