CART: Compatible Routing for Multimodal Continual Instruction Tuning

Ling-Xiang Zhang; Tao Hu; Da-Wei Zhou

cs.AIOct 15, 2026

CART: Compatible Routing for Multimodal Continual Instruction Tuning

Authors: Ling-Xiang Zhang, Tao Hu, Da-Wei Zhou

Organizations: School of Artificial Intelligence, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world applications often require them to continually acquire new capabilities from sequential tasks, i.e., Multimodal Continual Instruction Tuning (MCIT). In typical MLLMs, a projector is employed to align visual representations with the language model. However, existing methods primarily protect the language model from forgetting while continuously updating the projector across tasks. Such asymmetric adaptation can cause substantial representation drift in the projector, increasing the mismatch between the projector and language model and consequently exacerbating catastrophic forgetting. To address this issue, we propose Compatible Routing (CART), which explicitly coordinates the adaptation of the projector and language model. Specifically, CART maintains task-specific experts for both modules and combines them through weighted mixtures. CART jointly infers their weights by considering both expert relevance and cross-module compatibility, thereby promoting coordinated adaptation between the two modules. To further control model growth, we introduce relevance-guided expert consolidation, which merges experts exhibiting similar relevance profiles across samples within each module. Extensive experiments demonstrate that CART achieves state-of-the-art performance.

Figures & tables

Figure 1: Performance retention during sequential learning. We plot each task’s performance, normalized to its initial instruction-tuning peak. Even when the corresponding task-specific LoRA is retrieved, performance on previously learned tasks degrades substantially as the projector is continually updated on new tasks.

Figure 2: Updating the projector shows a trade-off between learning new tasks and retaining old ones. Although updating the projector improves performance ( ), learning later tasks can reduce these gains and lower performance on earlier tasks below the frozen-projector baseline ( ). Retraining the language model with the final projector fixed largely restores performance ( ), suggesting that the mismatch between the projector and language model is indeed an important cause of forgetting.

Figure 3: Overview of CART. We jointly train and preserve expert pairs $(\mathbf{E}_{\text{p}}^{(t)},\mathbf{E}_{\text{l}}^{(t)})$ , retain their distribution statistics, and consolidate similar pairs. At inference, Gaussian likelihoods yield relevance scores $R_{\text{p}}(\mathbf{x})$ and $R_{\text{l}}(\mathbf{x})$ . Alternating compatibility-aware updates produce routing weights $q_{\text{p}}(\mathbf{x})$ and $q_{\text{l}}(\mathbf{x})$ , which combine projector outputs into a shared visual representation and mix LLM LoRA branches for inference without task identities.

Methods	ScienceQA	TextVQA	ImageNet	GQA	VizWiz	Grounding	VQAv2	OCR-VQA	Last
Finetune	26.00	25.38	28.51	33.07	26.52	0.10	40.00	52.92	29.06
EWC	59.75	51.06	31.88	51.84	42.26	34.96	55.30	64.55	48.95
O-LoRA	72.56	57.66	62.84	59.14	48.43	58.97	63.21	63.31	60.77
MoELoRA	47.34	32.91	38.73	37.15	42.48	0.97	42.77	57.50	37.48
HiDe-LLaVA	73.20	56.92	69.28	61.33	50.76	59.18	67.12	64.76	62.82
ProgLoRA	74.84	51.83	83.90	49.93	53.87	31.19	62.71	64.44	59.09

Table 1: Main results on the CoIN benchmark with LLaVA-v1.5-7B as the backbone (higher is better). The best and second-best values are marked in bold and underline , respectively.

Methods	ImgNet-R	ArxivQA	VizWiz	IconQA	CLEVR	Flickr30k	Last
Finetune	58.03	77.63	44.39	67.40	61.77	58.22	61.24
EWC	39.05	77.88	43.24	45.33	39.72	55.94	50.19
O-LoRA	77.50	78.07	44.50	63.13	64.73	58.16	64.35
MoELoRA	70.07	77.70	44.69	50.03	54.03	57.34	58.98
HiDe-LLaVA	84.03	90.73	44.43	58.93	41.37	54.25	62.29
DISCO	88.88	94.25	47.52	69.50	60.75	56.32	69.54

Table 2: Main results on the UCIT benchmark with LLaVA-v1.5-7B as the backbone (higher is better). The best and second-best values are marked in bold and underline , respectively.

Variants	ImgNet-R	ArxivQA	VizWiz	IconQA	CLEVR	Flickr30k	Last	$\Delta$
Component Ablation
w/o Distribution-Aware Relevance	80.27	91.60	46.38	79.07	78.67	54.06	71.68	-3.99
Routing Strategy Comparison
Hard Routing	73.87	91.73	45.88	71.73	73.73	52.80	68.29	-7.38
Independent Routing	85.07	92.40	46.41	71.20	69.60	54.53	69.87	-5.80
Naive Joint Routing	86.33	91.33	45.99	76.33	75.67	55.13	71.80	-3.87

Table 3: Ablation study on the UCIT benchmark.

\Delta

denotes the average accuracy change relative to full CART. Best results are highlighted in bold .

Figure 4: Analysis of expert-pool capacity and CAVI hyperparameters on the six-task UCIT benchmark. The left panels compare consolidation strategies under different expert-pool limits, while the right panels examine compatibility strength and the number of CAVI iterations. Overall, stronger compatibility coupling improves retention, and performance stabilizes after only a few CAVI iterations.

Figure 5: Qualitative comparison with NCM-based routing. CART better attends to instruction-relevant visual evidence and produces correct answers.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Domain	Train	Test
GQA	Compositional VQA	20,000	3,000
ImageNet	Image classification	20,000	3,000
TextVQA	Scene-text VQA	34,602	3,000
OCRVQA	OCR-based VQA	20,000	3,000
REC-COCO	Referring expressions	20,000	3,000
VizWiz	Visual assistance	20,523	3,000

Appendix

Table 4: Dataset statistics for the eight-task CoIN benchmark.

Dataset	Domain	Train	Test
ImageNet-R	Artistic	24,000	3,000
ArxivQA	Academic figures	40,000	3,000
VizWiz	Visual assistance	40,000	3,000
IconQA	Icons	30,000	3,000
CLEVR-Math	Synthetic reasoning	40,000	3,000
Flickr30k	Captioning	40,000	3,000

Appendix

Table 5: Dataset statistics for the six-task UCIT benchmark.

Seed	SAME	DISCO	CART
1	72.93	68.76	75.18
10	72.38	69.13	75.57
100	72.56	68.81	75.75
1000	71.86	68.50	75.70
42 (original)	73.07	69.54	75.67
Average	72.56	68.95	75.58

Appendix

Table 6: Multi-seed results on UCIT. We report final average accuracy (%) over five random seeds; mean and standard deviation are computed across runs.

Metric	Original	Reverse	Fixed random
Final Avg. Accuracy (%)	75.68	74.36	75.43

Appendix

Table 7: Task-order robustness of CART on UCIT. Final average accuracy (%) is reported under the original, reverse, and fixed random task orders.

Figure 6: Accuracy–parameter efficiency of MCIT methods on UCIT. CART achieves a Pareto-optimal trade-off between Last Accuracy and adaptation parameters.

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 82 papers

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Mixture-of-Experts Models Multimodal Instruction Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1

Jun 1, 2026J/K move · Enter open · S save

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou

School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China

Mixture-of-Experts Models Multimodal Instruction Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.

Abstract

Continual Learning for VLMs

Explore similar work

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Progressive Multimodal Alignment for Continual Instruction Tuning

MAny: Merge Anything for Multimodal Continual Instruction Tuning