CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

Runze Liu; Naibin Gu; Mingxu Ai; Yuqing Li; Peng Fu; Zheng Lin; Weiping Wang

cs.AIAug 28, 2026

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

Authors: Runze Liu, Naibin Gu, Mingxu Ai, Yuqing Li, Peng Fu, Zheng Lin, Weiping Wang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Abstract

Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based capacity, but repeatedly learning and maintaining full LoRA experts leads to substantial parameter overhead. This raises a natural question: is full expert expansion necessary for every new task? To answer it, we analyze the SVD of task-specific LoRA updates and observe substantial overlap in their input- and output-side LoRA direction subspaces, with task-specific adaptation largely captured by lightweight coordinates over these subspaces. Motivated by this observation, we propose CoRe-MoE, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning. CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank, and for subsequent tasks trains only compact coordinate experts together with task-specific low-rank routers. Experiments on two representative MLLMs show that CoRe-MoE improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks. The code is publicly available at https://github.com/runzezz/CoRe-MoE.

Figures & tables

Figure 1: ArxivQA-to-ImageNet-R direction-subspace overlap. High SAR on both $V^{A}$ and $U^{B}$ indicates clear cross-task overlap.

Panel of Figure 2 — Figure 2: Effect of frozen LoRA direction bases on later-task adaptation.

Figure 3: Overview of CoRe-MoE pipeline. In the initial task, CoRe-MoE trains a vanilla LoRA-MoE expert bank and converts it into reusable direction bases and coordinate matrices through compact coordinate conversion. For subsequent tasks, the direction bases are frozen, and only compact coordinate experts and task-specific routers are trained. During inference, visual and textual prototypes guide task-aware routing, while token-level routing is performed in the corresponding compact coordinate space.

Method	Eval.	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k	Avg. (↑)	BWT final (↑)
Zero-shot	–	16.33%	53.83%	38.56	19.97%	20.40%	41.88	31.83%	–
Sequential Fine-Tuning	–	90.80%	93.07%	59.41	75.50%	72.70%	57.20	74.78%	–
LoRA-FT	Avg.	79.08%	80.85%	48.13	67.63%	55.10%	57.20	64.67%	-20.31%
LoRA-FT	Last	68.20%	75.43%	44.61	64.20%	37.50%	57.20	57.85%	-20.31%
HiDe	Avg.	88.64%	92.81%	49.25	67.84%	52.44%	53.36	67.39%	-5.98%
HiDe	Last	85.67%	93.07%	44.67	64.63%	50.40%	53.36	65.30%	-5.98%

Table 1: Main results on the UCIT benchmark with LLaVA-1.5-7B. Avg. and Last denote continual-average and final-after-last-task performance, respectively. BWT final is computed after the last task. Best and second-best continual results are marked in bold and underlined, and red numbers indicate gains over the second-best result.

Method	Eval.	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k	Avg. ( $\uparrow$ )	BWT final ( $\uparrow$ )
Zero-shot	–	28.30%	39.10%	42.72	31.17%	49.54	38.17%	–
Sequential Fine-Tuning	–	94.27%	91.23%	66.01	97.10%	60.93	82.51%	–
LoRA-FT	Avg.	86.17%	84.37%	59.40	95.83%	60.93	77.34%	-12.16%
LoRA-FT	Last	73.27%	79.90%	52.22	94.57%	60.93	72.18%	-12.16%
HiDe	Avg.	89.25%	83.92%	64.95	96.15%	60.42	78.94%	-6.27%
HiDe	Last	86.33%	80.87%	59.30	94.67%	60.42	76.32%	-6.27%

Table 2: Main results on the UCIT benchmark with Qwen2-VL-7B. Avg. and Last denote continual-average and final performance, respectively. BWT final is computed after the last task. Bold/underline mark the best/second-best continual learning results, and red numbers indicate gains over the second-best result.

Method	Avg. (↑)	Last (↑)	BWT (↑)
Vanilla LoRA-MoE ( $r=16$ )	73.21	73.21	-0.01
Budget-Matched LoRA-MoE	70.61	70.61	-0.01
CoRe-MoE w/ LR Router	71.20	71.20	-0.02
+ Ortho Init w/ LR Router	67.02	66.97	-0.19
CoRe-MoE w/ Full Router	71.54	71.52	-0.07
+ Ortho Init w/ Full Router	71.07	71.05	-0.09

Table 3: Ablation results of expert parameterization, routing strategies, and basis initialization on LLaVA-1.5-7B.

Figure 4: Effect of $\alpha$ on task routing accuracy after completing all six UCIT tasks. The coefficient $\alpha$ balances textual and visual similarities.

Metric	RAVICF	AIRFCV	FRCAVI	CVIFAR
Last	71.20%	69.79%	69.71%	69.90%
Avg.	71.20%	69.80%	69.73%	69.91%

Table 4: Ablation results under different task orders. R, A, V, I, C, and F denote ImageNet-R, ArxivQA, VizWiz-caption, IconQA, CLEVR-Math, and Flickr30k.

Figure 5: Continual learning phase parameter efficiency on UCIT. Bars denote average trainable parameters per later task from Tasks 2–6, and the line denotes cumulative additional stored overhead after six tasks.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Avg. ( $\uparrow$ )	Last ( $\uparrow$ )	BWT ( $\uparrow$ )
$\lambda_{\mathrm{aux}}=10^{-3}$ , lr= $2{\times}10^{-4}$	71.20	71.20	-0.02
$\lambda_{\mathrm{aux}}=0$ , lr= $2{\times}10^{-4}$	70.78	70.83	-0.03
$\lambda_{\mathrm{aux}}=10^{-3}$ , lr= $1{\times}10^{-4}$	68.51	68.50	-0.03
$\lambda_{\mathrm{aux}}=10^{-3}$ , lr= $5{\times}10^{-5}$	66.36	66.30	-0.03

Appendix

Table 5: Hyperparameter study on LLaVA-1.5-7B. Task 1 is trained with a fixed vanilla LoRA-MoE configuration using lr=

2{\times}10^{-4}

. We then fix the Task 1 result and tune the auxiliary load-balancing loss coefficient

\lambda_{\mathrm{aux}}

and learning rate for Task 2–Task 6.

Figure 6: Cumulative parameter overhead over the base model after each UCIT task.

Figure 7: Confusion matrix of CLIP-based task routing after completing all six UCIT tasks.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k
ImageNet-R	90.80%	–	–	–	–	–
ArxivQA	87.87%	93.07%	–	–	–	–
VizWiz	74.70%	88.90%	59.41	–	–	–
IconQA	76.37%	70.30%	44.33	75.50%	–	–
CLEVR-Math	76.53%	76.60%	44.18	63.20%	72.70%	–
Flickr30k	68.20%	75.43%	44.61	64.20%	37.50%	57.20

Appendix

Table 6: Detailed UCIT evaluation matrix of LLaVA-1.5-7B with LoRA-FT.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k
ImageNet-R	91.20%	–	–	–	–	–
ArxivQA	90.43%	92.77%	–	–	–	–
VizWiz	79.73%	91.17%	59.83	–	–	–
IconQA	80.87%	76.87%	53.32	76.00%	–	–
CLEVR-Math	78.33%	78.27%	50.67	68.63%	74.87%	–
Flickr30k	69.40%	78.37%	44.17	67.93%	48.90%	57.36

Appendix

Table 7: Detailed UCIT evaluation matrix of LLaVA-1.5-7B with MoELoRA.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k
ImageNet-R	91.47%	–	–	–	–	–
ArxivQA	90.13%	89.93%	–	–	–	–
VizWiz	88.53%	90.00%	46.53	–	–	–
IconQA	87.70%	79.07%	44.80	72.93%	–	–
CLEVR-Math	86.13%	78.60%	43.70	67.77%	68.07%	–
Flickr30k	84.37%	78.33%	44.54	64.93%	51.70%	57.41

Appendix

Table 8: Detailed UCIT evaluation matrix of LLaVA-1.5-7B with CL-MoE.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k
ImageNet-R	90.47%	–	–	–	–	–
ArxivQA	90.47%	92.13%	–	–	–	–
VizWiz	90.47%	92.13%	56.87	–	–	–
IconQA	90.47%	92.13%	56.87	65.73%	–	–
CLEVR-Math	90.47%	91.97%	56.87	65.73%	66.37%	–
Flickr30k	90.47%	91.97%	56.91	65.73%	66.37%	55.74

Appendix

Table 9: Detailed UCIT evaluation matrix of LLaVA-1.5-7B with CoRe-MoE.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	CLEVR-Math	Flickr30k
ImageNet-R	91.43%	–	–	–	–	–
ArxivQA	90.40%	93.87%	–	–	–	–
VizWiz	89.10%	92.47%	55.73	–	–	–
IconQA	88.10%	92.23%	49.65	72.57%	–	–
CLEVR-Math	87.03%	93.07%	46.68	66.33%	54.47%	–
Flickr30k	85.67%	93.07%	44.67	64.63%	50.40%	53.36

Appendix

Table 10: Detailed UCIT evaluation matrix of LLaVA-1.5-7B with HiDe.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k
ImageNet-R	94.27%	–	–	–	–
ArxivQA	93.00%	91.23%	–	–	–
VizWiz	85.40%	86.83%	66.01	–	–
IconQA	84.93%	79.50%	59.97	97.10%	–
Flickr30k	73.27%	79.90%	52.22	94.57%	60.93

Appendix

Table 11: Detailed UCIT evaluation matrix of Qwen2-VL-7B-Instruct with LoRA-FT.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k
ImageNet-R	94.20%	–	–	–	–
ArxivQA	93.13%	91.13%	–	–	–
VizWiz	82.23%	85.77%	66.32	–	–
IconQA	79.93%	79.17%	62.60	97.50%	–
Flickr30k	60.60%	79.07%	49.94	95.10%	60.92

Appendix

Table 12: Detailed UCIT evaluation matrix of Qwen2-VL-7B-Instruct with MoELoRA.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k
ImageNet-R	94.03%	–	–	–	–
ArxivQA	93.77%	95.23%	–	–	–
VizWiz	93.03%	95.17%	52.09	–	–
IconQA	92.77%	88.60%	51.74	95.03%	–
Flickr30k	90.57%	88.70%	51.53	91.93%	60.74

Appendix

Table 13: Detailed UCIT evaluation matrix of Qwen2-VL-7B-Instruct with CL-MoE.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k
ImageNet-R	94.13%	–	–	–	–
ArxivQA	94.13%	93.80%	–	–	–
VizWiz	94.13%	93.80%	65.52	–	–
IconQA	94.13%	93.87%	65.52	92.13%	–
Flickr30k	94.13%	93.87%	65.50	92.13%	60.71

Appendix

Table 14: Detailed UCIT evaluation matrix of Qwen2-VL-7B-Instruct with CoRe-MoE.

Training Step	ImageNet-R	ArxivQA	VizWiz	IconQA	Flickr30k
ImageNet-R	91.73%	–	–	–	–
ArxivQA	90.37%	88.40%	–	–	–
VizWiz	89.27%	85.70%	68.49	–	–
IconQA	88.53%	80.70%	67.07	97.63%	–
Flickr30k	86.33%	80.87%	59.30	94.67%	60.42

Appendix

Table 15: Detailed UCIT evaluation matrix of Qwen2-VL-7B-Instruct with HiDe.

Explore similar work

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University
CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Mixture-of-Experts Models Multimodal Instruction Tuning
School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia

Oct 9, 2026J/K move · Enter open · S save

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Xi Zhang, Boyu Diao, Zesen Chen, Ruiqi Liu+2

Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University

Continual Learning for VLMs Multimodal Instruction Tuning Low-Rank Adaptation Memory-Efficient Fine-Tuning

Open paper AskPDF arXiv

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.

Abstract

Explore similar work

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning