Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.
Figures & tables
Figure 1: Empirical analysis of cross-task expert redundancy on UCIT. Comparing native experts ( Before Swap ) with cross-task substitution ( After Overlap Swap ) shows that high-overlap experts preserve 90%∼100% of performance across tasks, confirming substantial functional redundancy while highlighting the need to preserve task-specific variations.
Figure 2: Overview of CoPre-MoE. Left: Task-specific low-rank experts and key-value routing pairs are trained on a frozen MLLM backbone. Middle: At task boundaries, functionally overlapping experts are consolidated into a fixed-capacity Generic Pool, while task-specific experts are factorized into shared bases and compact cores in a Private Pool. Right: Routing query q dynamically queries both pools to compose generic and private operators for inference.
Method
ImageNet-R
ArxivQA
VizWiz
IconQA
CLEVR
Flickr30k
A↑
BWT ↑
Zero-shot [ 26 ]
16.27
53.73
38.39
19.20
20.63
41.88
31.68
–
LoRA-FT [ 16 ]
58.03
77.63
44.39
67.40
61.77
58.22
61.24
-18.78
MoELoRA [ 3 ]
70.07
77.70
44.69
50.03
54.03
57.34
58.98
-14.63
SEFE [ 4 ]
80.83
78.00
47.01
69.63
65.83
57.92
66.54
-11.33
HiDe-LLaVA [ 13 ]
84.03
90.73
44.43
58.93
41.37
54.25
62.29
-9.20
SAME [ 47 ]
83.83
91.40
51.33
65.27
53.50
57.43
67.12
-2.41
Table 1: Final task accuracy and backward transfer on UCIT ( ↑ : higher is better). Bold and underline denote best and second-best results.
Method
ScienceQA
TextVQA
ImageNet
GQA
VizWiz
Grounding
VQAv2
OCR-VQA
A↑
BWT ↑
Zero-shot [ 26 ]
67.92
57.71
47.37
61.28
45.17
6.12
52.03
53.58
48.89
–
LoRA-FT [ 16 ]
26.00
25.38
28.51
33.07
26.52
0.10
40.00
52.92
29.06
-31.16
MoELoRA [ 3 ]
62.02
52.05
37.21
53.12
43.32
33.22
57.92
65.75
50.58
-19.04
SEFE [ 4 ]
75.35
58.66
83.10
54.25
48.85
16.75
65.35
66.25
58.57
-11.94
HiDe-LLaVA [ 13 ]
73.20
56.92
69.28
61.33
50.76
59.18
67.12
64.76
62.82
-6.38
SAME [ 47 ]
78.35
60.69
90.21
61.70
54.13
59.87
66.04
63.59
66.82
-4.23
Table 2: Final task accuracy and backward transfer on CoIN ( ↑ : higher is better). Bold and underline denote best and second-best results.
Setting
ImageNet-R
ArxivQA
VizWiz
IconQA
CLEVR
Flickr30k
A↑
BWT ↑
CoPre-MoE (Full model)
88.93
93.73
62.83
74.07
66.63
57.39
73.93
-0.10
w/o Dynamic role allocation
88.17
91.27
60.66
73.07
64.80
56.54
72.42
-0.52
w/o Activation-weighted merge
88.93
94.10
62.22
73.20
61.20
56.96
72.77
-0.02
w/o Multi-key retention
88.87
93.23
62.12
73.93
62.17
57.38
72.95
-0.45
Table 3: UCIT ablation study evaluating expert allocation, function consolidation, and routing mechanisms. Higher A and BWT are better ( ↑ ).
Figure 3: Backward transfer trajectory across sequential training stages t on UCIT. Points evaluate average BWT over previously learned tasks; terminal values report final retention.
Figure 4: t-SNE visualization on UCIT (Layer 16). (a) Baseline suffers from severe representational disorder across tasks. (b) Generic Pool forms a continuous shared manifold across tasks without task segregation. (c) Private Pool isolates task-specific features into well-separated, directional clusters. (d) Full CoPre-MoE model combines both pools into well-structured, discriminative task representations.
Figure 5: Private capacity on UCIT. Labels denote rank budgets; orange highlights the unchanged default rank 16. Each point is a full task-sequence run. Counts include Private factors only.
Figure 6: Sensitivity to routing orthogonality regularization weight λorth on UCIT in terms of final average score A and backward transfer BWT (default λorth=1.0 ).
Order
RAVICF
AIRFCV
IFRCAV
FCIVAR
A↑
73.93
73.39
71.83
72.90
BWT ↑
−0.10
0.85
0.07
1.70
Table 4: Task-order sensitivity on UCIT. R, A, V, I, C, F denote ImageNet-R, ArxivQA, VizWiz, IconQA, CLEVR, and Flickr30k.
Method
Trainable (M) ↓
Train (h) ↓
Latency (ms/sample) ↓
Throughput (Tokens/s) ↑
A↑
BWT ↑
MoELoRA
246.69
11.68
583.91
10.25
58.98
-14.63
SAME
73.29
10.42
404.99
14.30
67.12
-2.41
CoPre-MoE
78.12
8.15
326.13
17.34
73.93
-0.10
Table 5: Efficiency and retention comparison under identical hardware. Trainable counts adaptation parameters; Train denotes cumulative training hours; Latency and Throughput measure amortized per-sample time and output token rate; A and BWT denote final average score and backward transfer.
Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based capacity, but repeatedly learning and maintaining full LoRA experts leads to substantial parameter overhead. This raises a natural question: is full expert expansion necessary for every new task? To answer it, we analyze the SVD of task-specific LoRA updates and observe substantial overlap in their input- and output-side LoRA direction subspaces, with task-specific adaptation largely captured by lightweight coordinates over these subspaces. Motivated by this observation, we propose CoRe-MoE, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning. CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank, and for subsequent tasks trains only compact coordinate experts together with task-specific low-rank routers. Experiments on two representative MLLMs show that CoRe-MoE improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks. The code is publicly available at https://github.com/runzezz/CoRe-MoE.
Runze Liu, Naibin Gu, Mingxu Ai +4
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models. On SuperNI benchmark, spanning diverse sequential language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2 dataset, it scales effectively to multimodal visual reasoning, consistently reduces forgetting, and outperforms strong MoE baselines.
Yang Liu, Toan Nguyen, Flora D. Salim
School of Computer Science and Engineering University of New South Wales Australia
Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.
Xi Zhang, Boyu Diao, Zesen Chen +3
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University