Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Xi Zhang; Boyu Diao; Zesen Chen; Ruiqi Liu; Hangda Liu; Yongjun Xu

cs.LGOct 9, 2026

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

Authors: Xi Zhang, Boyu Diao, Zesen Chen, Ruiqi Liu, Hangda Liu, Yongjun Xu

Organizations: Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University

Abstract

Learning new tasks in multimodal continual instruction tuning can cause catastrophic forgetting of previously acquired knowledge. Although multi-expert LoRA distributes learning across experts, updates to a dense low-rank expert can still affect many output features. We introduce Monarch-MoE, which replaces the dense factors in LoRA experts with block-diagonal factors connected by a fixed permutation. This structure provides blockwise gradient isolation: individual block-pair updates affect only one output block, while connections between all input and output blocks are maintained. To make these compact experts practical on memory-limited hardware, we combine batched computation, fused forward operators, and an optimized backward pass. Across three continual-learning methods evaluated on MLLM-DCL and UCIT with LLaVA-1.5-7B, Monarch-MoE improves first-task retention while reducing adapter parameters by approximately 75% and 50%, respectively. In the main comparison, first-task forgetting is as low as 0.52 percentage points on MLLM-DCL and 0.10 on UCIT, with method-dependent trade-offs in final performance. Our implementation supports fine-tuning and inference on a single 24 GB RTX 4090, with lower measured memory use and competitive runtime in the evaluated settings. Together, these algorithmic and systems improvements make structured experts a practical option for continual adaptation on memory-limited hardware.

Figures & tables

Figure 1: Monarch factorization and channel permutation. (a)

M=LPR

connects all input–output block pairs. (b)

P

reorders the channels. The highlighted pair update changes only

y_{1}

at the expert output; full steps can update all pairs. Colors denote block pairs, not tasks.

	MLLM-DCL (5 tasks)						UCIT (6 tasks)
Method	Params	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$	$\mathrm{F}_{1}\downarrow$	Params	MFT $\uparrow$	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$	$\mathrm{F}_{1}\downarrow$
Existing methods (LoRA experts)
LoRA-FT †	$\sim$ 80	64.98	53.00	61.13	-14.97	8.67	$\sim$ 40	76.89	61.24	76.55	-18.78	33.97
MoELoRA †	$\sim$ 93	64.94	54.78	61.76	-12.71	1.55	$\sim$ 52	71.17	58.98	75.08	-14.63	21.16
CL-MoE †	$\sim$ 405	66.06	54.88	61.79	-13.97	7.74	$\sim$ 247	71.46	58.49	74.19	-15.56	24.97
O-LoRA †	$\sim$ 400	65.16	55.54	62.12	-12.03	4.61	$\sim$ 240	76.01	64.35	78.02	-13.99	13.90

Table 1: Full continual-learning results on MLLM-DCL and UCIT.

\dagger

: MCITlib baselines. Bold : best value per metric and benchmark among the compared methods. Parameters include adapter factors and routers (millions);

\sim

denotes a configuration estimate. Only the DCL standalone pair is parameter matched. Drop-in

\alpha

: CL-MoE/O-LoRA 16 on DCL and 64 on UCIT; DISCO 256 on both. See Appendix B.1 for counts and Appendix B.2 for metric definitions.

Figure 2: Retention and representation stability on DCL. Left: first-task RS scores after each training stage. Raising dense LoRA rank from 16 to 160 does not recover Monarch’s retention. Right: adapter-output linear CKA relative to the first checkpoint, using the same 256 RS examples at layer 16. Higher similarity indicates more stable representations at this measurement point; it does not by itself establish task isolation.

Expert	$\rho$	Params (M)	MFT	MFN	MAA	BWT	$\mathrm{F}_{1}$
LoRA $q=16$	16	164.5	67.48	48.03	54.16	-24.32	25.98
LoRA $q=160$	160	1603.6	69.16	47.27	53.67	-27.37	26.19
Monarch $m=8,r=2$	128	164.5	63.77	51.21	59.90	-15.70	3.01
Monarch $m=4,r=1$	16	44.5	64.05	51.23	60.29	-16.02	3.07

Table 2: DCL capacity controls.

\rho

is the per-expert rank bound. The 44.5M variant matches the dense rank bound, whereas the 164.5M variant matches parameters. Output scales differ (Appendix B.1 ).

$m$	$r$	$\rho=m^{2}r$	RS immediate	RS retained	Med	$\mathrm{F}_{1}$
1	8	8	79.35	76.06	52.91	3.29
2	4	16	79.17	76.07	52.03	3.10
4	2	32	78.95	76.63	51.45	2.32
8	1	64	78.77	77.03	50.48	1.74

Table 3: Two-task block-count probe at fixed factor budget.

mr=8

K=4

\alpha=8

and scale

s=1

. Increasing

m

changes the rank bound; each task is trained for one epoch.

Figure 3: Repeatability and directional partition diagnostics. Left: all three seeds for the equal-budget RS

\rightarrow

Med comparison, with mean

\pm

sample standard deviation (

n=3

); RS and Med use one and three epochs. Right: complete directional probe with

r=2

and

\alpha=2

, one epoch per task, and one run per configuration. Points show first-task forgetting; the Med column reports immediate new-task scores. Factor allocation and effective scale vary across partitions. The panels use different protocols and axis ranges and should not be compared as one experiment.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Initialization	MFT	MFN	MAA	BWT	$\mathrm{F}_{1}$
Zero	64.05	51.23	60.29	-16.02	3.07
SVD-Low	62.22	52.05	59.12	-12.71	1.58
SVD-High	45.62	24.96	30.01	-25.82	20.47

Appendix

Table 4: Blockwise SVD initialization, DCL. Standalone

K=4,m=4,r=1,\alpha=1

; the same 44.5M adapter budget.

Setting	Expert	$K$	Rank specification	$\alpha$	Params (M)
DCL standalone	LoRA	4	$q=16$	32	164.463 C
DCL capacity	LoRA	4	$q=160$	320 ∗	1603.633 C
DCL standalone	Monarch	4	$(8,2)$	1	164.463 C
DCL rank control	Monarch	4	$(4,1)$	1	44.532 C
DCL CL-MoE	Monarch	5	$(4,2)$	16	105.636 C
DCL O-LoRA	Monarch	5	$(4,2)$	16	99.942 C

Appendix

Table 5: Main expert configurations.

q

is a per-expert LoRA rank;

(m,r)

specifies symmetric Monarch.

\alpha

is the adapter’s implemented multiplier before rank normalization. Counts marked C come from archived checkpoint-count summaries; E denotes an estimate from configurations.

Expert	Seed 42	Seed 1234	Seed 2026	Mean $\pm$ std
LoRA $q=16$	34.05	32.07	31.28	$32.47\pm 1.43$
Monarch $(8,2)$	1.27	1.04	1.05	$1.12\pm 0.13$

Appendix

Table 6: First-task forgetting (pp) over three seeds on the RS

\rightarrow

Med prefix. Both models use the 164.5M-parameter standalone configurations. The last column reports mean

\pm

sample standard deviation (

n=3

); these are not repeated full DCL sequences.

$m_{h}$	$m_{w}$	$r$	$\alpha$	$s$	RS immediate	RS retained	Med	$\mathrm{F}_{1}$
1	1	2	2	1.00	78.98	76.72	51.54	2.26
4	1	2	2	0.25	78.70	76.81	49.61	1.89
8	1	2	2	0.13	78.42	76.99	49.79	1.43
1	4	2	2	1.00	78.95	76.53	51.41	2.42
1	8	2	2	1.00	78.90	76.45	51.33	2.45
1	1	1	1	1.00	78.88	76.53	50.34	2.35

Appendix

Table 7: Directional and fixed-

r

probes. Upper: output-only versus input-only partitions. Lower: symmetric

r=1

sweep, with increasing parameter count; it is not a same-budget experiment. The

(8,8,1)

row is the same run as Table 3 , not an independent replicate.

Expert	Task	E0	E1	E2	E3	Entropy	Dead (%)
Monarch	RS	25.83	24.25	24.63	25.29	.9998	.22
	Med	25.36	26.08	24.01	24.56	.9996	.78
	AD	25.71	26.26	24.42	23.60	.9994	1.67
	Sci	25.84	25.80	24.71	23.65	.9995	1.45
	Fin	26.04	26.14	24.77	23.05	.9991	2.34
LoRA	RS	25.89	25.78	23.81	24.52	.9996	.45

Appendix

Table 8: Archived router-utilization summary. Selection share is the percentage of top-2 expert selections. Entropy is normalized by

\log 4

. “Dead” is the reported percentage of unused layer–expert slots, not an expert-selection share.

Method / $\alpha$	MFT	MFN	MAA	BWT	$\mathrm{F}_{1}$
O-LoRA $16$	64.48	54.08	61.61	-12.99	1.80
O-LoRA $64$	58.97	59.46	57.63	0.61	-10.32
DISCO $16$	58.84	56.30	58.44	-3.17	7.90
DISCO $192$	63.14	62.68	63.70	-0.58	1.13
DISCO $256$	63.89	63.56	64.86	-0.41	0.52

Appendix

Table 9: DCL scale sensitivity. Complete runs at each indicated

\alpha

, with

m=4,r=2

and zero initialization. No separate validation-based scale selection is documented.

Method / $\alpha$	MFT	MFN	MAA	BWT	$\mathrm{F}_{1}$
CL-MoE $16$	69.50	55.14	73.36	-17.23	28.27
CL-MoE $64$	68.39	56.30	72.79	-14.51	19.23
O-LoRA $16$	69.31	57.61	75.16	-14.04	15.13
O-LoRA $64$	67.91	57.83	73.27	-12.11	7.66
DISCO $16$	63.94	61.01	76.22	-3.52	0.20
DISCO $192$	68.30	67.34	78.58	-1.15	0.13

Appendix

Table 10: UCIT scale sensitivity. Complete runs at each indicated

\alpha

, with

m=4,r=2

and zero initialization. No separate validation-based scale selection is documented.

Measurement	LoRA-MoE	Monarch-MoE
Steady training-step time (s)	11.907	9.964
Training throughput (samples/s)	1.344	1.606
Maximum training peak (recorded GB)	20.216	19.300
Inference forward time (s)	0.725	0.647
Inference mean peak (recorded GB)	16.205	16.186

Appendix

Table 11: Single-RTX-4090 DCL-RS measurements from the retained efficiency registry. Both standalone models have 164.5M adapter parameters, microbatch 4, and accumulation 4. Training memory is the maximum recorded step peak; inference memory is the mean forward-probe peak. GB follows the registry’s unit label. Forward probes are not autoregressive generation.

Setting	Samples/s	Peak (MB)	Exit code
Local LoRA $q=32,\alpha=64,K=5$	–	23,966	1
Monarch $m=4,r=2,\alpha=16,K=5$	1.039	23,410	0

Appendix

Table 12: Separate five-expert capacity probe on one RTX 4090 at microbatch 4 and sequence limit 2048. Peak-memory units follow the source field (MB). The dense run’s exit code does not identify its failure cause.

Setting	Adapter (M)	Samples/s	Latency (ms)	Peak (MiB)
LoRA-MoE	164.5	5.90	169.5	14,582
Monarch-MoE	44.5	5.42	184.5	14,352

Appendix

Table 13: Archived single-GPU generation benchmark with model-level compilation. Monarch uses

m=4,r=1,K=4

; LoRA uses

q=16,K=4

. Peak memory is recorded in MiB.

Explore similar work

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
Aug 28, 2026Runze Liu, Naibin Gu, Mingxu Ai +4Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
Mar 29, 2026Chongyang Zhao, Mingsong Li, Haodong Lu +1Continual Learning for VLMs Mixture-of-Experts Models
University of New South Wales (UNSW Sydney)

Aug 28, 2026J/K move · Enter open · S save

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

Runze Liu, Naibin Gu, Mingxu Ai, Yuqing Li+3

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

Continual Learning for VLMs Multimodal Instruction Tuning Mixture of Experts Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Open paper AskPDF arXiv

Continual multimodal instruction tuning requires multimodal large language models to acquire new task abilities sequentially while preserving previously learned knowledge. LoRA-MoE provides a promising solution by introducing expert-based capacity, but repeatedly learning and maintaining full LoRA experts leads to substantial parameter overhead. This raises a natural question: is full expert expansion necessary for every new task? To answer it, we analyze the SVD of task-specific LoRA updates and observe substantial overlap in their input- and output-side LoRA direction subspaces, with task-specific adaptation largely captured by lightweight coordinates over these subspaces. Motivated by this observation, we propose CoRe-MoE, a Compact Reusable MoE framework for parameter-efficient continual multimodal instruction tuning. CoRe-MoE extracts reusable input- and output-side direction bases from an initial expert bank, and for subsequent tasks trains only compact coordinate experts together with task-specific low-rank routers. Experiments on two representative MLLMs show that CoRe-MoE improves final average performance over the strongest competing baseline by up to 5.90 points, while using less than 1% of the trainable parameters required by sequential LoRA for later tasks. The code is publicly available at https://github.com/runzezz/CoRe-MoE.

Abstract

Explore similar work

CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models