On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models

Chongyang Zhao; Mingsong Li; Haodong Lu; Dong Gong

cs.LGMar 29, 2026

On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models

Authors: Chongyang Zhao, Mingsong Li, Haodong Lu, Dong Gong

Organizations: University of New South Wales (UNSW Sydney)

Abstract

Multimodal Continual Instruction Tuning aims to continually enhance Large Vision Language Models (LVLMs) by learning from new data without forgetting previously acquired knowledge. Mixture of Experts (MoE) architectures naturally facilitate this by incrementally adding new experts and expanding routers while keeping the existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing-drift: old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks. We analyze the failure mode at the token level and reveal the token's dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts, due to their ambiguous routing assignment during training. Motivated by this, we propose LLaVA-DyMoE, a dynamic MoE framework that incrementally expands the MoE with drift-aware token assignment. We characterize token types via their routing score distributions and apply targeted regularization. Specifically, a token-level assignment guidance steers ambiguous and old tokens away from new experts to preserve established routing patterns and alleviate routing-drift, while complementary routing score regularizations enforce expert-group separation and promote new-expert specialization. Extensive experiments demonstrate that our LLaVA-DyMoE effectively mitigates routing-drift-induced forgetting, achieving over a 7% gain in mean final accuracy and a 12% reduction in forgetting compared to baselines. The project page is https://zhaoc5.github.io/DyMoE.

Figures & tables

Figure 1: Performance comparison on the CoIN benchmark, showing per-task final accuracy and mean final accuracy (MFN).

Panel of Figure 2 — Figure 2: Routing-drift analysis in a controlled two-task learning experiment. After learning on the 1st task (SQA), we conduct the 2nd task (TextVQA) training using the baseline (default training) and three different token masking strategies based on token type. Throughout the training stages, we evaluate forgetting (decrease in task 1 accuracy) and new-task learning (improvement of task 2 accuracy). Polynomial regression-fitted curves are used for better visualization and readability of performance changes. The baseline (default training) refers to a scenario where each input token is assigned to all experts (including old frozen and new learnable ones). We then examine the role of each token group based on its routing score: (a) We only retain the contribution of tokens that have high affinity to the new expert group (termed “new tokens”). (b) We mask out the contribution of tokens with a high affinity to the old expert group (termed “old tokens”). (c) We only retain the contribution of tokens with a small affinity difference between the old and new expert group (termed “ambiguous tokens”).

Figure 3: Overview of our LLaVA-DyMoE method, which applies a dynamic MoE with LoRA experts to each layer of the language backbone in LLaVA. It is a two-fold regularization approach designed to resolve routing-drift-induced forgetting, based on our analysis of different token types in Sec. 3.3 . The right panel illustrates this high-level approach: as new tasks (Task $t$ ) arrive, the router and experts expand, creating a frozen “ old group ” and a trainable “ new group ”. Our Token Assignment Guidance (TAG) prevents routing-drift (red dashed arrow) by directing tokens to appropriate expert–router groups, complemented by our Routing Score Regularization (RSR) that encourage exclusive token-to-group routing and new-expert specialization. Our method regularizes the router behavior during training and imposes no constraints at inference, allowing seamless combination with other continual learning methods.

Method	Accuracy on Each Task (%)								Aggregate Results (%)
Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
LoRA	52.56	48.12	39.27	44.47	37.46	1.22	56.10	55.11	41.79	43.99	-23.12
MoELoRA [ 7 ]	72.01	46.89	44.75	42.79	28.22	3.31	55.74	57.72	43.93	43.92	-22.18
EWC [ 54 ]	59.11	47.21	39.88	45.12	35.33	2.72	56.29	41.21	40.86	43.75	-21.76
LWF [ 36 ]	62.32	48.66	51.45	45.84	43.76	0.24	54.96	44.63	43.98	44.89	-19.69
IncLoRA	73.33	44.32	54.59	44.07	25.93	4.49	54.91	58.55	45.02	43.12	-23.21

Table 1: Comparison with continual learning models on the CoIN benchmark.

Table 5

	Method	Accuracy on Each Task (%)								Aggregate Results (%)
	Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
None	MoELoRA [ 7 ]	72.01	46.89	44.75	42.79	28.22	3.31	55.74	57.72	43.93	43.92	-22.18
None	O-LoRA [ 63 ]	75.61	49.98	78.24	44.18	30.70	4.66	55.51	57.37	49.53	46.65	-17.54
+ ASD [ 8 ]	SEFE [ 8 ]	75.35	58.66	83.10	54.25	48.85	16.75	65.35	66.25	58.57	63.04	-10.45
+ ASD [ 8 ]	LLaVA-DyMoE (Ours)	74.60	55.24	93.80	53.45	55.00	25.50	63.95	62.85	60.55	62.26	-4.75
+ Replay	ProgLoRA [ 73 ]	74.84	51.83	83.90	49.93	53.87	31.19	62.71	64.44	59.09	62.38	-6.59

Table 5: LLaVA-DyMoE is compatible with data-based continual learning strategies.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task Order	Aggregate Results (%)
Task Order	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
Origin	57.03	57.70	-4.67
Reverse	56.67	57.34	-4.71
Alphabet	56.44	56.98	-4.92

Appendix

Table 6: Performance under varying task orders.

Instruction	Aggregate Results (%)
Instruction	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
Origin	57.03	57.70	-4.67
Diverse	56.90	57.44	-5.12
10Type	56.77	57.25	-4.87

Appendix

Table 7: Performance under distinct training instruction types.

Top-K	Method	Aggregate Results (%)
Top-K	Method	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
8	IncMoELoRA	48.55	49.87	-16.28
8	LLaVA-DyMoE	56.89	57.71	-5.12
16	IncMoELoRA	49.68	49.50	-16.67
16	LLaVA-DyMoE	57.03	57.70	-4.67

Appendix

Table 8: Ablations on top-K value.

Expert Number	Method	Aggregate Results (%)
Expert Number	Method	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
8	IncMoELoRA	48.39	50.62	-17.93
8	LLaVA-DyMoE	56.97	58.37	-5.78
16	IncMoELoRA	49.68	49.50	-16.67
16	LLaVA-DyMoE	57.03	57.70	-4.67

Appendix

Table 9: Ablations on expert number.

Expert Capacity	Method	Aggregate Results (%)
Expert Capacity	Method	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
1	IncMoELoRA	48.23	49.14	-15.79
1	LLaVA-DyMoE	56.78	57.34	-4.13
2	IncMoELoRA	49.08	49.25	-16.58
2	LLaVA-DyMoE	56.91	57.48	-4.62
4	IncMoELoRA	49.68	49.50	-16.67

Appendix

Table 10: Ablations on expert capacity.

Size	Method	Accuracy on Each Task (%)								Aggregate Results (%)
Size	Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
7B	IncMoELoRA	68.43	50.31	68.42	47.97	39.46	4.56	57.31	60.95	49.68	49.50	-16.67
7B	LLaVA-DyMoE (Ours)	76.25	53.86	95.80	48.40	52.35	9.25	58.30	62.00	57.03	57.70	-4.67
13B	IncMoELoRA	68.75	51.69	85.80	48.10	40.20	6.55	58.85	64.60	53.07	53.20	-14.23
13B	LLaVA-DyMoE (Ours)	78.75	56.24	96.05	55.85	53.20	13.85	64.05	65.15	60.39	61.25	-4.64

Appendix

Table 11: Performance across different model sizes.

Method	Accuracy on Each Task (%)								Aggregate Results (%)
Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
LLaVA-DyMoE	76.25	53.86	95.80	48.40	52.35	9.25	58.30	62.00	57.03	57.70	-4.67
+ Task Router	78.18	53.36	95.63	54.63	53.92	24.46	59.54	60.40	60.02	60.78	-1.73

Appendix

Table 12: LLaVA-DyMoE is compatible with task-level routing methods.

Replay Size	Method	Aggregate Results (%)
Replay Size	Method	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
200	ProgLoRA	59.09	62.38	-6.59
200	LLaVA-DyMoE	62.08	61.93	-1.55
500	ProgLoRA	59.14	62.74	-6.47
500	LLaVA-DyMoE	62.55	62.17	-1.00
1000	ProgLoRA	59.66	63.23	-6.21

Appendix

Table 13: LLaVA-DyMoE is compatible with data-based continual learning strategies. Table 5 in the main paper shows that the proposed token assignment regularization can work with replay techniques. This table shows the performance across different replay buffer sizes, with ProgLoRA [ 73 ] (containing replay) as a reference.

Method	Accuracy on Each Task (%)								Aggregate Results (%)
Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	MFN $\uparrow$	MAA $\uparrow$	BWT $\uparrow$
LLaVA-DyMoE	76.25	53.86	95.80	48.40	52.35	9.25	58.30	62.00	57.03	57.70	-4.67
+ Pruning $1/8$	76.04	53.91	96.10	48.16	52.51	9.27	58.21	61.79	57.00	57.62	-4.63
+ Pruning $1/4$	75.59	53.79	95.11	47.78	52.51	9.26	57.94	61.39	56.67	57.37	-4.48

Appendix

Table 14: Performance of LLaVA-DyMoE with expert pruning.

Panel of Figure 4 — Figure 4: Layer-wise expert activation on the CoIN benchmark. Activation frequency is shown for each expert group across layers, and circle size reflects how often an expert is activated.

Figure 5: Comparisons between baseline IncMoELoRA and LLaVA-DyMoE on cases after training on the final task. The first column shows cases from ScienceQA, the second column shows cases from ImageNet.

Method	Training Time on Each Task (min)								Average
Method	SQA	VQA Text	ImgNet	GQA	VizWiz	REF	VQA v2	VQA OCR	Average
IncMoELoRA	7.4	13.6	92.4	123.4	14.9	99.6	105.0	137.1	74.18
LLaVA-DyMoE (Ours)	7.4	14.6	95.5	127.6	15.8	103.2	109.6	145.7	77.43

Appendix

Table 15: Training time of LLaVA-DyMoE during sequential training.

Task	Original	Diverse	10Type
SQA	Answer with the option’s letter from the given choices directly	Answer with the option’s letter from the given choices directly	Answer with the option’s letter from the given choices directly Select the correct answer from the given choices and respond with the letter of the chosen option Determine the correct option from the provided choices and reply with its corresponding letter Pick the correct answer from the listed options and provide the letter of the selected option Identify the correct choice from the options below and respond with the letter of the correct option From the given choices, choose the correct answer and respond with the letter of that choice Choose the right answer from the options and respond with its letter Select the correct answer from the provided options and reply with the letter associated with it From the given choices, select the correct answer and reply with the letter of the chosen option Identify the correct option from the choices provided and respond with the letter of the correct option From the given choices, pick the correct answer and respond by indicating the letter of the correct option
VQA Text	Answer the question using a single word or phrase	Capture the essence of your response in a single word or a concise phrase	Answer the question with just one word or a brief phrase Use one word or a concise phrase to respond to the question Answer using only one word or a short, descriptive phrase Provide your answer in the form of a single word or a brief phrase Use a single word or a short phrase to respond to the question Summarize your response in one word or a concise phrase Respond to the question using a single word or a brief phrase Provide your answer in one word or a short, descriptive phrase Answer the question with a single word or a brief, descriptive phrase Capture the essence of your response in one word or a short phrase Capture the essence of your response in a single word or a concise phrase
ImgNet	Answer the question using a single word or phrase	Express your answer in a single word or a short, descriptive phrase	Express your answer in a single word or a short, descriptive phrase Provide your answer using a single word or a brief phrase Describe the content of the image using one word or a concise phrase Respond to the question with a single word or a short, descriptive phrase Classify the image content using only one word or a brief phrase Give your answer in the form of a single word or a concise phrase Use a single word or a short phrase to categorize the image content Express your answer with one word or a short, descriptive phrase Identify the type of content in the image using one word or a concise phrase Summarize your response in a single word or a brief phrase Use one word or a short phrase to classify the content of the image
GQA	Answer the question using a single word or phrase	Respond to the question briefly, using only one word or a phrase	Respond to the question with a single word or a short phrase Respond to the question using only one word or a concise phrase Answer the question with a single word or a brief phrase Respond with one word or a short phrase Provide your answer in the form of a single word or a concise phrase Respond to the question with just one word or a brief phrase Answer the question using a single word or a concise phrase Provide your response using only one word or a short phrase Respond to the question with a single word or a brief phrase Respond to the question using just one word or a concise phrase Answer the question with one word or a short phrase
VizWiz	Answer the question using a single word or phrase	Provide a succinct response with a single word or phrase	Answer the question using only one word or a concise phrase Respond to the question using only one word or a concise phrase Respond to the question with a single word or a brief phrase Provide your answer using just one word or a short phrase Respond with one word or a concise phrase Answer the question with just one word or a brief phrase Use a single word or a short phrase to answer the question Provide your answer in the form of one word or a brief phrase Reply to the question using one word or a concise phrase Answer with a single word or a short phrase Use one word or a brief phrase to answer the question
REF	Please provide the bounding box coordinate of the region this sentence describes	Please provide the bounding box coordinate of the region this sentence describes	Identify and provide the bounding box coordinates that match the description given in this sentence Extract and provide the bounding box coordinates based on the region described in the sentence Please provide the bounding box coordinate of the region this sentence describes Find and provide the bounding box coordinates for the region mentioned in the sentence Provide the coordinates of the bounding box that correspond to the region described in the sentence Give the bounding box coordinates as described in the sentence Determine and provide the bounding box coordinates based on the description in the sentence Identify and provide the coordinates of the bounding box described in the sentence Provide the coordinates for the bounding box based on the region described in the sentence Extract and provide the coordinates for the bounding box described in the sentence Identify and give the coordinates of the bounding box as described by the sentence

Appendix

Table 16: The list of instruction templates for each task [ 7 ] .

Explore similar work

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
May 8, 2026Mengxin Qin, Xiang Zhang, Xi Wang +3Vision-Language Models Fine-Tuning
School of Electronic Engineering, Xidian University Xi’an 710071, China
Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning
Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMs Multimodal Instruction Tuning
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · ShanghaiTech University

May 18, 2026J/K move · Enter open · S save

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

Yang Liu, Toan Nguyen, Flora D. Salim

School of Computer Science and Engineering University of New South Wales Australia

Continual Learning for LLMs Vision-Language Models Mixture-of-Experts Language Models Continual Learning for VLMs Catastrophic Forgetting

Open paper AskPDF arXiv

Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) architectures offer an efficient path to scaling, existing LoRA-based MoE continual learning methods still face a fundamental trade-off: they either isolate experts too aggressively, limiting knowledge transfer across tasks, or allow task-specific updates to overwrite important existing parameters, leading to severe forgetting. To address this, we propose CP-MoE, a continual learning framework built around a transient expert that captures early task-specific updates and guides their integration into stable experts. CP-MoE introduces a consistency-preserving routing bias, which uses the transient expert to estimate representation similarity with stable experts and steer routing towards more compatible expert selection, and a transient expert-guided regularisation mechanism, which selectively protects important historical parameters during merging. Together, these components reduce parameter interference and forgetting while preserving cross-task knowledge transfer. We validate CP-MoE on both unimodal and multimodal continual learning benchmarks with LLM-based and VLM-based MoE models. On SuperNI benchmark, spanning diverse sequential language tasks, CP-MoE achieves state-of-the-art performance and stronger zero-shot transfer to unseen tasks. On VQA v2 dataset, it scales effectively to multimodal visual reasoning, consistently reduces forgetting, and outperforms strong MoE baselines.

Abstract

Explore similar work

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models

Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning