Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework

Jingyang Qiao; Zhizhong Zhang; Xin Tan; Jingyu Gong; Yanyun Qu; Yuan Xie

cs.LGFeb 20, 2026

Continual-NExT: A Unified Comprehension And Generation Continual Learning Framework

Authors: Jingyang Qiao, Zhizhong Zhang, Xin Tan, Jingyu Gong, Yanyun Qu, Yuan Xie

Organizations: East China Normal University · Shanghai Innovation Institute · Shanghai Artificial Intelligence Laboratory · Xiamen University

Abstract

Dual-to-Dual MLLMs refer to Multimodal Large Language Models, which can enable unified multimodal comprehension and generation through text and image modalities. Although exhibiting strong instantaneous learning and generalization capabilities, Dual-to-Dual MLLMs still remain deficient in lifelong evolution, significantly affecting continual adaptation to dynamic real-world scenarios. One of the challenges is that learning new tasks inevitably destroys the learned knowledge. Beyond traditional catastrophic forgetting, Dual-to-Dual MLLMs face other challenges, including hallucination, instruction unfollowing, and failures in cross-modal knowledge transfer. However, no standardized continual learning framework for Dual-to-Dual MLLMs has been established yet, leaving these challenges unexplored. Thus, in this paper, we establish Continual-NExT, a continual learning framework for Dual-to-Dual MLLMs with deliberately-architected evaluation metrics. To improve the continual learning capability of Dual-to-Dual MLLMs, we propose an efficient MAGE (Mixture and Aggregation of General LoRA and Expert LoRA) method to further facilitate knowledge transfer across modalities and mitigate forgetting. Extensive experiments demonstrate that MAGE outperforms other continual learning methods and achieves state-of-the-art performance.

Figures & tables

Figure 1: Continual-NExT: A framework for lifelong multimodal learning in Dual-to-Dual MLLMs. The left part shows its multimodal generation capability. The middle part illustrates its development across diverse training stages. The right part presents its supported downstream tasks.

Figure 2: Overview of Continual-NExT, including tasks, task types, sizes, and examples.

Figure 3: Parameter update patterns across heterogeneous tasks from global output view (left) to local input view (right). VQAv2->ImageNet refers to first training on VQAv2 (Task 1) and then training on ImageNet (Task 2). Pixel value represents the absolute discrepancy between the parameters learned from Task 1 and Task 2.

Figure 4: Results of different LoRA settings.

Figure 5: Overview of the MAGE. We insert four types of LoRA into the MLLM, Image/Text General LoRA (G-LoRA) and Image/Text Expert LoRA (E-LoRA). Then, we use the parameter-wise EMA strategy to update LoRA corresponding to the specific input and output modality of current task, while freezing that unrelated to it.

Method	Venue	Accuracy on Each Task						Overall Results
Method	Venue	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )	Forgetting( $\downarrow$ )	New.ACC( $\uparrow$ )
Zero-Shot	-	26.64	28.38	19.76	13.78	25.54	58.76	28.81	-	-
LoRA ( Hu et al., 2022 )	ICLR’22	59.68	65.19	31.18	41.22	66.60	91.27	42.23	20.36	59.19
LoRA ( Hu et al., 2022 )	ICLR’22	18.40	61.01	30.60	19.38	32.70	91.27	42.23	20.36	59.19
MoELoRA ( Chen et al., 2024 )	NIPS’24	62.10	82.16	31.46	21.52	70.00	91.54	43.90	19.08	59.80
MoELoRA ( Chen et al., 2024 )	NIPS’24	25.32	63.57	30.59	16.04	36.32	91.54	43.90	19.08	59.80

Table 1: Avg.ACC, Forgetting, and New.ACC performance comparisons. Accuracy on each task refers to the performance after training on the current(first row)/final(second row) task. Bold fonts represent the best performance, and underlined fonts represent the second-best performance. Upper-Bound denotes once training all the datasets.

Figure 6: Visualized examples for each method after UCL on the Continual-NExT framework.

Method	Forgetting Type Metrics
Method	Avg.IUF( $\downarrow$ )	Avg.HAL( $\downarrow$ )	Avg.OTH( $\downarrow$ )
LoRA	9.07	11.11	0.18
MoELoRA	8.47	10.53	0.08
MAGE	4.42	7.48	0.36

Table 2: Comparisons on Avg.IUF, Avg.HAL and Avg.OTH between LoRA, MoELoRA, and MAGE.

Method	LoRA				MAGE
Tra.Task	Flickr30k	OCRVQA	Flickr30k	OCRVQA	Flickr30k	OCRVQA	Flickr30k	OCRVQA
Evl.Task	VQAv2	VQAv2	Flickr30k	Flickr30k	VQAv2	VQAv2	Flickr30k	Flickr30k
Trans.Type	Text $\rightarrow$ Text		Text $\rightarrow$ Image		Text $\rightarrow$ Text		Text $\rightarrow$ Image
Accuracy	20.10	50.26	31.18	31.26	43.56	57.29	31.26	32.02

Table 3: Knowledge transfer comparisons between LoRA and MAGE in Continual-NExT.

Method	Accuracy on Each Task						Overall Results
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )	Forgetting( $\downarrow$ )	New.ACC( $\uparrow$ )
MoE	25.32	63.57	30.59	16.04	36.32	91.54	43.90	19.08	59.80
EW	47.28	60.93	31.25	18.46	31.30	91.29	46.75	15.66	59.81
EW+GA	48.54	60.59	30.97	20.76	33.92	91.25	47.67	14.90	60.09
MAGE	45.42	68.02	31.44	27.28	34.10	91.19	49.58	12.26	59.79

Table 4: Ablation study on different method components and their influence on each task’s performance. The best and second-best results are marked in bold and underlined , respectively.

Method	Modality		Overall Results
Method	Text	Image	Avg.ACC( $\uparrow$ )	Forgetting( $\downarrow$ )	New.ACC( $\uparrow$ )
Default	43.71	61.32	49.58	12.26	59.79
Reverse	46.07	56.45	49.53	12.17	59.67
Alphabet	45.18	57.31	49.22	12.35	59.51

Table 5: UCL performance under different task orders.

Figure 7: Training and evaluation case visualization in Continual-NExT.

Figure 8: Comparison radar chart of Accuracy on each task between PEMA and DEMA.

Panel of Figure 9 — Figure 9: Results of distinct methods on each task.

Panel of Figure 10 — Figure 10: Prompt used for text generation consistency evaluation.

Panel of Figure 11 — Figure 11: Prompt used for image generation consistency evaluation.

Panel of Figure 12 — Figure 12: Prompt used for image editing consistency evaluation.

Metrics	LoRA	MoELoRA	EWC	LAE	PGP	CIA	RegLoRA	Ours(MAGE)	Upper-Bound
Avg.ACC	42.23 $\pm$ 0.83	43.90 $\pm$ 0.47	47.00 $\pm$ 0.85	45.25 $\pm$ 0.61	47.70 $\pm$ 0.52	47.99 $\pm$ 0.18	47.39 $\pm$ 0.67	49.58 $\pm$ 0.58	53.31 $\pm$ 0.49
Forgetting	20.36 $\pm$ 0.31	19.08 $\pm$ 0.22	15.85 $\pm$ 0.19	16.29 $\pm$ 0.28	13.54 $\pm$ 0.37	14.89 $\pm$ 0.16	15.05 $\pm$ 0.49	12.26 $\pm$ 0.13	-
New.ACC	59.19 $\pm$ 0.73	59.80 $\pm$ 0.50	60.21 $\pm$ 0.75	58.82 $\pm$ 0.38	58.97 $\pm$ 0.76	60.40 $\pm$ 0.55	59.94 $\pm$ 0.42	59.79 $\pm$ 0.63	-

Table 6: Comparison of Avg. ACC, Forgetting, and New ACC between our method and the baselines. The accuracy for each task is measured after training on the final task. We report the mean and standard deviation of each metric across different methods as

\text{mean}\pm\text{std}

. Results are obtained from three runs with distinct random seeds.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
MAGE	62.10	-	-	-	-	-	62.10
	43.56	68.35	-	-	-	-	55.96
	41.25	78.37	31.26	-	-	-	50.29
	57.29	75.18	32.02	37.09	-	-	50.40
	55.84	70.56	31.75	34.10	68.74	-	52.20

Table 7: Detailed raw performance after training on each task of our proposed MAGE .

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
LoRA	59.68	-	-	-	-	-	59.68
	20.10	65.19	-	-	-	-	42.65
	18.94	78.80	31.18	-	-	-	42.97
	50.26	71.32	31.26	41.22	-	-	48.52
	36.75	66.43	30.94	29.28	66.60	-	46.00

Table 8: Detailed raw performance after training on each task of LoRA baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
MoELoRA	62.10	-	-	-	-	-	62.10
	33.83	82.16	-	-	-	-	58.00
	28.74	85.86	31.46	-	-	-	48.69
	56.37	76.95	31.99	21.52	-	-	46.71
	42.81	68.53	31.21	18.15	70.00	-	46.14

Table 9: Detailed raw performance after training on each task of MoELoRA baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
EWC	59.68	-	-	-	-	-	59.68
	41.94	73.29	-	-	-	-	57.62
	41.92	89.41	31.11	-	-	-	54.15
	53.46	82.48	32.21	37.50	-	-	51.41
	47.98	67.31	32.15	29.47	68.40	-	49.06

Table 10: Detailed raw performance after training on each task of EWC baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
LAE	59.68	-	-	-	-	-	59.68
	42.28	77.49	-	-	-	-	59.89
	40.50	88.81	31.18	-	-	-	53.50
	55.50	83.29	32.04	27.48	-	-	49.58
	42.36	79.53	31.87	26.28	65.72	-	49.15

Table 11: Detailed raw performance after training on each task of LAE baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
PGP	59.68	-	-	-	-	-	59.68
	43.11	78.20	-	-	-	-	60.66
	41.83	83.56	31.40	-	-	-	52.26
	52.98	81.59	31.88	19.96	-	-	46.60
	45.45	77.76	30.32	18.40	73.20	-	49.03

Table 12: Detailed raw performance after training on each task of PGP baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
CIA	59.68	-	-	-	-	-	59.68
	45.66	68.95	-	-	-	-	57.31
	40.98	79.12	31.08	-	-	-	50.39
	51.33	77.38	31.50	41.06	-	-	50.32
	42.79	72.27	30.89	35.29	70.28	-	50.30

Table 13: Detailed raw performance after training on each task of CIA baseline.

Method	Accuracy on Each Task						Avg.ACC( $\uparrow$ )
Method	VQAv2	ImageNet	Flickr30k	OCRVQA	RefCOCO	HQEdit	Avg.ACC( $\uparrow$ )
RegLoRA	59.68	-	-	-	-	-	59.68
	45.23	73.31	-	-	-	-	59.27
	40.41	80.47	31.29	-	-	-	50.72
	52.17	70.72	32.06	36.92	-	-	47.97
	45.43	65.61	31.39	30.68	66.82	-	47.99

Table 14: Detailed raw performance after training on each task of RegLoRA baseline.

Figure 13: Visualized cases after training on each task of LoRA Fine-Tune baseline.

Cross-Modal Representation Transfer

See all 23 papers

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
Mar 15, 2026He Li, Yuhui Zhang, Xiaohan Wang +2LLM Fine-Tuning Multimodal Large Language Models
Stanford University · Tsinghua University
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMs Vision-Language Models
School of Computer Science and Engineering University of New South Wales Australia
Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models
May 14, 2026Yuehao Liu, Shanyan Guan, Weijia Zhang +4Continual Learning for LLMs Continual Learning for VLMs
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University · vivo Mobile Communication Co., Ltd.

Mar 15, 2026J/K move · Enter open · S save

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

He Li, Yuhui Zhang, Xiaohan Wang, Kaifeng Lyu+1

Stanford University · Tsinghua University

LLM Fine-Tuning Multimodal Large Language Models Catastrophic Forgetting Continual Learning for VLMs

Open paper AskPDF arXiv

The paper demonstrate that simple adjustments of the fine-tuning recipes of multimodal large language models (MLLM) are sufficient to mitigate catastrophic forgetting. On visual question answering, we design a 2x2 experimental framework to assess model performance across in-distribution and out-of-distribution image and text inputs. Our results show that appropriate regularization, such as constraining the number of trainable parameters or adopting a low learning rate, effectively prevents forgetting when dealing with out-of-distribution images. However, we uncover a distinct form of forgetting in settings with in-distribution images and out-of-distribution text. We attribute this forgetting as task-specific overfitting and address this issue by introducing a data-hybrid training strategy that combines datasets and tasks. Finally, we demonstrate that this approach naturally extends to continual learning, outperforming existing methods with complex auxiliary mechanisms. In general, our findings challenge the prevailing assumptions by highlighting the inherent robustness of MLLMs and providing practical guidelines for adapting them while preserving their general capabilities.

Abstract

Cross-Modal Representation Transfer

Explore similar work

Fine-tuning MLLMs Without Forgetting Is Easier Than You Think

CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Models