cs.CVNov 13, 2026

CoPre-MoE: Consolidating and Preserving MoE for Multimodal Continual Instruction Tuning

Authors: Tao Hu, Ling-Xiang Zhang, Da-Wei Zhou

Organizations: School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University

Abstract

Multimodal continual instruction tuning requires multimodal large language models to sequentially acquire new capabilities without catastrophic forgetting. Mixture-of-experts (MoE) architectures offer a promising way to mitigate task interference through expert specialization and combination. However, while existing MoE-based methods avoid overwriting shared parameters by allocating separate experts for each task, this progressive expansion incurs severe cross-task redundancy and linear parameter growth. Specifically, experts across tasks often learn overlapping functions alongside task-specific capabilities. For instance, ScienceQA and TextVQA both locate relevant image regions. To curb parameter redundancy without inducing forgetting, we propose Consolidating and Preserving MoE (CoPre-MoE). For each incoming task, CoPre-MoE first learns task-specific MoE to capture specialized capabilities. To eliminate redundancy and reduce parameter growth across tasks, we design a shared-and-private expert architecture. Specifically, we first evaluate functional overlap to selectively consolidate overlapping experts across tasks into a fixed-capacity generic expert pool. We then factorize the remaining task-specific experts into shared projection bases and compact task-specific cores within a private expert pool. To mitigate forgetting, we construct the router by pairing each expert with learnable retrieval keys, where historical keys remain frozen during continual updates while newly added keys are regularized with an orthogonality penalty. At inference, input-dependent routing dynamically combines both pools without test-time task labels. Extensive experiments demonstrate that CoPre-MoE achieves superior performance while substantially reducing parameter overhead.

Figures & tables

Explore similar work

CardsList
  1. CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning

    Aug 28, 2026Runze Liu, Naibin Gu, Mingxu Ai +4Continual Learning for VLMsMultimodal Instruction Tuning

  2. CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

    May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimContinual Learning for LLMsVision-Language Models

  3. Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuning

    Oct 9, 2026Xi Zhang, Boyu Diao, Zesen Chen +3Continual Learning for VLMsMultimodal Instruction Tuning