DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning

Xiwei Liu; Yulong Li; Feilong Tang; Imran Razzak

cs.LGMar 2, 2026

DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning

Authors: Xiwei Liu, Yulong Li, Feilong Tang, Imran Razzak

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Monash University

Abstract

Adapting Large Multimodal Models (LMMs) to real-world scenarios poses the dual challenges of learning from sequential data streams while handling frequent modality incompleteness, a task known as Continual Missing Modality Learning (CMML). However, existing works on CMML have predominantly relied on prompt tuning, a technique that struggles with this task due to cross-task interference between its learnable prompts in their shared embedding space. A naive application of Low-Rank Adaptation (LoRA) with modality-shared module will also suffer modality interference from competing gradients. To this end, we propose DeLo, the first framework to leverage a novel dual-decomposed low-rank expert architecture for CMML. Specifically, this architecture resolves modality interference through decomposed LoRA expert, dynamically composing LoRA update matrix with rank-one factors from disentangled modality-specific factor pools. Embedded within a task-partitioned framework that structurally prevents catastrophic forgetting, this expert system is supported by two key mechanisms: a Cross-Modal Guided Routing strategy to handle incomplete data and a Task-Key Memory for efficient, task-agnostic inference. Extensive experiments on established CMML benchmarks demonstrate that our method significantly outperforms state-of-the-art approaches. This highlights the value of a principled, architecturally-aware LoRA design for real-world multimodal challenges.

Figures & tables

Figure 1: Illustration of continual missing modality learning.

Method	Missing	Continual	Prompt	LoRA
Method	Modality	Learning	Tuning	LoRA
MAP ( Lee et al. 2023 )
MSP ( Jang et al. 2024 )
L2P ( Wang et al. 2022c )
DualPrompt ( Wang et al. 2022b )
InfLoRA ( Liang and Li 2024 )

Table 1: Comparison of related representative works by problem area and adaptation technique.

Figure 2: Overview of our proposed DeLo framework. The left panel illustrates the task-partitioned architecture for continual learning. To prevent catastrophic forgetting, only the expert modules and classifier for the current task $k$ are trainable, while the main backbone and all modules for previous tasks (1 to $k-1$ ) are frozen. The right panel details the dual decomposed LoRA expert for a given Task $k$ , which consists of Modality-Specific Factor Pools (Visual: $\mathcal{P}_{k}^{\mathrm{v}}=\{(a_{e}^{\mathrm{v}},b_{e}^{\mathrm{v}})\}_{e=1}^{E}$ and Textual: $\mathcal{P}_{k}^{\mathrm{t}}=\{(a_{e}^{\mathrm{t}},b_{e}^{\mathrm{t}})\}_{e=1}^{E}$ ). For each input, factors are dynamically selected from these pools to compose modality-specific weight adjustments $\Delta W^{\mathrm{v}}=\sum_{i=1}^{r}b_{i}^{\mathrm{v}}\otimes a_{i}^{\mathrm{v}}$ and $\Delta W^{\mathrm{t}}=\sum_{j=1}^{r}b_{j}^{\mathrm{t}}\otimes a_{j}^{\mathrm{t}}$ , whose sum forms the final update $\Delta W_{k}$ . The top of this panel shows our Cross-Modal Guided Routing for handling missing modalities, where the query from an available modality serves as a proxy for the missing one (e.g., $\hat{q}_{\mathrm{t}}:=q_{\mathrm{v}}$ for image-only input) and the Alignment Loss $\mathcal{L}_{\mathrm{align}}$ for complete data.

Figure 3: Comparison of LoRA parameterizations. (a) Conventional LoRA, where the weight adjustment matrix is represented as a product of two low-rank matrices (

BA

). (b) Decomposed LoRA, where the adjustment is expressed as a sum of

r

rank-one matrices, each formed by the outer product of a vector pair

\sum_{i=1}^{r}b_{i}\otimes a_{i}

$\eta$	#Image	#Text	MAP		MSP		L2P		DualPrompt		RebQ		DeLo
$\eta$	#Image	#Text	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )
$10\%$	$100\%$	$90\%$	20.66	82.50	21.45	80.03	34.09	5.80	59.56	3.38	68.67	9.50	77.19	10.21
	$90\%$	$100\%$	21.53	82.20	23.29	78.98	35.21	5.51	59.90	1.72	72.46	7.64	80.23	7.88
	$95\%$	$95\%$	22.84	80.13	22.12	79.35	34.00	6.55	58.18	6.32	71.06	8.22	78.12	10.32
$30\%$	$100\%$	$70\%$	16.57	83.77	21.37	79.95	30.40	6.53	51.86	6.46	62.06	10.76	74.08	9.69
	$70\%$	$100\%$	24.00	77.12	22.21	76.02	34.75	4.66	52.66	5.11	71.62	6.44	78.20	6.34

Table 2: Performance comparison on UPMC-Food101-CMML dataset . We report AP and FG based on classification accuracy. Missing Modality can occur in both training and testing. The best performance is in bold .

$\eta$	#Image	#Text	MAP		MSP		L2P		DualPrompt		RebQ		DeLo
$\eta$	#Image	#Text	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )
$10\%$	$100\%$	$90\%$	15.77	34.67	19.49	35.11	13.67	8.96	22.79	20.42	24.19	32.46	29.14	32.11
	$90\%$	$100\%$	16.63	33.30	19.01	34.89	13.54	8.26	24.82	24.09	27.35	28.88	34.77	30.36
	$95\%$	$95\%$	17.24	31.28	15.69	32.17	13.90	8.42	24.19	24.04	26.90	28.26	31.84	30.72
$30\%$	$100\%$	$70\%$	19.85	32.14	20.01	34.87	10.68	8.03	22.96	23.96	24.75	25.86	29.28	32.79
	$70\%$	$100\%$	17.19	31.09	18.31	32.45	10.98	7.27	20.98	19.67	26.68	27.89	34.56	32.38

Table 3: Performance comparison on the MM-IMDb-CMML dataset . We report AP and FG based on F1-Macro. Missing Modality can occur in both training and testing. The best performance is in bold .

Figure 4: (a) t-SNE visualization of the final representations learned by the Task-Key Memory for each of the 10 continual tasks in UPMC-Food101-CMML. (b) The distribution of cosine similarity between visual and textual query signals, evaluated on modality-complete data.

Method	AP ( $\uparrow$ )	FG ( $\downarrow$ )
DeLo ( $E=16,r=4$ )	67.42	9.41
DeLo-LoRA ( $r=4$ )	60.39	12.08
w/o Cross-Modal Guide	48.84	20.13
w/o Modality-Specific Factor Pool	56.52	13.76

Table 4: Component-wise analysis of the DeLo framework.

Factors	Rank	AP ( $\uparrow$ )	FG ( $\downarrow$ )	Trainable	Total
( $E$ )	( $r$ )	AP ( $\uparrow$ )	FG ( $\downarrow$ )	Parameters	Parameters
16	2	63.78	11.20	7.1M	182.8M
16	4	67.42	9.41	12.4M	235.9M
16	8	67.11	10.89	23.0M	342.1M
32	2	64.53	13.37	14.2M	253.6M
32	4	68.03	8.92	24.8M	359.8M

Table 5: Impact of factor pool size and selected rank on DeLo’s performance and parameter efficiency.

Figure 5: Comparison of selection frequency of each factor expert for (a) the vision pool and (b) the text pool.

$\mathcal{L}_{\mathrm{c}}$	$\mathcal{L}_{\mathrm{align}}$	$\mathcal{L}_{\mathrm{con}}$	AP ( $\uparrow$ )	FG ( $\downarrow$ )
✓	-	-	61.95	12.37
✓	-	✓	65.15	10.85
✓	✓	-	65.88	10.74
✓	✓	✓	67.42	9.41

Table 6: Impact of each loss component on performance.

Missing-Modality Learning

Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.

See all 170 papers

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning
May 10, 2026Xing Han, Shravan Chaudhari, Tanvi Ranade +2Cross-Modal Learning Healthcare
Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1
Residual-Guided Expert Specialization for Incomplete Multimodal Learning
Jun 29, 2026Seunghun Baek, Jihwan Park, Jaeyoon Sim +3Missing-Modality Learning Multimodal Robustness
Pohang University of Science and Technology (POSTECH), South Korea

May 10, 2026J/K move · Enter open · S save

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

Xing Han, Shravan Chaudhari, Tanvi Ranade, Rama Chellappa+1

Department of Computer Science Johns Hopkins University · Department of ECE Johns Hopkins University · Department of Computer Science Johns Hopkins University, Bayesian Health

Cross-Modal Learning Healthcare Sparse Mixture-of-Experts Multi-Task Learning Multimodal Learning Multimodal Understanding

Open paper AskPDF arXiv

Real-world model deployment across multiple domains requires multimodal models to operate under two complementary regimes: (1) multi-task pretraining, tasks are co-available at design time where related tasks could borrow representational strength from one another, (2) continual adaptation, in which new tasks emerge after deployment with previously unseen modality combinations. However, neither regime alone suffices: the pretraining task set is never exhaustive, while bypassing joint training forfeits the transfer gains and efficiency among co-trainable tasks. Sparse Mixture-of-Experts (MoE) is a natural fit for this dual requirement: sparse activation enables modular capacity expansion as new tasks arrive, while routing decouples modality-level computation from task-level composition. In this work, we propose a scalable MoE framework for multitask pretraining and continual learning across flexible modality combinations. The framework is designed to support training on multimodal tasks with diverse modality configurations by leveraging modality-specific routers that process tokens from each modality across tasks. Furthermore, it enables continual learning over sequential multimodal tasks within a fixed-capacity MoE by compressing accumulated expert knowledge into low-rank memory subspaces, while expanding only the lightweight routers. We validate the effectiveness of our method on multiple healthcare multimodal benchmarks. It demonstrates competitive multitask pretraining performance while alleviating catastrophic forgetting and improving parameter efficiency.

Abstract

Missing-Modality Learning

Explore similar work

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

MAny: Merge Anything for Multimodal Continual Instruction Tuning

Residual-Guided Expert Specialization for Incomplete Multimodal Learning