Mitigating Task Objective Conflicts for Continual Learning in Unified Multimodal Models

Jiaxing Zhang; Hao Tang

cs.CVJun 4, 2025

Mitigating Task Objective Conflicts for Continual Learning in Unified Multimodal Models

Authors: Jiaxing Zhang, Hao Tang

Organizations: Sichuan University, Chengdu, China · Peking University, Beijing, China

Abstract

Recent advances in unified autoregressive (AR) models have enabled multimodal understanding (MMU) and text-to-image generation (T2I) within a single architecture. However, in continual learning settings, their divergent objectives can induce task interference, hindering knowledge preservation and adaptation to new tasks. In this work, we investigate Task Objective Conflict (TOC) through theoretical and empirical analyses, revealing conflicting optimization directions at the task level but complementary interactions at the token level. These findings suggest that continual adaptation requires mitigating task interference while preserving beneficial cross-task knowledge sharing. To this end, we propose UniDecouple, a selective decoupling framework based on a Task-Aware Mixture-of-Experts (TA-MoE) architecture. Specifically, UniDecouple retains shared attention layers to preserve common contextual dependencies while introducing task-specific feed-forward network (FFN) experts to mitigate optimization conflicts. Furthermore, hierarchical expert routing enables adaptive expert selection, while hybrid expert collaboration leverages shared experts to facilitate cross-task knowledge transfer. A two-stage training strategy further balances task specialization and coordinated adaptation. Extensive experiments and ablation studies demonstrate that UniDecouple improves both multimodal understanding and image generation performance under a comparable per-token activation budget, achieving a favorable balance between knowledge preservation and task adaptation in continual learning.

Figures & tables

Figure 1: Comparison of different multimodal large language model (MLLM) architectures. (a) Single Image Encoder: pixel and semantic features are jointly encoded into a unified representation (e.g., Chameleon, EMU3, LWM, VILA-U, TokenFlow). (b) Decoupled Image Encoder: pixel-level and semantic-level features are separately processed, then integrated into the MLLM (e.g., Janus, JanusPro). (c) Decoupled AR (Ours): UniDecouple introduces task-specific experts via a Mixture-of-Experts (MoE) to resolve conflicts between semantic-level understanding and pixel-level generation.

Figure 2: a) Overview of UniDecouple . “Und.” and “Gen.” denote MMU and T2I. b) Task-Aware Mixture-of-Experts Layer . Hierarchical Expert Routing consists of a task-aware router and a dynamic-assignment (DA) router, while Hybrid expert collaboration refers to integration of task-specific experts and shared experts.

Figure 3: Illustration of the proposed two-stage training framework. Stage 1 derives highly specialized task-specific experts by freezing the shared self-attention layers and optimizing only FFN parameters under the autoregressive cross-entropy objective. Stage 2 replaces conventional FFNs with the TA-MoE layer, incorporating the pre-trained experts and performing end-to-end LoRA-based fine-tuning. This design enables effective task decoupling while maintaining cross-task knowledge sharing and parameter-efficient adaptation.

Panel of Figure 4 — Figure 4: Direct validation of task-level gradient conflict between MMU and T2I. (a) Gradient cosine similarity during joint training; the solid curve and shaded region indicate the mean and $\pm 1$ standard deviation over independent runs. Negative values indicate gradient conflict. (b) First-order prediction $-\eta\langle g_{U},g_{G}\rangle$ versus the observed MMU loss change after a T2I gradient step.

Panel of Figure 5 — Figure 5: Cross-task interference between MMU and T2I. (a) Mean token-pair gradient cosine similarity across normalized prediction-position bins, using the same samples as the task-level analysis. Values range from $-1$ to $1$ ; negative and positive values indicate conflict and complementarity, respectively. (b) Autoregressive cross-entropy loss curves showing cross-task trade-offs. (c) Single-task versus joint-training performance: blue, red, and green denote MMU-only, T2I-only, and joint training, respectively. Red annotations indicate relative changes defined in Eq. ( 33 ). GenEval scores are scaled by $100$ ; higher is better on all benchmarks.

Attention	$\langle g_{U},g_{G}\rangle$	$\cos(g_{U},g_{G})$	$C_{-}$	$C_{+}$	$C_{-}-C_{+}$
Full	$-0.7124\pm 0.1847$	$-0.0719\pm 0.0187$	$2.1387$	$1.4263$	$0.7124$
Causal	$-1.2664\pm 0.2137$	$-0.1327\pm 0.0219$	$2.6843$	$1.4179$	$1.2664$

Table 1: Gradient conflict under full and causal attention. Statistics are averaged over MMU–T2I mini-batch pairs and independent runs. The task-level inner product and cosine similarity are reported as mean

\pm

standard deviation across runs.

C_{-}

and

C_{+}

are the negative and positive raw token-pair interaction masses.

Type	Model	Params	POPE↑	MME-P↑	MMB↑	SEED↑	GQA↑	MMMU↑	MM-Vet↑
Und. Only	LlaVA-v1.5-Phi-1.5 Xie et al. (2024)	1.3B	84.1	1128.0	-	-	56.5	30.7	-
	MobileVLM Chu et al. (2023)	1.4B	84.5	1196.2	53.2	-	56.1	-	-
	MobileVLM-V2 Chu et al. (2024)	1.4B	84.3	1302.8	57.7	-	59.3	-	-
	MobileVLM Chu et al. (2023)	2.7B	84.9	1288.9	59.6	-	59.0	-	-
	MobileVLM-V2 Chu et al. (2024)	2.7B	84.7	1440.5	63.2	-	61.1	-	-
	LlaVA-Phi Zhu et al. (2024a)	2.7B	85.0	1335.1	59.8	-	-	-	28.9

Table 2: Comparison with state-of-the-arts on MMU benchmarks. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.

Type	Method	Single Obj.	Two Obj.	Counting	Colors	Position	Color Attri.	Overall
Gen. Only	LlamaGen Sun et al. (2024)	0.71	0.34	0.21	0.58	0.07	0.04	0.32
	LDM Rombach et al. (2022)	0.92	0.29	0.23	0.70	0.02	0.05	0.37
	SDv1.5 Rombach et al. (2022)	0.97	0.38	0.35	0.76	0.04	0.06	0.43
	PixArt- $\alpha$ Chen et al. (2023a)	0.98	0.50	0.44	0.80	0.08	0.07	0.48
	SDv2.1 Rombach et al. (2022)	0.98	0.51	0.44	0.85	0.07	0.17	0.50
	DALL-E2 Ramesh et al. (2022)	0.94	0.66	0.49	0.77	0.10	0.19	0.52

Table 3: Evaluation of text-to-image generation ability on GenEval benchmark. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.

Method	Global	Entity	Attribute	Relation	Other	Overall
SDv1.5 Rombach et al. (2022)	74.63	74.23	75.39	73.49	67.81	63.18
PixArt- $\alpha$ Chen et al. (2023a)	74.97	79.32	78.60	82.57	76.96	71.11
SDXL Podell et al. (2023)	83.27	82.43	80.91	86.76	80.41	74.65
Hunyuan-DiT Li et al. (2024b)	84.59	80.59	88.01	74.36	86.41	78.87
PixArt- $\Sigma$ Chen et al. (2024a)	86.89	82.89	88.94	86.59	87.68	80.54
Emu3-Gen Wang et al. (2024c)	85.21	86.68	86.84	90.22	83.15	80.60

Table 4: Performances on DPG-Bench. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.

Figure 6: Qualitative comparison between UniDecouple and JanusPro on multimodal understanding and generation.

Figure 7: Visualization of the average expert load distribution based on a random sample of 100 instances from MMU.

Variant	Attn. split	FFN split	Routing	MMMU $\uparrow$	POPE $\uparrow$	MM-Vet $\uparrow$	GenEval $\uparrow$	DPG-Bench $\uparrow$
JanusPro	No	No	–	36.3	86.2	39.8	0.73	82.63
Attention-only separation	Yes	No	Fixed task	36.4	86.2	39.7	0.71	82.66
FFN-only separation	No	Yes	Fixed task	36.9	86.4	41.8	0.73	82.68
Attention + FFN separation	Yes	Yes	Fixed task	35.7	85.7	39.1	0.69	81.52
UniDecouple	No	Yes	Learned token	41.7	87.2	46.7	0.76	84.11

Table 5: Ablation of module-level separation. All variants except Janus-Pro are initialized from the same Janus-Pro checkpoint and use identical training data, objectives, update counts, and evaluation protocols.

Model	Task-Aware Router	Shared	Understanding		Generation
Model	+ Group Loss	Expert	MMMU	POPE	Gen.eval
A	✗	✗	36.8	85.4	0.70
B	✓	✗	37.6	85.9	0.73
C	✓	✓	41.7	87.2	0.76

Table 6: Ablation study on Task-Aware Router (incl. Group Loss) and Shared Expert.

Model	MoE	Training	Expert Init.	Epochs $\downarrow$	MMMU $\uparrow$	POPE $\uparrow$	GenEval $\uparrow$
D	✓	Single-Stage	Pure	94	36.5	85.3	0.68
E	✓	Two-Stage	Stage 1	76	41.7	87.2	0.76

Table 7: Ablation study of the proposed two-stage training strategy and Stage-1 expert initialization.

G:S	POPE				GenEval	MMMU	MM-Vet	DPG-Bench
G:S	Overall	Random	Popular	Adversarial	GenEval	MMMU	MM-Vet	DPG-Bench
$1{:}0$	85.9	86.8	85.1	85.8	0.73	37.6	42.1	82.77
$0{:}1$	86.7	86.9	88.0	85.2	0.74	37.4	41.4	82.35
$1{:}1$	87.1	87.5	87.8	86.0	0.74	38.5	43.6	83.44
$2{:}1$	87.2	88.0	87.7	85.9	0.76	41.7	46.7	84.11
$3{:}1$	86.9	87.6	87.5	85.6	0.73	39.2	44.0	82.89

Table 8: Extended evaluation of different group-specific-to-shared expert (G:S) ratios. Results are averaged over three independent runs. Bold indicates the best result in each column.

Routing strategy	Grouping accuracy	POPE	GenEval	MMMU	DPG-Bench
Standard Top-2 MoE	–	$84.6$	$0.63$	$35.7$	$81.72$
Hard routing w/ Shared Expert	$100\%$	$86.8$	$0.74$	$37.2$	$83.03$
Hard routing w/o Shared Expert	$100\%$	$86.4$	$0.73$	$36.9$	$82.68$
Ours: learnable token-level routing	$\sim 76\%$	$\mathbf{87.2}$	$\mathbf{0.76}$	$41.7$	$84.11$

Table 9: Comparison of different routing strategies. Oracle hard routing uses the ground-truth sample-level task identity, whereas the proposed router makes token-level decisions from the current hidden representations.

Model	$d_{\mathrm{ff}}$	Total	Activated	Active Experts	MMMU	GenEval	POPE
Janus-Pro	5632	1.500B	1.500B	Dense	36.3	0.73	86.2
UniDecouple-3B	2880	2.795B	1.520B	$1+1$	41.7	0.76	87.2
UniDecouple-5B	5632	4.820B	2.330B	$1+1$	43.0	0.77	87.9

Table 10: Model capacity and benchmark performance. Parameter counts are in billions. Both UniDecouple variants activate one task-specific expert and one shared expert per token.

Model	Matched adaptation	MoE	MMMU	POPE	GenEval
Janus-Pro (original)	No	No	$36.3$	$86.2$	$0.73$
Janus-Pro + same data/LoRA	Yes	No	$37.4$	$86.7$	$0.74$
UniDecouple	Yes	Yes	$\mathbf{41.7}$	$\mathbf{87.2}$	$\mathbf{0.76}$

Table 11: Data-matched comparison with Janus-Pro. “Same data/LoRA” denotes adaptation on ShareGPT4V

+

MidJourney using the same data sampling, training schedule, optimizer, and rank-16 LoRA configuration as UniDecouple.

Task Interference

See all 12 papers

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding
Apr 9, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Cross-Modal Representation Learning Unified Multimodal Models
HKUST · Tencent Hunyuan
Lance: Unified Multimodal Modeling by Multi-Task Synergy
May 18, 2026Fengyi Fu, Mengqi Huang, Shaojin Wu +10Unified Multimodal Models Multi-Task Learning
Intelligent Creation Lab, ByteDance
Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea

Apr 9, 2026J/K move · Enter open · S save

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

Xiangyue Liu, Zijian Zhang, Miles Yang, Zhao Zhong+2

HKUST · Tencent Hunyuan

Cross-Modal Representation Learning Unified Multimodal Models Multimodal Pretraining Multimodal Generation Mixture-of-Experts Models

Open paper AskPDF arXiv

Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.

Abstract

Task Interference

Explore similar work

Symbiotic-MoE: Unlocking the Synergy between Generation and Understanding

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Transferability Between Understanding and Generation in Unified Multimodal Models