Recent advances in unified autoregressive (AR) models have enabled multimodal understanding (MMU) and text-to-image generation (T2I) within a single architecture. However, in continual learning settings, their divergent objectives can induce task interference, hindering knowledge preservation and adaptation to new tasks. In this work, we investigate Task Objective Conflict (TOC) through theoretical and empirical analyses, revealing conflicting optimization directions at the task level but complementary interactions at the token level. These findings suggest that continual adaptation requires mitigating task interference while preserving beneficial cross-task knowledge sharing. To this end, we propose UniDecouple, a selective decoupling framework based on a Task-Aware Mixture-of-Experts (TA-MoE) architecture. Specifically, UniDecouple retains shared attention layers to preserve common contextual dependencies while introducing task-specific feed-forward network (FFN) experts to mitigate optimization conflicts. Furthermore, hierarchical expert routing enables adaptive expert selection, while hybrid expert collaboration leverages shared experts to facilitate cross-task knowledge transfer. A two-stage training strategy further balances task specialization and coordinated adaptation. Extensive experiments and ablation studies demonstrate that UniDecouple improves both multimodal understanding and image generation performance under a comparable per-token activation budget, achieving a favorable balance between knowledge preservation and task adaptation in continual learning.
Figures & tables
Figure 1: Comparison of different multimodal large language model (MLLM) architectures. (a) Single Image Encoder: pixel and semantic features are jointly encoded into a unified representation (e.g., Chameleon, EMU3, LWM, VILA-U, TokenFlow). (b) Decoupled Image Encoder: pixel-level and semantic-level features are separately processed, then integrated into the MLLM (e.g., Janus, JanusPro). (c) Decoupled AR (Ours): UniDecouple introduces task-specific experts via a Mixture-of-Experts (MoE) to resolve conflicts between semantic-level understanding and pixel-level generation.
Figure 2: a) Overview of UniDecouple . “Und.” and “Gen.” denote MMU and T2I. b) Task-Aware Mixture-of-Experts Layer . Hierarchical Expert Routing consists of a task-aware router and a dynamic-assignment (DA) router, while Hybrid expert collaboration refers to integration of task-specific experts and shared experts.
Figure 3: Illustration of the proposed two-stage training framework. Stage 1 derives highly specialized task-specific experts by freezing the shared self-attention layers and optimizing only FFN parameters under the autoregressive cross-entropy objective. Stage 2 replaces conventional FFNs with the TA-MoE layer, incorporating the pre-trained experts and performing end-to-end LoRA-based fine-tuning. This design enables effective task decoupling while maintaining cross-task knowledge sharing and parameter-efficient adaptation.
Figure 4: Direct validation of task-level gradient conflict between MMU and T2I. (a) Gradient cosine similarity during joint training; the solid curve and shaded region indicate the mean and ±1 standard deviation over independent runs. Negative values indicate gradient conflict. (b) First-order prediction −η⟨gU,gG⟩ versus the observed MMU loss change after a T2I gradient step.
Figure 5: Cross-task interference between MMU and T2I. (a) Mean token-pair gradient cosine similarity across normalized prediction-position bins, using the same samples as the task-level analysis. Values range from −1 to 1 ; negative and positive values indicate conflict and complementarity, respectively. (b) Autoregressive cross-entropy loss curves showing cross-task trade-offs. (c) Single-task versus joint-training performance: blue, red, and green denote MMU-only, T2I-only, and joint training, respectively. Red annotations indicate relative changes defined in Eq. ( 33 ). GenEval scores are scaled by 100 ; higher is better on all benchmarks.
Attention
⟨gU,gG⟩
cos(gU,gG)
C−
C+
C−−C+
Full
−0.7124±0.1847
−0.0719±0.0187
2.1387
1.4263
0.7124
Causal
−1.2664±0.2137
−0.1327±0.0219
2.6843
1.4179
1.2664
Table 1: Gradient conflict under full and causal attention. Statistics are averaged over MMU–T2I mini-batch pairs and independent runs. The task-level inner product and cosine similarity are reported as mean ± standard deviation across runs. C− and C+ are the negative and positive raw token-pair interaction masses.
Type
Model
Params
POPE↑
MME-P↑
MMB↑
SEED↑
GQA↑
MMMU↑
MM-Vet↑
Und. Only
LlaVA-v1.5-Phi-1.5 Xie et al. (2024)
1.3B
84.1
1128.0
-
-
56.5
30.7
-
MobileVLM Chu et al. (2023)
1.4B
84.5
1196.2
53.2
-
56.1
-
-
MobileVLM-V2 Chu et al. (2024)
1.4B
84.3
1302.8
57.7
-
59.3
-
-
MobileVLM Chu et al. (2023)
2.7B
84.9
1288.9
59.6
-
59.0
-
-
MobileVLM-V2 Chu et al. (2024)
2.7B
84.7
1440.5
63.2
-
61.1
-
-
LlaVA-Phi Zhu et al. (2024a)
2.7B
85.0
1335.1
59.8
-
-
-
28.9
Table 2: Comparison with state-of-the-arts on MMU benchmarks. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.
Type
Method
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overall
Gen. Only
LlamaGen Sun et al. (2024)
0.71
0.34
0.21
0.58
0.07
0.04
0.32
LDM Rombach et al. (2022)
0.92
0.29
0.23
0.70
0.02
0.05
0.37
SDv1.5 Rombach et al. (2022)
0.97
0.38
0.35
0.76
0.04
0.06
0.43
PixArt- α Chen et al. (2023a)
0.98
0.50
0.44
0.80
0.08
0.07
0.48
SDv2.1 Rombach et al. (2022)
0.98
0.51
0.44
0.85
0.07
0.17
0.50
DALL-E2 Ramesh et al. (2022)
0.94
0.66
0.49
0.77
0.10
0.19
0.52
Table 3: Evaluation of text-to-image generation ability on GenEval benchmark. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.
Method
Global
Entity
Attribute
Relation
Other
Overall
SDv1.5 Rombach et al. (2022)
74.63
74.23
75.39
73.49
67.81
63.18
PixArt- α Chen et al. (2023a)
74.97
79.32
78.60
82.57
76.96
71.11
SDXL Podell et al. (2023)
83.27
82.43
80.91
86.76
80.41
74.65
Hunyuan-DiT Li et al. (2024b)
84.59
80.59
88.01
74.36
86.41
78.87
PixArt- Σ Chen et al. (2024a)
86.89
82.89
88.94
86.59
87.68
80.54
Emu3-Gen Wang et al. (2024c)
85.21
86.68
86.84
90.22
83.15
80.60
Table 4: Performances on DPG-Bench. ”Und.” and ”Gen.” denote ”understanding” and ”generation”, respectively. Model using external pretrained diffusion model are marked with †.
Figure 6: Qualitative comparison between UniDecouple and JanusPro on multimodal understanding and generation.
Figure 7: Visualization of the average expert load distribution based on a random sample of 100 instances from MMU.
Variant
Attn. split
FFN split
Routing
MMMU ↑
POPE ↑
MM-Vet ↑
GenEval ↑
DPG-Bench ↑
JanusPro
No
No
–
36.3
86.2
39.8
0.73
82.63
Attention-only separation
Yes
No
Fixed task
36.4
86.2
39.7
0.71
82.66
FFN-only separation
No
Yes
Fixed task
36.9
86.4
41.8
0.73
82.68
Attention + FFN separation
Yes
Yes
Fixed task
35.7
85.7
39.1
0.69
81.52
UniDecouple
No
Yes
Learned token
41.7
87.2
46.7
0.76
84.11
Table 5: Ablation of module-level separation. All variants except Janus-Pro are initialized from the same Janus-Pro checkpoint and use identical training data, objectives, update counts, and evaluation protocols.
Model
Task-Aware Router
Shared
Understanding
Generation
+ Group Loss
Expert
MMMU
POPE
Gen.eval
A
✗
✗
36.8
85.4
0.70
B
✓
✗
37.6
85.9
0.73
C
✓
✓
41.7
87.2
0.76
Table 6: Ablation study on Task-Aware Router (incl. Group Loss) and Shared Expert.
Model
MoE
Training
Expert Init.
Epochs ↓
MMMU ↑
POPE ↑
GenEval ↑
D
✓
Single-Stage
Pure
94
36.5
85.3
0.68
E
✓
Two-Stage
Stage 1
76
41.7
87.2
0.76
Table 7: Ablation study of the proposed two-stage training strategy and Stage-1 expert initialization.
G:S
POPE
GenEval
MMMU
MM-Vet
DPG-Bench
Overall
Random
Popular
Adversarial
1:0
85.9
86.8
85.1
85.8
0.73
37.6
42.1
82.77
0:1
86.7
86.9
88.0
85.2
0.74
37.4
41.4
82.35
1:1
87.1
87.5
87.8
86.0
0.74
38.5
43.6
83.44
2:1
87.2
88.0
87.7
85.9
0.76
41.7
46.7
84.11
3:1
86.9
87.6
87.5
85.6
0.73
39.2
44.0
82.89
Table 8: Extended evaluation of different group-specific-to-shared expert (G:S) ratios. Results are averaged over three independent runs. Bold indicates the best result in each column.
Routing strategy
Grouping accuracy
POPE
GenEval
MMMU
DPG-Bench
Standard Top-2 MoE
–
84.6
0.63
35.7
81.72
Hard routing w/ Shared Expert
100%
86.8
0.74
37.2
83.03
Hard routing w/o Shared Expert
100%
86.4
0.73
36.9
82.68
Ours: learnable token-level routing
∼76%
87.2
0.76
41.7
84.11
Table 9: Comparison of different routing strategies. Oracle hard routing uses the ground-truth sample-level task identity, whereas the proposed router makes token-level decisions from the current hidden representations.
Model
dff
Total
Activated
Active Experts
MMMU
GenEval
POPE
Janus-Pro
5632
1.500B
1.500B
Dense
36.3
0.73
86.2
UniDecouple-3B
2880
2.795B
1.520B
1+1
41.7
0.76
87.2
UniDecouple-5B
5632
4.820B
2.330B
1+1
43.0
0.77
87.9
Table 10: Model capacity and benchmark performance. Parameter counts are in billions. Both UniDecouple variants activate one task-specific expert and one shared expert per token.
Model
Matched adaptation
MoE
MMMU
POPE
GenEval
Janus-Pro (original)
No
No
36.3
86.2
0.73
Janus-Pro + same data/LoRA
Yes
No
37.4
86.7
0.74
UniDecouple
Yes
Yes
41.7
87.2
0.76
Table 11: Data-matched comparison with Janus-Pro. “Same data/LoRA” denotes adaptation on ShareGPT4V + MidJourney using the same data sampling, training schedule, optimizer, and rank-16 LoRA configuration as UniDecouple.
Empowering Large Multimodal Models (LMMs) with image generation often leads to catastrophic forgetting in understanding tasks due to severe gradient conflicts. While existing paradigms like Mixture-of-Transformers (MoT) mitigate this conflict through structural isolation, they fundamentally sever cross-modal synergy and suffer from capacity fragmentation. In this work, we present Symbiotic-MoE, a unified pre-training framework that resolves task interference within a native multimodal Mixture-of-Experts (MoE) Transformers architecture with zero-parameter overhead. We first identify that standard MoE tuning leads to routing collapse, where generative gradients dominate expert utilization. To address this, we introduce Modality-Aware Expert Disentanglement, which partitions experts into task-specific groups while utilizing shared experts as a multimodal semantic bridge. Crucially, this design allows shared experts to absorb fine-grained visual semantics from generative tasks to enrich textual representations. To optimize this, we propose a Progressive Training Strategy featuring differential learning rates and early-stage gradient shielding. This mechanism not only shields pre-trained knowledge from early volatility but eventually transforms generative signals into constructive feedback for understanding. Extensive experiments demonstrate that Symbiotic-MoE achieves rapid generative convergence while unlocking cross-modal synergy, boosting inherent understanding with remarkable gains on MMLU and OCRBench.
We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.
Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate transferability in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.
Jiwon Kang, Heeji Yoon, Jaewoo Jung +5
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea