Continual Adaptation Across Modalities: Learning and Forgetting in Unified Multimodal Models

Xinrui Wang; Zhihao Chen; Jia Tang; Alexandra Gomez-Villa; Bartlomiej Twardowski; Joost van de Weijer; Songcan Chen

cs.LGOct 28, 2026

Continual Adaptation Across Modalities: Learning and Forgetting in Unified Multimodal Models

Authors: Xinrui Wang, Zhihao Chen, Jia Tang, Alexandra Gomez-Villa, Bartlomiej Twardowski, Joost van de Weijer, Songcan Chen

Organizations: Nanjing University of Aeronautics and Astronautics · Computer Vision Center, Barcelona · EPFL · Shanghai Artificial Intelligence Laboratory ·

Abstract

Unified multimodal models (UMMs), which can process and generate diverse modalities within a single framework, have recently emerged as a powerful paradigm. However, when these models undergo continual learning on multiple tasks across different modalities, their learning and forgetting dynamics remain poorly understood. In this paper, we systematically investigate how architectural design and adaptation method shape intra-modal learning and cross-modal interference between the generation and understanding sides. Specifically, we report three findings. First, the severity of cross-modal interference decreases with the degree of modality-specific parameter isolation. Second, a within-model asymmetry: the generation side of a UMM is markedly more fragile than its understanding side. Third, what post-training adds is the first to be forgotten: purely post-training capabilities such as personalized generation collapse by 84% to 90%, while pretraining capabilities remain stable. Together, these results clarify how parameter isolation and post-training fragility shape UMMs' learning and forgetting dynamics, offering a starting point for developing continually evolving UMMs.

Figures & tables

Figure 1: The four UMM architectures studied (Chameleon, OmniDiffusion, JanusPro and BAGEL), redrawn in simplified unified style and ordered by parameter coupling (high to low).

Panel of Figure 2 — Figure 2: Cross-modal interference in UMMs. Left: Inter-modal interference. Top : generation to understanding. Middle : understanding to generic generation. Bottom : understanding to personalized generation. Right: personalized concept generations before and after the seven sequential understanding tasks (rows: reference image, generation before and after understanding training).

Figure 3: Intra-modal performance matrices (Understanding). Lower-triangle score matrices across the 7 continual understanding tasks: each column shows the state after one more trained task, so the main diagonal reflects forward plasticity and entries below it reflect retention (backward transfer). The MMBench/MMMU rows track general understanding via zero-shot probes, and the BWT row summarizes forgetting on the trained tasks, averaging the final score minus the score right after that task’s training (the diagonal). The probe row reports each probe’s final score minus its zero-shot baseline, tracking general understanding.

Figure 4: Mean generation quality of the first personalized concept (Q-Align, Q-Quality, CLIP-T, and CLIP-I) as the remaining 18 DreamBooth concepts are learned in sequence, per model.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Learning rate	LoRA rank	LoRA $\alpha$	Epochs	LoRA target modules
Chameleon	1e-4	16	32	1	q/k/v/o
OmniDiffusion	2e-5	16	32	1	q/k/v/o
JanusPro 7B	2e-4	16	32	1	q/k/v/o + aligner
BAGEL	2e-5	32	64	1	q/v/o/gate/up/down (no k_proj)

Appendix

Table 1: Sequential understanding: seven tasks trained in a fixed order, each on its training split. Within each model, all seven tasks share one configuration.

Model	Method	Concepts	Token num	Learning rate	LoRA rank	LoRA $\alpha$	Steps
Chameleon	Dreambooth	19	–	3e-5	16	32	100
Chameleon	YoPrompt	19	16	1e-3	–	–	4000
OmniDiffusion	Dreambooth	19	–	2e-5	16	32	800
OmniDiffusion	YoPrompt	19	16	1e-3	–	–	6000
JanusPro 7B	Dreambooth	19	–	1e-5	16	32	1000
JanusPro 7B	YoPrompt	19	16	5e-3	–	–	4000

Appendix

Table 2: Personalized generation: continual concept-by-concept training on 6 to 10 images per concept [ 1 ] , with a fixed number of steps per concept. YoPrompt optimizes a 16-token soft prompt instead of LoRA (marked –).

Axis	Metric	Protocol
Understanding, VQA	Accuracy (%)	six VQA benchmarks
Understanding, caption	SPICE $\times 100$	vizwiz
Understanding, general	Accuracy (%)	MMBench, MMMU as evaluation points
Generation	CLIP-I, CLIP-T	identity and text alignment
Generation	Q-Align, Q-Quality	semantic consistency and visual quality

Appendix

Table 3: Evaluation protocols.

Model	vizwiz	Fin	Sci	Med	RS	AD	OCR	MMBench	MMMU	Avg (VQA)
Bagel	16.00	60.07	86.32	46.84	49.42	29.04	85.58	87.98	52.60	59.55
JanusPro 7B	9.31	68.02	56.11	25.03	31.80	16.35	49.37	84.82	32.18	41.11
Chameleon	7.47	57.55	9.41	16.70	35.04	24.69	10.40	37.63	24.89	25.63
OmniDiffusion	3.76	48.06	40.83	26.22	35.45	24.85	5.07	50.94	23.11	30.08

Appendix

Table 4: Understanding across all seven tasks and the general benchmarks.

Figure 5: Per-task mean of the stage-over-stage

\Delta

at and after each task’s own training stage.

Figure 6: Generic generation performance before vs. after understanding, averaged across all concepts. Evaluated on 100 complex-conditioned text-to-image prompts (generated by an LLM).

Figure 7: Generic generation before vs. after the seven understanding tasks. Each row shows one of the 10 representative prompts from the 100-prompt evaluation set, printed in full at the row head; columns are the four models, each with before- and after-understanding panels.

Panel of Figure 8 — Figure 8: Personalized generation quality, averaged across all 19 concepts.

Figure 9: DreamBooth personalized generation before vs. after the seven understanding tasks, per model and metric (0–1 scale). Chameleon and JanusPro average 3 concepts (pre to T6, no re-personalization); BAGEL and OmniDiffusion average 19 concepts.

Figure 10: YoPrompt personalized generation before vs. after the seven understanding tasks, per model and metric (0–1 scale), averaged over 19 concepts.

Figure 11: Per-concept DreamBooth generations before vs. after the seven understanding tasks, all four models (three concepts each, coupling order), reference image on top of each column.

Figure 12: Per-concept YoPrompt generations before vs. after the seven understanding tasks, all four models (three concepts each, coupling order), reference image on top of each column.

Figure 13: Sequential generation, visualized. Generated images of the first personalized concept at successive DreamBooth checkpoints: after the 1st concept (the target itself, bo for Chameleon, brody for JanusPro, emma for BAGEL and will for OmniDiffusion) and after the 2nd, 4th, 11th and 19th concepts of the sequence were learned. Rows are models, with the reference image on the far left of each row.

Unified Multimodal Models

Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.

See all 295 papers

FlexUMM: Flexible Representation Learning for Unified Multimodal Models
Oct 15, 2026Yinyi Luo, Haoyue Bai, Yi Ge +3Task Interference Unified Multimodal Models
Carnegie Mellon University · William & Mary · University of Maryland
Transferability Between Understanding and Generation in Unified Multimodal Models
Jul 5, 2026Jiwon Kang, Heeji Yoon, Jaewoo Jung +5Unified Multimodal Models Multimodal Generation
KAIST AI, South Korea · Blynx, South Korea · Trillionlabs, South Korea
Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models
Aug 18, 2026Zongyang Qiu, Yihan Wu, Kaixuan Fan +2Unified Multimodal Models Multimodal Generation
The Hong Kong University of Science and Technology (Guangzhou) · Columbia University · MMLab, CUHK +1

Oct 15, 2026J/K move · Enter open · S save

FlexUMM: Flexible Representation Learning for Unified Multimodal Models

Yinyi Luo, Haoyue Bai, Yi Ge, Nilton Cesar Rojas Vales+2

Carnegie Mellon University · William & Mary · University of Maryland

Task Interference Unified Multimodal Models Multimodal Learning Conditional Computation

Open paper AskPDF arXiv

Unified multimodal models (UMMs) can integrate understanding and generation within a single backbone. However, it is commonly known that these two capabilities can cause conflicting representations, leading to a central question: how representation and computation should be shared across functionalities? Existing UMMs often rely on a largely fixed structure, where different inputs and functionalities follow the same depth and parameter-sharing pattern. We argue that this assumption is too restrictive: more effective unification requires sharing the computation that supports cross-functional communication while specializing the capacity that becomes task- or input-specific. We introduce FlexUMM, a flexible UMM framework that dynamically decides when to preserve shared computation, when to route specialized capacity, and when to stop further processing. Evidence shows that this flexible allocation improves UMMs along two complementary axes. It improves accuracy by reducing harmful competition between functionalities, and it improves efficiency by avoiding unnecessary computation once an input representation is sufficiently mature. Beyond simple architectural unification from the pre-trained models, FlexUMM works as a model-agnostic, functional plug-in to make existing UMMs more unified, stronger, and efficient. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/flexumm.

Abstract

Unified Multimodal Models

Explore similar work

FlexUMM: Flexible Representation Learning for Unified Multimodal Models

Transferability Between Understanding and Generation in Unified Multimodal Models

Where a New Concept Must Enter: Entry Point Gates Cross-Task Usability in Unified Multimodal Models