A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training

Kaichen Li; Zhilin Zhu; Jianhao Huang; Zhengqin Lai; Baochen Xiong; Zibo Shao; Yaguang Song; Linhui Xiao

cs.CVAug 26, 2026

A Visual Dependence-Aware Framework for Multimodal Unsupervised Continual Post-Training

Authors: Kaichen Li, Zhilin Zhu, Jianhao Huang, Zhengqin Lai, Baochen Xiong, Zibo Shao, Yaguang Song, Linhui Xiao,

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Harbin Institute of Technology, Shenzhen, China

Abstract

In this paper, we explore a novel task of Multimodal Unsupervised Continual Post-Training (MU-CPT), enabling deployed MLLMs to continually evolve from streaming unlabeled data. Existing unsupervised post-training methods for MLLMs typically optimize target tokens uniformly, overlooking their heterogeneous visual dependence (VD). However, we reveal that token-level VD is crucial for MU-CPT. Specifically, its structural distortion serves as an indicator of cross-modal catastrophic forgetting, and its inherent heterogeneity acts as a compass to guide new-task learning. Leveraging this property, we propose a Visual Dependence-Aware (VDA) framework with two main components. First, Visually Constrained Optimal Transport (VC-OT) formulates the VD structural distortion of old-task VD during new-task learning as an optimal transport problem to mitigate cross-modal forgetting. By designing a region-aware ground cost and a dependence-stratified transport penalty, it prevents global shifts in visual focus while strictly prohibiting visual reliance from degenerating into language bias. Second, Visually Modulated Adaptation (VMA) exploits VD heterogeneity to emphasize visually grounded new-task learning, promoting new-task plasticity. Together, our method simultaneously maintains old-task stability and new-task plasticity during challenging MU-CPT. Extensive experiments under our MU-CPT setting validate the effectiveness of VDA.

Figures & tables

Figure 1: Comparison between (a) Multimodal Continual Instruction-Tuning and (b) Multimodal Unsupervised Continual Post-Training (MU-CPT), where labeled answers are completely unavailable during training.

Figure 2: (a) Text tokens in multimodal data exhibit heterogeneous levels of visual dependence , where red and grey denote high and low visual dependence, respectively. (b) Learning new tasks leads to structural distortion of visual dependence on old tasks.

Figure 3: Overview of the proposed Visual Dependence-Aware (VDA) framework. (a) Visually Constrained Optimal Transport (VC-OT) preserves old-task VD strength and structure through region-aware and dependence-stratified transport. (b) Visually Modulated Adaptation (VMA) reweights incoming-task tokens according to their VD to promote visually grounded learning. Together, they effectively balance stability and plasticity in MU-CPT.

Frozen Backbone
Method	TextVQA	SciVQA	StockQA	GQA	DriveLM	PMC-VQA	AvgAcc $\uparrow$
Qwen2.5-VL-7B	79.1	52.9	53.3	60.0	24.7	53.5	53.9
Unsupervised Post-training Methods
LSMI (EMNLP’23)	79.5	63.2	62.5	68.3	29.0	49.7	58.7
SeVa (ACM MM’24)	79.9	57.3	60.7	64.0	27.3	54.3	57.2
ScPO (ICML’25)	80.3	61.7	64.4	66.0	28.6	51.6	58.8

Table 1: Comparison with existing Unsupervised Post-Training methods and Continual learning methods.

Method	AvgAcc $\uparrow$	AvgLA $\uparrow$	AvgF $\downarrow$
SEEKR-MLLM	60.1	62.5	2.9
CL-MoE	57.9	59.5	1.8
SEFE	53.6	61.4	9.4
DGG	57.6	60.8	3.9
VDA (Ours)	62.5	63.0	0.6

Table 2: Fine-grained comparison of new-task plasticity and old-task stability with continual learning methods.

$\mathcal{L}_{\mathrm{VMA}}$	$\mathcal{L}_{\mathrm{strength}}$	$\mathcal{L}_{\mathrm{struct}}$	AvgAcc $\uparrow$	AvgLA $\uparrow$	AvgF $\downarrow$
			51.0	61.7	12.8
✓			56.9	63.8	8.2
	✓		58.0	61.5	4.2
✓	✓		60.1	62.9	3.3
	✓	✓	59.8	60.3	0.7
✓	✓	✓	62.5	63.0	0.6

Table 3: Ablation study of the components in VDA.

Figure 4: RRAR trajectories on TextVQA throughout continual-learning.

VD Protection	VD Strength Drift (%) $\downarrow$	VD Distribution Drift $\downarrow$	AvgAcc $\uparrow$	AvgF $\downarrow$
No Protection	35.94	0.1052	56.9	8.2
Token-Level L1	5.08	0.0187	59.1	0.4
Mass+KL	14.89	0.0428	60.3	3.4
VC-OT	14.27	0.0385	62.5	0.6

Table 4: Comparison of different VD preservation strategies. VD Strength Drift measures the relative change in total VD. VD Distribution Drift measures the JSD between VD distributions.

Figure 5: Hyperparameter Analysis.

Continual Learning for VLMs

Continual learning for vision-language models studies how to adapt them to sequentially arriving data and tasks without losing earlier capabilities.

See all 82 papers

Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
CART: Compatible Routing for Multimodal Continual Instruction Tuning
Oct 15, 2026Ling-Xiang Zhang, Tao Hu, Da-Wei ZhouContinual Learning for VLMs Multimodal Large Language Models
School of Artificial Intelligence, Nanjing University · State Key Laboratory of Novel Software Technology, Nanjing University
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1

Jul 29, 2026J/K move · Enter open · S save

Progressive Multimodal Alignment for Continual Instruction Tuning

Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong+1

Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University+2

Cross-Modal Alignment Multimodal Large Language Models Continual Learning for VLMs Multimodal Instruction Tuning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.

Abstract

Continual Learning for VLMs

Explore similar work

Progressive Multimodal Alignment for Continual Instruction Tuning

CART: Compatible Routing for Multimodal Continual Instruction Tuning

MAny: Merge Anything for Multimodal Continual Instruction Tuning