LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

Muyao Yuan; Muyan Jiao; Jiangyong Ying; Weizhan Zhang; Yuanhong Zhang; Lan Ma; Yuan Gao; Haipeng Du

cs.CVAug 27, 2026

LLaVAFlow: Preserving Latent Alignment Flow for Parameter-Efficient Multimodal Fine-Tuning

Authors: Muyao Yuan, Muyan Jiao, Jiangyong Ying, Weizhan Zhang, Yuanhong Zhang, Lan Ma, Yuan Gao, Haipeng Du

Organizations: MOEKLINNS, and School of Computer Science and Technology Xi’an Jiaotong University Xi’an, China · E-surfing Vision Technology Co., Ltd China Telecom Hangzhou, China · China Telecom Xi’an, China

Abstract

While Multimodal Large Language Models (MLLMs) exhibit strong generalization, visual instruction tuning for downstream tasks inevitably causes catastrophic forgetting, impairing overall generalization. While existing methods regulate weight updates to reduce forgetting, they overlook the fundamental cross-modal alignment in MLLMs. Based on prior work and our observations, we argue that cross-modal alignment is implicitly captured in the information-compression trajectory. To preserve the alignment flow embedded in the trajectory, we propose LLaVAFlow, an information-theoretic distillation framework. First, we compress the mutual information between the extracted relations and MLLM embeddings, encouraging a learnable module to produce a refined alignment flow that benefits downstream tasks. Second, we maximize the mutual information between the extracted alignment flows of the pretrained and fine-tuned MLLMs, enabling the transfer of compact alignment information. Extensive experiments show that LLaVAFlow is an effective plug-and-play framework that preserves alignment flow and enhances both downstream performance and generalization.

Figures & tables

Figure 1. Motivation. We map visual tokens to semantic words along the forward pass using Logit Lens ( Nostalgebraist, 2020 ) . Existing methods ( Liang et al., 2025 ) disrupt the alignment within the feature flow, leading to incorrect predictions. By preserving the alignment flow, we effectively mitigate forgetting.

Figure 2. Overview of LLaVAFlow. LLaVAFlow is a distillation framework that effectively transfers alignment flow from a pretrained MLLM to its fine-tuned counterpart from an information-theoretic perspective. The framework comprises an Alignment Flow Module (AFM) and an Information Bottleneck Distillation (IBD) objective that jointly regulate MLLM fine-tuning: the Flow Refinement Loss drives the AFM to capture refined alignment flow, while the Alignment Preservation Loss effectively transfers it to the student. The T-MLLM (teacher) and S-MLLM (student) share the same AFM parameters.

Methods	ScienceQA							COCO-Caption
Methods	OKVQA	OCRVQA	GQA	TextVQA	Source	Target	Avg	OKVQA	OCRVQA	GQA	TextVQA	Source	Target	Avg
Zero-shot	58.00	66.20	61.94	58.21	61.09	70.22	65.65	58.00	66.20	61.94	58.21	61.09	40.34	50.71
LoRA Rank = 32
LoRA	56.99	65.25	61.52	56.64	60.10	87.57	73.84	48.60	66.00	56.73	43.91	53.81	128.46	91.14
LoRA+Ours	56.94	65.65	61.19	57.07	60.21	89.46	74.84	52.18	67.25	59.33	51.20	57.49	130.04	93.76
DoRA	55.10	61.05	59.68	54.18	57.50	88.94	73.22	49.69	66.50	57.62	45.81	54.90	128.26	91.58

Table 1. Comparison with state-of-the-art methods on the visual question answering task ScienceQA and the image captioning task COCO-Caption. Source indicates the average performance across upstream datasets, Target reports the performance on the fine-tuned dataset, and Avg is the mean of Source and Target .

Panel of Figure 3 — Figure 3. Alignment trends across layers. (a) and (b) show the average cosine similarity between features at the original vision and text embedding positions within each layer on the target and source datasets, respectively. (c) and (d) show the corresponding normalized Euclidean distance. All fine-tuned models are fine-tuned on ScienceQA ( Lu et al., 2022 ) . Our model preserves the pre-trained modality alignment in the shallow layers (1–25), indicating retention of general cross-modal alignment, while exhibiting divergence in the deeper layers as they adapt to task-specific objectives.

Figure 4. Comparison of semantic trajectories across models, obtained via Logit Lens ( Nostalgebraist, 2020 ) . Existing methods, such as LoRASculpt, fail to preserve alignment within the feature flow, leading to incorrect predictions. In contrast, our method retains most of the trajectory of the pre-trained LLaVA, effectively mitigating catastrophic forgetting. The trajectory is derived from the image patch indicated by the red box. Full trajectories are provided in Appendix B.1.

$\mathcal{L}_{FR}$	$\mathcal{L}_{AP}$	LoRA Rank $=32$			LoRA Rank $=64$
$\mathcal{L}_{FR}$	$\mathcal{L}_{AP}$	Target	Source	Avg	Target	Source	Avg
Zero-shot		70.22	61.09	65.65	70.22	61.09	65.65
✗	✗	87.57	60.10	73.84	88.49	55.67	72.08
✗	✓	89.08	60.04	74.56	87.86	58.71	73.28
✓	✗	89.37	60.05	74.71	88.52	57.11	72.81
✓	✓	89.46	60.21	74.84	89.46	58.13	73.79

Table 2. Ablation study of Flow Refinement Loss

\mathcal{L}_{FR}

and Alignment Preservation Loss

\mathcal{L}_{AP}

. All experiments are conducted with LoRA on the ScienceQA dataset.

Method	LoRA Rank $=32$			LoRA Rank $=64$
Method	Target	Source	Avg	Target	Source	Avg
Zero-shot	70.22	61.09	65.65	70.22	61.09	65.65
Scale	88.92	59.97	74.45	87.97	57.18	72.58
Linear	88.73	59.34	74.03	87.55	34.99	61.27
Attn-2	89.39	59.70	74.54	87.34	52.02	69.68
Attn-3	88.23	60.12	74.18	86.68	38.61	62.64

Table 3. Comparison of different alignment flow modeling strategies. All experiments are conducted with LoRA on the ScienceQA dataset. Attn-

L

denotes stacking

L{-}1

standard attention layers followed by AFM as the final scoring layer.

Panel of Figure 5 — Figure 5. Hyperparameter analysis of $\lambda_{FR}$ and $\lambda_{AP}$ . All results are obtained with LLaVAFlow applied.

Explore similar work

Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
MAny: Merge Anything for Multimodal Continual Instruction Tuning
Apr 15, 2026Zijian Gao, Wangwang Jia, Xingxing Zhang +6Continual Learning for VLMs Multimodal Instruction Tuning
College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · National Key Laboratory of Parallel and Distributed Computing, National University of Defense Technology +1
Curvature-Guided Mixing for MLLM Adaptation
Jun 23, 2026Jinglong Yang, Jiaxuan He, Wenjian Huang +2VLM Adaptation Multimodal Large Language Models
Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology · Department of Computer Science, City University of Hong Kong

Jul 29, 2026J/K move · Enter open · S save

Progressive Multimodal Alignment for Continual Instruction Tuning

Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong+1

Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University+2

Cross-Modal Alignment Multimodal Large Language Models Continual Learning for VLMs Multimodal Instruction Tuning

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding. In Multimodal Continual Instruction Tuning (MCIT), however, shifting visual distributions and evolving instruction semantics cause this shared projector to drift, leading to projector-level forgetting, an issue largely overlooked by methods that focus primarily on the LLM backbone. We introduce Progressive Multimodal Alignment (PMA), a framework that enables the projector to adapt continually while preserving previously learned alignment. PMA detects multimodal distribution shifts via a lightweight representation descriptor and progressively expands projector experts only when needed. An expandable router integrates expert outputs based on multimodal features, while the original pretrained projector is retained as a stable alignment anchor. This progressive mechanism balances stability and plasticity with sub-linear parameter growth and serves as a method-agnostic add-on to existing MCIT approaches. Extensive experiments on two recent MCIT benchmarks demonstrate that mitigating projector-level forgetting yields consistent gains over prior state-of-the-art methods when combined with PMA. Moreover, PMA scales across diverse MLLM backbones, demonstrating robust and broadly applicable MCIT performance.

Abstract

Explore similar work

Progressive Multimodal Alignment for Continual Instruction Tuning

MAny: Merge Anything for Multimodal Continual Instruction Tuning

Curvature-Guided Mixing for MLLM Adaptation