CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

Nan Zhou; Huiqun Wang; Yaoyan Zheng; Di Huang

cs.CVMar 22, 2026

CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models

Authors: Nan Zhou, Huiqun Wang, Yaoyan Zheng, Di Huang

Organizations: State Key Laboratory of Complex and Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University

Abstract

Multimodal large language models (MLLMs) achieve remarkable progress in cross-modal perception and reasoning, yet a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen? Despite the success of models such as LLaVA and Qwen-VL, inconsistent design choices and heterogeneous training setups hinder a unified understanding of visual fine-tuning (VFT) in MLLMs. Through a configuration-aligned benchmark, we find that existing VFT methods fail to consistently outperform the frozen baseline across multimodal tasks. Our analysis suggests that this instability arises from visual preference conflicts, where the context-agnostic nature of vision encoders induces divergent parameter updates under diverse multimodal context. To address this issue, we propose the Context-aware Visual Fine-tuning (CoVFT) framework, which explicitly incorporates multimodal context into visual adaptation. By integrating a Context Vector Extraction (CVE) and a Contextual Mixture-of-Experts (CoMoE) module, CoVFT decomposes conflicting optimization signals and enables stable, context-sensitive visual updates. Extensive experiments on 12 multimodal benchmarks demonstrate that CoVFT achieves state-of-the-art performance with superior stability. Notably, fine-tuning a 7B MLLM with CoVFT surpasses the average performance of its 13B counterpart, revealing substantial untapped potential in visual encoder optimization within MLLMs.

Figures & tables

Figure 1: (a) Performance comparison of distinct VFT methods on multimodal tasks, showing instability compared with Freeze baseline. (b) Illustration of the visual preference conflict phenomenon. Setup: we construct grounding and captioning tasks using identical images from the Visual Genome [ 30 ] dataset and train two independent MLLMs that differ only in their textual queries. (c) The L2 distance between the two vision encoders steadily increases during VFT, indicating growing divergence in learned, especially the deeper representations, which aligns with the classical finding that deeper layers tend to capture information with task-specific preferences [ 54 , 37 ] .

Figure 2: Illustration of the proposed context-aware visual fine-tuning (CoVFT) framework. Contextual vector extraction (CVE) generates a contextual vector $\bm{c}$ by aggregating multimodal cues through text-guided cross-attention. Contextual mixture-of-experts (CoMoE) injects $\bm{c}$ into the vision encoder via context-conditioned expert routing, enabling adaptive visual parameter updates.

	General				Knowledge & OCR			Vision-centric					Mean (%)
Method	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	G.	K.	V.	Avg.
LLaVA-1.5-7B †	1510.7	64.30	58.30	62.00	66.80	55.21	58.20	27.33	56.21	67.45	54.98	63.25	65.04	60.07	53.84	59.13
Freeze ∗	1473.7	67.87	60.30	63.07	69.31	55.76	58.53	28.00	56.73	63.73	49.61	60.50	66.23	61.20	51.71	58.93
Full fine-tuning	1510.2	67.44	59.88	63.92	67.67	56.51	59.70	27.33	55.82	65.71	51.97	60.00	66.69	61.29	52.17	59.29
VPT	1452.1	67.10	58.16	63.17	67.92	56.08	57.29	24.67	56.21	65.09	51.34	58.67	65.26	60.43	51.20	58.19
SVPT	1431.7	66.58	57.90	63.26	68.82	56.54	57.89	29.33	56.86	65.34	51.03	59.29	64.83	61.08	52.37	58.70

Table 1: Comparison of diverse VFT methods on 12 multimodal benchmarks under the unified LLaVA-1.5 [ 34 ] pre-training and instruction-tuning pipeline. The best and second-best VFT results are highlighted in bold and underline , respectively. “G.”, “K.”, “V.”, and “Avg.” denote the mean scores of the three task groups, and the overall average. Results marked with “

\dagger

” are obtained from LLaVA-1.5 or evaluating its official checkpoints, while “

*

” indicates our reproduction of LLaVA-1.5 Freeze baseline under the same configuration.

Ablation	General	Know.&OCR	Vision	Mean (%)
N/A	66.69	61.29	52.17	59.29
w.r.t. contextual vector
Image-only	66.60	61.69	53.17	59.77
Text-only	66.84	61.86	54.73	60.55
Concat $[\rm{I,T}]$	66.78	61.79	54.56	60.44
CVE	67.04	61.93	55.81	61.08

Table 2: Ablation study of the contextual vector design in CVE and the routing strategy in CoMoE under LLaVA-1.5-7B.

Figure 3: (a) PCA [ 42 ] visualization of contextual vectors extracted from the CVE module. A subset of 5,000 instruction samples is clustered via k -means [ 2 ] , showing clear semantic grouping aligned with distinct visual preference patterns. (b) Correlation between contextual similarity and inference similarity, computed as the cosine similarity of routing weights aggregated across CoMoE layers. The strong positive trend ( $r=0.76$ ) indicates that samples with similar contextual vectors yield similar expert activation patterns.

Figure 4: Analysis of data scalability. Following the two-stage LLaVA training pipeline, models are fine-tuned with varying proportions of the 665K multimodal instruction dataset during the second-stage instruction-tuning. We compare the performance of the baseline model with a frozen or full fine-tuned vision encoder and our CoVFT framework under different data scales.

MLLM	V. E. & LLM	VFT	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	Mean (%)
LLaVA-1.5	SigLIP-L/16	Freeze	1393.4	65.81	58.33	62.32	70.85	55.67	58.79	28.00	56.60	61.37	46.45	56.67	57.54
LLaVA-1.5	Vicuna-7B	CoVFT	1466.8	66.92	60.74	62.43	69.96	55.73	61.57	35.33	57.65	65.71	50.71	56.83	59.74
LLaVA-1.5	DINOv3-L/16	Freeze	1237.4	57.56	51.03	60.43	65.20	53.76	47.22	15.33	47.58	58.26	49.45	59.25	52.25
LLaVA-1.5	Vicuna-7B	CoVFT	1309.8	58.33	51.80	62.50	64.90	52.95	49.75	20.00	52.55	67.08	50.08	62.17	54.80
InternVL 2.0	InternViT-300M	Full ft.	1502.2	71.99	66.75	63.65	90.18	75.39	67.51	37.33	60.78	72.05	58.93	70.42	67.51
InternVL 2.0	Phi-3-mini	CoVFT	1527.0	75.95	69.33	64.47	94.25	77.46	71.16	44.00	62.35	72.80	60.35	72.83	70.11

Table 3: Generalization evaluation of CoVFT across different MLLM architectures on 12 multimodal benchmarks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Start	End	Num.	General	Know.&OCR	Vision	Mean (%)
	N/A		66.69	61.29	52.17	59.29
0	22	23	66.72	61.43	54.32	60.23
0	10	11	66.75	61.43	52.13	59.33
11	22	12	67.04	61.93	55.81	61.08
15	22	8	66.96	62.10	52.87	59.87
19	22	4	66.78	61.70	53.50	59.98

Appendix

Table 4: Ablation on the placement of CoMoE layers within the vision encoder. “Start” and “End” denote the layer range where CoMoE is inserted. “Num.” is the total number of CoMoE layers applied. The baseline (N/A) corresponds to full fine-tuning without CoMoE.

Num.	General	Know.&OCR	Vision	Mean (%)
2	66.24	61.63	55.16	60.47
4	67.04	61.93	55.81	61.08
8	66.12	61.73	56.03	60.82

Appendix

Table 5: Ablation on the number of experts in each CoMoE block.

# Experts	# Task types $\rightarrow$	3	6	9	12	15
Full ft.		45.49	47.66	49.01	51.79	52.17
2		48.09	49.55	51.09	53.14	55.16
4		47.36	48.71	52.27	54.72	55.81
8		46.94	48.35	52.01	54.50	56.03

Appendix

Table 6: Vision-centric task performance under different levels of task diversity and numbers of experts. Bold indicates the best configuration for each diversity level.

Figure 5: Cosine similarity between each gradient update and the dominant gradient direction during training under the standard instruction-tuning setting.

Method	G.	K.	V.	Mean (%)	Param. (B)
Freeze	66.23	61.20	51.71	58.93	6.76
LoRA ( $r$ =8)	65.93	60.86	52.45	59.04	6.76
LoRA ( $r$ =512)	66.09	60.90	52.67	59.20	6.86
CoVFT ( $N$ =4)	66.78	61.70	53.50	59.98	6.85
Full fine-tuning	66.69	61.29	52.17	59.29	7.06
CoVFT ( $N$ =12)	67.04	61.93	55.81	61.08	7.24

Appendix

Table 7: Comparison of diverse VFT methods under different trainable parameters.

N

denotes the number of CoMoE layers.

Method	Pre-training	Instruction-tuning	Total	Inference	Peak GPU memory
Freeze	2h 38m	4h 28m	7h 6m	77.04ms	41,352MB
Full ft.	2h 38m	5h 14m	7h 52m	77.04ms	44,036MB
CoVFT	2h 38m	5h 32m	8h 10m	88.17ms	49,974MB

Appendix

Table 8: Comparison of computational overhead across different VFT strategies. Training time is reported in hours and minutes; inference time is averaged per sample.

Figure 6: PCA [ 42 ] visualization of 5,000 contextual vectors extracted from the CVE module. Vectors are grouped using k -means [ 2 ] , and for each cluster, four nearest samples to the cluster centroid are shown to illustrate representative task patterns.

	General				Knowledge & OCR			Vision-centric					Mean (%)
Ablation	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	G.	K.	V.	Avg.
w.r.t. contextual vector
N/A	1510.2	67.44	59.88	63.92	67.67	56.51	59.70	27.33	55.82	65.71	51.97	60.00	66.69	61.29	52.17	59.29
Image-only	1492.2	68.56	61.17	62.04	68.96	56.80	59.32	26.00	56.99	66.71	55.13	61.00	66.60	61.69	53.17	59.77
Text-only	1505.6	69.07	60.22	62.77	68.72	57.35	59.52	30.66	56.99	68.82	57.35	59.83	66.84	61.86	54.73	60.55
Concat $[\rm I,T]$	1498.6	68.90	60.48	62.82	69.16	56.67	59.54	31.33	55.82	68.70	55.29	61.67	66.78	61.79	54.56	60.44

Appendix

Table 9: Per task results by various ablation settings in Table 2 of the main body.

	General				Knowledge & OCR			Vision-centric					Mean (%)
Method	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	G.	K.	V.	Avg.
100% data
Freeze	1473.7	67.87	60.30	63.07	69.31	55.76	58.53	28.00	56.73	63.73	49.61	60.50	66.23	61.20	51.71	58.93
Full fine-tuning	1510.2	67.44	59.88	63.92	67.67	56.51	59.70	27.33	55.82	65.71	51.97	60.00	66.69	61.29	52.17	59.29
CoVFT	1525.2	68.13	60.40	63.37	69.51	56.64	59.64	36.67	57.52	66.96	56.08	61.83	67.04	61.93	55.81	61.08
50% data

Appendix

Table 10: Per task results by various ablation settings in Figure 4 of the main body.

			General				Knowledge & OCR			Vision-centric					Mean (%)
Start	End	Num.	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	G.	K.	V.	Avg.
	N/A		1510.2	67.44	59.88	63.92	67.67	56.51	59.70	27.33	55.82	65.71	51.97	60.00	66.69	61.29	52.17	59.29
0	22	23	1508.4	68.08	60.55	62.84	67.92	56.93	59.45	30.67	55.69	67.20	55.29	62.75	66.72	61.43	54.32	60.23
0	10	11	1501.4	68.04	60.65	63.22	68.62	56.99	58.68	27.33	56.86	62.11	52.34	62.00	66.75	61.43	52.13	59.33
11	22	12	1525.2	68.13	60.40	63.37	69.51	56.64	59.64	36.67	57.52	66.96	56.08	61.83	67.04	61.93	55.81	61.08
15	22	8	1518.0	68.21	60.45	63.29	69.66	56.77	59.88	31.33	56.60	64.57	54.50	57.33	66.96	62.10	52.87	59.87

Appendix

Table 11: Per task results by various ablation settings in Table 4 of the supplementary material.

	General				Knowledge & OCR			Vision-centric					Mean (%)
Num.	MME ${}^{\text{P}}$	MMB ${}^{\text{en}}$	MMB ${}^{\text{cn}}$	GQA	SQA ${}^{\text{I}}$	AI2D	TextVQA	MMVP	RWQA	COCO	ADE	Omni3D	G.	K.	V.	Avg.
2	1478.6	67.61	60.91	62.51	69.06	56.25	59.59	32.00	58.04	68.57	55.77	61.42	66.24	61.63	55.16	60.47
4	1525.2	68.13	60.40	63.37	69.51	56.64	59.64	36.67	57.52	66.96	56.08	61.83	67.04	61.93	55.81	61.08
8	1476.2	67.61	60.48	62.57	68.62	56.99	59.58	30.67	56.73	70.68	58.14	63.92	66.12	61.73	56.03	60.82

Appendix

Table 12: Per task results by various ablation settings in Table 5 of the supplementary material.

VLM Adaptation

Vision-language model adaptation tailors pretrained image-text and multimodal models to new tasks and domains while retaining transferable capabilities.

See all 545 papers

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM Evaluation Multimodal Large Language Models
University of Copenhagen · University of Cambridge · 3ETH Zürich +1
Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
May 18, 2026Xinpeng Dong, Min Zhang, Kairong Han +3Object Hallucination in VLMs Multimodal Large Language Models
Zhejiang University · East China Normal University · Zhejiang University of Science and Technology
Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding
Feb 28, 2026Ying Liu, Yudong Han, Kean Shi +1Memory-Augmented VLMs Multimodal Large Language Models
Beijing Institute of Technology · Peking University · Yangtze Delta Region Academy of Beijing Institude of Technology, Jiaxing, China

Jul 28, 2026J/K move · Enter open · S save

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan+3

University of Copenhagen · University of Cambridge · 3ETH Zürich+1

VLM Evaluation Multimodal Large Language Models Multimodal Model Interpretability Multimodal Understanding Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Abstract

VLM Adaptation

Explore similar work

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models

Mema: Memory-Augmented Adapter for Enhanced Vision-Language Understanding