Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Li Ma; Yan Zhuang; Jiawen Deng; Fuji Ren

cs.CVOct 15, 2026

Seeing is Believing: Benchmarking and Mitigating the Interference of Language Priors on Visual Perception in MLLMs

Authors: Li Ma, Yan Zhuang, Jiawen Deng, Fuji Ren

Organizations: School of Computer Science and Engineering University of Electronic Science and Technology of China

Abstract

Language priors in MLLMs can override visual evidence, impairing visual perception. Existing benchmarks assume shared priors across models and rely solely on accuracy. However, cross-model prior agreement falls below 80%, and over 15% of items can be answered correctly without any image, causing accuracy to overestimate prior resistance. To address these limitations, we introduce ViSiP-Bench, a benchmark grounded in rare-but-real scenarios across six prior categories, with a model-aware dynamic prior induction protocol and three metrics that more precisely and continuously measure prior resistance than accuracy alone. Evaluating 24 models reveals that prior adherence is pervasive, does not improve with scale, and larger models tend to either strongly override their priors or firmly adhere to them. We therefore propose MAPS, which identifies each model's strongest priors and fine-tunes on factually grounded knowledge to soften excessive prior reliance without disrupting general capabilities.

Figures & tables

Figure 1: A representative failure case of language prior dependency in MLLMs.

Figure 2: Distribution of three sources of no-image correct responses across existing benchmarks.

Figure 3: Representative examples from each of the six prior categories in ViSiP-Bench. Green answers denote the correct visual answer, which conflicts with the expected answer under the text-only condition.

Benchmark	Venue	Size	Cate- gories	Image Source	Style Diversity	Prior Annotation	Decision-level Metric	Probability-level Metric
ROME	EMNLP’23	1,563	4	Counterfactual	✗	Fixed	Acc only	✗
HallusionBench	CVPR’24	527	2	Counterfactual	✗	Fixed	Acc only	✗
PvP	EMNLP’25	2,904	2	Counterfactual	✗	Fixed	Acc only	✗
VLind-Bench	NAACL’25	2,576	11	Counterfactual	✗	Fixed	Acc only	✗
ViLP	ICML’ 25	900	3	Counterfactual	✗	Fixed	Acc only	✗
ViSiP-Bench	Ours	4500	6	Rare-but-Real	✓	Dynamic	PAR + Filtered Acc	VIG

Table 1: Overview of benchmarks for language prior dependency. ✓ and ✗ denote presence and absence of the corresponding property, respectively.

Figure 4: Cross-model prior agreement matrix across six representative models.

Model	PAR $\downarrow$	Acc $\uparrow$	FC	$\mathbf{VIG_{avg}}$ $\uparrow$	$\mathbf{VIG_{med}}$ $\uparrow$
Open-source Models
InternVL3.5-2B	51.51	42.94	182	4.41	4.11
InternVL3.5-8B	47.17	44.64	156	7.13	7.48
InternVL3.5-14B	38.64	53.17	131	9.62	9.74
InternVL3.5-38B	30.04	63.01	102	10.12	10.31
InternVL3-2B	62.77	30.69	223	2.91	2.30

Table 2: Evaluation results on ViSiP-Bench. PAR, Filtered Accuracy (Acc), and Filtered Count (FC) are reported for all models;

\text{VIG}_{avg}

(average) and

\text{VIG}_{med}

(median) are reported for open-source models only as they require access to token-level probabilities.

\downarrow

denotes lower is better;

\uparrow

denotes higher is better. Best results are bolded , second-best results are underlined .

Panel of Figure 5 — Figure 5: Overall analysis of model performance on ViSiP-Bench. Larger models exhibit a VIG polarization effect (a), and category-level prior resistance remains consistent across models (b).

Figure 6: Overview of the MAPS pipeline.

Method	PAR $\downarrow$	Acc $\uparrow$	FC	$\mathbf{VIG_{avg}}$ $\uparrow$	$\mathbf{VIG_{med}}$ $\uparrow$
Base	42.67	51.29	147	10.64	10.82
Inference-time Methods
VFP	40.60	53.01	147	10.78	10.90
CoT	47.71	43.11	147	8.11	8.32
AS	42.11	52.08	147	10.88	11.22
VCD	39.15	55.48	147	12.11	13.08

Table 3: Comparison of inference-time and training-based methods on ViSiP-Bench evaluated on Qwen3-VL-8B-Instruct.

Method	Reasoning		General
	MMMU-Pro $\uparrow$	VisuLogic $\uparrow$	MMStar $\uparrow$	RealWorldQA $\uparrow$
Base	55.90	28.14	70.90	71.50
Post-hoc	41.88	21.50	56.18	53.08
NaPO	49.51	24.93	66.51	66.00
MAPS (Ours)	56.84	27.94	69.27	71.68

Table 4: Generalization results on reasoning and general vision benchmarks. All methods are evaluated on Qwen3-VL-8B-Instruct.

Threshold	PAR $\downarrow$	Acc $\uparrow$	FC	$\mathbf{VIG_{avg}}$ $\uparrow$	$\mathbf{VIG_{med}}$ $\uparrow$	MMMU-Pro $\uparrow$	MMStar $\uparrow$
$\alpha=0.50$	33.09	60.93	218	19.81	19.08	50.90	63.84
$\alpha=0.80$	34.24	60.10	181	19.41	19.90	54.17	66.81
$\alpha=0.90$	35.09	59.01	154	18.84	18.90	56.84	69.27
$\alpha=0.95$	38.41	54.09	150	17.77	18.13	56.10	71.02
$\alpha=1.00$	42.67	51.29	147	10.64	10.82	55.90	70.90

Table 5: Ablation study on the prior probing threshold

\alpha

. Results on both ViSiP-Bench and generalization benchmarks are reported.

\alpha=1.00

corresponds to the base model without any fine-tuning.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Model	Question	Reason for Categorization
Chance Success	Qwen3-VL-2B	When is the vernal equinox? (A. March 20 B. June 20 C. July 23 D. December 21)	Model assigns near-uniform probabilities: A: 28.3%, B: 24.1%, C: 26.8%, D: 20.8%, indicating no strong prior and random correct selection.
Low-quality Framing	—	What is unusual about the color of the rose in this image? (A. Red B. Black C. Pink D. White)	The phrase “unusual” implicitly hints at an atypical answer, guiding the model toward anomalous options.
Low-quality Framing	—	What color is the watermelon? (A. Red B. Green C. White D. Yellow)	The question does not specify flesh or rind, making the prior ambiguous and the correct answer uncertain without the image.
Anomalous Prior	Qwen2.5-VL-3B	What color is the dolphin? (A. Blue B. White C. Pink D. Grey)	Model assigns high probability to “Pink”, indicating it inherently holds the anomalous answer as its own prior.

Appendix

Table 6: Representative examples of the three sources of no-image correct responses.

Style	Generation Prompt
Photorealistic	A police officer in full uniform stands at the front of a classroom, sharing career experiences with attentive students seated at desks.
Anime	A color-blind boy proudly holds up his drawing of a figure with green skin, smiling with satisfaction, in a vibrant anime art style.
Oil painting	A young boy stares in disbelief at a perfectly square watermelon sitting on a wooden table, his mouth wide open in shock, rendered in the style of a classical oil painting.
Sci-fi	A boy uses his smartphone to crack open a walnut, pressing the screen edge against the shell with focused determination, in a futuristic sci-fi setting with neon lighting.

Appendix

Table 7: Generation prompts for each visual style in ViSiP-Bench.

Panel of Figure 7 — Figure 7: Representative examples of the four visual styles in ViSiP-Bench. Each image depicts a rare-but-real scenario designed to conflict with commonsense priors.

Model	Type	PAR $\downarrow$	Acc $\uparrow$	FC	$\mathbf{VIG_{avg}}$ $\uparrow$	$\mathbf{VIG_{med}}$ $\uparrow$
InternVL3.5-2B	base	51.51	42.94	182	4.41	4.11
InternVL3.5-2B	+MAPS	46.91	47.11	189	8.91	9.20
InternVL3.5-8B	base	47.17	44.64	156	7.13	7.48
InternVL3.5-8B	+MAPS	39.11	51.85	158	15.94	15.63
InternVL3-14B	base	48.40	41.33	161	6.21	6.33
InternVL3-14B	+MAPS	42.60	47.91	166	13.81	13.48

Appendix

Table 8: Cross-model results on ViSiP-Bench. MAPS consistently reduces PAR and increases Filtered Accuracy and VIG across all six models spanning three architecture families, while FC remains close to the base model, indicating that prior knowledge is not indiscriminately disrupted.

Model	Type	MMMU-Pro	VisuLogic	MMStar	RealWorldQA
InternVL3.5-2B	base	34.10	24.10	61.18	63.12
InternVL3.5-2B	+MAPS	33.91	24.15	62.05	62.49
InternVL3.5-8B	base	41.20	28.60	65.23	67.32
InternVL3.5-8B	+MAPS	41.13	28.71	64.79	67.51
InternVL3-14B	base	40.10	26.30	64.14	63.98
InternVL3-14B	+MAPS	40.23	26.41	64.87	63.48

Appendix

Table 9: Generalization results on reasoning (MMMU-Pro, VisuLogic) and general vision (MMStar, RealWorldQA) benchmarks across six models. MAPS preserves performance within a narrow margin for every model, with no consistent direction of degradation.

Model	Physical	Time & Space	Scientific	Customs & Culture	Psychology	Function
Open-source Models
InternVL3.5-2B	34.1	49.2	69.4	51.3	43.2	60.1
InternVL3.5-8B	30.2	44.1	63.2	47.1	39.4	55.2
InternVL3.5-14B	22.1	29.8	49.7	29.1	21.3	39.4
InternVL3.5-38B	16.3	21.4	38.9	21.4	19.7	21.1
InternVL3-2B	40.2	55.3	78.4	58.2	51.3	68.2

Appendix

Table 10: Per-category PAR results for all evaluated models. Scientific & Mathematic consistently exhibits the highest PAR values, while Physical Properties proves most amenable to visual override.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM Evaluation Multimodal Large Language Models
University of Copenhagen · University of Cambridge · 3ETH Zürich +1
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Jul 27, 2026Zichao Lin, Yifeng Xie, Bowen Qu +30Object Hallucination in VLMs Multimodal Large Language Models
Moonshot AI

Jul 28, 2026J/K move · Enter open · S save

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Jiaang Li, Chengzu Li, Zhaochong An, Yifei Yuan+3

University of Copenhagen · University of Cambridge · 3ETH Zürich+1

VLM Evaluation Multimodal Large Language Models Multimodal Model Interpretability Multimodal Understanding Visual Evidence Sensitivity in VLMs

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Abstract

Vision-Language Models

Explore similar work

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

BabyVision: Visual Reasoning Beyond Language

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models