Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Josh Barua; Kyle Mahowald; William Rudman

cs.CLOct 14, 2026

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Authors: Josh Barua, Kyle Mahowald, William Rudman

Organizations: The University of Texas at Austin

Abstract

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

Figures & tables

Figure 1: Overview. (Left) In the conflicting VQA task, a question is paired with a text passage and an image that support different answers (

a_{T}\neq a_{V}

); the VLM arbitrates between them and outputs a preference. (Middle) We independently degrade textual evidence (text noise level

p

) and visual evidence (Gaussian blur

\sigma

); as the reliability of each modality shifts, the VLM’s preference transitions between the text- and vision-supported answers. (Right) A two-parameter Bayesian model over the unimodal Bayes factors (

e_{T}

e_{V}

) and the prior belief (

\pi_{0}

) accurately predicts the joint preference

s_{J}

of the VLM.

Figure 2: Modality preferences shift sigmoidally with reliability and vary across datasets and VLMs. Mean

P_{J}

(probability of the text answer) as evidence from each modality is degraded. Within each panel, the left half is text noise

p

with blur

\sigma{=}0

, the right half is blur

\sigma

with text noise

p{=}0

, and the clean input

(p{=}0,\sigma{=}0)

sits at the center. Left: per-dataset curves for Qwen3.5-4B. Right: per-model curves averaged over all datasets.

Figure 3: Linear cue combination predicts the transition between text- and vision-dominated behavior. Rows show DocVQA, VQAv2, and RealworldQA. Columns show empirical Qwen3.5-4B preferences (left) and model predictions (right), averaged over held-out examples under the OOD protocol. Solid and dashed lines mark the empirical ( $s_{J}=0$ ) and predicted ( $\hat{s}_{J}=0$ ) transition boundaries, respectively.

	Predictor	KL $\downarrow$	$R^{2}$ $\uparrow$
Qwen2.5	Best Single Probe	0.161	0.609
	Calibrated Naive Bayes	0.097	0.717
	Linear Cue Combination	0.057	0.852
	Bilinear Cue Combination	0.058	0.851
	Unconstrained Linear	0.058	0.852
Qwen3.5	Best Single Probe	0.240	0.452

Table 1: Goodness of fit of each predictor to VLM modality preferences under the OOD protocol.

R^{2}

is computed between the predicted

\hat{s}_{J}

and observed

s_{J}

; KL is the divergence between the observed and predicted answer distributions. Metrics are averaged equally across DocVQA, VQAv2, and RealworldQA. Predictors above the inner rule have at most two free parameters (

\beta_{T}

\beta_{V}

). Predictors below it add expressivity: a cross-modal interaction term (bilinear) or a free prior coefficient and intercept (unconstrained linear). Bold marks the best predictor above the rule.

	Fit Grid	Test Grid	$R^{2}$
Qwen2.5	relevance	relevance	0.850
	$\times$ occlusion	$\times$ occlusion	0.850
	text noise	relevance	0.852
	$\times$ blur	$\times$ occlusion	0.852
Qwen3.5	relevance	relevance	0.806
	$\times$ occlusion	$\times$ occlusion	0.806

Table 2: Linear cue combination and its weights generalize across reliability manipulations.

R^{2}

on joint log-odds, averaged equally across datasets. For each VLM, the first row refits weights on the new grid; the second tests generalization by transferring weights from text noise

\times

blur without refitting. Both use the OOD protocol and identical test examples on relevance

\times

occlusion.

Model	Dataset	Example	Cell	Clean
		$\beta_{T}/\beta_{V}$ [95% CI]		$\|\bar{e}_{T}\|/\|\bar{e}_{V}\|$
Qwen2.5-VL	DocVQA	$1.063_{[1.027,1.102]}$	$0.988_{[0.951,1.025]}$	1.36
	VQAv2	$1.449_{[1.375,1.514]}$	$1.273_{[1.209,1.336]}$	1.40
	RealworldQA	$1.436_{[1.311,1.519]}$	$1.092_{[0.991,1.187]}$	2.66
Qwen3.5	DocVQA	$0.661_{[0.632,0.699]}$	$0.651_{[0.619,0.683]}$	1.44
	VQAv2	$0.563_{[0.539,0.626]}$	$0.580_{[0.551,0.609]}$	2.03

Table 3: Evidence ratio and integration bias. Integration bias above one indicates greater weighting of text evidence in the fitted rule. An evidence ratio above one indicates a larger magnitude of mean text evidence. Example-level fits predict individual scores; cell-level fits predict cell means, and serve as a robustness check. Bracketed subscripts give pointwise 95% percentile intervals (Appendix C.4 ). The evidence ratio uses the absolute values of signed mean evidence on clean held-out inputs (

\sigma=0

p=0

), with absolute values taken after averaging.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Total	Train	Validation	Test
DocVQA	1,000	700	150	150
VQAv2	1,000	700	150	150
RealworldQA	765	535	114	116
Total	2,765	1,935	414	416

Appendix

Table 4: Underlying example counts and split sizes for the per-dataset predictor comparisons. Counts exclude repeated degradation variants. Under the ID protocol, every example contributes

36

cells; under the OOD protocol, training and validation examples contribute

11

boundary cells each, and test examples contribute

25

interior cells each.

Panel of Figure 4 — Figure 4: Representative conflicting-modality examples from each dataset. The text is generated so that the text-supported answer differs from the vision-supported answer ( $a_{T}\neq a_{V}$ ).

Figure 5: Pixel occlusion for the VQAv2 ski-brand example. Starting from the pixel-normalized image, we replace a fraction

r

of pixels with black using nested masks: every pixel masked at one level remains masked at all higher levels. At

r=0

the image is intact; at

r=1

it is entirely black.

Level	Passage
0 (original)	The skier is using skis with the brand name “Rossignol” clearly visible on the side.
1	The skier is using skis with the brand name Rossignol faintly visible on the side.
2	The skier is using gear showing partial lettering that looks like the start of Rossignol.
3	The athlete raced downhill equipped with well-known French winter sports gear from a traditional brand.
4	Someone mentioned that the skier might be testing some European manufacturer’s newly designed alpine skis.
5	The skier sped down the icy slope while expertly maneuvering between the bright red gates.

Appendix

Table 5: Textual relevance degradation for the VQAv2 ski-brand example. These are the original passage and the five generated variants used in the experiments, in their generation order. Support for

a_{T}=\textit{Rossignol}

shifts from a clearly visible brand name to faint and partial lettering, broader manufacturer cues, and finally a passage with no brand information.

Panel of Figure 6 — Figure 6: Full RealworldQA conversion prompt. The original image is supplied before this text in the same request. The placeholders {question} and {answer} are replaced by the source example’s question and answer. The resulting JSON supplies the rewritten question, visual answer, conflicting passage, and text-supported answer.

Panel of Figure 7 — Figure 7: Prompt for generating textual relevance degradations. Braced placeholders are filled for each example with the question, text-supported answer, original passage, and word-count bounds; line breaks are adjusted for display. For an original passage of $n$ whitespace-separated words, the bounds are $\max(1,n-d)$ and $n+d$ , where $d=\max(1,\operatorname{round}(0.1n))$ .

Probe	Template
Prior $\pi_{0}$	[Question][Answer]
Text $s_{T}$	[Text][Question][Answer]
Vision $s_{V}$	[Image][Question][Answer]
Joint $s_{J}$	[Image][Text][Question][Answer]

Appendix

Table 6: Prompt templates for the four probes. Bracketed blocks denote content, with native chat delimiters omitted. The answer is scored in the assistant turn; Gemma3 additionally uses the assistant prefixes described in Appendix B.3 .

Model	Dataset	$s_{T}{>}0$	$s_{V}{<}0$
Qwen2.5-VL-3B	DocVQA	0.954	0.914
	VQAv2	0.935	0.828
	RealworldQA	0.922	0.733
Qwen3.5-4B	DocVQA	0.954	0.897
	VQAv2	0.938	0.836
	RealworldQA	0.958	0.799

Appendix

Table 7: Probe-sign agreement on clean inputs (

\sigma{=}0,p{=}0

). We report the fraction of examples for which the text-only probe favors the text-supported answer (

s_{T}>0

) and the image-only probe favors the benchmark visual answer (

s_{V}<0

Model	Retention	Overall	Low	Mid	High
Qwen2.5-VL-3B	0.799	0.813	0.671	0.828	0.939
Qwen3.5-4B	0.750	0.820	0.686	0.854	0.921
Gemma3-4B	0.767	0.795	0.637	0.834	0.915
GLM-4.6V-Flash	0.781	0.753	0.628	0.813	0.818

Appendix

Table 8: Agreement between MCQ choice and

\operatorname{sign}(s_{J})

, pooled across datasets and the full

(\sigma,p)

grid. Each cell is presented with both letter assignments; Retention is the fraction of pairs where the VLM selects the same answer string at both positions. Low, Mid, and High split retained pairs by

|s_{J}|

tertiles.

Figure 8: Visualization of the two evaluation protocols on the $6{\times}6$ degradation grid. In-distribution: every $(\sigma,p)$ cell contributes both training and test examples; the split is across example IDs, depicted by the two-triangle shading of each cell. Out-of-distribution: training is restricted to the boundary $\{(\sigma,0)\}\cup\{(0,p)\}$ with unimodal degradation, and evaluation is on the held-out interior $\{(\sigma,p):\sigma>0,\,p>0\}$ with combined degradation.

		DocVQA		VQAv2		RealworldQA
Model	Predictor	KL $\downarrow$	$R^{2}$ $\uparrow$	KL $\downarrow$	$R^{2}$ $\uparrow$	KL $\downarrow$	$R^{2}$ $\uparrow$
Qwen2.5-VL-3B	Best Single Probe	0.091	0.718	0.311	0.393	0.081	0.716
	Calibrated Naive Bayes	0.025	0.904	0.194	0.519	0.073	0.728
	Linear Cue Combination	0.017	0.918	0.124	0.756	0.030	0.882
	Bilinear Cue Combination	0.017	0.919	0.124	0.756	0.032	0.876
	Unconstrained Linear	0.018	0.920	0.125	0.754	0.030	0.883

Appendix

Table 9: Per-dataset OOD results for all predictors in Table 1 . KL is the mean divergence from the observed to the predicted two-answer distribution;

R^{2}

compares predicted and observed joint log-odds. Bold marks the best of the three predictors above the inner rule.

Figure 9: Qwen2.5-VL-3B: empirical (top) and linear cue-combination predictions (bottom) across the three datasets, using OOD-fitted weights. All $36$ cells are displayed on held-out examples; weights are estimated only on training boundary cells and OOD metrics use the $25$ interior cells. Solid and dashed contours mark empirical and predicted $s_{J}=0$ boundaries, respectively.

Figure 10: Gemma3-4B: empirical (top) and linear cue-combination predictions (bottom) across the three datasets, using OOD-fitted weights. All $36$ cells are displayed on held-out examples; weights are estimated only on training boundary cells and OOD metrics use the $25$ interior cells. Solid and dashed contours mark empirical and predicted $s_{J}=0$ boundaries, respectively.

Figure 11: GLM-4.6V-Flash: empirical (top) and linear cue-combination predictions (bottom) across the three datasets, using OOD-fitted weights. All $36$ cells are displayed on held-out examples; weights are estimated only on training boundary cells and OOD metrics use the $25$ interior cells. Solid and dashed contours mark empirical and predicted $s_{J}=0$ boundaries, respectively.

Model	Dataset	Bilinear cue combination			Unconstrained linear
Model	Dataset	$\beta_{T}$	$\beta_{V}$	$\beta_{I}$	$\beta_{T}$	$\beta_{V}$	$\beta_{\pi}$	$\beta_{0}$
Qwen2.5-VL-3B	DocVQA	$0.836$	$0.772$	$0.0063$	$0.824$	$0.804$	$1.051$	$0.076$
	VQAv2	$0.854$	$0.590$	$-0.0001$	$0.847$	$0.632$	$1.104$	$0.185$
	RealworldQA	$0.834$	$0.623$	$-0.0187$	$0.838$	$0.624$	$1.123$	$0.120$
Qwen3.5-4B	DocVQA	$0.543$	$0.818$	$0.0009$	$0.527$	$0.804$	$0.650$	$0.130$
	VQAv2	$0.441$	$0.728$	$0.0109$	$0.400$	$0.782$	$0.661$	$0.217$

Appendix

Table 10: Fitted coefficients of the bilinear and unconstrained-linear predictors in Table 1 . These are the saved example-level OOD fits, estimated on the

11

training boundary cells with validation checkpoint selection. Bilinear cue combination predicts

\hat{s}_{J}=\beta_{T}e_{T}+\beta_{V}e_{V}+\beta_{I}e_{T}e_{V}+\pi_{0}

, fixing the prior coefficient to one and the intercept to zero. Unconstrained linear predicts

\hat{s}_{J}=\beta_{T}e_{T}+\beta_{V}e_{V}+\beta_{\pi}\pi_{0}+\beta_{0}

, with all four coefficients free. Here

e_{T}=s_{T}-\pi_{0}

and

e_{V}=s_{V}-\pi_{0}

after input clipping. To express the stored raw-probe regression in this evidence parameterization, we use

\beta_{T}=w_{T}

\beta_{V}=w_{V}

\beta_{\pi}=w_{\pi}+w_{T}+w_{V}

, and

\beta_{0}=w_{0}

. Gemma3-4B uses the same measured prior and answer-prefilled evidence probes as Table 12 .

		Frozen (text noise $\times$ blur)			Refit (relevance $\times$ occlusion)
Model	Dataset	$\beta_{T}$	$\beta_{V}$	$\beta_{T}/\beta_{V}$	$\beta_{T}$	$\beta_{V}$	$\beta_{T}/\beta_{V}$
Qwen2.5-VL-3B	DocVQA	0.831	0.782	1.063	0.817	0.803	1.018
	VQAv2	0.855	0.590	1.449	0.859	0.550	1.561
	RealworldQA	0.841	0.586	1.436	0.806	0.567	1.423
Qwen3.5-4B	DocVQA	0.542	0.820	0.661	0.619	0.892	0.694
	VQAv2	0.426	0.756	0.563	0.471	0.809	0.582

Appendix

Table 11: Frozen and refitted modality weights. Example-level linear cue-combination weights underlying Tables 2 and 18 . Frozen weights are fitted on the text noise

\times

blur boundary and transferred unchanged; refitted weights are estimated on the relevance

\times

occlusion boundary, using the same example partitions and fitting procedure. Ratios are computed before rounding. Across all

21

VLM–dataset pairs, the absolute relative ratio change,

|r_{\mathrm{refit}}/r_{\mathrm{frozen}}-1|

with

r=\beta_{T}/\beta_{V}

, has a median of

4.5\%

and a maximum below

10\%

Model	Dataset	Example-level		Cell-level [95% CI]		Mean evidence		Prior
Model	Dataset	$\beta_{T}$	$\beta_{V}$	$\beta_{T}$	$\beta_{V}$	$\overline{e}_{T}$	$\overline{e}_{V}$	$\overline{\pi}_{0}$	$\mathrm{SD}(\pi_{0})$
Qwen2.5-VL-3B	DocVQA	$0.831$	$0.782$	$0.973$ $[0.960,0.987]$	$0.985$ $[0.956,1.018]$	$3.739$	$-2.747$	$0.052$	$1.958$
	VQAv2	$0.855$	$0.590$	$1.068$ $[1.045,1.093]$	$0.840$ $[0.803,0.878]$	$4.780$	$-3.422$	$0.041$	$2.463$
	RealworldQA	$0.841$	$0.586$	$0.993$ $[0.965,1.019]$	$0.909$ $[0.846,0.985]$	$3.205$	$-1.203$	$0.044$	$1.820$
Qwen3.5-4B	DocVQA	$0.542$	$0.820$	$0.699$ $[0.679,0.721]$	$1.074$ $[1.041,1.109]$	$4.357$	$-3.029$	$0.168$	$1.020$
	VQAv2	$0.426$	$0.756$	$0.545$ $[0.524,0.568]$	$0.940$ $[0.908,0.978]$	$6.130$	$-3.012$	$-0.152$	$1.105$

Appendix

Table 12: Weights, evidence, and prior in linear cue combination. Example-level weights are the saved OOD fits. Cell-level weights fit the same nonnegative two-weight model to the

11

training boundary-cell means with equal weight per cell, fixed prior coefficient one, and no intercept. Brackets give pointwise 95% percentile intervals from

2{,}000

whole-example refit bootstraps (Appendix C.4 ). The evidence means and prior mean and standard deviation are computed on clean held-out test inputs (

\sigma=0

p=0

), matching the evidence ratios in Table 3 . Standard deviations use divisor

N

, the number of held-out examples. Input probes are clipped to

[-30,30]

before subtracting the prior. Gemma uses its measured knowledge-prefilled prior and answer-prefilled evidence probes.

Model	Predictor	KL $\downarrow$	$R^{2}$ $\uparrow$
Qwen2.5-VL-3B	Best Single Probe	0.242	0.494
	Calibrated Naive Bayes	0.121	0.695
	Linear Cue Combination	0.073	0.827
	Bilinear Cue Combination	0.074	0.826
	Unconstrained Linear	0.074	0.828
Qwen3.5-4B	Best Single Probe	0.364	0.192

Appendix

Table 13: ID replication of Table 1 , averaging the three dataset-level metrics equally. KL is the mean divergence from the observed to the predicted two-answer distribution;

R^{2}

compares predicted and observed joint log-odds. Bold marks the best of the three predictors above the inner rule.

Model	Predictor	KL $\downarrow$	$R^{2}$ $\uparrow$
Qwen2.5-VL-3B	Linear Cue Combination	0.073	0.827
Qwen2.5-VL-3B	Random Forest	0.077	0.820
Qwen3.5-4B	Linear Cue Combination	0.089	0.794
Qwen3.5-4B	Random Forest	0.082	0.809
Gemma3-4B	Linear Cue Combination	0.364	0.672
Gemma3-4B	Random Forest	0.421	0.674

Appendix

Table 14: ID comparison of random forest and linear cue combination, using the same protocol as Table 13 and averaging the three dataset-level metrics equally. KL is the mean divergence from the observed to the predicted two-answer distribution;

R^{2}

compares predicted and observed joint log-odds. Bold marks the better predictor within each VLM and metric.

Model	Predictor	KL $\downarrow$	$R^{2}$ $\uparrow$
Qwen2.5-VL-7B	Best Single Probe	0.231	0.591
	Calibrated Naive Bayes	0.116	0.732
	Linear Cue Combination	0.064	0.865
	Bilinear Cue Combination	0.064	0.866
	Unconstrained Linear	0.065	0.865
Qwen3.5-9B	Best Single Probe	0.094	0.272

Appendix

Table 15: OOD predictor comparison for the three larger VLMs, replicating Table 1 . Metrics average the three datasets equally. KL is the mean divergence from the observed to the predicted two-answer distribution;

R^{2}

compares predicted and observed joint log-odds. Bold marks the best of the three predictors above the inner rule.

Model	Dataset	Example-level		Cell-level [95% CI]
Model	Dataset	$\beta_{T}$	$\beta_{V}$	$\beta_{T}$	$\beta_{V}$
Qwen2.5-VL-7B	DocVQA	$0.801$	$0.824$	$0.929$ $[0.914,0.944]$	$0.996$ $[0.972,1.024]$
	VQAv2	$0.695$	$0.698$	$0.846$ $[0.820,0.873]$	$0.854$ $[0.823,0.887]$
	RealworldQA	$0.732$	$0.714$	$0.871$ $[0.837,0.901]$	$0.995$ $[0.945,1.055]$
Qwen3.5-9B	DocVQA	$0.613$	$0.897$	$0.725$ $[0.711,0.740]$	$1.085$ $[1.060,1.113]$
	VQAv2	$0.466$	$0.809$	$0.572$ $[0.548,0.595]$	$1.014$ $[0.977,1.054]$

Appendix

Table 16: Fitted cue weights for larger VLMs. Example-level weights are the saved OOD fits. Cell-level weights fit the same nonnegative two-weight model to the

11

training boundary-cell means with equal weight per cell, fixed prior coefficient one, and no intercept. Brackets give pointwise 95% percentile intervals from

2{,}000

whole-example refit bootstraps (Appendix C.4 ). Gemma uses its measured knowledge-prefilled prior and answer-prefilled evidence probes.

Model	Dataset	Example	Cell	Clean
		$\beta_{T}/\beta_{V}$ [95% CI]		$\|\bar{e}_{T}\|/\|\bar{e}_{V}\|$
Qwen2.5-VL-7B	DocVQA	$0.972\;[0.939,1.003]$	$0.933\;[0.901,0.964]$	1.20
	VQAv2	$0.995\;[0.952,1.040]$	$0.991\;[0.950,1.030]$	1.31
	RealworldQA	$1.025\;[0.965,1.087]$	$0.876\;[0.818,0.935]$	2.42
Qwen3.5-9B	DocVQA	$0.683\;[0.667,0.709]$	$0.668\;[0.646,0.690]$	1.46
	VQAv2	$0.576\;[0.552,0.630]$	$0.564\;[0.538,0.591]$	2.01

Appendix

Table 17: Integration bias and evidence ratio in larger VLMs. Ratios above one indicate greater text weighting or stronger text evidence. Brackets give pointwise 95% percentile intervals from

2{,}000

whole-example refit bootstraps (Appendix C.4 ). Example fits repeat validation checkpoint selection; cell fits recompute the

11

training boundary-cell means in each resample. Evidence ratios are the absolute values of signed mean evidence on clean held-out inputs (

\sigma=0

p=0

), with absolute values taken after averaging.

Model	Refit $R^{2}$	Transferred $R^{2}$
Qwen2.5-VL-7B	0.822	0.830
Qwen3.5-9B	0.759	0.757
Gemma3-12B	0.688	0.689

Appendix

Table 18: Weight transfer for the larger VLMs, replicating Table 2 . Refit estimates weights on the boundary of relevance

\times

pixel occlusion; Transferred freezes weights estimated on the boundary of text noise

\times

blur. Both are evaluated on the same held-out interior examples of the new grid. Values average the three dataset-level

R^{2}

scores equally.

Evaluation	VLM	Checkpoint identifier
Main	Qwen2.5-VL-3B	Qwen/Qwen2.5-VL-3B-Instruct
Main	Qwen3.5-4B	Qwen/Qwen3.5-4B
Main	Gemma3-4B	google/gemma-3-4b-it
Main	GLM-4.6V-Flash	zai-org/GLM-4.6V-Flash
Larger	Qwen2.5-VL-7B	Qwen/Qwen2.5-VL-7B-Instruct
Larger	Qwen3.5-9B	Qwen/Qwen3.5-9B

Appendix

Table 19: Checkpoint identifiers for the VLMs evaluated in this paper. Gemma3 checkpoints are loaded from local copies of the listed repositories.

Explore similar work

Sep 22, 2026

Same evidence, different judgments: Evidence noncommutative in vision/speech-text conflicts

For multimodal large language models, when images or speech conflict with accompanying text, measured text reliance can entangle modality preference with evidence position. Earlier studies of text bias often used a fixed evidence order or moved task instructions with the evidence, leaving the contribution of order unclear. In this paper, we use a paired comparison that keeps the instructions and evidence content fixed and swaps only the positions of the two sources to quantify this potential influence. Across vision and speech models, placing an image or recording after conflicting text consistently shifts answers toward its content. We also revisit previous studies and analyze why their experimental settings can lead to misleading conclusions. These findings reveal cross-modal evidence noncommutativity: the same evidence can lead to different judgments when its order changes, and placing perceptual evidence later can increase the model's reliance on its content.

Zhuoyun Li, Boxuan Wang, Xiaowei Huang +1

School of Computer Science and Informatics, University of Liverpool, United Kingdom

Multimodal Large Language Models VLM Evaluation Multimodal Model Evaluation

Ask PDF

Sep 28, 2026

When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models

Despite substantial progress across downstream applications, vision-language models (VLMs) remain susceptible to language bias, often prioritizing linguistic cues over visual evidence and consequently producing incorrect predictions. Prior studies have proposed various approaches to understanding and mitigating language bias in VLMs, yet their findings often conflict due to the difficulty of tracing how language bias propagates within black-box VLMs. Building on the word completion task, we trace how language bias propagates through VLM inference by (1) proposing a diagnostic framework that decomposes the inference process into four distinct yet interdependent stages to trace the propagation of language bias; and (2) examining how two key factors underlying language bias, i.e., linguistic priors and cross-modal coverage, evolve across these stages and ultimately give rise to incorrect predictions. The linguistic prior captures the strength of statistical bias induced by the language model component of a VLM and represents the origin of language bias, whereas cross-modal coverage measures the extent to which linguistic cues cover the visual content. By decomposing inference into four stages and characterizing the interplay between linguistic priors and cross-modal coverage across these stages, we propose a systematic framework for tracing the propagation of language bias throughout the inference process; and uncover the underlying mechanism of language bias by revealing the interplay between linguistic priors and cross-modal coverage.

Yizhou Fang, Siyue Chen, Zimo Qi +3

University of Waterloo · Independent Researcher · Johns Hopkins University +3

VLM Evaluation Vision-Language Models VLM Interpretability Linguistic Bias in Language Models

Ask PDF

Aug 17, 2026

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model's preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at https://github.com/Ro-netizen004/multimodal-arbitration-artifact.

Rodela Ghosh, Aviral Gupta, Guangjing Wang

University of South Florida

Vision-Language Models VLM Evaluation

Ask PDF