Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Rodela Ghosh; Aviral Gupta; Guangjing Wang

cs.CLAug 17, 2026

Which Source Wins? Task-Dependent Reliance in Vision-Language Models

Authors: Rodela Ghosh, Aviral Gupta, Guangjing Wang

Organizations: University of South Florida

Abstract

Vision-language models (VLMs) combine images and text, but when the two conflict and one becomes harder to read, it is unclear how a model shifts its reliance between them. We study this modality reallocation with a controlled setup: we degrade either the image or the text across four levels of legibility while keeping the other clean, and track how the model's preference changes. We build conflicts from GSM8K and SVAMP by pairing the rendered image of one arithmetic problem with the text of another, so the two sources support different answers. We also introduce ChartQA-Conflict, a manually reviewed benchmark of 229 chart-report conflicts with matched chart and table-image representations. We evaluate six open-weight VLMs using both generated answers and a length-normalized conditional log-likelihood margin. On GSM8K and SVAMP, five of six models shift more strongly away from degraded text than from degraded images. On ChartQA-Conflict, all six likelihood-scored models exhibit the opposite pattern, shifting more strongly away from the degraded visual source. This reversal persists after calibrating for unimodal accuracy loss and after replacing charts with plain table images. Two frontier API models, GPT-5.6-Luna and Gemini-3.5-Flash, behaviorally replicate the ChartQA-Conflict reversal, with GPT-5.6-Luna also matching the arithmetic direction. These results show that modality reliance in VLMs is not fixed, but varies across tasks, evidence structures, models, and evaluation settings. The source code is available at https://github.com/Ro-netizen004/multimodal-arbitration-artifact.

Figures & tables

Figure 1: The VLM is prompted to solve a problem based on two independent information sources. Source A is an image, which renders one problem; Source B is the text of a different problem.

Panel of Figure 2 — Figure 2: Experimental logic. We pair the rendered image of one problem with the text of another so the two sources support different answers ( $a_{I}$ vs. $a_{T}$ ). A role-neutral prompt labels them Source A/B, counterbalanced across items. From the clean pair we degrade one source at a time across four legibility levels while the other stays clean, and score every trial by both the generated source choice and the length-normalized CLL preference margin.

Panel of Figure 3 — Figure 3: Legibility ladders for the same conflict item. Top (image arm): the rendered image (Source A) is degraded from clean (L0) to heavy blur $+$ noise (L5) with the text clean. Bottom (text arm): the conflicting text (Source B) is corrupted at the character level with the image clean. L0/L2/L4/L5 mark increasing corruption within each arm, matched by level index, not absolute severity.

Figure 4: Paired asymmetry $A=R_{T}-R_{I}$ per model and setting; whiskers are 95% paired-bootstrap CIs. $A>0$ indicates a stronger response to text/report degradation, $A<0$ to image/chart degradation. On GSM8K and SVAMP the asymmetry is positive for five of six models; on ChartQA-Conflict it reverses to negative for all six. Full per-model values are in Table 1 .

Benchmark	Model	$R_{I}$	$R_{T}$	$A$ (paired)	Paired-bootstrap 95% CI for $A$
GSM8K	Qwen2-VL-2B	$+0.063$	$+1.193$	$+1.107$	$[+1.031,+1.191]$
	Qwen2.5-VL-7B	$+0.855$	$+1.013$	$+0.059$	$[+0.000,+0.181]$
	Idefics3-8B	$+0.750$	$+4.844$	$+3.877$	$[+3.540,+4.250]$
	LLaVA-OneVision-7B	$+0.022$	$+1.982$	$+1.950$	$[+1.829,+2.099]$
	LLaVA-1.6-7B	$+0.006$	$+0.645$	$+0.616$	$[+0.531,+0.670]$
	Phi-3.5-Vision	$+0.064$	$+1.403$	$+1.310$	$[+1.184,+1.390]$

Table 1: Reliance on the clean source when the other is degraded (neutral prompt).

R_{I}

R_{T}

is the median reallocation toward the clean text/image when the image/text is degraded (chart/report on ChartQA-Conflict);

A

is the median within-item contrast

R_{T,i}-R_{I,i}

, with

A>0

(

A<0

) indicating stronger reallocation under text (image) degradation. As medians are not additive,

A

need not equal

R_{T}-R_{I}

. Intervals are paired-bootstrap 95% CIs (10,000 resamples;

n=229

for ChartQA-Conflict).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Revision
Qwen/Qwen2-VL-2B-Instruct	895c3a49bc3fa70a340399125c650a463535e71c
Qwen/Qwen2.5-VL-7B-Instruct	cc594898137f460bfe9f0759e9844b3ce807cfb5
HuggingFaceM4/Idefics3-8B-Llama3	fddb4ff79181e55a994674777e06cd5456ce3dc3
llava-hf/llava-onevision-qwen2-7b-ov-hf	0d50680527681998e456c7b78950205bedd8a068
llava-hf/llava-v1.6-mistral-7b-hf	2424fdd47412fccc66d91719126b420e9fbd7065
microsoft/Phi-3.5-vision-instruct	12b77fb40b63a2c73c68243d3f767aab688a1b2a

Appendix

Table 2: Hugging Face checkpoints and pinned commit revisions (InternVL2-8B is used for behavioral analysis only).

Model	$n$	$R_{I}$	$R_{T}$	$A$	95% CI
GPT-5.6-Luna	173	$0.925$	$0.000$	$-0.925$	$[-0.971,-0.873]$
Gemini-3.5-Flash	171	$0.836$	$0.064$	$-0.772$	$[-0.854,-0.684]$

Appendix

Table 3: Generated-answer contrasts on ChartQA-Conflict.

R_{I}

R_{T}

: within-item reallocation toward the clean report/chart when the chart/report is degraded;

A=R_{T}-R_{I}

, negative

A

indicating stronger reallocation under chart degradation. 95% CIs are paired-bootstrap (10,000 resamples); both paired permutation tests

p<.001

Model	Clean src.	L0	L2	L4	L5
GPT-5.6-Luna	Report	3.0	3.5	96.3	95.4
	Chart	97.0	98.5	97.5	97.4
Gemini-3.5-Flash	Report	14.3	14.4	43.3	93.6
	Chart	85.7	94.3	95.8	97.3

Appendix

Table 4: Frontier-model ChartQA-Conflict trajectories: behavioral preference (%), among attributable generated answers, for the source that remains clean. Report rows track report-following as the chart is degraded; Chart rows track chart-following as the report is degraded. Arms coincide at L0; the report-degradation arm reuses the image-arm L0 generations, so the L0 values are complementary.

Model	Image	Text
Idefics3-8B	4008/4656 (86.1%)	2033/2603 (78.1%)
Phi-3.5-Vision	1268/1487 (85.3%)	820/1099 (74.6%)
Qwen2-VL-2B	1404/1934 (72.6%)	1009/1721 (58.6%)
Qwen2.5-VL-7B	2436/3893 (62.6%)	2695/3915 (68.8%)
LLaVA-OneVision-7B	3447/4000 (86.2%)	2082/2772 (75.1%)
LLaVA-1.6-7B	1821/2331 (78.1%)	719/989 (72.7%)

Appendix

Table 5: Agreement between generated source choice and CLL-margin sign (role-neutral arithmetic runs). Cells give count agreeing / attributable observations, with the agreement percentage below. Model rows pool GSM8K and SVAMP over all four legibility levels; the arm-pooled row also pools across models.

Benchmark	Analysis	$\beta_{3}$	SE	95% CI	$p$
GSM8K	Item level, model-clustered	$+2.705$	0.906	$[+0.375,+5.034]$	.031
	Item level, cell-clustered	$+2.705$	0.828	$[+1.082,+4.327]$	.0011
	Item level, 1% winsorized, model-clustered	$+2.429$	0.666	$[+0.716,+4.143]$	.015
	Cell-median aggregation	$+1.779$	0.967	$[-0.191,+3.749]$	.075
SVAMP	Item level, model-clustered	$+3.273$	0.979	$[+0.756,+5.790]$	.020
	Item level, cell-clustered	$+3.273$	1.103	$[+1.111,+5.435]$	.0030

Appendix

Table 6: Task-accuracy-adjusted channel interactions under the role-neutral arithmetic prompt. Positive

\beta_{3}

means that reallocation per unit of lost unimodal accuracy is larger under text degradation. The item-level regression estimates a mean shift; the cell-median analysis is a coarser sensitivity analysis with a different estimand.

Benchmark	Model	Image slope $b_{I}$	Text slope $b_{T}$	$D=b_{T}-b_{I}$	Bootstrap 95% CI for $D$
GSM8K	Qwen2-VL-2B	$+0.131$	$+0.957$	$+0.827$	$[+0.748,+0.899]$
	Qwen2.5-VL-7B	$+1.872$	$+0.880$	$-0.993$	$[-1.149,-0.785]$
	Idefics3-8B	$+0.731$	$+4.100$	$+3.369$	$[+3.010,+3.603]$
	LLaVA-OV-7B	$+0.021$	$+1.648$	$+1.627$	$[+1.503,+1.722]$
	LLaVA-1.6-7B	$+0.006$	$+0.570$	$+0.564$	$[+0.500,+0.623]$
	Phi-3.5	$+0.064$	$+1.144$	$+1.079$	$[+0.989,+1.161]$

Appendix

Table 7: Per-model reallocation slopes on the rendered-text arithmetic benchmarks after calibration by proportional unimodal task-accuracy loss. Slopes are fitted through the origin over L0, L2, L4, and L5.

b_{I}

is movement toward clean text per unit of lost image-only accuracy, and

b_{T}

is movement toward the clean image per unit of lost text-only accuracy. Positive

D=b_{T}-b_{I}

indicates greater reallocation under text degradation. Confidence intervals condition on the observed accuracy losses and use 10,000 bootstrap resamples of the matched conflict items.

Model	Image slope $b_{I}$	Text slope $b_{T}$	$D=b_{T}-b_{I}$	Bootstrap 95% CI for $D$
Qwen2-VL-2B	$+2.897$	$+1.740$	$-1.157$	$[-1.355,-0.895]$
Qwen2.5-VL-7B	$+3.138$	$+2.003$	$-1.134$	$[-1.309,-0.784]$
Idefics3-8B	$+7.215$	$+4.901$	$-2.314$	$[-3.456,-1.379]$
LLaVA-OV-7B	$+2.637$	$+1.575$	$-1.062$	$[-1.330,-0.731]$
LLaVA-1.6-7B	$+1.891$	$+1.516$	$-0.375$	$[-0.555,-0.229]$
Phi-3.5-Vision	$+3.411$	$+2.810$	$-0.601$	$[-0.951,-0.216]$

Appendix

Table 8: Calibrated slopes on ChartQA-Conflict.

b_{I}

measures reallocation per unit of lost chart-only accuracy, and

b_{T}

measures reallocation per unit of lost report-only accuracy. Slopes are fitted through the origin over L0, L2, L4, and L5. Negative

D=b_{T}-b_{I}

indicates stronger reallocation per unit of lost chart accuracy. All six models exceed the prespecified

0.10

clean-accuracy floor and are calibrated; every

D

is negative with a 95% CI excluding zero. Intervals are paired-bootstrap 95% confidence intervals over matched items using 10,000 resamples.

Model	$A_{\mathrm{chart}}$	$A_{\mathrm{table}}$ [95% CI]	table $-$ chart (paired) [95% CI]	$p_{\mathrm{perm}}$
Qwen2-VL-2B	$-1.221$	$-0.970$ $[-1.217,-0.813]$	$+0.125$ $[-0.040,+0.369]$	$0.079$
Qwen2.5-VL-7B	$-1.362$	$-1.701$ $[-1.936,-1.523]$	$-0.335$ $[-0.481,-0.108]$	$<.001$
Idefics3-8B	$-2.175$	$-2.003$ $[-2.688,-1.572]$	$+0.404$ $[-0.085,+1.257]$	$0.107$
LLaVA-OneVision-7B	$-1.156$	$-0.904$ $[-1.084,-0.656]$	$+0.098$ $[-0.028,+0.342]$	$0.253$
LLaVA-1.6-7B	$-0.280$	$-0.386$ $[-0.677,-0.166]$	$-0.079$ $[-0.329,+0.046]$	$0.278$
Phi-3.5-Vision	$-0.963$	$-0.700$ $[-0.840,-0.370]$	$+0.355$ $[+0.098,+0.655]$	$0.016$

Appendix

Table 9: Chart-versus-table control on ChartQA-Conflict (

n=229

per model).

A_{\mathrm{chart}}

is the median asymmetry with the original chart (as in Table 1 );

A_{\mathrm{table}}

is the median asymmetry when the visual source is the official source table rendered as a plain image. The last column is the within-item paired difference (table

-

chart); because it is a within-item median, it need not equal

A_{\mathrm{table}}-A_{\mathrm{chart}}

from the two columns. All intervals are 95% paired-bootstrap CIs (10,000 resamples);

p_{\mathrm{perm}}

is a two-sided paired permutation test. Every

A_{\mathrm{table}}

interval excludes zero, so the reversal persists under the table representation. The chart-versus-table difference is not systematic: four of six intervals include zero, and the two significant changes (Qwen2.5-VL-7B, Phi-3.5-Vision) point in opposite directions.

GSM8K (neutral)
Model	$\alpha{=}0$	$0.5$	$1$
Qwen2-VL-2B	$+2.648$	$+1.723$	$+1.107$
Qwen2.5-VL-7B	$+0.109$	$+0.053$	$+0.060$
Idefics3-8B	$+4.250$	$+4.063$	$+3.877$
LLaVA-OneVision-7B	$+4.969$	$+3.183$	$+1.950$
LLaVA-1.6-7B	$+1.432$	$+0.928$	$+0.616$

Appendix

Table 10: Median arm asymmetry under alternative candidate-length normalizations (

\alpha\in\{0,0.5,1\}

) for role-neutral GSM8K, role-neutral SVAMP, and ChartQA-Conflict. Positive values indicate stronger reallocation under text degradation; negative values, stronger reallocation under image (chart) degradation. All entries are significant (Wilcoxon signed-rank,

p<0.05

). Within every cell the sign and significance are preserved across

\alpha

; the exponent affects only the estimated magnitude.

Figure 5: A ChartQA-Conflict item as given to the VLM. The question asks for California’s 2019 poverty rate. Source A is the original chart, whose 2019 value supports $a_{I}{=}11.8$ ; Source B is the evidence-bearing report from our dataset, perturbed to support $a_{T}{=}14.8$ . A reply of $11.8$ is chart-following, $14.8$ is report-following; Source A/B labels are counterbalanced across items. The answer-identification annotations shown below the sources are explanatory overlays and were not included in the model input. Chart from ChartQA ( Masry et al., 2022 ) .

Model	$n$	$R_{I}$	$R_{T}$	$A$	95% CI	$p_{W}$
Qwen2-VL-2B	113	$+.929$	$+.053$	$-.876$	$[-.956,-.788]$	$5.63\times 10^{-21}$
Qwen2.5-VL-7B	188	$+.957$	$+.037$	$-.920$	$[-.968,-.862]$	$1.10\times 10^{-36}$
Idefics3-8B	162	$+.722$	$+.247$	$-.475$	$[-.605,-.340]$	$7.98\times 10^{-10}$
LLaVA-OneVision-7B	93	$+.925$	$+.043$	$-.882$	$[-.968,-.785]$	$5.45\times 10^{-18}$
LLaVA-1.6-7B	73	$+.808$	$+.178$	$-.630$	$[-.795,-.452]$	$5.92\times 10^{-8}$
InternVL2-8B	105	$+.286$	$+.676$	$+.391$	$[+.219,+.562]$	$4.51\times 10^{-5}$

Appendix

Table 11: Generated-answer ChartQA-Conflict contrasts after excluding the item that failed the report-entailment audit. Values are complete-case mean changes in source-choice indicators, not CLL margins. Negative

A=R_{T}-R_{I}

indicates stronger reallocation under chart degradation. Intervals are paired bootstrap 95% CIs and

p_{W}

is the two-sided Wilcoxon signed-rank test.

Model	$A_{\mathrm{original}}$	$A_{\mathrm{neutral}}$	Neutral–original	95% CI	Wilcoxon $p$
Qwen2-VL-2B	$+0.863$	$+1.107$	$+0.190$	$[+0.125,+0.250]$	$9.02\times 10^{-16}$
Qwen2.5-VL-7B	$+0.813$	$+0.060$	$-0.375$	$[-0.475,-0.290]$	$1.90\times 10^{-27}$
Idefics3-8B	$+4.438$	$+3.877$	$-0.281$	$[-0.625,+0.000]$	$2.39\times 10^{-4}$
LLaVA-OneVision-7B	$+1.596$	$+1.950$	$+0.215$	$[+0.155,+0.259]$	$3.56\times 10^{-26}$
Phi-3.5-Vision	$+1.375$	$+1.310$	$+0.000$	$[-0.085,+0.041]$	$0.112$

Appendix

Table 12: Prompt-framing sensitivity on matched GSM8K items. Positive

A

indicates stronger reallocation under text degradation. The neutral-minus-original contrast is computed within item, so it need not equal the difference between the two displayed medians. Intervals are paired bootstrap 95% confidence intervals, and

p

is from a two-sided Wilcoxon signed-rank test.

Explore similar work

Jun 9, 2026

Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.

Pratham Singla, Shivank Garg, Vihan Singh +1

Lossfunk · Indian Institute of Technology Roorkee · Raeth AI

VLM Evaluation Vision-Language Models Vision-Language Grounding

Ask PDF

Oct 14, 2026

Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.

Josh Barua, Kyle Mahowald, William Rudman

The University of Texas at Austin

Vision-Language Models Modality Gap in VLMs

Ask PDF

Apr 10, 2026

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of

ρ=

0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.

Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2

Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.

Vision-Language Grounding Faithfulness in VLMs VLM Interpretability

Ask PDF