Retrieval-augmented vision-language models (VLMs) answer from two sources of evidence that can each fail: the retrieved text can be wrong or noisy, and the image can be degraded or contradicted. We introduce GIB-200K, 200,000 image-question items built from COCO and Visual Genome, each expanded into controlled stress conditions: hard-negative retrieval noise (1-20 documents), visual ambiguity (Gaussian and motion blur, low-resource crops) and cross-modal conflict with counterfactual flip pairs. Instead of only measuring degradation, we ask how models combine the two modalities. From ideal Bayesian cue combination we derive a testable prediction that yields formal definitions of Knowledge Inertia (text over-weighted relative to the model's own image evidence) and Sensory Capture (the reverse), and we estimate the weights with a measurement-error-corrected regression whose recovery of known weights we verify. A counterfactual flip decomposition separates modality bias from concept priors, and a Grounding Illusion Score measures how much clean accuracy is not grounded in the image. We evaluate 4 open models on one frozen, seeded evaluation split of 2,000 GIB-200K items. All models are accurate on clean inputs (92.5%-94.2%) but adverse accuracy falls to 41.3%-59.9%. The two LLaVA models over-weight the text (Knowledge Inertia, w_T/w_I = 1.229 and 1.322), whereas InternVL3 favours the image (w_T/w_I = 0.680); between 34.8% and 71.4% of the clean answers that are correct are abandoned when the document lies. A training-free consistency check that compares the answer with and without retrieval trades coverage for accuracy under misleading documents, while falling back to the image fails when the image is heavily degraded. All per-instance outputs, code, and an audit that recomputes every reported open-model number are released.
Figures & tables
Axis
Condition
Levels evaluated (full benchmark)
clean
image + gold document
–
non-RAG baseline
image, no document
–
retrieval noise
gold + k hard negatives
k ∈ {1, 5, 20} ({1, 3, 5, 10, 20})
visual ambiguity
Gaussian blur, no document
σ∈ {2, 4, 8, 16} px ({1, 2, 4, 8, 16})
horizontal motion blur
{10, 40} px ({5, 10, 20, 40})
low-resource crop
{25, 5}% of object visible ({50, 25, 10, 5})
Table 1: Stress conditions per item. Levels evaluated in this paper (all fixed before any model was run); the benchmark also defines the full level set given in brackets.
Model
Acc clean
Acc adverse
ECE adv.
R
TI
GIS
internvl3
0.93 [0.92, 0.94]
0.60 [0.58, 0.61]
0.21 [0.20, 0.23]
0.51 [0.49, 0.53]
0.20 [0.16, 0.24]
0.35 [0.33, 0.37]
llava05b
0.93 [0.91, 0.94]
0.41 [0.40, 0.43]
0.20 [0.19, 0.21]
0.36 [0.34, 0.37]
2.71 [2.48, 2.95]
0.71 [0.69, 0.73]
llava16
0.94 [0.93, 0.95]
0.49 [0.47, 0.50]
0.30 [0.29, 0.32]
0.36 [0.35, 0.38]
3.72 [3.24, 4.29]
0.62 [0.60, 0.65]
qwen2vl
0.94 [0.93, 0.95]
0.55 [0.53, 0.56]
0.16 [0.14, 0.17]
0.49 [0.47, 0.51]
0.58 [0.52, 0.64]
0.48 [0.46, 0.50]
Table 2: Overall results on all items (95% bootstrap CIs, images as clusters). R = (acc adverse / acc clean) × (1 − ECE adverse); TI = Trust Index; GIS = Grounding Illusion Score.
Figure 1: Vulnerability heatmaps: accuracy drop (points) relative to the clean condition when the image is blurred (rows) and k hard-negative documents are added (columns).
Model
k=0
k=1
k=5
k=20
decay rate
retained AUC
super-additive drop (p)
internvl3
0.927
0.812
0.693
0.487
-0.56 [-0.59, -0.52]
0.77 [0.75, 0.78]
0.43 [0.41, 0.46] (0.0002)
llava05b
0.925
0.701
0.381
0.202
-0.85 [-0.90, -0.82]
0.54 [0.53, 0.56]
0.17 [0.15, 0.19] (0.0002)
llava16
0.939
0.759
0.544
0.281
-0.79 [-0.83, -0.75]
0.64 [0.62, 0.65]
0.24 [0.22, 0.26] (0.0002)
qwen2vl
0.942
0.800
0.584
0.366
-0.73 [-0.77, -0.69]
0.68 [0.67, 0.70]
0.33 [0.31, 0.35] (0.0002)
Table 3: Retrieval-noise dose–response (free-form accuracy with k hard negatives next to the gold document; decay = log-odds per doubling of 1 + k) and the blur × retrieval interaction (super-additive drop; sign-flip permutation p).
Figure 2: Accuracy without documents under Gaussian blur, motion blur and low-resource crops (95% bootstrap bands); triangles mark resolution thresholds.
Figure 3: Cue-combination weights with 95% CIs. The star is the ideal observer (1, 1); points above the diagonal over-weight text (Knowledge Inertia), below it the image (Sensory Capture).
Model
pairs
text-consistent
image-consistent
concept prior
internvl3
816
0.15 [0.13, 0.18]
0.52 [0.48, 0.55]
0.33 [0.30, 0.36]
llava05b
816
0.56 [0.53, 0.60]
0.16 [0.13, 0.18]
0.28 [0.25, 0.31]
llava16
816
0.41 [0.38, 0.44]
0.19 [0.16, 0.21]
0.41 [0.37, 0.44]
qwen2vl
816
0.18 [0.16, 0.21]
0.47 [0.43, 0.50]
0.35 [0.32, 0.38]
Table 5: Counterfactual flip decomposition (share of flip pairs).
Figure 4: Visual share of attention per layer for the same images with a correct vs. a misleading document (solid), and for conflict items split by whether the answer followed the text (dotted).
Figure 5: Eight real failure cases (two per stress condition; green = correct, red = wrong under that stress). Retrieval noise and conflict: the models copy a wrong claim from a document; blur and crop: the image no longer determines the answer. The report’s gallery lists, per condition, the first four items (by id) that at least one model answered correctly on the clean image but not under stress; we show two per condition, preferring those that fool most models (not a random sample).
Model
condition
base acc
Δ image fallback
Δ confidence
abstain: sel. acc / coverage
internvl3
conflict
0.54 [0.52, 0.56]
0.19 [0.17, 0.21]
0.11 [0.10, 0.13]
0.82 / 0.62
internvl3
ret k20
0.49 [0.47, 0.51]
0.24 [0.22, 0.26]
0.18 [0.16, 0.20]
0.85 / 0.53
llava05b
conflict
0.24 [0.22, 0.25]
0.44 [0.42, 0.47]
0.22 [0.20, 0.24]
0.66 / 0.35
llava05b
ret k20
0.20 [0.18, 0.22]
0.48 [0.46, 0.50]
0.35 [0.33, 0.37]
0.66 / 0.28
llava16
conflict
0.32 [0.30, 0.34]
0.43 [0.41, 0.45]
0.18 [0.16, 0.19]
0.81 / 0.39
llava16
ret k20
0.28 [0.26, 0.30]
0.47 [0.44, 0.49]
0.42 [0.40, 0.44]
0.79 / 0.34
Table 6: Training-free consistency gate: change in accuracy when the answer with documents disagrees with the image-only answer ( Δ vs. no gate; 95% CI).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
ADS, all layers
ADS, late layers
AUROC (low share → follows text)
n follow text / image
internvl3
-0.012 [-0.013, -0.011]
-0.038 [-0.040, -0.036]
0.69
699 / 1301
llava05b
0.005 [0.004, 0.005]
0.010 [0.009, 0.012]
0.77
1405 / 595
llava16
-0.006 [-0.007, -0.006]
-0.011 [-0.013, -0.010]
0.71
1244 / 756
qwen2vl
-0.013 [-0.014, -0.012]
-0.039 [-0.041, -0.036]
0.77
948 / 1052
Appendix
Table 7: Attention shift, clean → conflict (paired per item; negative = more visual attention under conflict) and AUROC of a low layer-averaged visual share (conflict condition) for predicting that the answer follows the text.
Model
errors
Knowl. Inertia
Sens. Capture
Reas. Collapse
Perceptual
internvl3
15039
0.47
0.01
0.25
0.27
llava05b
18087
0.54
0.01
0.15
0.30
llava16
16396
0.55
0.01
0.18
0.27
qwen2vl
15786
0.53
0.00
0.19
0.27
Appendix
Table 8: Failure taxonomy: share of wrong free-form answers per class (all stress conditions).
Model
condition
base acc
Δ image fallback
Δ confidence
abstain: sel. acc / coverage
internvl3
grid g16 k0
0.97 [0.96, 0.98]
-0.53 [-0.55, -0.50]
-0.03 [-0.04, -0.02]
0.96 / 0.46
llava05b
grid g16 k0
0.95 [0.94, 0.96]
-0.51 [-0.53, -0.48]
-0.09 [-0.11, -0.08]
0.92 / 0.47
llava16
grid g16 k0
0.95 [0.94, 0.96]
-0.48 [-0.50, -0.46]
-0.03 [-0.04, -0.02]
0.93 / 0.50
qwen2vl
grid g16 k0
0.97 [0.96, 0.98]
-0.47 [-0.49, -0.45]
-0.03 [-0.03, -0.02]
0.97 / 0.51
Appendix
Table 9: Training-free consistency gate: change in accuracy when the answer with documents disagrees with the image-only answer ( Δ vs. no gate; 95% CI). grid_g16_k0 = correct document, heavily blurred image.
Model
Δ acc clean
Δ acc adverse
Δ follows text
internvl3
0.01 [-0.04, 0.08]
0.06 [-0.02, 0.13]
-0.03 [-0.13, 0.08]
llava05b
0.01 [-0.05, 0.07]
0.01 [-0.06, 0.07]
-0.02 [-0.12, 0.08]
llava16
0.01 [-0.04, 0.07]
0.05 [-0.03, 0.12]
-0.00 [-0.11, 0.10]
qwen2vl
0.00 [-0.04, 0.06]
0.04 [-0.04, 0.12]
-0.01 [-0.12, 0.10]
Appendix
Table 10: Contamination check: difference between items from COCO train2017 (likely seen in instruction tuning) and val2017.
Vision-language models must reconcile conflicting text and images. Prior work has characterized the resulting behavior in terms of modality biases--e.g., a "blind faith in text"--but why preferences vary across models, domains, and input conditions remains unclear. We develop a predictive account from a Bayesian perspective, positing that VLMs combine conditionally independent evidence from each modality with prior beliefs, subject to modality-specific biases. The resulting two-parameter cue combination model accurately predicts VLM preferences under unseen combinations of input degradation and transfers to new forms of unreliability without refitting. Separating evidence strength from integration bias explains how VLMs can follow text even while weighting visual evidence more heavily, reconciling reports of text and vision dominance under a single combination rule.
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.
Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5
Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.
Pratham Singla, Shivank Garg, Vihan Singh +1
Lossfunk · Indian Institute of Technology Roorkee · Raeth AI