Retrieval-augmented vision-language models (VLMs) answer from two sources of evidence that can each fail: the retrieved text can be wrong or noisy, and the image can be degraded or contradicted. We introduce GIB-200K, 200,000 image-question items built from COCO and Visual Genome, each expanded into controlled stress conditions: hard-negative retrieval noise (1-20 documents), visual ambiguity (Gaussian and motion blur, low-resource crops) and cross-modal conflict with counterfactual flip pairs. Instead of only measuring degradation, we ask how models combine the two modalities. From ideal Bayesian cue combination we derive a testable prediction that yields formal definitions of Knowledge Inertia (text over-weighted relative to the model's own image evidence) and Sensory Capture (the reverse), and we estimate the weights with a measurement-error-corrected regression whose recovery of known weights we verify. A counterfactual flip decomposition separates modality bias from concept priors, and a Grounding Illusion Score measures how much clean accuracy is not grounded in the image. We evaluate 4 open models on one frozen, seeded evaluation split of 2,000 GIB-200K items. All models are accurate on clean inputs (92.5%-94.2%) but adverse accuracy falls to 41.3%-59.9%. The two LLaVA models over-weight the text (Knowledge Inertia, w_T/w_I = 1.229 and 1.322), whereas InternVL3 favours the image (w_T/w_I = 0.680); between 34.8% and 71.4% of the clean answers that are correct are abandoned when the document lies. A training-free consistency check that compares the answer with and without retrieval trades coverage for accuracy under misleading documents, while falling back to the image fails when the image is heavily degraded. All per-instance outputs, code, and an audit that recomputes every reported open-model number are released.
Figures & tables
Axis
Condition
Levels evaluated (full benchmark)
clean
image + gold document
–
non-RAG baseline
image, no document
–
retrieval noise
gold + k hard negatives
k ∈ {1, 5, 20} ({1, 3, 5, 10, 20})
visual ambiguity
Gaussian blur, no document
σ∈ {2, 4, 8, 16} px ({1, 2, 4, 8, 16})
horizontal motion blur
{10, 40} px ({5, 10, 20, 40})
low-resource crop
{25, 5}% of object visible ({50, 25, 10, 5})
Table 1: Stress conditions per item. Levels evaluated in this paper (all fixed before any model was run); the benchmark also defines the full level set given in brackets.
Model
Acc clean
Acc adverse
ECE adv.
R
TI
GIS
internvl3
0.93 [0.92, 0.94]
0.60 [0.58, 0.61]
0.21 [0.20, 0.23]
0.51 [0.49, 0.53]
0.20 [0.16, 0.24]
0.35 [0.33, 0.37]
llava05b
0.93 [0.91, 0.94]
0.41 [0.40, 0.43]
0.20 [0.19, 0.21]
0.36 [0.34, 0.37]
2.71 [2.48, 2.95]
0.71 [0.69, 0.73]
llava16
0.94 [0.93, 0.95]
0.49 [0.47, 0.50]
0.30 [0.29, 0.32]
0.36 [0.35, 0.38]
3.72 [3.24, 4.29]
0.62 [0.60, 0.65]
qwen2vl
0.94 [0.93, 0.95]
0.55 [0.53, 0.56]
0.16 [0.14, 0.17]
0.49 [0.47, 0.51]
0.58 [0.52, 0.64]
0.48 [0.46, 0.50]
Table 2: Overall results on all items (95% bootstrap CIs, images as clusters). R = (acc adverse / acc clean) × (1 − ECE adverse); TI = Trust Index; GIS = Grounding Illusion Score.
Figure 1: Vulnerability heatmaps: accuracy drop (points) relative to the clean condition when the image is blurred (rows) and k hard-negative documents are added (columns).
Model
k=0
k=1
k=5
k=20
decay rate
retained AUC
super-additive drop (p)
internvl3
0.927
0.812
0.693
0.487
-0.56 [-0.59, -0.52]
0.77 [0.75, 0.78]
0.43 [0.41, 0.46] (0.0002)
llava05b
0.925
0.701
0.381
0.202
-0.85 [-0.90, -0.82]
0.54 [0.53, 0.56]
0.17 [0.15, 0.19] (0.0002)
llava16
0.939
0.759
0.544
0.281
-0.79 [-0.83, -0.75]
0.64 [0.62, 0.65]
0.24 [0.22, 0.26] (0.0002)
qwen2vl
0.942
0.800
0.584
0.366
-0.73 [-0.77, -0.69]
0.68 [0.67, 0.70]
0.33 [0.31, 0.35] (0.0002)
Table 3: Retrieval-noise dose–response (free-form accuracy with k hard negatives next to the gold document; decay = log-odds per doubling of 1 + k) and the blur × retrieval interaction (super-additive drop; sign-flip permutation p).
Figure 2: Accuracy without documents under Gaussian blur, motion blur and low-resource crops (95% bootstrap bands); triangles mark resolution thresholds.
Figure 3: Cue-combination weights with 95% CIs. The star is the ideal observer (1, 1); points above the diagonal over-weight text (Knowledge Inertia), below it the image (Sensory Capture).
Model
pairs
text-consistent
image-consistent
concept prior
internvl3
816
0.15 [0.13, 0.18]
0.52 [0.48, 0.55]
0.33 [0.30, 0.36]
llava05b
816
0.56 [0.53, 0.60]
0.16 [0.13, 0.18]
0.28 [0.25, 0.31]
llava16
816
0.41 [0.38, 0.44]
0.19 [0.16, 0.21]
0.41 [0.37, 0.44]
qwen2vl
816
0.18 [0.16, 0.21]
0.47 [0.43, 0.50]
0.35 [0.32, 0.38]
Table 5: Counterfactual flip decomposition (share of flip pairs).
Figure 4: Visual share of attention per layer for the same images with a correct vs. a misleading document (solid), and for conflict items split by whether the answer followed the text (dotted).
Figure 5: Eight real failure cases (two per stress condition; green = correct, red = wrong under that stress). Retrieval noise and conflict: the models copy a wrong claim from a document; blur and crop: the image no longer determines the answer. The report’s gallery lists, per condition, the first four items (by id) that at least one model answered correctly on the clean image but not under stress; we show two per condition, preferring those that fool most models (not a random sample).
Model
condition
base acc
Δ image fallback
Δ confidence
abstain: sel. acc / coverage
internvl3
conflict
0.54 [0.52, 0.56]
0.19 [0.17, 0.21]
0.11 [0.10, 0.13]
0.82 / 0.62
internvl3
ret k20
0.49 [0.47, 0.51]
0.24 [0.22, 0.26]
0.18 [0.16, 0.20]
0.85 / 0.53
llava05b
conflict
0.24 [0.22, 0.25]
0.44 [0.42, 0.47]
0.22 [0.20, 0.24]
0.66 / 0.35
llava05b
ret k20
0.20 [0.18, 0.22]
0.48 [0.46, 0.50]
0.35 [0.33, 0.37]
0.66 / 0.28
llava16
conflict
0.32 [0.30, 0.34]
0.43 [0.41, 0.45]
0.18 [0.16, 0.19]
0.81 / 0.39
llava16
ret k20
0.28 [0.26, 0.30]
0.47 [0.44, 0.49]
0.42 [0.40, 0.44]
0.79 / 0.34
Table 6: Training-free consistency gate: change in accuracy when the answer with documents disagrees with the image-only answer ( Δ vs. no gate; 95% CI).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Model
ADS, all layers
ADS, late layers
AUROC (low share → follows text)
n follow text / image
internvl3
-0.012 [-0.013, -0.011]
-0.038 [-0.040, -0.036]
0.69
699 / 1301
llava05b
0.005 [0.004, 0.005]
0.010 [0.009, 0.012]
0.77
1405 / 595
llava16
-0.006 [-0.007, -0.006]
-0.011 [-0.013, -0.010]
0.71
1244 / 756
qwen2vl
-0.013 [-0.014, -0.012]
-0.039 [-0.041, -0.036]
0.77
948 / 1052
Appendix
Table 7: Attention shift, clean → conflict (paired per item; negative = more visual attention under conflict) and AUROC of a low layer-averaged visual share (conflict condition) for predicting that the answer follows the text.
Model
errors
Knowl. Inertia
Sens. Capture
Reas. Collapse
Perceptual
internvl3
15039
0.47
0.01
0.25
0.27
llava05b
18087
0.54
0.01
0.15
0.30
llava16
16396
0.55
0.01
0.18
0.27
qwen2vl
15786
0.53
0.00
0.19
0.27
Appendix
Table 8: Failure taxonomy: share of wrong free-form answers per class (all stress conditions).
Model
condition
base acc
Δ image fallback
Δ confidence
abstain: sel. acc / coverage
internvl3
grid g16 k0
0.97 [0.96, 0.98]
-0.53 [-0.55, -0.50]
-0.03 [-0.04, -0.02]
0.96 / 0.46
llava05b
grid g16 k0
0.95 [0.94, 0.96]
-0.51 [-0.53, -0.48]
-0.09 [-0.11, -0.08]
0.92 / 0.47
llava16
grid g16 k0
0.95 [0.94, 0.96]
-0.48 [-0.50, -0.46]
-0.03 [-0.04, -0.02]
0.93 / 0.50
qwen2vl
grid g16 k0
0.97 [0.96, 0.98]
-0.47 [-0.49, -0.45]
-0.03 [-0.03, -0.02]
0.97 / 0.51
Appendix
Table 9: Training-free consistency gate: change in accuracy when the answer with documents disagrees with the image-only answer ( Δ vs. no gate; 95% CI). grid_g16_k0 = correct document, heavily blurred image.
Model
Δ acc clean
Δ acc adverse
Δ follows text
internvl3
0.01 [-0.04, 0.08]
0.06 [-0.02, 0.13]
-0.03 [-0.13, 0.08]
llava05b
0.01 [-0.05, 0.07]
0.01 [-0.06, 0.07]
-0.02 [-0.12, 0.08]
llava16
0.01 [-0.04, 0.07]
0.05 [-0.03, 0.12]
-0.00 [-0.11, 0.10]
qwen2vl
0.00 [-0.04, 0.06]
0.04 [-0.04, 0.12]
-0.01 [-0.12, 0.10]
Appendix
Table 10: Contamination check: difference between items from COCO train2017 (likely seen in instruction tuning) and val2017.