Organizations: Department of Biomedical Engineering, Purdue University, West Lafayette, IN, USA · Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, Indianapolis, IN, USA · Department of Computer Science, University of Toronto, Toronto, ON, Canada
Vision-language models answer questions about images they were never shown, a failure called mirage. Such an answer can be correct without being justified by the evidence, and the difference matters wherever a model's output is checked by someone who did not see its input. We locate the failure in the training objective: cross-entropy separates a model that reads the image from one that ignores it by only the conditional mutual information between answer and image, and no training example rewards declining. We formalise support completion, TIDE, which also shows each training question with a blank image, a same-domain wrong image and an out-of-domain image, with targets that decline, and prove that under TIDE every image-blind predictor is suboptimal by at least 1.04 nats per example, that the optimum declines exactly on the counterfactual kinds while keeping its behaviour on valid images, and that for any finite model the error of its answer-or-decline decision, declining on valid input plus answering on invalid input, is bounded below by how well its representation separates valid from invalid evidence. On Qwen2.5-VL-7B under a frozen held-out protocol with two judges, the three-condition predecessor of TIDE reduced the invalid-evidence mirage rate from 0.759 to 0.097 and the blank-image rate from 0.700 to 0.003 at a false-abstention cost of 0.126; label-free grounding raised the rate to 0.859, a likelihood-advantage contrast moved it only to 0.718 because its empty-image term carries no gradient, and grounding applied after support completion re-opened the mirage. The four-condition protocol on three base models, with seeds and component ablations, is in progress and not reported here.
Figures & tables
Figure 1: Mirage, and support completion. A question with (A) the image it is about, (B) a blank image and (C) a real image of the right kind that is not the one asked about: a stock model answers in all three cases, and only the first answer rests on evidence. (D) Support completion (TIDE) trains the same question with its valid image and three counterfactual inputs whose targets decline; only the language model’s adapters are trained, and evaluation uses the same four roles on groups that never enter training. No measured quantity is shown.
judge U (response only)
rule B (gpt-5.5, image aware)
cell
I-MR
blank
in-dom.
out-dom.
TC-LIA MR
IA-IMR
acc.
answered
false abst.
E0 Base
0.759
0.700
0.813
0.764
0.569
0.431
0.577
0.982
0.018
TIDE only (three-condition)
0.097
0.003
0.233
0.054
0.072
0.088
0.559
0.874
0.126
E1 Grounding
0.859
0.646
0.962
0.969
0.644
0.799
0.377
0.774
0.226
E2 Grounding then TIDE (three-cond.)
0.094
0.000
0.236
0.046
0.070
0.086
0.526
0.849
0.151
E3 TIDE then Grounding (three-cond.)
0.615
0.000
0.936
0.908
0.461
0.490
0.423
0.821
0.179
Table 1: The measured record: one model, the component-and-order grid plus E5, one frozen held-out test. Qwen2.5-VL-7B-Instruct, seed 0, protocol v4.1 final test (390 groups, 1,560 rows); the TIDE stage is the three-condition predecessor. Left: judge U, I-MR over the 1,170 invalid-evidence rows, the rate per role (390 rows each) and TC-LIA MR (denominator 1,560). Right: rule B (gpt-5.5), IA-IMR on the same rows, and accuracy, answered rate and false abstention on the 390 real rows. No judge reply was unparseable; E5 was judged with judge U only. Bold: the support-completion cells.
Figure 2: Mirage rate by cell and role (measured, v4.1 final test). Bars: judge U with 95% Wilson intervals; hollow squares: rule B (gpt-5.5). Support completion (the TIDE-only cell and E2, Grounding then TIDE) empties the blank role, nearly empties the untrained out-of-domain role and leaves the in-domain role as the residual; grounding after TIDE (E3) keeps blank at zero and re-opens both unrelated-image roles.
Figure 3: Training order (measured, v4.1). (a) Per-role mirage rate for the four trained cells. (b) Teacher-forced loss of the E3 model on ten train-owned probe keys before and after its grounding stage: the decline targets are forgotten (a diagnostic, not a held-out result).
Figure 4: Stress pilot (measured, DEV, 100 groups): IA-IMR by prompt and cell. Bars: primary judge (gpt-5.5) with 95% Wilson intervals; diamonds: secondary judge. A dev pilot, not a final-test result.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
aspect
earlier pipeline (superseded; no number reused)
protocols v4.1 and v5 (this paper)
MCQ options
never reached the model (1,131 train / 399 test groups)
rendered once, visibly; the MCQ target is the option letter
multi-image records
cut to the first image (477 train / 121 test groups)
every image at its own marker; all roles count-matched
gold answers
collapsed to the first string (9 of 10 TextVQA annotators discarded)
per-domain target policy; scoring keeps every gold string
counterfactual images
assigned before the split: 803 of 1,000 test groups had their real image in training
content-component split first, counterfactuals after, from a different component
decoding
hard cap of 64 tokens (stock and grounding cells truncated on 71 to 95% of rows)
greedy, cap 3,072, frozen after a dev-only audit
artifact identity
a file name
a protocol record; a mismatch is a hard failure
Appendix
Table 2: The earlier pipeline against the frozen protocols. No number from the left column is reused.
Figure 5: Leakage-safe construction of the protocol v5 tiers. Whole content components are assigned to tiers before any counterfactual is drawn; counterfactuals come from the same tier and a different component.
fact
value
protocol versions
v4.1 (measured study); v5.0 (frozen 2026-10-09; runs in progress)
eligible groups (five recorded exclusions)
4,746
content components (shared real-image content or identical question string)
Table 3: Split, holdout and training-set facts for protocols v4.1 and v5.
blind responses (model mean)
paired complete cases (model mean)
≥1 model correct blind
domain
n
formats
answer
decline
empty
blind acc.
image acc.
gain
observed
guessing ref.
Chest radiographs
451
251 yes/no, 200 OE
0.48
0.36
0.16
0.31
0.67
0.36
276 (0.61)
0.54
DocVQA
500
494 OE, 6 yes/no
0.39
0.43
0.19
0.12
0.89
0.77
41 (0.08)
0.01
InfoVQA
500
500 OE
0.61
0.23
0.16
0.34
0.88
0.54
166 (0.33)
0.00
MedXpertQA-MM
500
500 MC
0.80
0.03
0.17
0.41
0.43
0.01
304 (0.61)
0.67
MicroVQA
500
500 MC
0.86
0.00
0.14
0.57
0.60
0.03
369 (0.74)
0.71
Appendix
Table 4: Text-sufficiency of the ten domains, five API models, image removed (2026-07 pipeline, all questions). Row-level formats (MC multiple choice, yes/no without options, OE other open-ended); model-mean composition of the blind responses with every evaluated question as the denominator; model-mean blind and with-image accuracy and their difference on paired complete cases; the number and fraction of questions on which at least one model was correct without the image, with the fraction the same rule would flag if the questions with a chance level were guessed uniformly (guessing ref.) and, for the total, the leave-one-model-out range. Descriptive; not a protocol result.
pairs or cells
n
Pearson r
Spearman ρ
full benchmark, all 50 pairs (primary)
50
-0.41
-0.43
same, 40 pairs without GPT-5.5
40
-0.69
-0.72
same, rate conditional on non-empty responses
50
-0.67
-0.72
same, empty responses counted incorrect in the gain
50
-0.52
-0.59
paired gain vs explicit-decline rate, 50 pairs
50
+0.66
+0.70
within multiple-choice cells (25 cells, 5 domains)
25
-0.07
-0.03
Appendix
Table 5: Exploratory associations between the substantive blind-answer rate and the paired accuracy gain across (model, domain) pairs or (model, domain, format) cells. Observations are clustered by model and domain; no hypothesis test is claimed.
Figure 6: Text-sufficiency of the ten domains (five API models, image removed; descriptive). (a) Per domain, model mean: the blind responses split into substantive answers, explicit declines and empty output, next to the fraction of questions on which at least one model was correct without the image and the fraction the same rule would flag if the questions with a chance level were guessed uniformly. (b) Substantive blind-answer rate against the accuracy gain from the image on paired complete cases, one point per (model, domain) pair; the dashed line is the pooled fit over all 50 pairs (Pearson −0.411 ); GPT-5.5 is hollow because a fraction 0.789 of its blind responses are empty.
Figure 7: Where each stage acts, and how the cells are composed. TIDE trains LoRA adapters on the language model’s attention projections on the completed support and never reads a grounding score; Grounding trains only the vision projector with a label-free cosine contrast and never reads an answer. E1 and E3 Stage B share one grounding budget, candidate pool and negative schedule, so E2 and E3 differ in order only.
TIDE (the adapter of TIDE only, E2 stage 2 and E3 Stage A)
LoRA rank 8, α 16, dropout 0, on the language model’s q/k/v/o projections; lr 10−4 , 800 steps, accumulation 4; per-example token-mean cross-entropy; v5: 12,000 examples from 3,000 train groups (real image to the gold answer; blank image to the blank decline; in-domain and out-of-domain wrong images to the shared unrelated decline); v4.1: 9,000 examples, out-of-domain never trained
Grounding (E1; E3 Stage B)
vision projector only, language model and vision encoder frozen, no answer read; joint softmax over the cosine scores of one anchor against five candidate image sets (v5: real, in-domain wrong, out-of-domain wrong and two further negatives; v4.1: real, in-domain wrong, three in-batch real images), temperature 0.2, batch 16, lr 10−4 , 300 fixed steps, fp32 master weights with a bf16 forward; pool 2,750 source groups / 2,747 anchors; E1 and E3 Stage B share one negative schedule
E2 Grounding then TIDE
the E1 projector, frozen, then a fresh TIDE adapter (no adapter is inherited)
E3 TIDE then Grounding
the TIDE adapter (Stage A), then Stage B grounding on the E1 pool and schedule
E5 evidence-advantage contrast (v4.1 diagnostic)
projector only, from the released weights; row-wise cross-entropy over logp(a∣q,vk)−logp(a∣q,∅) across the five candidates, temperature 1.0, batch 16, 300 fixed steps, fp32 masters
generation (every cell)
greedy, max_new_tokens 3,072, one renderer (options visible once, every image at its marker), pinned fast processor, bf16, seed 0
selection
fixed final step for every stage; no early stopping and no checkpoint selection; the 250-group held-match probe is a diagnostic only
Appendix
Table 6: Recipes of every cell, read from the stored protocol records.
rule-B judge model
agreement
κ
precision
recall
F1
gpt-5.5 (in sample)
1.000
1.000
1.000
1.000
1.000
gpt-4o-mini
0.974
0.874
1.000
0.800
0.889
Appendix
Table 7: Rule-B judge models on the 38 audited rows (agreement and Cohen’s κ on non-real rows, precision and recall of the mirage label). In sample for gpt-5.5.
Figure 8: Design of the blinded human validation of both judges. Not run; no number exists.
cell
real rows (390)
counterfactual rows (1,170)
string rule
exact
judge
agree
decline
refusal wd.
decline
near
refusal wd.
empty
F
M
41F+43M
95% CI
only
proxy
real / cf.
E0 Base
0
58
0
0
491
0
0.000
1.000
0.750
[0.750, 0.750]
0.750
0.516
383/390, –
TIDE only
49
0
1,056
1
0
0
0.126
0.097
0.104
[0.090, 0.118]
0.104
0.104
390/390, 1,057/1,057
E1 Grounding
0
23
0
0
194
1
0.000
1.000
0.750
[0.750, 0.750]
0.750
0.389
302/390, –
E2 Grounding then TIDE
59
0
1,060
0
0
0
0.151
0.094
0.108
[0.094, 0.122]
0.108
0.108
390/390, 1,060/1,060
E3 TIDE then Grounding
5
1
398
4
1
0
0.013
0.656
0.495
[0.490, 0.501]
0.497
0.177
325/390, 402/402
Appendix
Table 8: The decision statistic of Proposition 3 , v4.1 final test. decline : the response equals one of the two trained v4.1 decline strings or begins with their shared clause “I can’t answer this question” ( near ); answer : anything else, including refusals in other words ( refusal wording : a fixed list of refusal phrases, counted for description only and entering no rate), hedges and empty outputs. Hedged replies (rule B’s P1 and P2 both yes) are answers under the rule; none occurs in the support-completion cells. F is the decline count on the 390 real rows over 390, M the answer count on the 1,170 counterfactual rows over 1,170; the interval is a paired question-group bootstrap (2,000 resamples of the 390 groups, the four roles of a group kept together, percentile); exact only counts the near matches as answers. Counts come from exact string operations on the stored generation files; no row was read by a person. Judge proxy : 41(1−P1real)+43P1cf. from rule B (gpt-5.5), whose judge sees the question, the captions and the role; descriptive only. Agree : real rows on which the string decline and the judge’s “abstained” coincide, and counterfactual rows declined under the string rule that rule B also calls not a mirage.
cell
Mblank
Mid
Mod
Minv
I-MR
TC-LIA MR
E0 Base
273
317
298
888
0.7590
0.5692
TIDE only (3-cond.)
1
91
21
113
0.0966
0.0724
E1 Grounding
252
375
378
1005
0.8590
0.6442
E2 Grounding then TIDE
0
92
18
110
0.0940
0.0705
E3 TIDE then Grounding
0
365
354
719
0.6145
0.4609
E5 evidence-advantage
256
304
280
840
0.7179
0.5385
Appendix
Table 9: Judge U, v4.1 final test: mirage counts per invalid role (390 rows each), their sum over the 1,170 invalid-evidence rows, I-MR and TC-LIA MR (denominator 1,560).
judge model
cell
blank
in-dom.
out-dom.
IA-IMR
IA-FMR
acc.
answered
false abst.
unparseable
gpt-5.5 (primary)
E0 Base
0.718
0.331
0.244
0.431
0.323
0.577
0.982
0.018
0
gpt-5.5 (primary)
TIDE only
0.003
0.208
0.054
0.088
0.066
0.559
0.874
0.126
0
gpt-5.5 (primary)
E1 Grounding
0.631
0.869
0.897
0.799
0.599
0.377
0.774
0.226
0
gpt-5.5 (primary)
E2 Grounding then TIDE
0.000
0.215
0.044
0.086
0.065
0.526
0.849
0.151
0
gpt-5.5 (primary)
E3 TIDE then Grounding
0.000
0.726
0.744
0.490
0.367
0.423
0.821
0.179
0
gpt-4o-mini (secondary)
E0 Base
0.580
0.526
0.374
0.493
0.370
0.528
0.941
0.059
0
Appendix
Table 10: Rule B, v4.1 final test, both judge models: mirage rate per invalid role, IA-IMR (1,170 rows), IA-FMR (1,560 rows), real-image accuracy, answered rate and false abstention (390 rows), and unparseable replies.
cell
role
P1 substantive answer
P2 acknowledges problem
P3 about shown image
P4 describes absent content
E0 Base
real
0.982
0.082
0.949
0.867
blank
0.756
0.349
–
0.564
in-domain
0.695
0.487
0.749
0.272
out-of-domain
0.595
0.659
0.741
0.164
TIDE only
real
0.874
0.126
0.726
0.451
blank
0.003
0.997
–
0.000
Appendix
Table 11: Probe yes-rates per role, rule B, gpt-5.5. P3 is not asked on blank rows.
contrast (v4.1 final test, 1,170 invalid rows)
Δ I-MR, judge U [95% CI]
Δ IA-IMR, rule B [95% CI]
TIDE only minus E0 Base
-0.662 [-0.692, -0.632]
-0.343 [-0.380, -0.308]
E2 minus E0 Base
-0.665 [-0.696, -0.636]
-0.344 [-0.381, -0.311]
E2 minus TIDE only
-0.003 [-0.018, 0.014]
-0.002 [-0.016, 0.014]
E1 Grounding minus E0 Base
+0.100 [0.073, 0.127]
+0.368 [0.338, 0.397]
E3 minus TIDE only
+0.518 [0.497, 0.538]
+0.402 [0.376, 0.425]
E5 minus E0 Base
-0.041 [-0.068, -0.014]
–
Appendix
Table 12: Paired group-level bootstrap contrasts (2,000 resamples over the 390 question groups) for the differences the main text reports.
I-MR, judge U
IA-IMR, rule B
in-dom., U
false abst.
domain (117 invalid, 39 real rows)
E0
TIDE
E2
E3
E0
TIDE
E2
E3
TIDE
E2
TIDE
E2
Chest radiographs
0.63
0.23
0.23
0.63
0.26
0.17
0.17
0.49
0.56
0.59
0.15
0.21
DocVQA
0.58
0.04
0.03
0.57
0.14
0.03
0.01
0.32
0.08
0.08
0.05
0.03
InfoVQA
0.81
0.00
0.01
0.64
0.28
0.00
0.01
0.50
0.00
0.03
0.03
0.08
MedXpertQA-MM
0.89
0.09
0.08
0.61
0.64
0.09
0.08
0.49
0.23
0.18
0.10
0.13
MicroVQA
0.92
0.10
0.09
0.56
0.83
0.10
0.09
0.49
0.31
0.28
0.08
0.08
Appendix
Table 13: Domain breakdown, v4.1 final test (117 invalid rows and 39 real rows per domain): pooled mirage rates under both judges, the in-domain rate and the false-abstention rate of the two support-completion cells. Spearman rank correlation across domains between in-domain mirage and false abstention: TIDE only 0.820 , E2 0.579 .
Figure 9: I-MR per domain (measured, v4.1). Stock model, TIDE only and Grounding then TIDE.
rule B, gpt-5.5 (primary)
rule B, gpt-4o-mini (secondary)
cell
prompt
IA-IMR
blank
in-dom.
out-dom.
acc.
answered
IA-IMR
blank
in-dom.
out-dom.
acc.
answered
E0 Base
P0
0.457
0.720
0.360
0.290
0.550
0.970
0.503
0.560
0.580
0.370
0.460
0.950
P1
0.920
1.000
0.910
0.850
0.500
1.000
0.710
0.510
0.850
0.770
0.490
0.900
P2
0.340
0.530
0.340
0.150
0.450
0.980
0.380
0.410
0.510
0.220
0.390
0.960
E1 Grounding
P0
0.773
0.640
0.830
0.850
0.250
0.790
0.457
0.480
0.560
0.330
0.210
0.770
P1
0.990
1.000
0.990
0.980
0.320
0.930
0.653
0.510
0.880
0.570
0.300
0.800
Appendix
Table 14: Mirage Stress Test V1, DEV pilot, 100 groups: every pre-registered quantity, both judge models. Preliminary; not a benchmark result.
diagnostic (train-owned keys; not a held-out result)
value
E3 Stage B (grounding after TIDE): teacher-forced loss on the model’s own decline targets, 10 train-owned probe keys, before / after
0.021 / 1.675
same, on the real-image gold targets, before / after
0.332 / 2.611
E1 Grounding: held-match accuracy on the 250 train-owned probe pairs at the fixed final step
0.924
E5: contrastive loss on 64 train-owned probe anchors, before / after 300 steps
0.760 / 0.145
E5: mean evidence advantage of the gold answer (nats), before / after
+4.96 / +12.05
E5: candidate top-1 rate, before / after
0.750 / 0.938
Appendix
Table 15: Training-time diagnostics (train-owned keys; not held-out results).
Figure 10: E5 on train-owned probe anchors: per-step contrastive loss (left) and the candidate top-1 and evidence-positive rates before and after 300 steps (right).
DEV group chest_vqa / 361, canonical prompt P0, stress-pilot generations
input
E0 Base
E2 Grounding then TIDE (v4.1, three-condition)
real image
The hyperintensities surrounding the aorta in this CT image are likely to be fat deposits. In CT scans, fat appears as bright or hyperintense signals compared to other … [answered]
lymph nodes [answered]
blank image
The hyperintensities surrounding the aorta in the image you provided are likely due to fat deposits. In MRI images, fat appears as bright signals (hyperintense) compared … [mirage]
I can’t answer this question because no image was provided. I need to see the image to answer. [not mirage]
in-domain wrong
The hyperintensities surrounding the aorta in this CT image are likely to be related to the presence of air or gas. In CT scans, air and gas appear as bright … [mirage]
I can’t answer this question from this image - it does not show what the question is asking about. [not mirage]
out-of-domain wrong
The image you provided appears to be a whiteboard with notes related to a database schema and some HTTP methods. It does not contain any medical images or information … [not mirage]
I can’t answer this question from this image - it does not show what the question is asking about. [not mirage]
Appendix
Table 16: Outputs on one dev group (stress pilot, P0), with the rule-B verdict (gpt-5.5).
three-condition predecessor of TIDE (out-of-domain not trained); superseded for the v5 method but measured and traceable; reported as the predecessor study
question-only correctness of the ten domains (B-Clean union count with row-level format, chance-level and leave-one-model-out references) and the exploratory association between substantive blind answering (denominator: all evaluated questions; empty responses a separate category) and the paired complete-case accuracy gain on the full benchmark; B-Clean subset as a selection-conditioned sensitivity; descriptive context for Sec. 4.1; audited in docs/POC_V3_COMPARISON.md
blank is trivially separable; nothing rewards declining on wrong images
A3
Real plus in-domain wrong-image abstention
sft_mode real_id (6,000 examples), 800 steps
in-domain declining transfers partly to blank and out-of-domain inputs
A4
Full support completion (TIDE, four conditions)
equals TIDE-only (internal e1) under v5
the four-condition optimum declines on every counterfactual kind (Prop. 2)
A5
Generic refusal-SFT baseline
3,000 real examples plus 9,000 refusal examples whose inputs are text-only unanswerable or out-of-scope questions with the real image kept (no counterfactual images); same decline strings; 800 steps
a refusal style without counterfactual images is not an evidence check
Appendix
Table 19: Component ablations A0 to A7: composition and hypothesis. A4 is the TIDE-only cell; A0 is E0. Matched optimizer steps, LoRA budget and decline strings; separate from the training-order grid.
aspect
this paper
Mirage Probes
goal
remove mirage by changing the training objective; prove what the optimum does
detect mirage responses from internal activations; characterise mechanisms
mirage label
a judge’s verdict on the generated response given the role: response-only (judge U) and image-aware (rule B); invalid inputs are constructed (blank, in-domain wrong, out-of-domain)
consistency between the with-image and the without-image generation (bag-of-words cosine above 0.7), uncertainty detected by regular expressions; no wrong-image condition
models
Qwen2.5-VL-7B measured; LLaVA-OneVision-7B and InternVL3.5-8B in progress
Ovis2.5-2B and Qwen3-VL-32B; GLM-4.6-flash dropped for too few non-mirage responses
data
ten domains, frozen leakage-safe tiers, 390 groups measured (1,560 rows)
VQA-RAD, MMMU-Pro, MedXpertQA; MicroVQA dropped for too few non-mirage responses
intervention
TIDE (support completion) and the grounding and contrast baselines; held-out behavioural evaluation
none evaluated; discussion of text-distribution cleaning and representational interventions
mechanistic claim
the decision lives in the mapping from representation to response (order grid, gradient cancellation); the residual is bounded by validity separability (Prop. 3)
two mechanisms, spurious internal images and textual biases, stated as a hypothesis
Appendix
Table 20: Methodological comparison with Mirage Probes (arXiv:2606.13870v1).
Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.
Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5
Vision-language models (VLMs) can produce confident answers without relevant visual evidence, a failure mode known as mirage reasoning (Asadi et al., 2026). To that end, we study pre-release mirage detection: deciding whether a VLM answer should be released or withheld. Our model-agnostic method, Text-Conditioned Layer-wise Internal Alignment (TC-LIA), tracks question-image alignment across the layers of a frozen CLIP ViT-H/14 encoder, summarizing patch-text alignment by final similarity, late-layer top-k alignment, early-to-late gain, and slope. TC-LIA is purely unsupervised (fixed projections, fixed scoring weights, no labels, no training) and already delivers strong detection independently. Additionally, when combined with blank/noise detection, domain routing, and VLM self-assessment, it forms an ensemble whose supervised training improves performance but is an optional add-on. On 19,004 samples spanning ten VQA domains, fourteen state-of-the-art VLMs exhibit 57.3-75.0% base mirage rates. Our proposed TC-LIA alone cuts this to 7.5% with 83.5% Related/Unrelated/Blank-Noise classification accuracy, and the ensemble reaches 84.3-88.4% accuracy with 5.9-7.2% mirage rates (best joint result: 88.4% accuracy, 6.4% mirage rate). Notably, an ensemble trained on a single backbone transfers well to unseen backbones, with the best-transferring source staying within 1.2% accuracy points of per-backbone training across thirteen held-out VLMs.
Md. Shaown Miah, S. M. Taiabul Haque, Syed Ishtiaque Ahmed +1
Indiana University Indianapolis · University of Toronto
Vision-language models (VLMs) perform well on visual question answering with high-quality images but struggle when questions require knowledge beyond what is clearly and directly visible. In such settings, uncertainty quantification should not only indicate whether the model is likely to fail but also diagnose why it is uncertain, across dimensions such as perception, entity recognition, and knowledge retrieval. While prior work has focused on individual failure modes in isolation or treated incorrect answers as monolithic failures, we propose a unified framework for disentangling these failure modes and investigate whether pre-generation signals can predict these failure sources. Across a range of datasets and model families, we find a consistent pattern in VLM errors: some failures arise from visual or recognition bottlenecks, while others persist after the relevant entity is identified. Our main finding is that these failure sources can be predicted before decoding: recognition-related failures are best captured by visual-token representations, while failures that remain after recognition are better captured by prompt-conditioned hidden states. This pre-generation signal enables efficient failure-source prediction before the model produces an answer, allowing uncertain cases to be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.