Teaching Vision-Language Models to Decline: Support Completion Against Mirage

Alfaj Uddin Ahmed; Md. Shaown Miah; Md Arid Hasan; Syed Ishtiaque Ahmed; Sayeed Shafayet Chowdhury

cs.CVOct 12, 2026

Teaching Vision-Language Models to Decline: Support Completion Against Mirage

Authors: Alfaj Uddin Ahmed, Md. Shaown Miah, Md Arid Hasan, Syed Ishtiaque Ahmed, Sayeed Shafayet Chowdhury

Organizations: Department of Biomedical Engineering, Purdue University, West Lafayette, IN, USA · Department of Computer Science, Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, Indianapolis, IN, USA · Department of Computer Science, University of Toronto, Toronto, ON, Canada

Abstract

Vision-language models answer questions about images they were never shown, a failure called mirage. Such an answer can be correct without being justified by the evidence, and the difference matters wherever a model's output is checked by someone who did not see its input. We locate the failure in the training objective: cross-entropy separates a model that reads the image from one that ignores it by only the conditional mutual information between answer and image, and no training example rewards declining. We formalise support completion, TIDE, which also shows each training question with a blank image, a same-domain wrong image and an out-of-domain image, with targets that decline, and prove that under TIDE every image-blind predictor is suboptimal by at least 1.04 nats per example, that the optimum declines exactly on the counterfactual kinds while keeping its behaviour on valid images, and that for any finite model the error of its answer-or-decline decision, declining on valid input plus answering on invalid input, is bounded below by how well its representation separates valid from invalid evidence. On Qwen2.5-VL-7B under a frozen held-out protocol with two judges, the three-condition predecessor of TIDE reduced the invalid-evidence mirage rate from 0.759 to 0.097 and the blank-image rate from 0.700 to 0.003 at a false-abstention cost of 0.126; label-free grounding raised the rate to 0.859, a likelihood-advantage contrast moved it only to 0.718 because its empty-image term carries no gradient, and grounding applied after support completion re-opened the mirage. The four-condition protocol on three base models, with seeds and component ablations, is in progress and not reported here.

Figures & tables

Figure 1: Mirage, and support completion. A question with (A) the image it is about, (B) a blank image and (C) a real image of the right kind that is not the one asked about: a stock model answers in all three cases, and only the first answer rests on evidence. (D) Support completion (TIDE) trains the same question with its valid image and three counterfactual inputs whose targets decline; only the language model’s adapters are trained, and evaluation uses the same four roles on groups that never enter training. No measured quantity is shown.

	judge U (response only)					rule B (gpt-5.5, image aware)
cell	I-MR	blank	in-dom.	out-dom.	TC-LIA MR	IA-IMR	acc.	answered	false abst.
E0 Base	0.759	0.700	0.813	0.764	0.569	0.431	0.577	0.982	0.018
TIDE only (three-condition)	0.097	0.003	0.233	0.054	0.072	0.088	0.559	0.874	0.126
E1 Grounding	0.859	0.646	0.962	0.969	0.644	0.799	0.377	0.774	0.226
E2 Grounding then TIDE (three-cond.)	0.094	0.000	0.236	0.046	0.070	0.086	0.526	0.849	0.151
E3 TIDE then Grounding (three-cond.)	0.615	0.000	0.936	0.908	0.461	0.490	0.423	0.821	0.179

Table 1: The measured record: one model, the component-and-order grid plus E5, one frozen held-out test. Qwen2.5-VL-7B-Instruct, seed 0, protocol v4.1 final test (390 groups, 1,560 rows); the TIDE stage is the three-condition predecessor. Left: judge U, I-MR over the 1,170 invalid-evidence rows, the rate per role (390 rows each) and TC-LIA MR (denominator 1,560). Right: rule B (gpt-5.5), IA-IMR on the same rows, and accuracy, answered rate and false abstention on the 390 real rows. No judge reply was unparseable; E5 was judged with judge U only. Bold: the support-completion cells.

Figure 2: Mirage rate by cell and role (measured, v4.1 final test). Bars: judge U with 95% Wilson intervals; hollow squares: rule B (gpt-5.5). Support completion (the TIDE-only cell and E2, Grounding then TIDE) empties the blank role, nearly empties the untrained out-of-domain role and leaves the in-domain role as the residual; grounding after TIDE (E3) keeps blank at zero and re-opens both unrelated-image roles.

Figure 3: Training order (measured, v4.1). (a) Per-role mirage rate for the four trained cells. (b) Teacher-forced loss of the E3 model on ten train-owned probe keys before and after its grounding stage: the decline targets are forgotten (a diagnostic, not a held-out result).

Figure 4: Stress pilot (measured, DEV, 100 groups): IA-IMR by prompt and cell. Bars: primary judge (gpt-5.5) with 95% Wilson intervals; diamonds: secondary judge. A dev pilot, not a final-test result.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

aspect	earlier pipeline (superseded; no number reused)	protocols v4.1 and v5 (this paper)
MCQ options	never reached the model (1,131 train / 399 test groups)	rendered once, visibly; the MCQ target is the option letter
multi-image records	cut to the first image (477 train / 121 test groups)	every image at its own marker; all roles count-matched
gold answers	collapsed to the first string (9 of 10 TextVQA annotators discarded)	per-domain target policy; scoring keeps every gold string
counterfactual images	assigned before the split: 803 of 1,000 test groups had their real image in training	content-component split first, counterfactuals after, from a different component
decoding	hard cap of 64 tokens (stock and grounding cells truncated on 71 to 95% of rows)	greedy, cap 3,072, frozen after a dev-only audit
artifact identity	a file name	a protocol record; a mismatch is a hard failure

Appendix

Table 2: The earlier pipeline against the frozen protocols. No number from the left column is reused.

Figure 5: Leakage-safe construction of the protocol v5 tiers. Whole content components are assigned to tiers before any counterfactual is drawn; counterfactuals come from the same tier and a different component.

fact	value
protocol versions	v4.1 (measured study); v5.0 (frozen 2026-10-09; runs in progress)
eligible groups (five recorded exclusions)	4,746
content components (shared real-image content or identical question string)	3,141
v4.1: train / dev / final test groups	3,000 / 484 / 390 (39 per domain; 1,560 rows)
v4.1: TIDE training examples (three-condition)	9,000 (fingerprint a2a16a8fe2eb )
v4.1: split / counterfactual / final-test fingerprints	be1e7673f79a / v4.0:1f894eacffbd / a84e616b5c82

Appendix

Table 3: Split, holdout and training-set facts for protocols v4.1 and v5.

			blind responses (model mean)			paired complete cases (model mean)			$\geq 1$ model correct blind
domain	$n$	formats	answer	decline	empty	blind acc.	image acc.	gain	observed	guessing ref.
Chest radiographs	451	251 yes/no, 200 OE	0.48	0.36	0.16	0.31	0.67	0.36	276 (0.61)	0.54
DocVQA	500	494 OE, 6 yes/no	0.39	0.43	0.19	0.12	0.89	0.77	41 (0.08)	0.01
InfoVQA	500	500 OE	0.61	0.23	0.16	0.34	0.88	0.54	166 (0.33)	0.00
MedXpertQA-MM	500	500 MC	0.80	0.03	0.17	0.41	0.43	0.01	304 (0.61)	0.67
MicroVQA	500	500 MC	0.86	0.00	0.14	0.57	0.60	0.03	369 (0.74)	0.71

Appendix

Table 4: Text-sufficiency of the ten domains, five API models, image removed (2026-07 pipeline, all questions). Row-level formats (MC multiple choice, yes/no without options, OE other open-ended); model-mean composition of the blind responses with every evaluated question as the denominator; model-mean blind and with-image accuracy and their difference on paired complete cases; the number and fraction of questions on which at least one model was correct without the image, with the fraction the same rule would flag if the questions with a chance level were guessed uniformly (guessing ref.) and, for the total, the leave-one-model-out range. Descriptive; not a protocol result.

pairs or cells	$n$	Pearson $r$	Spearman $\rho$
full benchmark, all 50 pairs (primary)	50	-0.41	-0.43
same, 40 pairs without GPT-5.5	40	-0.69	-0.72
same, rate conditional on non-empty responses	50	-0.67	-0.72
same, empty responses counted incorrect in the gain	50	-0.52	-0.59
paired gain vs explicit-decline rate, 50 pairs	50	+0.66	+0.70
within multiple-choice cells (25 cells, 5 domains)	25	-0.07	-0.03

Appendix

Table 5: Exploratory associations between the substantive blind-answer rate and the paired accuracy gain across (model, domain) pairs or (model, domain, format) cells. Observations are clustered by model and domain; no hypothesis test is claimed.

Figure 6: Text-sufficiency of the ten domains (five API models, image removed; descriptive). (a) Per domain, model mean: the blind responses split into substantive answers, explicit declines and empty output, next to the fraction of questions on which at least one model was correct without the image and the fraction the same rule would flag if the questions with a chance level were guessed uniformly. (b) Substantive blind-answer rate against the accuracy gain from the image on paired complete cases, one point per (model, domain) pair; the dashed line is the pooled fit over all 50 pairs (Pearson

-0.411

); GPT-5.5 is hollow because a fraction

0.789

of its blind responses are empty.

Figure 7: Where each stage acts, and how the cells are composed. TIDE trains LoRA adapters on the language model’s attention projections on the completed support and never reads a grounding score; Grounding trains only the vision projector with a label-free cosine contrast and never reads an answer. E1 and E3 Stage B share one grounding budget, candidate pool and negative schedule, so E2 and E3 differ in order only.

TIDE (the adapter of TIDE only, E2 stage 2 and E3 Stage A)	LoRA rank 8, $\alpha$ 16, dropout 0, on the language model’s q/k/v/o projections; lr $10^{-4}$ , 800 steps, accumulation 4; per-example token-mean cross-entropy; v5: 12,000 examples from 3,000 train groups (real image to the gold answer; blank image to the blank decline; in-domain and out-of-domain wrong images to the shared unrelated decline); v4.1: 9,000 examples, out-of-domain never trained
Grounding (E1; E3 Stage B)	vision projector only, language model and vision encoder frozen, no answer read; joint softmax over the cosine scores of one anchor against five candidate image sets (v5: real, in-domain wrong, out-of-domain wrong and two further negatives; v4.1: real, in-domain wrong, three in-batch real images), temperature 0.2, batch 16, lr $10^{-4}$ , 300 fixed steps, fp32 master weights with a bf16 forward; pool 2,750 source groups / 2,747 anchors; E1 and E3 Stage B share one negative schedule
E2 Grounding then TIDE	the E1 projector, frozen, then a fresh TIDE adapter (no adapter is inherited)
E3 TIDE then Grounding	the TIDE adapter (Stage A), then Stage B grounding on the E1 pool and schedule
E5 evidence-advantage contrast (v4.1 diagnostic)	projector only, from the released weights; row-wise cross-entropy over $\log p(a\mid q,v_{k})-\log p(a\mid q,\emptyset)$ across the five candidates, temperature 1.0, batch 16, 300 fixed steps, fp32 masters
generation (every cell)	greedy, max_new_tokens 3,072, one renderer (options visible once, every image at its marker), pinned fast processor, bf16, seed 0
selection	fixed final step for every stage; no early stopping and no checkpoint selection; the 250-group held-match probe is a diagnostic only

Appendix

Table 6: Recipes of every cell, read from the stored protocol records.

rule-B judge model	agreement	$\kappa$	precision	recall	$F_{1}$
gpt-5.5 (in sample)	1.000	1.000	1.000	1.000	1.000
gpt-4o-mini	0.974	0.874	1.000	0.800	0.889

Appendix

Table 7: Rule-B judge models on the 38 audited rows (agreement and Cohen’s

\kappa

on non-real rows, precision and recall of the mirage label). In sample for gpt-5.5.

Figure 8: Design of the blinded human validation of both judges. Not run; no number exists.

cell	real rows (390)		counterfactual rows (1,170)				string rule				exact	judge	agree
	decline	refusal wd.	decline	near	refusal wd.	empty	$F$	$M$	$\tfrac{1}{4}F+\tfrac{3}{4}M$	95% CI	only	proxy	real / cf.
E0 Base	0	58	0	0	491	0	0.000	1.000	0.750	[0.750, 0.750]	0.750	0.516	383/390, –
TIDE only	49	0	1,056	1	0	0	0.126	0.097	0.104	[0.090, 0.118]	0.104	0.104	390/390, 1,057/1,057
E1 Grounding	0	23	0	0	194	1	0.000	1.000	0.750	[0.750, 0.750]	0.750	0.389	302/390, –
E2 Grounding then TIDE	59	0	1,060	0	0	0	0.151	0.094	0.108	[0.094, 0.122]	0.108	0.108	390/390, 1,060/1,060
E3 TIDE then Grounding	5	1	398	4	1	0	0.013	0.656	0.495	[0.490, 0.501]	0.497	0.177	325/390, 402/402

Appendix

Table 8: The decision statistic of Proposition 3 , v4.1 final test. decline : the response equals one of the two trained v4.1 decline strings or begins with their shared clause “I can’t answer this question” ( near ); answer : anything else, including refusals in other words ( refusal wording : a fixed list of refusal phrases, counted for description only and entering no rate), hedges and empty outputs. Hedged replies (rule B’s P1 and P2 both yes) are answers under the rule; none occurs in the support-completion cells.

F

is the decline count on the 390 real rows over 390,

M

the answer count on the 1,170 counterfactual rows over 1,170; the interval is a paired question-group bootstrap (2,000 resamples of the 390 groups, the four roles of a group kept together, percentile); exact only counts the near matches as answers. Counts come from exact string operations on the stored generation files; no row was read by a person. Judge proxy :

\tfrac{1}{4}(1-\text{P1}_{\text{real}})+\tfrac{3}{4}\,\overline{\text{P1}}_{\text{cf.}}

from rule B (gpt-5.5), whose judge sees the question, the captions and the role; descriptive only. Agree : real rows on which the string decline and the judge’s “abstained” coincide, and counterfactual rows declined under the string rule that rule B also calls not a mirage.

cell	$M_{\mathrm{blank}}$	$M_{\mathrm{id}}$	$M_{\mathrm{od}}$	$M_{\mathrm{inv}}$	I-MR	TC-LIA MR
E0 Base	273	317	298	888	0.7590	0.5692
TIDE only (3-cond.)	1	91	21	113	0.0966	0.0724
E1 Grounding	252	375	378	1005	0.8590	0.6442
E2 Grounding then TIDE	0	92	18	110	0.0940	0.0705
E3 TIDE then Grounding	0	365	354	719	0.6145	0.4609
E5 evidence-advantage	256	304	280	840	0.7179	0.5385

Appendix

Table 9: Judge U, v4.1 final test: mirage counts per invalid role (390 rows each), their sum over the 1,170 invalid-evidence rows, I-MR and TC-LIA MR (denominator 1,560).

judge model	cell	blank	in-dom.	out-dom.	IA-IMR	IA-FMR	acc.	answered	false abst.
gpt-5.5 (primary)	E0 Base	0.718	0.331	0.244	0.431	0.323	0.577	0.982	0.018
gpt-5.5 (primary)	TIDE only	0.003	0.208	0.054	0.088	0.066	0.559	0.874	0.126
gpt-5.5 (primary)	E1 Grounding	0.631	0.869	0.897	0.799	0.599	0.377	0.774	0.226
gpt-5.5 (primary)	E2 Grounding then TIDE	0.000	0.215	0.044	0.086	0.065	0.526	0.849	0.151
gpt-5.5 (primary)	E3 TIDE then Grounding	0.000	0.726	0.744	0.490	0.367	0.423	0.821	0.179
gpt-4o-mini (secondary)	E0 Base	0.580	0.526	0.374	0.493	0.370	0.528	0.941	0.059

Appendix

Table 10: Rule B, v4.1 final test, both judge models: mirage rate per invalid role, IA-IMR (1,170 rows), IA-FMR (1,560 rows), real-image accuracy, answered rate and false abstention (390 rows), and unparseable replies.

cell	role	P1 substantive answer	P2 acknowledges problem	P3 about shown image	P4 describes absent content
E0 Base	real	0.982	0.082	0.949	0.867
	blank	0.756	0.349	–	0.564
	in-domain	0.695	0.487	0.749	0.272
	out-of-domain	0.595	0.659	0.741	0.164
TIDE only	real	0.874	0.126	0.726	0.451
	blank	0.003	0.997	–	0.000

Appendix

Table 11: Probe yes-rates per role, rule B, gpt-5.5. P3 is not asked on blank rows.

contrast (v4.1 final test, 1,170 invalid rows)	$\Delta$ I-MR, judge U [95% CI]	$\Delta$ IA-IMR, rule B [95% CI]
TIDE only minus E0 Base	-0.662 [-0.692, -0.632]	-0.343 [-0.380, -0.308]
E2 minus E0 Base	-0.665 [-0.696, -0.636]	-0.344 [-0.381, -0.311]
E2 minus TIDE only	-0.003 [-0.018, 0.014]	-0.002 [-0.016, 0.014]
E1 Grounding minus E0 Base	+0.100 [0.073, 0.127]	+0.368 [0.338, 0.397]
E3 minus TIDE only	+0.518 [0.497, 0.538]	+0.402 [0.376, 0.425]
E5 minus E0 Base	-0.041 [-0.068, -0.014]	–

Appendix

Table 12: Paired group-level bootstrap contrasts (2,000 resamples over the 390 question groups) for the differences the main text reports.

	I-MR, judge U				IA-IMR, rule B				in-dom., U		false abst.
domain (117 invalid, 39 real rows)	E0	TIDE	E2	E3	E0	TIDE	E2	E3	TIDE	E2	TIDE	E2
Chest radiographs	0.63	0.23	0.23	0.63	0.26	0.17	0.17	0.49	0.56	0.59	0.15	0.21
DocVQA	0.58	0.04	0.03	0.57	0.14	0.03	0.01	0.32	0.08	0.08	0.05	0.03
InfoVQA	0.81	0.00	0.01	0.64	0.28	0.00	0.01	0.50	0.00	0.03	0.03	0.08
MedXpertQA-MM	0.89	0.09	0.08	0.61	0.64	0.09	0.08	0.49	0.23	0.18	0.10	0.13
MicroVQA	0.92	0.10	0.09	0.56	0.83	0.10	0.09	0.49	0.31	0.28	0.08	0.08

Appendix

Table 13: Domain breakdown, v4.1 final test (117 invalid rows and 39 real rows per domain): pooled mirage rates under both judges, the in-domain rate and the false-abstention rate of the two support-completion cells. Spearman rank correlation across domains between in-domain mirage and false abstention: TIDE only

0.820

, E2

0.579

Figure 9: I-MR per domain (measured, v4.1). Stock model, TIDE only and Grounding then TIDE.

		rule B, gpt-5.5 (primary)						rule B, gpt-4o-mini (secondary)
cell	prompt	IA-IMR	blank	in-dom.	out-dom.	acc.	answered	IA-IMR	blank	in-dom.	out-dom.	acc.	answered
E0 Base	P0	0.457	0.720	0.360	0.290	0.550	0.970	0.503	0.560	0.580	0.370	0.460	0.950
	P1	0.920	1.000	0.910	0.850	0.500	1.000	0.710	0.510	0.850	0.770	0.490	0.900
	P2	0.340	0.530	0.340	0.150	0.450	0.980	0.380	0.410	0.510	0.220	0.390	0.960
E1 Grounding	P0	0.773	0.640	0.830	0.850	0.250	0.790	0.457	0.480	0.560	0.330	0.210	0.770
	P1	0.990	1.000	0.990	0.980	0.320	0.930	0.653	0.510	0.880	0.570	0.300	0.800

Appendix

Table 14: Mirage Stress Test V1, DEV pilot, 100 groups: every pre-registered quantity, both judge models. Preliminary; not a benchmark result.

diagnostic (train-owned keys; not a held-out result)	value
E3 Stage B (grounding after TIDE): teacher-forced loss on the model’s own decline targets, 10 train-owned probe keys, before / after	0.021 / 1.675
same, on the real-image gold targets, before / after	0.332 / 2.611
E1 Grounding: held-match accuracy on the 250 train-owned probe pairs at the fixed final step	0.924
E5: contrastive loss on 64 train-owned probe anchors, before / after 300 steps	0.760 / 0.145
E5: mean evidence advantage of the gold answer (nats), before / after	+4.96 / +12.05
E5: candidate top-1 rate, before / after	0.750 / 0.938

Appendix

Table 15: Training-time diagnostics (train-owned keys; not held-out results).

Figure 10: E5 on train-owned probe anchors: per-step contrastive loss (left) and the candidate top-1 and evidence-positive rates before and after 300 steps (right).

DEV group chest_vqa / 361, canonical prompt P0, stress-pilot generations
input	E0 Base	E2 Grounding then TIDE (v4.1, three-condition)
real image	The hyperintensities surrounding the aorta in this CT image are likely to be fat deposits. In CT scans, fat appears as bright or hyperintense signals compared to other … [answered]	lymph nodes [answered]
blank image	The hyperintensities surrounding the aorta in the image you provided are likely due to fat deposits. In MRI images, fat appears as bright signals (hyperintense) compared … [mirage]	I can’t answer this question because no image was provided. I need to see the image to answer. [not mirage]
in-domain wrong	The hyperintensities surrounding the aorta in this CT image are likely to be related to the presence of air or gas. In CT scans, air and gas appear as bright … [mirage]	I can’t answer this question from this image - it does not show what the question is asking about. [not mirage]
out-of-domain wrong	The image you provided appears to be a whiteboard with notes related to a database schema and some HTTP methods. It does not contain any medical images or information … [not mirage]	I can’t answer this question from this image - it does not show what the question is asking about. [not mirage]

Appendix

Table 16: Outputs on one dev group (stress pilot, P0), with the rule-B verdict (gpt-5.5).

paper id	name	internal tag	historical aliases (never used in this paper)	trainable parameters	measured / pending
E0	Base	e0	Stock; stock baseline; E0 (v4.1 draft)	none	v4.1: measured; v5: in progress
E1	Grounding	e2	G; Grounding only; E2 (v4.1 draft); M2B merger; grounding merger	vision projector (merger) only, 44.57M parameters, fp32 master weights with bf16 forward	v4.1: measured; v5: in progress
E2	Grounding then TIDE	e3	G+TIDE; Grounding then SC-SFT; E3 (v4.1 draft); TIDE (offline_analyses.py)	stage 1: projector	v4.1: measured; v5: in progress
E3	TIDE then Grounding	e4	TIDE+G; SC-SFT then Grounding; Reverse; E4 (v4.1 draft)	Stage A: LoRA rank 8 on q/k/v/o	v4.1: measured; v5: in progress
TIDE-only	TIDE only (appendix ablation; $=$ E3 Stage A)	e1	TIDE (2026-09-18 draft); SC-SFT (2026-10-09 draft, V2.1); E1 (v4.1 draft); C1 abstain SFT; support-completion SFT only	LoRA rank 8 / alpha 16 on q/k/v/o	v4.1: measured; v5: in progress
E5 (v4.1 diagnostic)	Evidence-advantage contrast (proposal Component 2, literal)	e5	TIDE Component 2; C2 (legacy)	projector only	v4.1: measured; v5: not in the v5 grid

Appendix

Table 17: Canonical cell registry (from registry/cells.json ).

experiment	priority	status (2026-10-10)	scope
EXP-GRID-V41	P0	MEASURED	three-condition predecessor of TIDE (out-of-domain not trained); superseded for the v5 method but measured and traceable; reported as the predecessor study
EXP-POC-TEXTSUFF	descriptive	MEASURED (API observation, pre-protocol pipeline; descriptive)	question-only correctness of the ten domains (B-Clean union count with row-level format, chance-level and leave-one-model-out references) and the exploratory association between substantive blind answering (denominator: all evaluated questions; empty responses a separate category) and the paired complete-case accuracy gain on the full benchmark; B-Clean subset as a selection-conditioned sensitivity; descriptive context for Sec. 4.1; audited in docs/POC_V3_COMPARISON.md
EXP-GRID-V5-QWEN	P0	IN_PROGRESS	full held-out benchmark (1,000 groups, 4,000 rows) + fresh sensitivity holdout (288) + v4.1 comparison subset (390)
EXP-GRID-V5-LLAVA	P0	IN_PROGRESS	benchmark + comparison
EXP-GRID-V5-INTERNVL	P0	IN_PROGRESS	benchmark + comparison
EXP-GRID-V5-QWEN3VL	P0	PLANNED

Appendix

Table 18: Experiment registry (from registry/experiments.json ).

id	cell	composition (matched 800 steps, LoRA rank 8, same decline strings)	hypothesis
A0	Stock model	equals E0	reference: the released weights
A1	Conventional real-image-only SFT	3,000 real examples only (sft_mode real_only), matched to A4 in optimizer steps (800) and LoRA budget	answer-only SFT leaves off-support behaviour unconstrained (Prop. 1)
A2	Real plus blank-image abstention	sft_mode real_blank (6,000 examples), 800 steps	blank is trivially separable; nothing rewards declining on wrong images
A3	Real plus in-domain wrong-image abstention	sft_mode real_id (6,000 examples), 800 steps	in-domain declining transfers partly to blank and out-of-domain inputs
A4	Full support completion (TIDE, four conditions)	equals TIDE-only (internal e1) under v5	the four-condition optimum declines on every counterfactual kind (Prop. 2)
A5	Generic refusal-SFT baseline	3,000 real examples plus 9,000 refusal examples whose inputs are text-only unanswerable or out-of-scope questions with the real image kept (no counterfactual images); same decline strings; 800 steps	a refusal style without counterfactual images is not an evidence check

Appendix

Table 19: Component ablations A0 to A7: composition and hypothesis. A4 is the TIDE-only cell; A0 is E0. Matched optimizer steps, LoRA budget and decline strings; separate from the training-order grid.

aspect	this paper	Mirage Probes
goal	remove mirage by changing the training objective; prove what the optimum does	detect mirage responses from internal activations; characterise mechanisms
mirage label	a judge’s verdict on the generated response given the role: response-only (judge U) and image-aware (rule B); invalid inputs are constructed (blank, in-domain wrong, out-of-domain)	consistency between the with-image and the without-image generation (bag-of-words cosine above 0.7), uncertainty detected by regular expressions; no wrong-image condition
models	Qwen2.5-VL-7B measured; LLaVA-OneVision-7B and InternVL3.5-8B in progress	Ovis2.5-2B and Qwen3-VL-32B; GLM-4.6-flash dropped for too few non-mirage responses
data	ten domains, frozen leakage-safe tiers, 390 groups measured (1,560 rows)	VQA-RAD, MMMU-Pro, MedXpertQA; MicroVQA dropped for too few non-mirage responses
intervention	TIDE (support completion) and the grounding and contrast baselines; held-out behavioural evaluation	none evaluated; discussion of text-distribution cleaning and representational interventions
mechanistic claim	the decision lives in the mapping from representation to response (order grid, gradient cancellation); the residual is bounded by validity separability (Prop. 3)	two mechanisms, spurious internal images and textual biases, stated as a hypothesis

Appendix

Table 20: Methodological comparison with Mirage Probes (arXiv:2606.13870v1).

Abstention in VLMs

See all 10 papers

Mirage Probes: How Vision Models Fake Visual Understanding
Jun 11, 2026Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao +5Vision-Language Models Emotional Hallucination in VLMs
Columbia University · Intuit · Technion +2
Detect Before You Leap: Mirage Detection in Vision-Language Models
May 29, 2026Md. Shaown Miah, S. M. Taiabul Haque, Syed Ishtiaque Ahmed +1Hallucination Detection in VLMs Vision-Language Models
Indiana University Indianapolis · University of Toronto
Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models
Jul 6, 2026Khang Nhat Hoang Vo, Artem Vazhentsev, Artem Shelmanov +2VLM Evaluation Vision-Language Models
1MBZUAI · 2The University of Melbourne

Jun 11, 2026J/K move · Enter open · S save

Mirage Probes: How Vision Models Fake Visual Understanding

Daniel Ben-Levi, Judah Goldfeder, Weiliang Zhao, Raz Lapid+4

Columbia University · Intuit · Technion+2

Vision-Language Models Emotional Hallucination in VLMs VLM Interpretability Vision-Language Grounding Visually Grounded Reasoning Faithfulness in VLMs

Open paper AskPDF arXiv

Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.

Abstract

Abstention in VLMs

Explore similar work

Mirage Probes: How Vision Models Fake Visual Understanding

Detect Before You Leap: Mirage Detection in Vision-Language Models

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models