Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust

Faisal Mohammad; Mohamed Younis

cs.CVOct 15, 2026

Shift-Reliance Alignment: Ranking Vision-Language Models by the Cues They Trust

Authors: Faisal Mohammad, Mohamed Younis

Organizations: University of Maryland, Baltimore County · Johns Hopkins University Applied Physics Laboratory

Abstract

Vision-language models (VLMs) are usually chosen from leaderboards, which give one ranking for every application. On questions about objects that a scene suggests but does not contain, however, the model with the lowest error on the POPE hallucination benchmark is not the one with the lowest error on such questions. For yes/no questions about object presence, we show that the best model for a target domain can be predicted without running any candidate on that domain, given 650 to 1,200 labeled target items that calibrate a detector. We measure each model's reliance on a cue, a feature such as scene context or text in the image, by editing 100 labeled source images so that the cue suggests the wrong answer, and comparing with a matched control edit. Reliance predicts which VLMs are misled by physical sticker attacks (Spearman .93 on nine models not used to design the score) and which hallucinate on unedited COCO photographs, beyond what their tendency to answer yes explains. Shift-Reliance Alignment (SRA) scores a model by its source error on the labeled source questions plus its reliance multiplied by the shift. The shift is the proportion of target questions on which the cue misleads, and a detector estimates it on the unlabeled target images. SRA selects the model with the lowest score. In pre-registered tests on new images where most questions mislead, it lowered the error of the selected model from 30.4% under selection by source error to 18.8% on Objects365 and OpenImages but, against its prediction, not on COCO. On CueDial, a benchmark we release that varies how often a cue misleads, selection by model confidence falls more than 40 percentage points behind the best model when overlaid text denies a visible object, and SRA stays within 3 points.

Figures & tables

Figure 1: The same models rank differently on two kinds of test set. (a) An uncurated test set asks about random objects, here a present TV and an absent fork, and the $18$ models used to develop the score ( Sec. 4.1 ) ranked by error on $32$ uncurated test sets. (b) A plausible test set asks only about objects the scene suggests, half of them absent, here a cellphone and a remote that are not there, and the same models ranked by hallucination on one plausible test set, Spearman $-.15$ between the two rankings.

Figure 2: How SRA chooses between two models on Waterbirds [ 64 ] , Qwen2.5-VL-3B ( $A$ ) and LLaVA-1.5-7B ( $B$ ). (a, b) Misleading text changes $A$ ’s answer but not $B$ ’s, and a misleading hint in the prompt changes $B$ ’s answer but not $A$ ’s. (c) The reliance of each model and the shift $\hat{\mathbf{d}}=(1,.5,0)$ of a target domain over background, image text, and prompt. Each unit of shift along $\hat{\mathbf{d}}$ , with unit direction $\mathbf{u}$ , adds $.31$ error to $A$ and $.04$ to $B$ . (d) Predicted error grows from each source error along $\hat{\mathbf{d}}$ , the two lines cross at $t^{*}$ , and no single ranking is right on both sides of it. SRA selects $B$ , whose realized error is lower ( $.488$ against $.538$ , Appendix B ).

Figure 3: The SRA pipeline on COCO test sets of mostly misleading questions. (0) Partners, the categories most associated with the queried object, are mined from COCO and define the misleading questions ( Sec. 3.2 ). (1) Reliance is a candidate’s error on labeled source images after the object is erased, minus its error on an unrelated absent object in the same image ( Sec. 3.3 ). (2) A detector flags target questions whose object appears missing, and its error rates, measured on labeled items disjoint from the test sets, correct the flagged proportion ( Sec. 3.4 ). The marks ✓ and $\times$ show whether the detector is right, which the method never sees. (3) SRA selects $B$ although $A$ has the lower source error ( Eq. 1 ).

	CueDial	POPE [ 40 ]	AMBER [ 78 ]	DASH-B [ 3 ]	SCAM [ 81 ]	Ranked from Within [ 73 ]	LOVM [ 94 ]
Controlled proportion of misleading questions	✓	$\circ$	–	–	$\circ$	–	–
Cue types	text, context	context	context	context	text	–	–
Source datasets	6	3	1	web images	1	9 (visual question answering)	23
Answer formats	yes/no	yes/no	yes/no, free text	yes/no	two-way choice	choice, open	classification
Model answers per item	✓ ( $40$ models)	–	–	–	–	–	$\circ$
Scores	selectors	models	models	models	models	rankers	rankers, estimators

Table 1: CueDial against related benchmarks. Only CueDial sets the proportion of misleading questions by design, releases every model’s answer on every item, and scores selectors across that proportion.

\circ

marks partial support, since SCAM pairs each sticker photo with a clean copy, POPE’s adversarial split raises the proportion without setting it, and LOVM releases accuracies, not answers.

Figure 4: Three CueDial questions from three families , with how many of the $30$ candidates answer each wrongly. Below, a test set for the text cue at $d=0$ , $.25$ , and $.5$ , shown with $20$ questions. Half the answers are yes.

Figure 5: Reliance from source edits predicts failure on existing benchmarks. (a) DASH-B’s

1{,}341

images without the queried object. (b) SCAM’s physical stickers. Hollow points are the nine held-out models ( Sec. 4.1 ), and LLaVA-1.5 cannot read the pasted label.

type	sets	images and questions	misleading
uncurated	$32$	random objects, four corpora	${\approx}.02$
constructed	$10$	edited AMBER and COCO images	$.50$
COCO misleading	$4$	train2017, absent, partners present	$.80$
OpenImages	$2{+}3$	validation split, misleading and ordinary	$.80$ , ${<}.05$
Objects365	$4{+}3$	one train shard, misleading and ordinary	$.80$ , ${\leq}.01$
SCAM	$6$	sticker photos, clean and pasted copies	varies

Table 2: The

87

test sets and the proportion of their questions that mislead.

2{+}3

means misleading plus ordinary test sets, the latter about random objects of the same corpus,

4{+}2

plausible plus ordinary ones, and – a proportion not measured. CueDial’s test sets, the

20

pre-registered test sets, and the six calibrated DASH-B test sets of Sec. E.10 are not among the

87

( Appendices G and E.3 ).

Figure 6: Reliance measured on edits predicts hallucination on unedited images, and reliance on one cue says little about reliance on the other. (a) Hallucination on the COCO hallucination set with each model’s answer bias subtracted,

18

VLMs, with a least-squares fit and bootstrap band. (b) Text reliance

R_{\mathrm{text}}

on overlays that deny the object against context reliance

R_{\mathrm{ctx}}

for the same models, colored as in (a) with OneVision under Qwen, its backbone (Spearman

-.34

95\%

confidence interval (CI)

[-.76,.20]

never runs the model on the test images
	COCO $\uparrow$	DASH-B $\uparrow$		SCAM stickers $\uparrow$
score	halluc. ( $18$ )	all $30$	$9$ held-out	all $30$	$9$ held-out
answer bias	$+.53$	$+.70$	$+.98$	$+.36$	$+.55$
error on present objects	$-.55$	$-.52$	$-.92$	$-.08$	$-.42$
reliance $R_{f}$ (ours)	$+.78$	$+.59$	$+.80$	$+.76$	$+.93$
$R_{f}$ , partial (ours)	$+.77$	$+.47$	$-.03$	$+.73$	$+.90$

Table 3: Reliance predicts hallucination without using the test images. Spearman correlation between a per-model score and the hallucination rate on the COCO hallucination set (

18

development models), the false “yes” rate on DASH-B, or the rise in false “yes” answers under SCAM’s stickers. Reliance is context reliance, and on SCAM reliance on a pasted word naming another object. The partial row holds answer bias fixed. Error on present objects correlates negatively because it rewards models that answer yes readily. Partials over nine models are fragile. Red marks a correlation at or below zero.

	test-set type			misleading test sets
selector	uncurated	constructed	plausible	COCO	Open- Images	Objects365
source error	$1.6$	$10.9$	$5.7$	$2.3$	$13.6$	$12.5$
label-shift reweighted	$1.6$	$10.9$	$5.7$	$4.9$	$21.2$	$13.0$
POPE error	$1.3$	$14.4$	$7.2$	$3.4$	$21.2$	$13.4$
fixed POPE choice	$4.5$	$13.2$	$2.2$	$2.5$	$0.8$	$0.4$
least answer bias	$5.0$	$13.6$	$4.3$	$3.8$	$7.2$	$6.4$

Table 4: SRA has the lowest regret on the constructed test sets. Mean regret in points,

21

candidates, over the

32

uncurated,

10

constructed, and

6

plausible test sets of Tab. 2 and over the misleading test sets of each corpus. The reference block runs every candidate on the test set, and the best single model, Qwen3-VL-4B, is never green. Table 14 adds the mixed and POPE test sets.

Figure 7: SRA switches to a different model once misleading questions are common, and on OpenImages and Objects365 it then stays close to the best model. Regret of each selector on test sets resampled from each corpus’s misleading test sets and calibration items at each proportion of misleading questions,

21

candidates, with SRA given the true proportion. In the gray regions nearly all items share one label.

Figure 8: Under overlays that deny a visible object, confidence-based selectors fall far behind and SRA does not. Mean regret on CueDial over

30

test sets at each proportion of misleading questions,

30

candidates and audited labels. SRA estimates that proportion with OCR in (a–d) and is given it in (e–h), whose corpora are those of Fig. 7 plus LVIS. (c, d) Physical and pasted stickers, where the curves of SRA and source error overlap.

		Tab. 2 sets		new images
corpus	selector	error $\downarrow$	regret $\downarrow$	error $\downarrow$	regret $\downarrow$
	source error	$6.4$	$2.3$	$6.6$	$2.4$
	avg. conf.	$4.8$	$0.6$	$5.3$	$1.2$
	labeled items	$4.8$	$0.7$	$5.3$	$1.2$
	random	$7.8$	$3.7$	$7.7$	$3.6$
COCO	SRA (ours)	$6.6$	$2.5$	$6.4$	$2.3$

Table 5: On test sets where

80\%

of questions mislead, SRA selects close to the best model on OpenImages and Objects365 and ties source error on COCO.

21

candidates, error in percent, regret in points. Italic rows use target data, namely average confidence on the test set and selection by error on

400

labeled items drawn from the test sets themselves (none where the sets hold fewer). Labeled items drawn like the test sets ( Tab. 13 ) trail SRA by

1

4

points at

100

items or fewer on OpenImages and Objects365, come within

0.4

points at

650

, and beat it on COCO from

100

on. On new images, only the choices of SRA and source error were fixed in advance, and SRA’s reduction over source error has a

95\%

CI above zero on OpenImages and Objects365 and spanning zero on COCO, and pooled over these two corpora’s new images, error falls from

30.4\%

18.8\%

Figure 9: Without running any candidate on the target, SRA matches the nuclear norm and leads the rest in worst-case mean regret. Each selector’s largest

W_{s}

over the eight CueDial families (named above each bar), public split,

30

candidates, SRA given the true proportion on the context families. Post hoc item-level bootstrap intervals separate SRA from every selector but the nuclear norm here, and on the hidden split the nuclear norm’s mean

W_{s}

over families is lower by

1.7

points (

[0.6,2.9]

, Appendix G ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	graded		constructed
selector	regret $\downarrow$	$\tau$ $\uparrow$	regret $\downarrow$	$\tau$ $\uparrow$
clean accuracy	$.020$	$+.07$	$.027$	$+.31$
mean reliance	–	–	$.027$	$+.45$
random	$.215$	–	$.234$	–
reliance term only (ours)	$.016$	$+.62$	$.023$	$+.73$

Appendix

Table 6: Waterbirds drop prediction and selection. The graded columns give the regret, in drop units, of the model with the smallest predicted drop and the pooled Kendall

\tau

between predicted and realized drops, over the informative graded targets (those whose realized drops have a standard deviation across models of at least

.10

, the text and prompt targets at proportions

.75

and

1

). The constructed columns give the regret and the mean per-target

\tau

over the four informative constructed targets (text, background and text, prompt, background and prompt). Dashes mark selectors we did not evaluate in a setting. Our row is the reliance term alone. Green marks the best value per column, and yellow the reliance term where it is not the best.

variant	Pearson $\uparrow$	Spearman $\uparrow$
raw	$+.75$	$+.78$
answer bias subtracted	$+.89$	$+.86$
residuals given bias	$+.88$	$+.81$
leave-one-family-out	$+.69$ – $.77$	$+.73$ – $.79$
within-family only	$+.62$	$+.78$

Appendix

Table 7: Ablations of the prediction result (

18

VLMs, reliance measured by edits against hallucination on unedited images). The bootstrap

95\%

CI for the Pearson correlation with answer bias subtracted is

[.77,.96]

, and resampling clustered by model family gives

[.78,.96]

. The leave-one-family-out row gives the range over families. Residuals are those of each variable after regressing on answer bias. The leave-one-family-out and within-family rows use the raw outcome, and families group the two OneVision models with Qwen, their language backbone. Green marks the column maximum, and yellow the chosen variant where it is not.

reliance estimate	interaction $r$ $\uparrow$	precision $\uparrow$
single template	$.37\,/\,.53$	$.41\,/\,.33$
averaged over four templates	$.64\,/\,.76$	$.52\,/\,.46$

Appendix

Table 8: Reversal test outcomes on the AMBER test sets (left of each slash) and the constructed COCO test sets (right). Interaction

r

is the correlation with observed error after removing model and test-set means, using the shift estimated from unlabeled images. Precision uses the known composition.

95\%

bootstrap CIs over models for precision on AMBER are

[.21,.58]

and

[.32,.72]

(rows in order), and for interaction on constructed COCO

[+.03,+.79]

and

[+.49,+.89]

. The permutation nulls for precision are

.16

and

.14

(

95

th percentile

.22

on AMBER). Green marks the best value in a column.

Figure 10: Text reliance depends on how the text is drawn. Each column is one overlay template (boxed bold “no obj here”, red banner sentence, bare label “obj: absent”, plain box sentence), and each row is one model, sorted by mean. The bare label is the weakest template for most models, and full sentences mislead many more. The Qwen family responds to every template, the bare label included.

Figure 11: The reliance term orders the models once misleading questions are common. Spearman correlation within each test set of the reliance term with realized error against the estimated shift (type means,

95\%

whiskers). Above a shift of about

0.2

it orders the models, and source error orders them on POPE.

	OpenImages	Objects365
split	validation	one train shard
query vocabulary	$25$ categories a COCO object detector detects	$29$ mapped categories
per category	$\geq 50$ verified present, $\geq 50$ verified absent	$\geq 40$ misleading, $\geq 40$ present
partners mined over	$423$ categories	$365$ categories
calibration set	$400$ ( $150$ misleading, $250$ present)	$400$
random absent items	$250$	$250$

Appendix

Table 9: Protocol on the two new corpora. For Objects365 we fixed one shard of the train split, chosen from annotation counts alone. The random absent items give the detector’s signal rates on that item type.

test set	$\hat{d}$	best	SRA	source	$\rho$
Objects365 A	$.65$	$.175$	$.180$	$.310$	$+.75$
Objects365 B	$.65$	$.170$	$.180$	$.285$	$+.52$
Objects365 C	$.63$	$.175$	$.180$	$.265$	$+.68$
Objects365 D	$.41$	$.165$	$.165$	$.310$	$+.17$
OpenImages A	$.82$	$.175$	$.205$	$.320$	$+.52$
OpenImages B	$.67$	$.205$	$.205$	$.340$	$+.63$

Appendix

Table 10: Test on new images ,

21

candidates. The misleading test sets A–D and the mean over the three ordinary ones, with the detector’s estimated proportion of misleading questions

\hat{d}

(true proportion

.80

on A–D), the error of the best model and of the choices of SRA and source error, and the Spearman correlation

\rho

between the SRA score and the realized error. Predictions of the reduction of error over source error, pooled and within each corpus, of a mean

\rho

above zero within the misleading sets, and of the choices on the ordinary sets were fixed in advance and held, also with all

30

candidates (paired bootstrap over items,

B{=}10{,}000

test set	$\hat{d}$	best	SRA	source	$\rho$
COCO A	$1.00$	$.045$	$.080$	$.050$	$+.17$
COCO B	$.84$	$.025$	$.035$	$.050$	$+.58$
COCO C	$1.00$	$.040$	$.065$	$.065$	$+.58$
COCO D	$1.00$	$.045$	$.055$	$.070$	$+.54$
COCO E	$.84$	$.045$	$.075$	$.085$	$+.58$
COCO F	$.59$	$.050$	$.075$	$.075$	$+.33$

Appendix

Table 11: Test on new COCO images ,

21

candidates, columns as in Tab. 10 .

source error	SRA	sets	lower	reduction (points)
InternVL3-14B	Qwen2.5-VL-7B	$14$	$12$	$5.5$
InternVL3-14B	InternVL3-1B	$8$	$8$	$11.2$
InternVL3-14B	Qwen3-VL-2B	$6$	$6$	$2.1$
InternVL3-14B	tie of two	$1$	$1$	$8.0$

Appendix

Table 12: Model pairs behind the selections SRA changed ,

21

candidates. Mean error reduction in points over the test sets of each pair.

Figure 12: Selection regret against the number of labeled items on test sets where most questions mislead. Each variant refits on a random subsample of the labeled items,

1{,}000

draws per budget (one at the full budget), with bands from the 10th to the 90th percentile. The detector used on OpenImages and Objects365 also consumes random absent items, counted in its budget, so its curve extends to

650

. The dashed line is selection by source error, which uses no annotation. Ties are scored as the mean of the tied models. Reweighted accuracy also uses detector items that its budget does not count.

		labeled items
corpus	selector	$24$	$50$	$100$	$200$	$400$	$650$
	labeled items	$2.9$	$2.5$	$2.1$	$1.8$	$1.7$	$1.8$
	source error	$2.4$	$2.4$	$2.4$	$2.4$	$2.6$	$2.7$
COCO	SRA (ours)	$2.4$	$2.4$	$2.4$	$2.4$	$2.6$	$2.7$
	labeled items	$5.1$	$3.8$	$2.7$	$1.9$	$1.5$	$1.8$
	source error	$13.0$	$13.0$	$13.1$	$13.1$	$13.3$	$13.8$

Appendix

Table 13: Selection regret, in points, when the labeled items come from the test sets themselves .

21

candidates on the misleading test sets of each corpus, test sets of Tab. 2 and new images pooled. The labeled items are drawn at the test sets’ proportion of misleading questions, and every selector is scored on the remaining items,

1{,}000

draws per budget. Ties are scored as the mean of the tied models. The

650

column is the detector’s label budget on OpenImages and Objects365 ( Tab. 9 ). On COCO, whose detector uses

1{,}200

items, labeled items have regret

2.4

at that budget against

3.3

for SRA and for source error.

	test-set type (number of test sets)					misleading test sets
selector	uncurated ( $32$ )	constructed ( $10$ )	plausible ( $6$ )	mixed ( $4$ )	POPE ( $3$ )	COCO	OpenImages	Objects365
source error	$.016$	$.109$	$.057$	$.023$	$.010$	$.023$	$.136$	$.125$
label-shift reweighted	$.016$	$.109$	$.057$	$.023$	$.010$	$.049$	$.212$	$.130$
POPE error	$.013$	$.144$	$.072$	$.028$	$.005^{\ddagger}$	$.034$	$.212$	$.134$
least answer bias	$.050$	$.136$	$.043$	$.039$	$.036$	$.038$	$.072$	$.064$
random	$.043$	$.102$	$.050$	$.035$	$.028$	$.037$	$.111$	$.087$

Appendix

Table 14: Selection by test-set type, eight types. Mean regret,

21

candidates, as in Tab. 4 without the fixed POPE choice, adding the mixed and POPE test sets and average confidence and the nuclear norm after Platt scaling. Raw nuclear norm, which runs every candidate, never does worse than selecting by source error in any column of Tab. 4 , but does on POPE. POPE scores are in sample for selecting by POPE error ( ‡ ). Colors as in Tab. 4 , with the random row as the red threshold and the constant row never green.

selector	source labels	detector on target	run on target	target annot.
source error	✓	–	–	–
ATC, DoC	✓	–	all	–
COT	✓	–	all	yes/no balance
confidence-based rows of Tab. 16	–	–	all	–
model agreement, majority vote	–	–	all	–
accuracy, calibration set	–	–	calibration set	✓

Appendix

Table 15: What each estimator and SRA variant uses. Labeled source items, unlabeled target images read by the selector’s own detector, candidate answers on target items, and labeled target items (the calibration set).

selector (regret $\downarrow$ )	uncurated	constructed	plausible	mixed	POPE	COCO misleading	OpenImages	Objects365
source error	$.016$	$.109$	$.057$	$.023$	$.010$	$.023$	$.136$	$.125$
ATC	$.015$	$.147$	$.063$	$.013$	$.004$	$.023$	$.120$	$.098$
DoC	$.016$	$.045$	$.057$	$.023$	$.010$	$.023$	$.180$	$.136$
average confidence	$.019$	$.071$	$.018$	$.004$	$.013$	$.006$	$.120$	$.070$
entropy	$.017$	$.071$	$.019$	$.004$	$.013$	$.005$	$.120$	$.070$
MaNo, softmax only	$.017$	$.071$	$.015$	$.004$	$.013$	$.005$	$.120$	$.070$

Appendix

Table 16: Label-free selectors on eight test-set types ,

21

candidates, mean regret of each selector’s chosen model, with ties scored as the mean of the tied models. Inputs as in Tab. 15 . The SRA row repeats Tabs. 4 and 14 (

21

candidates) and the SRA row of Tab. 5 . The last two columns are the misleading test sets of the new corpora. The constant row, a reference rather than a selector, always selects Qwen3-VL-4B and is never colored green. The constant row is worse than a random selection on the constructed test sets (

.165

against

.102

). Selecting by POPE error always picks InternVL3-8B, the model with the lowest error on POPE’s three splits ( ‡ in sample). Green marks the best value in a column, red a regret above the random row’s, and yellow an SRA regret that is neither.

corpus	test sets	predicted	realized	audited	MAE	$r$
COCO	Tab. 2	$37.0$	$7.8$	$6.9$	$29.2$	$.53$
	new images	$35.2$	$7.7$	–	$27.5$	$.60$
OpenImages	Tab. 2	$32.1$	$28.7$	$19.2$	$4.6$	$.52$
	new images	$31.1$	$28.3$	–	$3.8$	$.56$
Objects365	Tab. 2	$31.3$	$26.4$	$20.4$	$5.2$	$.67$
	new images	$27.1$	$25.4$	–	$3.2$	$.60$

Appendix

Table 17: Predicted against realized error on the misleading test sets, mean over the

21

candidates, in percent. MAE is the mean absolute difference between predicted and realized error, and

r

the Pearson correlation over candidates. The audited column removes the items the label audit flagged, on the test sets of Tab. 2 , and MAE and

r

use the original labels.

family	source error	SRA	ATC	DoC	avg. conf.	nuclear	agreement
context, COCO	$.022$ / $.024$	$.056$ / $.065$	$.034$ / $.051$	$.038$ / $.044$	$.031$ / $.026$	$.031$ / $.025$	$.025$ / $.026$
context, LVIS	$.088$ / $.115$	$.053$ / $.068$	$.032$ / $.059$	$.030$ / $.057$	$.021$ / $.064$	$.021$ / $.060$	$.023$ / $.067$
context, OpenImages	$.096$ / $.081$	$.063$ / $.094$	$.038$ / $.050$	$.007$ / $.030$	$.094$ / $.055$	$.094$ / $.050$	$.091$ / $.049$
context, Objects365	$.045$ / $.059$	$.037$ / $.055$	$.066$ / $.069$	$.047$ / $.065$	$.040$ / $.025$	$.033$ / $.024$	$.033$ / $.040$
text, AMBER	$.167$ / $.124$	$.013$ / $.028$	$.406$ / $.424$	$.406$ / $.423$	$.315$ / $.051$	$.069$ / $.036$	$.172$ / $.148$
text, COCO	$.259$ / $.256$	$.027$ / $.012$	$.435$ / $.418$	$.434$ / $.418$	$.434$ / $.378$	$.053$ / $.026$	$.325$ / $.296$

Appendix

Table 18: Public and hidden splits. Worst-case mean regret

W_{s}

over proportions of misleading questions, public split / hidden split,

30

candidates, audited labels. SRA uses the proportion read by OCR on the text families and the true proportion on the context families. Green marks the lowest public

W_{s}

in a row, red a

W_{s}

more than

.10

above that of source error, and yellow SRA where it is neither.

family	source error	SRA	ATC	DoC	avg. conf.	nuclear	agreement
context, COCO	$.022$ / $.024$	$.049$ / $.049$	$.035$ / $.059$	$.049$ / $.046$	$.063$ / $.070$	$.058$ / $.068$	$.025$ / $.027$
context, LVIS	$.090$ / $.118$	$.043$ / $.062$	$.033$ / $.063$	$.041$ / $.061$	$.028$ / $.062$	$.028$ / $.062$	$.027$ / $.074$
context, OpenImages	$.096$ / $.081$	$.063$ / $.095$	$.038$ / $.047$	$.066$ / $.041$	$.047$ / $.047$	$.046$ / $.044$	$.085$ / $.078$
context, Objects365	$.045$ / $.060$	$.056$ / $.071$	$.066$ / $.073$	$.085$ / $.094$	$.103$ / $.085$	$.081$ / $.046$	$.036$ / $.039$
text, AMBER	$.167$ / $.124$	$.028$ / $.028$	$.406$ / $.424$	$.406$ / $.423$	$.317$ / $.307$	$.069$ / $.061$	$.186$ / $.189$
text, COCO	$.259$ / $.256$	$.027$ / $.012$	$.435$ / $.418$	$.434$ / $.418$	$.336$ / $.374$	$.060$ / $.026$	$.322$ / $.325$

Appendix

Table 19: CueDial over all

40

models , as in Tab. 18 . Green marks the lowest public

W_{s}

in a row, red a

W_{s}

more than

.10

above that of source error, and yellow SRA where it is neither.

test-set type	items	audited	flagged wrong	est. wrong
COCO misleading	$800$	$8$	$6$	$1.5\%$
mixed	$800$	$30$	$29$	$4.4\%$
Objects365 misleading	$800$	$63$	$54$	$7.5\%$
Objects365 plausible	$1{,}200$	$52$	$41$	$4.2\%$
Objects365 ordinary	$600$	$2$	$2$	$1.1\%$
OpenImages misleading	$250$	$31$	$24$	$10.3\%$

Appendix

Table 20: Label audit by test-set type. Items nearly all models answer against their label (audited), how many the auditor flags as wrong, and the estimated proportion of wrong labels in the type, which adds the control error rate of the type’s audit on the items not audited (

.008

for most types,

.054

for POPE and uncurated). The auditor flags

722

of the

837

audited items as wrongly labeled and

107

as unclear, against

6

wrong of the

218

random controls it could decide (

1

126

for most types,

5

92

for POPE and uncurated), and CueDial drops these

835

items.

error $\downarrow$ , original / audited	COCO	OpenImages	Objects365
source error	$.064$ / $.054$	$.312$ / $.214$	$.303$ / $.244$
SRA, detector calibrated (ours)	$.066$ / $.057$	$.184$ / $.091$	$.181$ / $.114$
SRA regret $\downarrow$	$.025$ / $.024$	$.008$ / $.014$	$.004$ / $.005$

Appendix

Table 21: Selections on the misleading test sets with audited labels. Error of each selector’s choice on the misleading test sets of Tab. 5 ,

21

candidates, with the original labels and after removing the items the audit flagged. The choices are those of Tab. 5 , and every model’s error falls, most on OpenImages. Green marks, per corpus, the lower error with the original labels, and yellow the SRA value where it is not the lower.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Jun 9, 2026

Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.

Pratham Singla, Shivank Garg, Vihan Singh +1

Lossfunk · Indian Institute of Technology Roorkee · Raeth AI

VLM Evaluation Vision-Language Models Vision-Language Grounding

Ask PDF

Jun 16, 2026

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

Multimodal Foundation Models are increasingly used as reasoning agents, making reliability, knowing when a model may hallucinate, critical. A common intuition, which we call the Attention-Confidence Assumption, holds that reliability follows from "structural" visual perception: tight attention on relevant regions should signal a trustworthy answer, while scattered attention signals confusion. We challenge this through the VLM Reliability Probe (VRP), a systematic cross-family study of reliability signals in contemporary Vision-Language Models (VLMs). We introduce structural-attention metrics, cluster counts (C_k) and spatial entropy (H_s), to quantify the visual encoder's gaze, and track its evolution (Delta H_s) across layers. This reveals a "Symbolic Detachment": models often "Early Lock" visual features only to diffuse attention later, severing early perception from final generation. Contrary to the grounding hypothesis, we find a "Cluster Failure": spatial attention has near-zero correlation (R approx 0.001) with accuracy. Instead, reliability is a phenomenon of generation dynamics and internal-state distributions. Self-Consistency, the agreement rate across sampled reasoning paths, is the dominant predictor of truth (R = 0.429). Scaling causal interventions exposes a sharp architectural divergence: LLaVA locks its prediction in a fragile late-stage bottleneck, whereas PaliGemma and Qwen2-VL distribute reliability globally, staying resilient even when ~50% or more of their most predictive layer is destroyed. For current VLMs, reliability signals are detached from visual grounding maps and are best inferred from generation-time dynamics and hidden-state probes.

Logan Mann, Yi Xia, Ajit Saravanan +6

Algoverse AI Research · University of California, Santa Barbara · University of California, Berkeley +1

VLM Evaluation VLM Robustness Vision-Language Models VLM Interpretability Confidence Estimation for VLMs

Ask PDF

May 5, 2026

Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits

A pervasive intuition holds that vision-language models (VLMs) are most trustworthy when their attention maps look sharp: concentrated attention on the queried region should imply a confident, calibrated answer. We test this Attention-Confidence Assumption directly. We instrument three open-weight VLM families (LLaVA-1.5, PaliGemma, Qwen2-VL; 3-7B parameters) with a unified mechanistic pipeline -- the VLM Reliability Probe (VRP) -- that compares attention structure, generation dynamics, and hidden-state geometry against a single correctness label. Three results emerge. (i) Attention structure is a near-zero predictor of correctness (R_pb(C_k,y)=0.001, 95% CI [-0.034,0.036]; R_pb(H_s,y)=-0.012, [-0.047,0.024] on a pooled n=3,090 split), even though attention remains causally necessary for feature extraction (top-30% patch masking drops accuracy by 8.2-11.3 pp, p<0.001). (ii) Reliability becomes legible later in the computation: a single hidden-state linear probe reaches AUROC>0.95 on POPE for two of three families, and self-consistency at K=10 is the strongest behavioral predictor we measure at 10x inference cost (R_pb=0.43). (iii) Causal neuron-level ablations expose a sharp architectural split with direct monitor-design implications: late-fusion LLaVA concentrates reliability in a fragile late bottleneck (-8.3 pp object-identification accuracy after top-5 probe-neuron ablation), whereas early-fusion PaliGemma and Qwen2-VL distribute it widely and absorb destruction of ~50% of their peak-layer hidden dimension with <=1 pp degradation. The takeaway is narrow but consequential: in 3-7B VLMs, reliability is read more reliably off hidden-state geometry, layer-wise margin formation, and sparse late-layer circuits than off attention-map sharpness.

Logan Mann, Ajit Saravanan, Ishan Dave +4

UC Santa Barbara · UC Berkeley · NVIDIA +2

VLM Evaluation VLM Interpretability Attention Mechanisms Causal Interpretability Confidence Estimation for VLMs

Ask PDF

Abstract

VLM Evaluation

Explore similar work

Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

Where Reliability Lives in Vision-Language Models: A Mechanistic Study of Attention, Hidden States, and Causal Circuits