DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting

Qiuyi; Zhang; Zhen Xu; Jing Gu; Seojin Bang; Arnav Mohanty Das; Yaning Hu; Gautam Kumar

cs.AIOct 26, 2026

DecoyCount: Evaluating Visual-Reasoning Misalignment in Counting

Authors: Qiuyi, Zhang, Zhen Xu, Jing Gu, Seojin Bang, Arnav Mohanty Das, Yaning Hu, Gautam Kumar,

Organizations: Richard · Elorian AI · Mercor

Abstract

Visual counting benchmarks report whether vision-language models produce the correct number, but not whether errors arise from perception or reasoning. We introduce DecoyCount, a benchmark built from 1,660 real-world images with designed decoys across seven mechanisms, along with paired prompts: a neutral question, a matched alignment cue, and a novel multi-hop, multi-image question that situates the same counting problem in a realistic scenario. Using this design, we find that counting failures are induced partly by visual misalignment: models can execute the intended counting process when cue-guided but fail to select it unaided, and we term this difference in accuracy the reasoning-alignment gap. Despite being highly capable on general reasoning, frontier models remain below 50% accuracy on our benchmark, while alignment gaps range from 11.8-22.1 percentage points; strikingly, we observe that more capable models, greater thinking budgets, and agentic processes often induce larger gaps. Because humans may bring context that models lack, we test whether textual underspecification contributes to misalignment using a scenario-based multi-hop condition without explicit hinting, but this added context does not improve aggregate performance. Our analysis concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning, leaving misalignment with human visual expectations as a substantial component of the remaining deficit.

Figures & tables

Figure 1: DecoyCount exposes interpretable counting failures across seven mechanisms. One item per family: the neutral question with its ground-truth count (blue) and pre-registered decoy (red), the alignment cue (italic) naming the intended counting process without revealing the answer, and an abridged contextual question (purple) that implies the rule through a scenario instead. Most contextual items also supply a second image, not shown here. The final panel summarizes the 2,203 decoys over 1,660 images.

Benchmark	Image Decoy	Taxonomy	Cue	Multi-hop	Multi-image
TallyQA ( 2019 )	—	simple / complex only	—	—	—
CountBench ( 2023 )	—	—	—	—	—
PixMo-Count ( 2025 )	—	—	—	—	—
CountQA ( 2025 )	—	—	—	—	—
VLMCountBench ( 2025 )	—	by composition level	generic	—	—
HoloCount ( 2026 )	—	20 skill subsets, 3 tiers	—	comparison only	—

Table 1: DecoyCount combines adversarial multimodal decoys, alignment cues, and realistic multi-hop, multi-image counting. Prior benchmarks cover individual elements; DecoyCount pairs 2,203 pre-registered decoys across seven mechanisms with per-item alignment cues and bridge questions, including 842 multi-image items. “Prior only” denotes a canonical-count decoy.

Figure 2: High capability does not imply a small reasoning-alignment gap. Solid bars show neutral accuracy and striped extensions show the gain from the matched alignment cue; bold outlines mark the best neutral score in each family. GPT-6-astra leads every family but retains large reasoning-alignment gaps in five of seven. The smaller Structure and Estimate gaps instead identify failures that guidance does not readily repair ( Section 5.1 ); exact values and decoy-hit rates are in Tables 12 and 11 .

Model	Neutral	Cued	multi-hop	Neutral	Cued	$\Delta R$
	Accuracy			Decoy-hit
GPT-6-astra	43.17	65.26	43.10	40.00	16.71	+22.09
Gemini-3.8-flash	32.71	44.48	32.75	36.83	21.00	+11.77
Qwen3.8-max	28.31	49.90	32.94	50.04	25.30	+21.59
Gemini-3.1-pro-preview	25.30	38.55	27.49	39.62	23.13	+13.25
Claude-Fable-5.1	25.22	38.01	27.49	44.34	27.03	+12.79

Table 2: The strongest neutral model also has the largest gap. GPT-6-astra leads neutral accuracy at

43.17\text{\,}\mathrm{\%}

but gains another 22.09 percentage points when cued; Qwen3.8-max shows a similarly large 21.59-point gap. Thus greater capability can improve what a model executes under guidance without aligning what it selects unaided. The multi-hop condition uses the valid paired subset and is not directly comparable with neutral or cued accuracy ( Appendix A ); bold marks column maxima.

Figure 3: Scale and inference effort improve accuracy without closing the reasoning-alignment gap. The model-size sweep (left) and thinking-token sweep (right) compare neutral and cued exact-match accuracy under controlled Qwen configurations. From 4B to 27B, cued accuracy grows faster than neutral accuracy and

\Delta R

widens by 4.52 points; enabling native thinking increases it by 11.84 points before it plateaus. Shaded distances and labels are

\Delta R

in percentage points; “native” denotes unbounded Qwen thinking.

Gemini-3.8-flash
Condition	Acc.	vs. neutral	% of matched	Decoy-hit	Compl. tok.
Neutral	32.71	—	—	36.83	5297
Generic cue	34.54	$+$ 1.83	0 16	33.21	6258
Mismatched cue	34.86	$+$ 2.15	0 18	32.27	5407
Matched cue	44.48	$\mathbf{+11.77}$	100	21.00	6263
GPT-6-astra

Table 3: Generic and mismatched cues recover at most 18% of the matched gain: improvement depends on naming the actual trap, not merely eliciting more effort. Tokens include reasoning.

System	Comparison	$\Delta R_{\rm base}$	$\Delta R_{\rm tool}$	$\Delta^{2}R$
Qwen3.8-Flash-Next	bare $\to+$ CI ∗	19.46	26.63	$\mathbf{+7.17}$
Qwen3.8-Flash-Next	$+$ CI $\to+$ CI $+$ SAM3	26.63	22.11	$\mathbf{-4.52}$
GPT-6-sol	no tools $\to+$ CI	19.34	21.93	$\mathbf{+2.59}$
Claude-Opus-5.5	no tools $\to+$ CI	11.97	14.45	$\mathbf{+2.48}$

Table 4: General agentic harnesses widen the gap; targeted SAM3 narrows it. Change on the full release (points): green narrows the gap; red widens it. CI is provider-native execution; ∗ also changes the agent. Details: Table 16 .

Figure 4: Scenario grounding resolves some ambiguity but not the full alignment gap. For each family, the top panel reports losses among neutral-correct items and rescues among neutral-wrong items; the bottom panel shows the resulting change in accuracy. Context rescues only 11.6–32.6% of neutral failures and helps most on Duplicate, where the scenario clarifies what counts as one usable object. Text, Structure, and Estimate decline, showing that added context does not generally replace a targeted alignment cue. “Lost” denotes neutral-correct

\to

multi-hop-wrong; “rescued” denotes neutral-wrong

\to

multi-hop-correct. Exact transition counts remain in Table 21 .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Family	How the decoy is induced	What the matched cue tests
Duplicate	One physical instance appears more than once through a reflection, screen, or depiction.	Whether the model distinguishes entities from their representations.
Text	A printed numeral offers an answer-like shortcut.	Whether the model counts visible objects rather than transcribing text.
Wrong Unit	Groups, containers, or subparts invite counting at the wrong granularity.	Whether the model resolves the unit requested by the question.
Prior	A familiar arrangement suggests a canonical count unsupported by the image.	Whether visual evidence overrides a memorized or expected answer.
Lookalike	Target-like distractors blur the category boundary.	Whether the model maintains the intended object class while counting.
Structure	A regular row, column, or lattice invites an incorrect structural shortcut.	Whether the model verifies the arrangement instead of extrapolating a pattern.

Appendix

Table 5: The taxonomy localizes where counting can fail. Each family pre-registers both an answer-like decoy and a targeted intervention, allowing the paired evaluation to distinguish the reasoning-alignment gap from residual perceptual and other reasoning failures.

Decoy family	Decoys	Items	Scene domain Items
Estimate	489	258	Everyday 337
Lookalike	369	291	Public & outdoor 274
Structure	303	257	Documents, screens & artwork 244
Duplicate	273	252	Charts, maps & scientific 226
Prior	268	254	Retail & product 202
Wrong Unit	268	228	Observation 194

Appendix

Table 6: Release composition (

n=1,660

). Every item has at least one decoy; 392 have multiple, yielding 2,203 registered decoys. Items contribute once to every family represented by their decoys, whereas scene domains are mutually exclusive.

Model	Acc.	Decoy-hit	Acc.	Repaired	Broken	$p$	released
	Neutral		Cued	Paired neutral $\to$ cued			Bites
GPT-6-astra	43.17	40.00	65.26	437	0 71	$2.6\times 10^{-65}$	66.7
Gemini-3.8-flash	32.71	36.83	44.48	311	116	$1.2\times 10^{-21}$	56.0
Qwen3.8-max	28.31	50.04	49.90	422	0 63	$2.8\times 10^{-66}$	55.3
Gemini-3.1-pro	25.30	39.62	38.55	354	134	$6.0\times 10^{-24}$	60.6
Claude-Fable-5.1	25.22	44.34	38.01	305	0 93	$2.0\times 10^{-27}$	50.6

Appendix

Table 7: Matched cues repair 2.6–6.7

\times

more items than they break and remove 43–67% of neutral decoy hits. Repaired and broken are paired flips;

p

is exact McNemar.

Thinking	Neutral run	Cued run	Neutral	Cued	$\boldsymbol{\Delta R}$
Off	48345	49270	12.65	21.61	8.96
1,920	48347	49271	18.86	36.93	18.07
6,048	48348	49272	25.04	44.88	19.84
30,000	48343	49269	26.33	48.03	21.71
Native	48342	49268	27.17	47.97	20.80

Appendix

Table 8: Qwen3.8 FlashNext effort sweep. Accuracy is in percent and

\Delta R

in percentage points, averaged over three rollouts per item. “Native” means unbounded Qwen thinking, so the gap peaks at the 30k cap rather than at the endpoint. Run IDs identify the complete paired artifacts in s3://elorian-eval .

Figure 5: Scale reduces miscellaneous errors faster than decoy-aligned errors. The stacked bars (left) partition neutral responses into correct, exact-decoy, and other-error outcomes; the line (right) reports the decoy share among all remaining errors. From 4B to 27B, accuracy rises by 5.30 points and other errors fall by 12.29 points, yet exact-decoy answers rise by 6.98 points. Accordingly, the decoy share of errors increases by 12.02 points (paired 95% CI [10.30, 13.74]).

Temperature	Neutral acc.	$\Delta$ vs. 1.0
0.3	28.05	$+0.88$
0.6	28.05	$+0.88$
1.0	27.17	—
1.5	11.93	$-15.24$

Appendix

Table 9: Neutral exact-match accuracy for Qwen3.8 FlashNext at native unbounded thinking, averaged over three repeats (runs 48349, 48350, 48342, and 48351). Temperature is flat through 1.0; at 1.5,

12.73\text{\,}\mathrm{\%}

of rollouts return no integer.

		Qwen3.8-max			GPT-6-astra
Provenance	$n$	Neutral	Cued	$\Delta R$	Neutral	Cued	$\Delta R$
Original photography	432	41.7 / 37.7	56.7 / 18.5	$+15.0$	53.0 / 28.0	70.1 / 12.0	$+17.1$
Public domain, as found	555	30.1 / 47.6	53.5 / 22.0	$+23.4$	41.1 / 41.4	67.2 / 13.3	$+26.1$
Edited	673	18.4 / 62.4	41.6 / 35.1	$+23.2$	37.6 / 48.7	61.7 / 22.3	$+24.1$
Count-matched against original photography (same $y$ ; $\Delta$ accuracy)
Public domain, as found	320	$-9.4$	$-7.8$		$-5.0$	$-5.0$

Appendix

Table 10: Accuracy / decoy-hit by provenance (

\mathrm{\%}

). Bottom rows count-match against commissioned originals; differences and

\Delta R

are in percentage points. Editing changes accuracy but not

\Delta R

	GPT-6-astra		Gemini-3.8-fl.		Qwen3.8-max		Gemini-3.1-pro		Fable-5.1		Kimi-k3
Family	N.	C.	N.	C.	N.	C.	N.	C.	N.	C.	N.	C.
Duplicate	61.24	17.86	36.38	17.72	61.11	24.21	66.67	30.69	38.49	16.14	60.71	30.42
Wrong Unit	51.90	20.61	38.30	22.66	58.77	27.49	43.86	29.53	52.05	34.36	54.39	34.21
Prior	48.56	22.18	49.74	27.43	68.50	40.29	40.16	24.28	72.97	46.85	73.88	52.10
Lookalike	46.05	21.08	35.74	19.36	47.19	26.23	44.90	28.29	39.29	27.61	46.05	32.42
Text	38.89	11.71	56.31	28.08	70.72	25.53	37.54	15.02	54.50	26.13	84.38	54.80

Appendix

Table 11: Per-family neutral/cued decoy-hit, six models and three repeats (

\mathrm{\%}

). Bold marks the largest neutral-to-cued reduction in each family.

Family	$n$	GPT-6	G-3.8-fl.	Qwen-max	G-3.1-pro	Fable-5.1	Kimi-k3	Mean
Duplicate	252	+43.92	+16.67	+35.98	+36.24	+22.75	+30.03	+30.93
Text	222	+22.22	+23.72	+36.94	+16.97	+17.27	+24.32	+23.57
Wrong Unit	228	+31.73	0 +7.75	+29.68	+16.52	+15.79	+21.93	+20.57
Prior	254	+26.12	+19.82	+20.73	0 +8.66	+17.85	+17.72	+18.48
Lookalike	291	+20.85	0 +8.13	+15.12	0 +9.97	0 +6.64	0 +7.79	+11.42
Structure	257	0 +2.20	0 +5.56	0 +9.56	0 +4.91	0 +6.20	0 +7.11	+5.92

Appendix

Table 12: Per-family

\Delta R

(percentage points). An item with decoys from multiple families contributes once to each applicable family; multiple decoys from the same family do not duplicate the item. Bold marks the cross-model mean used to order families.

Family	Neutral	Cued	Gap	Errors on decoy
Text	31.1	54.6	$+$ 23.6	82.8
Duplicate	30.2	61.1	$\mathbf{+30.9}$	77.5
Prior	22.7	41.2	$+$ 18.5	76.3
Wrong Unit	24.2	44.8	$+$ 20.6	65.8
Lookalike	28.6	40.0	$+$ 11.4	60.5
Structure	33.3	39.2	0 $+$ 5.9	48.6

Appendix

Table 13: Outcomes pooled over six models and three repeats (

\mathrm{\%}

). Bold marks the largest and smallest gap and decoy capture; the last column is the share of wrong neutral answers on-decoy.

Subset	$n$	Neutral	Visible	$\Delta$	Repaired	Broken
All items	1,660	32.89	39.58	+6.69	246	135
Duplicate	252	49.21	56.75	+7.54	35	16
Estimate	258	26.74	20.54	-6.20	18	34
Lookalike	291	33.68	38.49	+4.81	35	21
Prior	254	27.17	40.16	+12.99	47	14
Structure	257	32.30	31.52	-0.78	31	33

Appendix

Table 14: Global wording sensitivity of Gemini-3.8-flash on the frozen release (

\mathrm{\%}

). Family memberships overlap.

		Gemini-3.8-flash				GPT-6-astra
Decoy family	$n$	Neut.	Gen.	Mism.	Match.	Neut.	Gen.	Mism.	Match.
Duplicate	252	50.26	48.02	46.16	66.93	25.93	23.54	28.57	69.84
Wrong Unit	228	34.94	34.65	36.11	42.69	31.58	33.48	34.36	63.30
Prior	254	28.08	35.70	32.41	47.90	37.40	47.64	37.66	63.52
Lookalike	291	32.65	31.27	34.02	40.78	35.17	35.17	36.08	56.01
Text	222	29.58	40.69	41.29	53.30	53.45	64.56	63.36	75.68

Appendix

Table 15: Per-family accuracy under four cue conditions, two models and three repeats on the frozen release (

\mathrm{\%}

). Bold marks the matched cue, the best condition in every row.

Model	Arm	Neutral	Cued	$\Delta R$	Rollouts
Qwen3.8-Flash-Next	bare	28.31	47.77	19.46	1
Qwen3.8-Flash-Next	scaffold ∗	21.67	41.39	19.72	1
Qwen3.8-Flash-Next	$+$ CI	30.00	56.63	26.63	1
Qwen3.8-Flash-Next	$+$ CI $+$ SAM3	33.61	55.72	22.11	1
GPT-6-sol	no tools	27.31	46.65	19.34	3
GPT-6-sol	$+$ CI	35.72	57.65	21.93	1

Appendix

Table 16: All tool and no-tool experiments, with raw exact-match accuracy and reasoning-alignment gap (accuracy in percent; gap in percentage points). CI denotes provider-native code execution (GPT code_interpreter ; Claude code_execution ); bold marks the fixed-agent CI+SAM3 arm. ∗ The scaffold exposes no usable image tool; the earlier GPT-5.6 study reported only the full-set neutral arm.

Family	$n$	CI $\to$ CI+SAM3	GPT-6 CI	Opus CI
Estimate	258	$\mathbf{-10.8}$	$\mathbf{+1.2}$	$\mathbf{+1.6}$
Structure	257	$\mathbf{-8.6}$	$\mathbf{+4.1}$	$\mathbf{+11.8}$
Text	222	$\mathbf{-5.4}$	$\mathbf{+1.1}$	$\mathbf{+0.6}$
Prior	254	$\mathbf{-2.8}$	$\mathbf{+6.6}$	$\mathbf{+1.3}$
Lookalike	291	$\mathbf{-2.1}$	$\mathbf{+3.9}$	$\mathbf{+3.5}$
Wrong Unit	228	$\mathbf{-0.9}$	$\mathbf{-0.8}$	$\mathbf{+1.2}$

Appendix

Table 17: Tool-induced change in the reasoning-alignment gap by family (

\Delta^{2}R

, percentage points). CI

\to

CI+SAM3 holds the Qwen agent fixed; GPT-6 and Opus compare no tools with provider-native CI. Green narrows the gap; red widens it. Signed values preserve the distinction in grayscale. No-tools values are three-rollout means; tool arms are one rollout.

Figure 6: Tool effects depend on the agentic surface. Blue circles and orange squares show the tool-induced gains under neutral and cued prompts, respectively; their difference is

\Delta^{2}R

, the change in the reasoning-alignment gap. Adding SAM3 to the same CI agent narrows the gap by 4.52 points, and the coding-agent contrast narrows it by 2.99 points. The interpreter contrast widens it by 7.17 points but also changes the agent, so it is not a tool-only comparison.

Model	Condition	Mean	$\sigma$	Spread	Min. gap	Runs
GPT-6-astra	Neutral	43.17	0.64	1.27	1.79	3
	Cued	65.26	0.52	1.02	1.44	3
	multi-hop	43.10	0.10	0.19	0.27	3
Gemini-3.8-flash	Neutral	32.71	0.18	0.36	0.50	3
	Cued	44.48	0.54	0.96	1.50	3
	multi-hop	32.75	0.97	1.82	2.69	3

Appendix

Table 18: Run-to-run variance, three independent repeats per cell (

\mathrm{\%}

). Min. gap is

1.96\sigma\sqrt{2}

; the pooled

1.98\text{\,}\mathrm{\%}

threshold is used throughout.

Figure 7: DecoyCount preserves broad counting rankings while exposing substantial new headroom. Each model is positioned by its CountQA exact match (left) or FSC147 within-10% accuracy (right), with DecoyCount exact match on the vertical axis. Blue and orange points are the neutral and cued prompts; each connector therefore visualizes that model’s reasoning-alignment gap. Strong standard-counting scores coexist with low neutral DecoyCount accuracy, while the large cue lifts show that part of the deficit is alignment rather than counting capability alone.

Model	multi-hop acc.	$n$	multi-hop acc.	$n$	Gap
	Neutral count correct		Neutral count wrong
GPT-6-astra	69.97	0 679	21.87	0 859	$+$ 48.10
Gemini-3.8-flash	62.40	0 518	17.67	1020	$+$ 44.73
Qwen3.8-max	66.00	0 448	19.37	1091	$+$ 46.63
Gemini-3.1-pro	45.55	0 401	21.12	1138	$+$ 24.43
Claude-Fable-5.1	60.39	0 398	16.01	1141	$+$ 44.38

Appendix

Table 19: Accuracy under the multi-hop condition, conditioned on the paired neutral count, three repeats (

\mathrm{\%}

);

n=1539

(see Exclusions, Appendix A ).

Decoy family	GPT-6	G-3.8-fl.	Qwen-max	G-3.1-pro	Fable-5.1	Kimi-k3
Duplicate	$+$ 13.3	0 $+$ 5.4	$+$ 16.5	$+$ 24.6	0 $+$ 7.6	0 $+$ 9.0
Wrong Unit	0 $+$ 1.9	0 $-$ 3.0	0 $+$ 6.4	0 $+$ 8.5	0 $+$ 1.6	0 $+$ 5.8
Lookalike	0 $+$ 0.6	0 $+$ 1.9	0 $+$ 4.2	0 $+$ 5.7	0 $+$ 4.5	0 $+$ 1.6
Text	$-$ 15.7	0 $-$ 6.3	0 $-$ 2.2	$-$ 15.2	0 $-$ 9.5	0 $+$ 1.4
Prior	0 $-$ 0.4	0 $+$ 0.1	0 $+$ 3.9	0 $-$ 7.0	0 $+$ 2.8	0 $+$ 3.9
Structure	0 $-$ 7.7	0 $-$ 3.5	0 $-$ 0.6	0 $-$ 5.0	0 $+$ 1.5	0 $+$ 2.2

Appendix

Table 20: Difference between multi-hop and neutral accuracy by family and model, three repeats (

\mathrm{\%}

);

n=1539

(see Exclusions, Appendix A ). The first three rows are unit-settling families.

	Rates		Contribution to the delta
Decoy family	Lost	Rescued	Cost	Gain	Net
Duplicate	34.6	32.6	$-$ 0 9.6	$+$ 22.3	$+$ 12.7
Wrong Unit	41.8	18.1	$-$ 10.1	$+$ 13.6	0 $+$ 3.5
Lookalike	43.3	21.2	$-$ 12.0	$+$ 15.1	0 $+$ 3.1
Text	52.6	11.6	$-$ 15.8	0 $+$ 7.9	0 $-$ 7.9
Prior	44.4	14.1	$-$ 0 9.9	$+$ 10.4	0 $+$ 0.5

Appendix

Table 21: Transitions under the multi-hop condition by family, averaged over six models (

\mathrm{\%}

). With neutral accuracy

A

, survival

R

, and rescue

W

\Delta=-A(1-R)+(1-A)W

Visually Grounded Reasoning

Visually grounded reasoning links visual evidence to multi-step conclusions and actions, aiming to make multimodal systems accurate, auditable, and robust.

See all 342 papers

Apr 11, 2026

Counting to Four is still a Chore for VLMs

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

MBZUAI

Attention Intervention in VLMs Vision-Language Models VLM Evaluation VLM Interpretability

Ask PDF

May 18, 2026

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly (

98.9\%

and

97.5\%

), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from

91.6\%

on factual to

49.5\%

on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a ``System-1''-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.

Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti +4

King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh

VLM Evaluation Vision-Language Models Counterfactual Evaluation of VLMs Object Counting Visually Grounded Reasoning Attention Mechanisms

Ask PDF

Feb 23, 2025

Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs

Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. Current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks. Does it guarantee their performance on these foundational visual capabilities? To systematically investigate this gap, we introduce VisFacor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFacor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 55.9%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition.

Jen-Tse Huang, Dasen Dai, Jen-Yuan Huang +7

CUHK · PKU · CUHKSZ +3

VLM Evaluation Vision-Language Models Visual Spatial Reasoning

Ask PDF

Abstract

Visually Grounded Reasoning

Explore similar work

Counting to Four is still a Chore for VLMs

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs