Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

Yingxin Jia; Hui Geng; Wenxin Yang; Mohan Zhang

cs.AIOct 20, 2026

Give the Model a Finger: Developmentally Aligned Evaluation of Counting in Vision-Language Models

Authors: Yingxin Jia, Hui Geng, Wenxin Yang, Mohan Zhang

Organizations: Independent Researcher

Abstract

Vision-language models (VLMs) are exact on small sets and drift into estimation on larger ones, a profile reminiscent of children before they master counting. Children cross the barrier with epistemic actions: pointing at each item while reciting number words, moving counted items aside, partitioning the array. We give VLMs the same actions as dumb image tools with no visual intelligence of their own in a training-free agent loop. Our central claim is about measurement, not intervention: a model that counts by marking leaves a normally hidden serial procedure on the image, and the developmental principles of counting turn those marks into an interpretable audit of serial visual reasoning: every tag is classified against point-level ground truth as on-target, duplicate, phantom, or omitted, the answer is checked against the number of tags, and order irrelevance is tested by forcing opposite traversal orders. Across 9 models, 682 images, and 7 harness conditions we find that (i) the exact-counting range is a model-specific signature spanning nearly two orders of magnitude (R80 from 3 to 150 dots); (ii) as an intervention the finger mostly fails: tags help 4 of 9 models on small-count photographs, hurt imprecise pointers, and rarely beat a verbal routine or a blind point list; what the loop buys is an auditable trace, not accuracy; (iii) as measurement it succeeds: a tag-perturbation probe shows that precise pointers read their answers off the tags, pointing precision tracks which models benefit, dense scenes fail by omission, and models rarely revise after seeing their own tags; and (iv) the capacities dissociate: models follow an instructed traversal order yet often change their answer when it reverses, and children's regular-over-scattered advantage appears only under the routine and survives lattices that defeat a rows-times-columns shortcut.

Figures & tables

Figure 1: The CountAgent loop, drawn with real artifacts from one trajectory (Gemini 3.1 Flash-Lite, FSC-147, 25 apples). Each turn the VLM emits one JSON action; the harness executes it with dumb PIL primitives, re-renders the working image, and returns it with a verification prompt, for at most $T$ tool calls. The final tags are audited against point ground truth, decomposing a wrong count into principle violations; here the model draws 24 tags, all on-target, misses one apple, and answers 24 against a ground truth of 25.

DotArrays (synthetic, $n\leq 60$ )
model	direct	CoT	list	tag	full toolbox
Qwen3-VL-8B	52.0	22.0 ↓	26.0	38.0 ↓	32.0 ↓
Qwen3-VL-32B	48.0	28.0 ↓	44.0	36.0	34.0
Gemini 3.1 Flash-Lite	36.0	58.0 ↑	54.0	50.0 ↑	52.0 ↑
GPT-5 mini	48.0	40.0	40.0	18.0 ↓	28.0 ↓
Gemini 3.7 Flash	100.0	98.0	94.0	94.0	96.0

Table 1: Exact-count accuracy (%) per model and arm on the common subset every model saw (240 images: 50 DotArrays, 100 FSC-147, 90 PixMo-Count); best arm per row in bold;

\uparrow

\downarrow

mark CI-solid changes against direct (95 % paired bootstrap CI over images excludes zero; all intervals in Table 8 ); unparseable answers count as wrong; tier-2/3 models ran fewer arms. MAE, tag+crop , and tiles are in Tables 12 and 13 .

model	$R_{80}$	point. err	omitted	duplicate	phantom	card. (%)	order (pp)
Qwen3-VL-8B	6 [6,7]	11.9 [10.6,12.9]	46.6 [42.6,50.5]	10.6 [7.8,14.0]	45.3 [37.3,54.4]	95.1 [93.0,96.9]	+51.5 [41.2,61.8]
Qwen3-VL-32B	7 [6,9]	8.7 [7.6,10.3]	30.8 [26.9,34.6]	29.7 [22.6,37.5]	85.7 [70.8,101.3]	81.0 [77.3,84.5]	+15.4 [8.1,23.2]
Gemini 3.1 Flash-Lite	7 [4,9]	5.1 [4.0,7.0]	27.6 [24.1,30.9]	3.9 [2.4,5.9]	21.2 [17.6,24.9]	88.5 [85.8,91.3]	+21.7 [13.6,30.5]
GPT-5 mini	7 [3,12]	40.4 [38.3,43.8]	79.4 [76.7,81.9]	2.2 [1.7,2.8]	31.9 [28.3,35.7]	64.7 [60.1,69.1]	–
Gemini 3.7 Flash	150 [100,200]	1.6 [1.5,1.8]	14.5 [11.2,17.9]	1.3 [0.6,2.2]	12.6 [9.6,15.7]	89.9 [87.1,92.5]	+0.7 [-1.5,3.3]
Gemini 3.1 Pro	150 [50,200]	10.7 [7.7,12.8]	22.1 [15.9,28.4]	1.4 [0.5,2.6]	22.4 [16.6,28.3]	89.3 [84.0,94.0]	–

Table 2: Capacity profile per model, with 95 % bootstrap CIs over images: counting range

R_{80}

under direct (

\geq

: right-censored at the largest

n

tested); pointing error of the tag arm (median over images of the per-image median on-target tag-to-object distance, in thousandths of the shorter side; DotArrays + FSC-147); omitted objects, duplicate and phantom tags per 100 ground-truth objects ( tag , DotArrays + FSC-147); cardinality (announced count

=

number of tags, %); order effect (excess disagreement from reversing the traversal beyond the same-order repeat control, pp; Table 9 ). Tier-3 models ran the common subset only.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

developmental prediction	operationalisation	verdict
small sets are exact without any routine [ Trick and Pylyshyn, 1994 ]	accuracy vs. $n$ : ceiling for $n\leq 4$	supported ( Section 5.1 )
pointing supports one-to-one tracking [ Alibali and DiRusso, 1999 ]	tag vs. direct , conditioned on pointing precision	precise pointers only ( Section 5.3 )
keeping-track errors are skips, rarely double counts [ Fuson, 1988 ]	omission/duplicate/phantom audit	supported ( Section 5.3 )
abstraction: anything can be counted [ Gelman and Gallistel, 1978 ]	mixed shapes and colours; red discs among blue	hurts estimation more than the routine ( Section 5.4 )
regular arrays are easier than scattered ones [ Potter and Levy, 1968 ]	regular/random/clustered; rotated, hexagonal, holed lattices	only under the routine ( Section 5.4 )
cardinality: the last tag names the set size [ Gelman and Gallistel, 1978 ]	announced answer $=$ number of tags?	model-dependent ( Section 5.3 )

Appendix

Table 3: The developmental predictions our protocol operationalises. One-to-one correspondence and cardinality are audited directly; abstraction, arrangement, and order effects are behavioural comparisons; stable order is supplied by the harness ( Table 5 ), so it is scaffolded, not tested.

method	marks come from	vision modules in loop	sees & revises marks	audit of the marks
Set-of-Mark [ Yang et al., 2023 ]	segmenter	segmenter	no	none
Autonomous Imag. [ Liu et al., 2025 ]	model cursor	SAM2, inpainting	yes	none
SketchVLM [ Collins et al., 2026 ]	model coords	none (rendering)	yes	none
Point-then-count [ Alghisi et al., 2026 ]	model coords	none (unrendered)	no	grounding
LVLM-Count [ Fetrat Qharabagh et al., 2024 ]	detector	detector	no	none
ABACUS [ Mondal et al., 2026 ]	trained model	trained counter	no	none

Appendix

Table 4: Positioning relative to prior mark-based methods (cells summarise the published descriptions; “grounding” = pointing accuracy is analysed, but wrong counts are not decomposed into principle violations). Our distinction is not that a VLM can make and inspect marks, it is that the marks are used as a measurement trace: audited point-by-point against ground truth and read through the developmental principles of counting.

tool	signature	child strategy	principle(s)
tag	points $\to$ numbered tags on the working image	pointing while reciting “1, 2, 3”	one-to-one, stable order (scaffolded), cardinality
crop	box $\to$ enlarged view (tags visible)	leaning in; partitioning	one-to-one in dense regions
erase	points $\to$ greyed-out items	moving counted items aside	one-to-one (no double count)
grid	on $\to$ coordinate grid overlay	organised row-by-row scanning	pointing precision; partition
tiles (scripted)	$k{\times}k$ overlapping tiles, per-tile points, merge	counting by groups	cardinality by addition
answer	$N$	announcing the last number word	cardinality

Appendix

Table 5: Epistemic tools, the child strategy each mirrors, and the counting principle each supports. All tools are PIL operations with no visual intelligence. Coordinates are in a normalised

[0,1000]^{2}

frame of the original image. The harness itself supplies the stable-order component (tags are numbered consecutively across calls), so stable order is scaffolded, not independently tested.

model	$\tau$ scale	on-target	omitted	duplicate	phantom
Grok 4.20	0.75	19.0	81.0	0.6	66.9
Grok 4.5	0.75	35.5	64.5	2.4	125.4
Gemini 3.1 Pro	0.75	73.1	26.9	0.4	28.2
Claude Sonnet 5	0.75	14.1	85.9	3.2	68.1
Qwen3-VL-8B	0.75	46.4	53.6	6.6	56.3
Gemini 3.7 Flash	0.75	82.5	17.5	0.6	16.3

Appendix

Table 6: Sensitivity of the audit to the matching threshold (tag arm, DotArrays + FSC-147): rates per 100 ground-truth objects.

Figure 2: Exact accuracy versus

n

for the two models whose

R_{80}

is right-censored on the main battery (

n\leq 60

), continued onto the XL battery (

n

up to 200, Appendix D ); the shaded band is the subitizing range, the dotted line the 80 % criterion. Both cliffs appear near 150; the verbal routine (CoT) collapses before direct estimation does.

Figure 3: Paired change in exact accuracy of each arm over direct on the common subset, with 95 % paired bootstrap CIs over images (

B{=}10^{4}

; numbers in Table 8 ). With 50–100 images per cell most single effects are within noise; CI-solid are Flash-Lite’s gains and the imprecise pointers’ losses on DotArrays, Qwen3-VL-32B’s gain on PixMo-Count, and CoT for Flash-Lite.

Figure 4: Left: gain of tag over direct on the common subset against pointing error (median tag-to-object distance of on-target tags, DotArrays + FSC-147); the models that gain are the models that point. Right: the same comparison in cost–accuracy space, each arrow running from a model’s direct answer (blue dot) to its tag loop (green improves, red hurts, grey within one point); the loop costs roughly

3\times

more per image.

model	direct	CoT	tag	full
Qwen3-VL-8B	6 [6,7]	5 [3,7]	10 [7,15]	12 [4,15]
Qwen3-VL-32B	7 [6,9]	7 [6,10]	12 [12,15]	12 [12,20]
Gemini 3.1 Flash-Lite	7 [4,9]	15 [15,25]	20 [12,25]	20 [15,25]
GPT-5 mini	7 [3,12]	7 [4,12]	3 [1,5]	5 [0,5]
Gemini 3.7 Flash	150 [100,200]	80 [60,120]	100 [50,150]	100 [50,100]
Gemini 3.1 Pro	150 [50,200]	150 [150,200]	100 [100,150]	100 [80,120]

Appendix

Table 7: Counting range

R_{80}

per model and arm on the DotArrays main battery (for Gemini 3.7 Flash and Gemini 3.1 Pro including the XL extension to

n{=}200

), with 95 % bootstrap confidence intervals (images resampled within each level,

B{=}10^{3}

\geq

marks right-censored values: the isotonic accuracy fit is still

\geq 80\,\%

at the largest tested

n

	DotArrays			FSC-147			PixMo-Count
model	CoT	tag	full	CoT	tag	full	CoT	tag	full
Qwen3-VL-8B	-30 [-44,-16]	-14 [-26,-4]	-20 [-34,-6]	+1 [-6,+8]	+6 [-2,+14]	+1 [-4,+6]	+2 [-9,+13]	-4 [-17,+8]	-9 [-21,+3]
Qwen3-VL-32B	-20 [-32,-8]	-12 [-26,+2]	-14 [-28,+0]	-4 [-14,+5]	+7 [-2,+16]	+1 [-7,+9]	+10 [-1,+21]	+13 [+4,+22]	+13 [+4,+22]
Gemini 3.1 Flash-Lite	+22 [+8,+36]	+14 [+4,+26]	+16 [+4,+28]	+18 [+7,+29]	-1 [-11,+9]	+6 [-4,+16]	+8 [+0,+16]	+0 [-9,+9]	+7 [-2,+16]
GPT-5 mini	-8 [-24,+8]	-30 [-44,-16]	-20 [-36,-4]	+0 [-6,+6]	+0 [-6,+6]	-4 [-10,+2]	-1 [-11,+10]	-1 [-11,+9]	-6 [-16,+3]
Gemini 3.7 Flash	-2 [-6,+0]	-6 [-14,+0]	-4 [-10,+0]	+0 [-7,+7]	-7 [-15,+1]	-5 [-13,+3]	+0 [-3,+3]	-2 [-7,+2]	-6 [-12,+1]

Appendix

Table 8: Paired change in exact accuracy (percentage points) of each arm over direct on the common subset, with 95 % paired bootstrap confidence intervals over images (

B{=}10^{4}

). Bold: the interval excludes zero.

model	$\bar{\tau}_{\rightarrow}$	$\bar{\tau}_{\leftarrow}$	agree ↔	agree =	order effect	acc →	acc ←	acc (tag)	agree rot	rot. effect	acc rot
Qwen3-VL-8B	+0.56	-0.39	13.2	64.7	+51.5 [+41,+62]	27.9	14.7	38.2	30.1	+34.6 [+24,+45]	36.0
Qwen3-VL-32B	+0.50	-0.47	24.3	39.7	+15.4 [+8,+23]	44.1	26.5	45.6	–	–	–
Gemini 3.1 Flash-Lite	+0.71	-0.59	43.8	65.4	+21.7 [+14,+31]	60.3	47.1	54.4	55.9	+9.6 [+1,+18]	63.2
Gemini 3.7 Flash	+0.96	-0.86	86.0	86.8	+0.7 [-1,+3]	94.1	92.6	94.1	86.8	+0.0 [-4,+4]	94.9

Appendix

Table 9: Traversal-order experiment (order-irrelevance principle) on the 68 DotArrays images of the ablation subset: the tag loop with an instructed scan order, left-to-right/top-to-bottom ( tag → ) versus right-to-left/bottom-to-top ( tag ← ), each run twice.

\bar{\tau}

is the mean Kendall correlation between tag sequence and vertical position (positive = top-down), showing that the instructed order is realised; agree ↔ = same final count under opposite orders (mean over the four run pairs); agree = = same final count when the same order is run twice (repeat control; mean over both orders); order effect = agree =

-

agree ↔ , the excess disagreement attributable to reversing the traversal, with a 95 % paired bootstrap CI over images; acc: exact accuracy under each order and under the uninstructed tag loop on the same images. Rotation control: the reading-order arm on the image rotated by

180^{\circ}

visits the dots in the reverse sequence under the natural instruction; agree rot is its agreement with the unrotated reading-order runs and the rotation effect its excess disagreement over the repeat control.

model	arm	acc (%)	MAE	omit	dup	phantom	calls	cost
Gemini 3.1 Flash-Lite	direct	29.4	2.2	–	–	–	1.0	0.05
Gemini 3.1 Flash-Lite	tag	54.4	5.7	14.6	3.3	21.6	2.9	0.67
Gemini 3.1 Flash-Lite	tag-1	51.5	4.2	43.6	2.7	6.1	29.3	7.79
Gemini 3.7 Flash	direct	100.0	0.0	–	–	–	1.0	0.24
Gemini 3.7 Flash	tag	94.1	0.1	2.3	0.5	4.0	2.8	0.41
Gemini 3.7 Flash	tag-1	85.3	0.5	67.9	0.1	0.1	17.7	5.95

Appendix

Table 10: Strict seriality ( tag-1 ): one point per tag call, so every mark is placed only after the previous one is seen, versus direct and the batched tag loop (

T{=}6

) on the same DotArrays ablation images. tag-1 runs with a budget of

T{=}80

tool calls (the largest array has 60 dots), so no object goes untagged by construction; omissions are the model abandoning the traversal. Error rates are per 100 ground-truth objects; cost in US cents per image.

Panel of Figure 5 — Figure 5: Developmental signatures. (a) The largest $n$ up to which the isotonic fit of exact accuracy stays $\geq 80\,\%$ (shaded: subitizing range; tier-3 models were run on a subset of $n$ ; the two Geminis include the XL extension, and $\geq$ marks ranges still censored at the largest tested $n$ ; CIs in Table 7 ). (b) Exact accuracy on regular, random, and clustered arrays for the models that saw all three arrangements.

Figure 6: A real trajectory, re-rendered from the stored actions: Gemini 3.1 Pro on an FSC-147 photograph with 46 tree logs. The direct answer is 32; three rounds of tagging reach 46, and the audit of the final tags against the dot annotations finds 44 on-target tags, one duplicate, one phantom, and two omissions. Panels show the working image after each tool call.

Figure 7: Second real trajectory: Gemini 3.1 Flash-Lite with the full toolbox on a 50-dot regular array. direct says 56; the model calls grid , tags 21 dots, then the remaining 29, and answers 50 with every tag on target.

Figure 8: Exact-count accuracy versus number of dots on the DotArrays main battery (three arrangements, four seeds per

n

), one panel per model, one line per harness arm; the shaded band marks the subitizing range

n\leq 4

. Tier-2 models ran fewer arms; tier-3 models ran a subset of

n

(two arrangements, two seeds per level).

Figure 9: Principle audit for the tag arm: composition of the model’s tags per 100 ground-truth objects on DotArrays and FSC-147. On-target tags plus omitted objects sum to 100; duplicates and phantoms are extra tags; numbers above a bar give totals that exceed the axis.

Figure 10: Principle audit for all point-producing arms ( list , tag , full , tiles ), as Figure 9 .

model	arm	on-target	omitted	duplicate	phantom	card. (%)	prec.
Qwen3-VL-8B	list	69.4	30.6	35.7	56.3	76.7	16.2
	tag	53.4	46.6	10.6	45.3	94.7	14.7
	full	48.0	52.0	13.3	66.9	85.6	17.0
	tiles	54.1	45.9	10.9	32.9	100.0	18.9
Qwen3-VL-32B	list	80.6	19.4	12.7	34.9	58.0	9.7
	tag	69.2	30.8	29.7	85.7	80.2	13.1

Appendix

Table 11: Principle audit on images with point ground truth (DotArrays + FSC-147). Per 100 ground-truth objects: on-target tags, omissions (skipped objects), duplicates (second tag on a tagged object), phantoms (tag on nothing). Card.: fraction of trajectories whose announced count equals the number of tags. Prec.: median tag-to-object distance of on-target tags in thousandths of the shorter image side (lower is better).

Figure 11: Ablations: exact accuracy on the ablation subset with bootstrap 95 % CIs.

DotArrays (synthetic, $n\leq 60$ )
model	direct	CoT	list	tag	tag+crop	full toolbox	tiles
Qwen3-VL-8B	52.0 (5.0)	22.0 (71.5)	26.0 (57.4)	38.0 (8.3)	34.0 (9.2)	32.0 (17.4)	2.0 (9.5)
Qwen3-VL-32B	48.0 (2.7)	28.0 (5.8)	44.0 (3.9)	36.0 (28.8)	38.0 (17.3)	34.0 (24.5)	10.0 (6.7)
Gemini 3.1 Flash-Lite	36.0 (2.0)	58.0 (4.7)	54.0 (3.0)	50.0 (6.0)	58.0 (3.6)	52.0 (3.3)	54.0 (1.6)
GPT-5 mini	48.0 (2.2)	40.0 (1.7)	40.0 (2.3)	18.0 (4.0)	–	28.0 (2.6)	–
Gemini 3.7 Flash	100.0 (0.0)	98.0 (0.1)	94.0 (0.4)	94.0 (0.2)	–	96.0 (0.1)	–

Appendix

Table 12: All seven arms on the common subset (240 images): exact-count accuracy (%) with MAE in parentheses; best arm per row in bold. tiles is not reported on PixMo-Count: its geometric merge only removes near-coincident points, so objects larger than the tile overlap are counted in every tile they touch ( Section 5.2 ).

DotArrays (synthetic, $n\leq 60$ )
model	direct	CoT	list	tag	tag+crop	full toolbox	tiles
Qwen3-VL-8B	57.4 (2.8)	41.9 (67.1)	50.4 (19.9)	58.1 (4.8)	59.2 (5.4)	54.4 (11.2)	11.8 (5.9)
Qwen3-VL-32B	60.7 (1.6)	53.7 (2.7)	66.5 (6.6)	68.0 (17.5)	65.1 (49.4)	64.3 (12.5)	15.1 (5.3)
Gemini 3.1 Flash-Lite	52.9 (1.2)	77.2 (3.0)	75.7 (1.9)	73.9 (2.5)	75.4 (4.8)	76.5 (2.3)	64.0 (0.8)
GPT-5 mini	57.7 (1.5)	54.4 (1.4)	53.3 (1.4)	36.8 (2.4)	–	39.7 (2.3)	–
Gemini 3.7 Flash	97.1 (0.0)	99.3 (0.0)	95.2 (0.2)	98.2 (0.0)	–	97.4 (0.1)	–

Appendix

Table 13: Exact-count accuracy (%) with MAE in parentheses on the FULL image sets (272 DotArrays, 200 FSC-147, 180 PixMo-Count) for the models that ran them; tier-3 models only saw the common subset, so their rows repeat Table 1 . tiles on PixMo-Count is omitted for the reason given in Table 12 .

model	arm	acc (MAE)	on-target	dup.	phantom	calls
Qwen3-VL-32B	direct	32.2 (6.1)	–	–	–	1.0
	list (blind points)	32.9 (16.2)	76.2	12.8	39.0	1.0
	tag, no nudge	35.0 (26.1)	64.5	29.2	92.8	3.4
	plain marks	35.0 (12.5)	65.7	42.2	98.6	3.6
	tag	35.7 (23.3)	66.7	33.5	98.8	3.4
	tag+grid	34.3 (22.4)	67.2	16.1	85.8	3.1

Appendix

Table 14: Ablations on the ablation subset (143 images: DotArrays

n\geq 5

and FSC-147). Exact accuracy (%) with MAE; duplicates and phantoms per 100 ground-truth objects; mean tool calls.

model	arm	8-15	16-25	26-40	41-60	61-80
Qwen3-VL-8B	direct	40.0 (3.4)	10.0 (3.1)	5.0 (5.0)	2.5 (16.6)	2.5 (20.4)
	list	45.0 (7.2)	15.0 (50.4)	5.0 (59.6)	2.5 (57.9)	2.5 (52.6)
	tag	32.5 (5.1)	17.5 (5.8)	10.0 (21.0)	2.5 (39.4)	0.0 (32.1)
	full	35.0 (2.5)	17.5 (12.2)	5.0 (28.2)	0.0 (23.4)	2.5 (40.0)
	tiles	5.0 (7.8)	5.0 (8.3)	2.5 (12.7)	2.5 (25.5)	0.0 (39.9)
Qwen3-VL-32B	direct	60.0 (2.5)	27.5 (2.7)	15.0 (4.1)	10.0 (12.6)	2.5 (18.0)

Appendix

Table 15: FSC-147 by count bin: exact accuracy (%) with MAE.

model	arm	2	3	4	5	6	7	8	9	10
Qwen3-VL-8B	direct	95.0 (0.1)	95.0 (0.1)	75.0 (0.6)	75.0 (0.5)	60.0 (0.5)	65.0 (0.3)	45.0 (0.7)	50.0 (0.7)	35.0 (1.1)
	list	80.0 (0.2)	75.0 (0.2)	70.0 (8.3)	75.0 (2.2)	25.0 (4.7)	45.0 (13.8)	55.0 (3.6)	55.0 (5.0)	45.0 (11.4)
	tag	90.0 (0.1)	100.0 (0.0)	65.0 (0.7)	40.0 (0.8)	40.0 (1.4)	50.0 (0.6)	55.0 (0.5)	60.0 (0.8)	35.0 (0.6)
	full	90.0 (0.1)	90.0 (0.1)	55.0 (9.2)	45.0 (0.9)	40.0 (5.9)	65.0 (2.1)	35.0 (1.8)	45.0 (1.6)	25.0 (1.7)
	tiles	30.0 (1.6)	15.0 (2.0)	15.0 (3.0)	35.0 (2.0)	15.0 (2.8)	10.0 (4.0)	15.0 (2.9)	30.0 (3.8)	10.0 (5.3)
Qwen3-VL-32B	direct	95.0 (0.1)	80.0 (0.3)	85.0 (0.3)	80.0 (0.2)	65.0 (0.3)	65.0 (0.5)	65.0 (0.7)	60.0 (0.6)	30.0 (0.9)

Appendix

Table 16: PixMo-Count by ground-truth count: exact accuracy (%) with MAE.

model	arm	main	hetero	distractor	dense	small
Qwen3-VL-8B	direct	64.4 (2.7)	37.5 (1.4)	25.0 (2.9)	33.3 (4.8)	25.0 (4.8)
	list	59.7 (13.2)	12.5 (34.0)	18.8 (31.1)	8.3 (38.6)	16.7 (88.8)
	tag	68.5 (3.0)	25.0 (3.9)	18.8 (6.5)	0.0 (12.2)	25.0 (29.3)
	full	62.5 (7.5)	37.5 (7.0)	25.0 (21.6)	0.0 (26.6)	25.0 (52.1)
	tiles	12.0 (5.8)	6.2 (3.5)	18.8 (2.5)	0.0 (9.3)	16.7 (12.3)
Qwen3-VL-32B	direct	65.7 (1.3)	87.5 (0.1)	37.5 (1.4)	25.0 (3.6)	0.0 (6.6)

Appendix

Table 17: DotArrays factor sets: exact accuracy (%) with MAE. hetero = mixed shapes/colours (abstraction), distractor = red among blue, dense = touching discs, small = 6-px discs.

model	arm	agree (%)	acc original	acc mirrored
Gemini 3.1 Flash-Lite	direct	36.4	25.2	25.9
Gemini 3.1 Flash-Lite	full toolbox	38.5	44.1	44.1
Gemini 3.1 Flash-Lite	tag	39.2	39.9	45.5
Qwen3-VL-32B	direct	67.8	32.2	27.3
Qwen3-VL-32B	full toolbox	33.6	33.6	31.5
Qwen3-VL-32B	tag	32.9	35.7	32.2

Appendix

Table 18: Mirror consistency (an indirect invariance probe motivated by the order-irrelevance principle; the direct traversal-order test is in Table 9 ): agreement between the count on the original image and on its horizontal mirror, and accuracy on each, for the ablation subset.

model	= direct	$\rho$	= direct	$\rho$	= direct	$\rho$	card. (%)	differ	closer (%)
	DotArrays		FSC-147		PixMo-Count
Qwen3-VL-8B	44 [38,50]	+0.14	10 [6,15]	+0.03	43 [36,50]	+0.01	91 [89,93]	84/901	96 [92,100]
Qwen3-VL-32B	51 [45,57]	-0.43	20 [14,25]	-0.19	77 [71,83]	+0.44	80 [77,83]	180/902	87 [82,92]
Gemini 3.1 Flash-Lite	52 [46,58]	+0.11	16 [12,22]	+0.20	66 [59,73]	+0.49	91 [89,92]	89/943	69 [58,78]
GPT-5 mini	31 [26,37]	+0.13	17 [12,22]	+0.38	55 [48,62]	+0.35	66 [63,69]	304/903	86 [82,90]
Gemini 3.7 Flash	92 [88,95]	+0.04	52 [44,58]	+0.51	94 [90,97]	+0.62	90 [88,92]	92/924	55 [45,65]

Appendix

Table 19: Do the tags follow the answer, or the answer the tags? For every image that ran both direct and tag : the fraction of images on which the model drew exactly as many tags as its direct answer (= direct, %, with 95 % CI) and the Spearman correlation

\rho

between the per-image tag-count error and the direct error. Right: cardinality (announced count

=

number of tags; tag and full , DotArrays + FSC-147), the number of trajectories where they differ, and how often the announced count is then closer to the truth than the tag count (closer, %).

model	data	pert.	$n$	follows image	follows own	repaired	acc. (%)	shift	$\downarrow$ / $\uparrow$
Qwen3-VL-8B	DotArrays	$-1$	55	100 [100,100]	0 [0,0]	45 [33,58]	29	+1.56 [-3.37,6.62]	40/33
	DotArrays	$+1$	31	97 [90,100]	3 [0,10]	0 [0,0]	10	+8.52 [1.24,16.55]	28/62
	FSC-147	$-1$	54	98 [94,100]	0 [0,0]	15 [6,24]	2	+5.48 [-3.40,15.98]	58/34
	FSC-147	$+1$	40	100 [100,100]	0 [0,0]	0 [0,0]	8	+15.16 [3.16,30.49]	27/70
Gemini 3.1 Flash-Lite	DotArrays	$-1$	68	91 [84,97]	1 [0,4]	81 [71,90]	57	-1.75 [-4.72,0.87]	25/22
	DotArrays	$+1$	35	94 [86,100]	0 [0,0]	40 [23,57]	23	+1.54 [-7.69,14.31]	46/37

Appendix

Table 20: Tag-perturbation probe. After the model’s first tag call the harness silently removes one on-target tag (

-1

) or adds one tag on a still-untagged object (

+1

) before rendering the working image; tags are renumbered, so the image and the harness message agree and only the model’s own emission differs.

n

: trajectories in which the perturbation could be applied (a

+1

needs an untagged object). follows image: announced count

=

number of tags visible at the end; follows own: announced count

=

number of points the model emitted (and

\neq

visible); repaired: the removed object was re-tagged by the end, or the model started over with replace (

-1

) / started over (

+1

); shift: announced count minus the unperturbed tag answer on the same image (mean, 95 % CI);

\downarrow/\uparrow

: fraction of images on which the answer moved down/up. Models that read their tags should shift by about

-1

unless they repair.

model	direct	$\geq 40$	any	CoT	setting	tokens	DotArrays	FSC-147	PixMo	$\Delta$ (pooled)	$R_{80}$
	reasoning tokens (low effort)				thinking-off control
Qwen3-VL-8B	0	0	0%	0	–	–	–	–	–	–	–
Qwen3-VL-32B	0	0	0%	0	–	–	–	–	–	–	–
Gemini 3.1 Flash-Lite	122	120	100%	133	off	0	53 $\to$ 60	20 $\to$ 18	74 $\to$ 72	+1.8 [-1.1,4.8]	7 $\to$ 4
GPT-5 mini	128	352	97%	256	minimal	0	58 $\to$ 38	12 $\to$ 19	69 $\to$ 58	-9.4 [-13.2,-5.4]	7 $\to$ 2
Gemini 3.7 Flash	480	1720	100%	596	minimal	500	97 $\to$ 99	48 $\to$ 45	90 $\to$ 88	-0.8 [-2.5,0.9]	150 $\to$ 120

Appendix

Table 21: What direct measures. Left: hidden reasoning tokens reported by the provider for the single-turn arms at the “low” effort used throughout (median over all images;

\geq 40

: median on DotArrays with

n\geq 40

; any: fraction of direct replies with at least one reasoning token). Right: the thinking-off control, the same direct prompt with hidden reasoning disabled (Flash-Lite: enabled=false ; GPT-5 mini: effort minimal , which yields zero reasoning tokens) or minimised where the endpoint refuses to disable it (Gemini 3.7 Flash: effort minimal ); exact accuracy on the same images and the paired change (pp, 95 % CI), and

R_{80}

on DotArrays.

model	arm	regular	random	rotated	hexagonal	holed	regular $-$ random	lattices $-$ random
Qwen3-VL-8B	direct	45	45	35	35	15	+0 [-30,30]	-17 [-42,8]
	tag	60	25	5	25	25	+35 [5,65]	-7 [-28,13]
Gemini 3.1 Flash-Lite	direct	35	25	10	25	35	+10 [-20,35]	-2 [-25,20]
	tag	85	55	30	85	90	+30 [5,55]	+13 [-12,38]
GPT-5 mini	direct	40	40	0	10	25	+0 [-30,30]	-28 [-50,-7]
	tag	10	20	0	10	30	-10 [-30,10]	-7 [-27,12]

Appendix

Table 22: Arrangement effect on lattices that defeat a rows-times-columns shortcut. Exact accuracy (%) on the main-battery near-regular and random arrays at

n\in\{12,20,30,40,50\}

(20 images each) and on three new lattice sets with the same

n

, jitter, and seeds-per-level (20 images each): a square lattice rotated by 20–40 ∘ , a hexagonal lattice, and a square lattice with about a quarter of its cells left empty. Right: the regular-minus-random and (rotated+hexagonal+holed)-minus-random differences with 95 % bootstrap CIs (pp).

model	arm	images	tags	$r$	slope	to dot (px)	to centre (px)	closer to dot (%)
Qwen3-VL-8B	list	40	919	0.63	0.88	15.1	24.2	67
	tag	40	990	0.55	0.80	16.0	21.4	64
Qwen3-VL-32B	list	40	1084	0.96	0.95	3.8	13.6	89
	tag	40	1075	0.85	0.90	5.2	14.1	81
Gemini 3.1 Flash-Lite	list	40	1082	0.98	0.98	1.7	13.7	95
	tag	40	1070	0.86	0.89	3.3	14.1	80

Appendix

Table 23: Grid-guessing check on the main-battery near-regular arrays (

n\geq 9

; each dot is displaced from its lattice cell centre by a jitter of up to

\pm 15\,\%

of the cell). For every on-target tag: Pearson correlation and regression slope between the dot’s jitter and the tag’s offset from the same cell centre (a model that emits an even grid has

r\approx 0

); median distance from the tag to the dot versus to the cell centre (px); and the fraction of tags closer to the dot than to the centre.

model	DotArrays	FSC-147	PixMo-Count
Qwen3-VL-8B	+12.0 [2,22]	+1.0 [-6,8]	+4.4 [-7,16]
Qwen3-VL-32B	-8.0 [-20,4]	+5.0 [-4,14]	-2.2 [-10,6]
Gemini 3.1 Flash-Lite	-4.0 [-16,8]	-9.0 [-17,-1]	-1.1 [-11,9]
GPT-5 mini	-22.0 [-36,-10]	-2.0 [-8,4]	-3.3 [-13,7]
Gemini 3.7 Flash	+0.0 [-8,8]	-2.0 [-10,6]	-1.1 [-6,3]

Appendix

Table 24: Seeing the marks versus listing them: paired change in exact accuracy of tag over list (blind pointing) on the common subset, percentage points with 95 % paired bootstrap CIs over images. Bold: the interval excludes zero.

model	identifier	$/M in/out	direct	CoT	list	tag	tag+crop	full	tiles	total
Qwen3-VL-8B	qwen/qwen3-vl-8b-instruct	0.12/0.45	0.01	0.16	0.16	0.19	0.34	0.15	0.12	7.34
Qwen3-VL-32B	qwen/qwen3-vl-32b-instruct	0.10/0.42	0.01	0.06	0.04	0.19	0.16	0.23	0.09	4.95
Gemini 3.1 Flash-Lite	google/gemini-3.1-flash-lite	0.25/1.50	0.06	0.13	0.13	0.35	0.37	0.40	0.27	11.13
GPT-5 mini	openai/gpt-5-mini	0.25/2.00	0.07	0.12	0.11	0.31	–	0.41	–	6.66
Gemini 3.7 Flash	google/gemini-3.7-flash	0.38/1.88	0.19	0.25	0.16	0.40	–	0.42	–	9.26
Gemini 3.1 Pro	google/gemini-3.1-pro-preview	2.00/12.00	1.08	1.24	–	3.61	–	3.33	–	22.23

Appendix

Table 25: Models (OpenRouter identifiers, list prices in USD per million input/output tokens on 2026-08-18) and mean cost per image in US cents for each arm; the last column is the total spend per model in USD.

Figure 12: Final working images from representative trajectories (re-rendered from stored actions), one per failure category: a case the tag loop fixed, a duplicate-heavy case, a phantom-heavy case, an omission-heavy case, and a cardinality-inconsistent case (announced count far from the number of tags). Titles give model, arm, ground truth, announced count and the audit.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 738 papers

Apr 11, 2026

Counting to Four is still a Chore for VLMs

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

MBZUAI

Attention Intervention in VLMs Vision-Language Models VLM Evaluation VLM Interpretability

Ask PDF

May 18, 2026

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

Do Vision-Language Models (VLMs) count what they see, or what they know about the world? We introduce CounterCount, a diagnostic dataset of paired factual and counterfactual (CF) images, in which the CF image contains an atypical count of a subject's attribute (e.g., a rabbit with four ears), together with annotations localizing the counted attributes. Humans answer both types of images nearly perfectly (

98.9\%

and

97.5\%

), but across 16 VLMs (14 open-weight and 2 closed-source), average accuracy drops from

91.6\%

on factual to

49.5\%

on CF images, with most errors stemming from responses that match the canonical count, therefore revealing a bias towards knowledge priors rather than reliance on visual information. Based on these findings, we perform an analysis that reveals two interesting insights behind this gap. First, models exhibit a ``System-1''-like response behavior, falling back on canonical counts after a shallow recognition of the subject. Second, model activations partially contain the visual evidence necessary to answer correctly, yet models underutilize it, attending less to the counted attributes when they provide biased answers. Input manipulations that break visual context, introduce cueing or reasoning, or direct amplification of the model's internal attention to relevant regions can reduce such biases. CounterCount goes beyond showing that VLMs fall back on knowledge priors, revealing that once a familiar context is recognized, models favor prior knowledge over available visual evidence rather than failing to encode that evidence.

Reem Alzahrani, Bushra Bin Hemid, Hassan Alshanqiti +4

King Abdullah University of Science and Technology, Thuwal, Saudi Arabia. · University of Edinburgh

VLM Evaluation Vision-Language Models Counterfactual Evaluation of VLMs Object Counting Visually Grounded Reasoning Attention Mechanisms

Ask PDF

Jul 10, 2026

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs

Despite strong performance on many multimodal tasks, vision-language models (VLMs) still struggle with basic object counting. We investigate whether this reflects missing internal knowledge or a gap between internal representations and verbalized outputs. Training simple probes on activations from four VLMs across five counting datasets reveals that nonlinear probes can reliably detect counting errors, suggesting that VLMs often encode the correct count even when they output the wrong answer. SVCCA analysis shows that probes trained on ground-truth counts and probes trained on model outputs occupy a partially shared activation subspace but read out along misaligned directions. We further validate our findings using a causal steering intervention, proving that strengthening the direction of count-identified probes does improve model counting performance. Motivated by this result, we propose a detector-guided self-correction method that selectively re-prompts the model only when an internal error detector predicts failure. This simple inference-time intervention improves counting accuracy by up to 15.6 absolute percentage points, without any parameter updates. Our results establish activation-based error probing as both a practical tool for improving VLM counting and a mechanistic lens on the gap between internal knowledge and model outputs.

Ahmed Oumar El-Shangiti, Abzal Nurgazy, Hilal AlQuabeh +2

MBZUAI · Imperial College London

Vision-Language Models VLM Interpretability Self-Correction in VLMs

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Counting to Four is still a Chore for VLMs

CounterCount: Diagnosing Knowledge Bias in How Vision-Language Models Count Attributes

The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs