From Seeing to Reporting: A Theory of Reliable Visual Set Generation in Vision-Language Models

Junzhe Shi; Yuan Gan; Shida Jiang

cs.CVOct 18, 2026

From Seeing to Reporting: A Theory of Reliable Visual Set Generation in Vision-Language Models

Authors: Junzhe Shi, Yuan Gan, Shida Jiang

Organizations: Systems Engineering, University of California, Berkeley, Berkeley, CA, USA · Department of AI, Quotr AI, Emeryville, CA, USA

Abstract

Vision-language models (VLMs) are asked to report what they see: to count objects, box them, parse a screen or pick a click target (GUI grounding). We argue that reporting is part of the visual task, not merely its formatting. In a dense scene with 173 shapes, GPT-6.1 lists 173 records that match every shape, yet the counts it states in the same response sum to 188. From Seeing to Reporting (FSR) is a theory of visual reporting and a set of inference-time procedures that improve the reports of frozen VLMs. It models reporting as a chain from responses to decoded records, constructed sets and task answers, linked by task-compatible matching. The chain gives exact relations for deleting records, averaging their coordinates and switching readouts, and it splits the error of one answer chosen among several calls into missing candidates, lost representatives and wrong choices. These relations guide procedures for counting, detection, document layout, screen parsing and GUI grounding that use the task specification and a shipped configuration, without requiring labels from the target task. On 50 dense synthetic scenes, a one-call reporting-and-decoding protocol cuts per-type count error by 85.2% for GPT-6.1 and 98.8% for Claude Sonnet 5.5. On 600 ScreenSpot-Pro items, a GUI-grounding tier that combines three models lowers the miss rate from 13.67% for GPT-6.1's two-stage zoom to 9.33%, against 12.67% for three such zooms combined by a cluster vote at about the same cost. A plain vote over the same three models misses 12.33%: the other models supply the candidates, and selecting among them turns those candidates into fewer misses. We release the protocols, the stored responses and the analysis code.

Figures & tables

Figure 1: Three repairs that change what a model delivers. (a) One response of GPT-6.1 to a dense synthetic scene (image 28 of SCP-dense, our dense synthetic counting scenes; 173 shapes of five classes): the list count , the records of each class in the model’s list counted by code, matches the truth, while the stated count the model wrote in the same response sums to 188; each of the 173 records matches a different shape of the same class (Section 7.2 ). (b) For the instruction “only allow values from a list” in an Excel menu, a single click by Gemini 3.8 lands on “Time”; a second call on a zoomed crop (ReGround, Li et al., 2025b ) lands inside the target “List” (green box). On 200 ScreenSpot-Pro items, two-stage zoom cuts Gemini 3.8’s miss rate by 65% (Section 7.5 ). (c) Miss rate against cost per item at list prices on 600 ScreenSpot-Pro items: GPT-6.1’s two-stage zoom, three such zooms with a cluster vote, a crop-first baseline in the style of GUI-Lens ( Fu et al., 2026 ) , a cluster vote over three models’ zoomed points, and FSR’s GUI-grounding tier (Algorithm 2 ), which selects among the same points with a scorer and a comparison; every item is in the denominator. Paired differences with intervals are in Section 7.7 : the tier’s difference from the crop-first baseline has an interval that reaches 0, and its differences from the other flows exclude 0.

Figure 2: FSR for visual reporting with vision-language models. Top lane: the reporting chain of the default pipeline (Algorithm 1 ), from the image, task and budget to the answer, with the repair at each step (blue cards) and the result that governs it. Bottom lane: the same four steps for GUI grounding with three models (Algorithm 2 ), and where each kind of miss arises: $F$ , no correct candidate after the calls; $L$ , a correct candidate but no correct representative after construction; $W$ , a correct representative that readout does not choose. The miss rate is $F+L+W$ (Proposition 8 ). Calls determine coverage, what the responses contain; decoding, construction and readout determine what is delivered from it. $F$ , $L$ and $W$ locate a miss; an operation can change more than one of them. Judge: choosing between configurations, for which the shipped defaults need no target labels. Detect: errors flagged without labels; in GUI grounding the alarm triggers the comparison. Every item is scored, and an item with no point counts as a miss.

Term	Meaning
call, response	One request to a model API, and the raw text it returns; every attempted call is stored and costed.
interface	Everything that defines a call except the image: model, reasoning setting ( lowest or default ), output cap, prompt, output schema, and the view of the image that is sent.
target	A true object in the image, given by the annotation (a point or box with a class, or the box of the element a click should hit).
record	One object as reported by the model: a class name and a location (a point or a box).
list	The sequence of records in one response. A list-first prompt asks for the list before any counts; a count-first prompt asks for counts first.
stated count	The numbers the model writes for how many objects of each class it found (for example the field counts_by_type ).

Table 1: Terms used in this paper.

Task	Scoring compatibility	Association across calls	Records that are not valid
SCP (synthetic, 100 images)	same class, within 30 px	not used	class unknown after the alias map: kept for status, excluded from the delivered set
SCP-dense (50 images)	same class, within 20 px	not used	as for SCP
Counts, one class (PixMo-Count, FSC-147; Section 7.2 )	absolute error of the count against the annotated number; no matching	not used	every decoded point is counted
Counts, several classes (30 DOTA windows, 50 DocLayNet pages; Section 7.2 )	per-class count error; DOTA objects marked difficult or less than half inside the window are not targets and there are no ignore regions, so records on them are counted like any other record; list recall (diagnostic only): a record centre inside a true box of the same class, one to one	not used	class unknown after the declared alias map: kept in the response, not counted
Detection boxes (SKU-110K, DOTA, DocLayNet)	same class, IoU $\geq 0.75$ ; unmatched records in ignore regions not scored	same class, IoU $\geq 0.5$	label outside the vocabulary or box outside the image: kept for audit, excluded from the delivered set, counted as invalid
FSC-147 points	within half the mean short side of the three exemplar boxes	within the same distance	point outside the image: kept for audit, excluded and counted as invalid

Table 2: Task contracts. Scoring compatibility decides matches with the truth; association decides which records of different calls are the same object, and uses no labels. Coordinates are mapped from each model’s native frame to image pixels before scoring. The detection rows and the multi-class count row use overlapping datasets but different images and different contracts.

Result	Quantity	Decision	Check
Intersection threshold (Prop. 2 )	marginal matching rate $p_{1}$ of unconfirmed records	confirm and delete for set error on over-reporting sources, keep a single call on under-reporting sources; label rule of Step 7	from 5 labelled images the threshold predicts the sign on the other images in 83% of draws; the Step 7 rule lowers the loss against the better decision from 0.43 (family default alone) to 0.17 errors per image (§ 7.4 )
Geometry term (Prop. 3 )	selection and geometry terms	coordinate averaging is a separate switch: on by default, the deletion-only variant in the menu	averaging helps in 37 of 54 pairs and hurts in 13; the threshold’s only two failures on averaged intersections are geometry reversals (§ 7.4 )
Two readouts (Prop. 4 )	gap between stated count and list count, without labels	bounds what switching readouts can gain, without labels	on aerial and document images, for GPT-6.1 and Sonnet 5.5, the gap is 0.1 to 7.4 against errors of 5 to 46, and switching readouts changes little (§ 7.2 )
Partner (Prop. 5 )	confirmation rates $h_{+}$ , $h_{-}$	a partner must reject more false than true records; cross-model partners in the menu	cross-model partners separate the two better in 32 of 36 comparisons (§ 7.4 )
Coverage (Prop. 9 )	rescue rate $\phi$ on the menu’s misses	three different models, not three calls of one	a GPT-6.1 repeat rescues 18%, then 7.5%; Sonnet 5.5 29%, Gemini 3.8 37%; given GPT-6.1’s 82 misses, rates from the 260 GUI items give 46 items with no correct candidate among the 600, against 43 observed (§ 7.7 )
Three sources (Prop. 8 )	$F$ , $L$ , $W$	the steps of Algorithm 2 ; three sources, not four	a fourth source lowers $F$ from 43 to 36 and raises $L$ from 3 to 7 and $W$ from 10 to 11 (§ 7.7 )

Table 3: What each result is used for: the quantity it measures, the decision it makes in the method, and the check in Section 7 .

Loss and family	Transferred construction	Final default	Evidence and reason
$E$ , over-reporting families (detection, layout, screen parsing)	three-call unanimity, averaged coordinates (chosen in all six folds)	the same; list interface, lowest setting	Significantly better in 9 of 15 cells, significantly worse in none, not significant in 6.
$E$ , under-reporting family (dense points, FSC-147)	three-call unanimity, averaged coordinates	single call, no deletion; list interface, lowest setting	The transferred construction is worse in 2 of 3 cells. Deleting unconfirmed records raises $E$ when their marginal matching rate $p_{1}$ is above 1/2 (Proposition 2 ); on FSC-147, $p_{1}$ is 0.39 to 0.90 and deletion hurts in 7 of 9 pairs (Section 7.4 ).
F1	three-call union (five folds) or majority (one fold)	single call; on detection sources, interface and setting per model (GPT-6.1: regions, lowest; Sonnet 5.5: regions, default; Gemini 3.8: list, default); on screen sources, list interface, lowest setting	The transferred construction is better in 2 of 18 cells, worse in 4 and not significant in 12. A configuration chosen on three detection sources is better on the fourth in 5 of 12 cells and worse in 1 (Section 7.9 ).
Count	not part of the transfer	single call, list-first request; GPT-6.1 and Sonnet 5.5 lowest, Gemini 3.8 default setting; list count, with the stated count as declared fallback	Sections 7.2 and 7.3 .
Click (hit in the target)	not part of the transfer	the user’s model, two-stage zoom; GPT-6.1 and Sonnet 5.5 default, Gemini 3.8 lowest setting; Algorithm 2 when the budget covers its planned worst case	On 600 ScreenSpot-Pro items the default setting lowers Sonnet 5.5’s miss rate by 4.0 points and does not change GPT-6.1’s significantly; Algorithm 2 lowers the miss rate by 4.33 points against GPT-6.1’s zoom (Section 7.7 ).

Table 4: Shipped defaults for one model (Algorithm 2 combines three). Transferred construction : the best construction on the other five sources, applied to the evaluated source (Section 7.9 ). Final default : what the pipeline ships. They coincide for set error on over-reporting families and differ elsewhere; the last column gives the reason. The prompts are in Appendix D , the settings of the GUI-grounding tier in Algorithm 2 , and every value in the released configuration file.

Failed assumption	How we know	Revised assumption and direct check	New consequence and test
The intersection threshold, applied to intersections with averaged coordinates, predicts whether confirmation helps.	Wrong sign in 2 of 54 base/partner pairs.	The threshold governs deletion only; coordinate averaging is a separate term (Proposition 2 , Proposition 3 ). Exact identity: both terms add up to the observed change on every image of all 54 pairs.	The averaging term can take either sign and can reverse the result. Observed: positive in 37 pairs, negative in 13; it reverses the deletion-only result in two pairs, which accounts for both former exceptions (Section 7.4 ). The deletion term, in turn, can be predicted from $p_{1}$ on 5 labelled images in 83% of draws (Section 7.4 ).
Similar quality is what makes a second model a useful confirming partner.	On ScreenParse the three models have similar set error (16.2, 13.3, 16.0), yet among constructions with averaged coordinates the best cross-model construction is not significantly different from the best same-model one ( $+1.8\%$ , $[-3.0,7.6]$ ): similar aggregate quality does not establish a benefit.	A partner helps if it rejects more of the base model’s false records than it loses of its true ones (Proposition 5 ). Direct check across pairs: cross-model partners separate the two better than same-model partners in 32 of 36 comparisons; the 4 exceptions are partners that confirm few of the base model’s true records.	Conditional consequence: a base with many true and few false records gains only if the partner keeps almost all of its true records ( $n_{+}(1-h_{+}^{B})$ dominates $G$ ), and a weaker partner usually keeps fewer, so cross-model confirmation is expected to do worst where the base model is strongest. Observed association across transfer cells: all 5 significantly worse cells are cells where the base model is strongest, and 20 of the 24 cells where it is not strongest improve (Section 7.9 ).
A count written first, or given in the prompt, pulls the list length like a quota for every model.	Giving a count moves list length with slope at most 0.004 for GPT-6.1 and Sonnet 5.5 on SCP and SCP-dense (controlled comparison).	The pull depends on the model and on how reliable its enumeration is. Direct check: slope 0.17 to 0.20 on PixMo, and 1.63 for Gemini 3.8 on SCP-dense, where its lists degenerate.	Asking for the list before the counts should not hurt and should help where enumeration is reliable. Observed in a controlled comparison under the same mapped readout: list-first lists beat count-first lists for all three models ( $-39\%$ , $-92\%$ , $-25\%$ ; Section 7.2 ). The list-first prompt also adds the row-by-row scan and the allowed class names, so this is a test of the protocol, not of order alone.
Gemini 3.8 has a coordinate spread that a different coordinate convention reduces.	Pixel versus grid convention: $+2\%$ , n.s. (controlled comparison).	Current models make no coordinate-frame error on SCP. Direct check: systematic offset at most 0.3 pixels, slope 1.000, spread 0.9 to 1.5 pixels, against a 30-pixel tolerance.	References that correct coordinates (grids, ticks) should not improve localization. Observed in controlled comparisons: localization changes by at most 0.23 pixels in every arm. Any effect of an overlay must come through how the model writes its list (next row).
A grid helps Gemini 3.8 by improving localization.	The grid lowers set error by 3.5 per image at the lowest setting, but localization does not change, and at the default setting the grid makes detection slightly worse (controlled comparisons).	The grid acts on the record format: its gain falls on records that omit their class field. Association within lists: a record omits its class more often when the previous record has the same class (logistic regression controlling for position in the list: $+0.70$ $[0.57,0.84]$ in log-odds, arm fixed effects, intervals clustered by image). This path explains only a small part of the grid’s gain.	An order that interleaves classes should lower omissions, and grouping by class should raise them. Observed in a controlled comparison: a row-by-row scan lowers set error by 11.5 per image against 3.5 for the grid, and grouping by class raises it (Section 7.3 ).

Table 5: Revisions of the model. Each row records the failed assumption and the evidence against it, the revised assumption and its direct check, and the new consequence with its test. The kind of evidence is named in each cell.

Experiment	What changes	Calls
Readout	13 prompt arms on SCP, SCP-dense, PixMo	7,900
Readout, full protocol	SCP-dense; three models at the lowest setting, Gemini 3.8 also at the default (16k and 32k caps)	250
Multi-class readout	count-first vs. list-first, DOTA and DocLayNet	574
Grid	grids, ticks, cells, scan, type names	1,340
Tiling	count-first, list-only, $2\times 2$ tiles (FSC-147)	1,800
Detection	list / regions $\times$ two settings, 4 datasets	6,523

Table 6: All experiments with model calls, grouped (67,364 calls, including repairs and smoke tests); the registry in Appendix C itemizes them.

	One call	Common remedy	Second common remedy	Ours (Section 5 )
Repair the readout: dense counting (SCP-dense, 50 images, per-type count error; cost per image)
GPT-6.1	10.84 ($0.029)	8.32 ( $0.089),$ -$ 23.2%	n/a	1.60 ( $0.025),$ -$ 85.2%
Sonnet 5.5	35.02 ($0.031)	34.56 ( $0.094),$ -$ 1.3%	n/a	0.42 ( $0.033),$ -$ 98.8%
Gemini 3.8	45.68 ($0.026)	45.22 ( $0.076),$ -$ 1.0%	n/a	52.80 ( $0.019),$ +$ 15.6%
Repair the view: clicks (ScreenSpot-Pro items with all four calls stored: 199, 196 and 200 items for GPT-6.1, Sonnet 5.5 and Gemini 3.8; miss rate; cost per item)
GPT-6.1	11.1% ($0.0061)	10.5% ( $0.0182),$ -$ 4.5%	11.1% ($0.0123), 0.0%	9.6% ( $0.0083),$ -$ 13.6%

Table 7: Against common practice, from stored calls only. Columns give one call, the usual remedies (self-consistency: median of three stated counts, or a vote of three clicks; uncertainty-gated zoom: zoom only when two clicks disagree; majority vote and equal-weight box fusion over three calls), and our default. Relative changes are against one call, lower is better. Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction. Our count readout costs about as much as one call and our click protocol less than a three-call vote. For Gemini 3.8, “ours” is the list-first protocol at its lowest setting; the method runs Gemini 3.8 at its default setting, which at the same 16k cap gives a readout error of 24.52 (Section 7.3 ). For F1 our default is one call; over the 15 over-reporting cells, majority vote and box fusion change F1 by a median of

-0.3\%

and

+0.2\%

. Per-cell set errors are in the released analysis outputs.

Effect	Comparison	GPT-6.1	Sonnet 5.5	Gemini 3.8 (lowest)
Pure readout, count-first	stated $\to$ raw list count	10.84 $\to$ 2.64 $-$ 75.6% [ $-$ 83.2, $-$ 65.8]	35.02 $\to$ 7.80 $-$ 77.7% [ $-$ 91.3, $-$ 58.4]	45.68 $\to$ 129.04 $+$ 182.5% [163.3, 202.3]
Pure readout, list-first	stated $\to$ raw list count	10.10 $\to$ 1.60 $-$ 84.2% [ $-$ 88.2, $-$ 79.6]	18.52 $\to$ 5.42 $-$ 70.7% [ $-$ 76.9, $-$ 64.8]	46.56 $\to$ 95.04 ( $n=45$ ) $+$ 104.2% [72.7, 139.5]
Alias map, count-first	raw $\to$ mapped list count	2.64 $\to$ 2.64 (no change)	7.80 $\to$ 5.42 $-$ 30.5% [ $-$ 76.3, $-$ 12.7]	129.04 $\to$ 129.04 (no change)
Alias map, list-first	raw $\to$ mapped list count	1.60 $\to$ 1.60 (no change)	5.42 $\to$ 0.42 $-$ 92.3% [ $-$ 97.0, $-$ 86.2]	96.78 $\to$ 96.78 (no change)
Protocol	mapped list count, count-first $\to$ list-first	2.64 $\to$ 1.60 $-$ 39.4% [ $-$ 57.6, $-$ 13.5]	5.42 $\to$ 0.42 $-$ 92.3% [ $-$ 97.7, $-$ 44.4]	129.04 $\to$ 96.78 $-$ 25.0% [ $-$ 32.7, $-$ 17.6]
Full method	count-first stated $\to$ readout	10.84 $\to$ 1.60 $-$ 85.2% [ $-$ 89.1, $-$ 80.3]	35.02 $\to$ 0.42 $-$ 98.8% [ $-$ 99.5, $-$ 98.1]	45.68 $\to$ 52.80 $+$ 15.6% [ $-$ 5.3, 39.4]

Table 8: Readout effects kept separate (SCP-dense, 50 images, lowest setting, one call; per-type count error, lower is better; paired image bootstrap 95% intervals). The raw list count counts the records of each class before the alias map, the mapped list count after it. Pure readout : stated count

\to

raw list count of the same response. Alias map : raw

\to

mapped list count of the same list. Protocol : mapped list count of a count-first response

\to

that of a list-first response. Full method : stated count of a count-first response

\to

readout of Section 5 on a list-first response, the effect of the reporting and decoding protocol. Totals : mean absolute difference between the list length and the true total, and between the stated total and the true total. Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction.

Figure 3: One response, two answers. GPT-6.1 is asked to list every shape in a dense synthetic scene (SCP-dense image 28) and then give counts by type “for the shapes you listed”. (a) The scene has 173 shapes. (b) A detail of the scene with the records of the model’s list as dots, one colour per class: each record matches a different shape of the same class within 20 pixels. (c) The response itself, abridged; coordinates are on the model’s native 0 to 1000 grid. The list has 173 records, so the list count of every class is exact, while the counts the model states after it sum to 188, a per-type error of 21. Across 50 such scenes, list length misses the true total by 0.04 objects on average and the stated total misses by 7.78 (Table 8 ).

Figure 4: Same-response readout across datasets (count-first prompts, lowest reasoning setting, one call; 100 images per dataset, 50 on SCP-dense). Bars: relative change of count error from the stated count to the raw list count of the same response, with paired image-bootstrap 95% intervals; negative means the list count is better. Blue: interval below 0; orange: interval above 0; grey and n.s.: interval includes 0. Error is per type on SCP and SCP-dense and total on PixMo-Count and FSC-147; numbers under dataset names are mean objects per image. Absolute errors are in Table B.1 . The list count is better on the multi-class synthetic scenes for GPT-6.1 and Sonnet 5.5; Gemini 3.8’s lists degenerate there, so reading them is worse (Section 7.3 ).

Images	Model	List recall (%)	Readout gap	Error	Change, % [95% CI]	Error	Change, % [95% CI]
				Same response		Full method
DOTA	GPT-6.1	88	2.2	40.5 $\to$ 40.8	$+$ 0.7 [ $-$ 5.0, 4.5]	40.7	$+$ 0.4 [ $-$ 17.7, 16.4]
	Sonnet 5.5	68	7.4	45.8 $\to$ 39.7	$-$ 13.5 [ $-$ 34.7, $-$ 0.6]	38.6	$-$ 15.7 [ $-$ 38.2, 7.7]
	Gemini 3.8	26	44.2	33.2 $\to$ 63.9	$+$ 92.4 [54.2, 145.0]	39.2	$+$ 18.0 [ $-$ 13.4, 61.2]
	Gemini 3.8, default	n/a	n/a	n/a	n/a	23.9	$-$ 28.2 [ $-$ 48.6, 0.1]
DocLayNet	GPT-6.1	87	0.1	4.9 $\to$ 4.9	$+$ 0.4 [ $-$ 1.6, 2.2]	4.8	$-$ 2.4 [ $-$ 12.4, 4.5]

Table 9: Readout on aerial and document images with several classes: 30 DOTA windows (71 objects and 3.3 classes per image on average) and 50 DocLayNet pages (25 elements, 5.0 classes); one call per image and arm, lowest setting unless stated, 16k cap. Per-type count error, lower is better; paired image bootstrap 95% intervals. Difficult DOTA objects and objects less than half inside the window are not targets, so records on them are counted like any other record, for every readout alike. List recall : share of true objects matched by a record centre inside a true box of the same class (list-first arm). Readout gap : mean per-type difference between the stated count and the list count of the same count-first response, which bounds how much switching readouts can change the error. Same response : stated count

\to

list count of the count-first response. Full method : readout of the list-first response (list count, or the stated count if the list is flagged) against the stated count of the count-first response; the Gemini 3.8 default row is the shipped configuration. Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction.

Setting, cap	Readout error	List-count error	Set error $E$	Unknown records	$E$ , unknown kept	Flagged (%)	JSON fail (%)	Hit cap (%)	Reasoning (%)	Output tokens	Cost ($)
lowest, 16k	52.80	96.78	103.46	184.76	288.22	90	72	4	0	4,796	0.95
default, 16k	24.52	25.90	34.58	4.90	39.48	12	70	8	100	8,831	1.71
default, 32k	15.44	20.14	28.26	8.84	37.10	6	78	0	98	9,487	1.83

Table 10: Gemini 3.8 on SCP-dense under the list-first protocol (50 images, one call each). At a fixed 16k output cap, the default reasoning setting changes the count readout error by

-

53.6% [

-

72.4,

-

29.5]; raising the cap to 32k changes it by a further

-

37.0% [

-

62.4, 7.7] (not significant). Readout error: per-type count error of the Step-6 readout (stated count if the list is flagged, list count otherwise). List-count error: the same for the list count alone. Set error

E

scores the delivered records, those whose class is valid after the alias map (equal class, 20 px); unknown-class records are excluded from the delivered set and counted in the next column; the column after it is a sensitivity that keeps them and scores them as unmatched. Flagged: share of lists with at least one degeneracy flag. JSON fail: share of responses that do not parse as the requested object (records are then recovered by the declared salvage step). Hit cap: share of responses that stopped at the output cap. Reasoning: share of calls with reasoning tokens. Output tokens: mean per call. Cost: all 50 calls, counting every attempted call. The setting comparison is cap-matched, not cost-matched: output tokens and cost rise with reasoning.

Source	Sign from 5 labels (%)	Deletion helps	$p_{1}$ range	Selection range	Geometry $+$ / $-$	Geometry range	Sign flips
DocLayNet	74	6	0.11 to 0.64	$-$ 2.2 to 2.1	6 / 2	$-$ 1.1 to 3.1	1
DOTA	98	9	0.05 to 0.22	4.4 to 16.3	7 / 2	$-$ 1.5 to 3.9	0
FSC-147	81	2	0.39 to 0.90	$-$ 21.8 to 0.3	8 / 0	0.0 to 0.4	0
SKU-110K	96	9	0.00 to 0.39	10.8 to 68.2	7 / 2	$-$ 14.1 to 13.8	0
GroundCUA	86	6	0.06 to 0.72	$-$ 16.8 to 37.7	4 / 5	$-$ 7.7 to 15.4	1
ScreenParse	63	8	0.19 to 0.50	0.0 to 2.1	5 / 2	$-$ 0.5 to 0.9	0

Table 11: Confirmation by a partner call, split into selection and geometry (Propositions 1 and 3 ). Each source has 9 base/partner pairs (3 base models

\times

same-model or one of two other models), 60 images, one call each; selection, geometry and their total are per-image reductions of

E

(positive is better). Both identities hold exactly on every image of every pair. Sign from 5 labels :

p_{1}

is estimated on 5 labelled images and Proposition 2 predicts whether deletion lowers

E

on the other 55; share of correct predictions over 200 draws per pair, averaged over the pairs (Section 7.4 ). Deletion helps : pairs whose selection term is positive. Geometry

+

-

: pairs whose geometry term is positive / negative. Sign flips : pairs in which the total with averaged coordinates has the opposite sign of the deletion-only selection term. Per-pair values are in Table B.2 .

Figure 5: Selection and the deletion threshold. (a) Selection gain of deleting unconfirmed records, as a reduction of set error

E

per image (positive: deletion lowers

E

), against the marginal matching rate

p_{1}

of the deleted records, for the 54 base/partner pairs (symmetric log scale; grey circles: same-model partner; blue squares: cross-model partner; open markers: screen sources). With equal error costs the gain per image equals

\mathbb{E}[\Delta c](1-2p_{1})

(Proposition 2 ), so its sign follows the side of

p_{1}=1/2

; the panel shows where each pair falls. (b) Reduction of

E

relative to one call when selecting among three calls of one model (detection sources, 60 images per cell): the constrained oracle, the cross-fitted logistic selector, and the logistic selector fitted on 10 labelled images (median of 40 draws). GPT, Son., Gem.: GPT-6.1, Sonnet 5.5, Gemini 3.8.

Figure 6: Risk-coverage of click alarms (260 items; point estimates). An alarm sets an item aside when the acted-on point and a second point for the same item are farther apart than

\delta

of the longer image side; the system acts on the other items. Acceptance coverage is the share of items acted on, and the miss rate when acting the share of those items whose point is outside the target; lower and further right is better. Each curve traces

\delta\in\{0.5,1,1.5,2,3,5,8\}\%

, and a smaller

\delta

sets more items aside. Grey: the second point comes from a second call of the same model; blue: from the partner, the cheapest other model (Gemini 3.8 for GPT-6.1 and Sonnet 5.5, Sonnet 5.5 for Gemini 3.8). Solid lines and filled markers: first clicks on the whole screenshot; dashed lines and open markers: both points after two-stage zoom. Rings: the shipped default

\delta=1.5\%

(Table 12 ). Diamonds: acting on every item. The cross-model alarm reaches lower miss rates than the same-model alarm, at the price of setting more items aside.

Model	Partner	Alarm	Miss rate, all items (%) $\downarrow$	Acceptance coverage (%) $\uparrow$	Miss rate when acting (%) $\downarrow$	Errors caught (%) $\uparrow$
GPT-6.1	Gemini 3.8	same model, unzoomed	8.9	95.8	7.6	17
		cross-model, unzoomed	8.9	85.8	4.0	61
		same model, after zoom	7.7	94.2	4.1	50
		cross-model, after zoom	7.7	87.7	2.6	70
Sonnet 5.5	Gemini 3.8	same model, unzoomed	17.3	91.1	12.2	36
		cross-model, unzoomed	17.3	78.5	6.4	71

Table 12: Disagreement alarms for clicks (ScreenSpot-Pro and ScreenSpot-v2, 260 items; partner set by a fixed rule as the cheapest other model; point estimates). The alarm compares the acted-on point with a second point for the same item and fires when the two are farther apart than

\delta=1.5\%

of the longer image side; in this risk-coverage analysis the item is then set aside instead of acted on (in Algorithm 2 it triggers a comparison instead). Unzoomed alarms compare first clicks; after zoom, the acted-on point is the zoomed first click, and the second point is the same model’s zoomed second click or the partner’s zoomed first click. Miss rate, all items: miss rate when acting on every item. Acceptance coverage: share of items acted on. Miss rate when acting: share of acted-on items whose point is outside the target. Errors caught: share of the acted-on prediction’s misses that the alarm sets aside. Other values of

\delta

are in Figure 6 , and miss rates at matched acceptance coverage in Table B.3 .

(a) Flows
Flow	Misses	No point	Miss rate (%)	Calls	Cost / item ($)
GPT-6.1 zoom, lowest	86	4	14.33	1.99	0.0081
Sonnet 5.5 zoom, lowest	104	15	17.33	1.98	0.0057
Gemini 3.8 zoom, lowest	88	0	14.67	2.00	0.0054
Sonnet 5.5 zoom, default	80	0	13.33	2.00	0.0066
GPT-6.1 zoom, default	82	2	13.67	2.00	0.0085

Table 13: Choosing one click among several calls, on 600 ScreenSpot-Pro items that share no item or screenshot with the 260 click items of the GUI experiment. (a) Every flow, with every item in the denominator; a flow that returns no point counts as a miss, which happened only in single-model flows (column No point). Calls and cost: what the flow needs per item, every attempt counted, at list prices. The three-model flows use GPT-6.1 and Sonnet 5.5 at the default setting and Gemini 3.8 at the lowest, unless marked “all lowest”. Cluster vote: the largest cluster of points within

\delta

wins, ties go to the earliest model (GPT-6.1, Sonnet 5.5, Gemini 3.8), and the earliest point of the winning cluster is returned; computed from the stored calls, no scorer. (b) Paired comparisons:

\Delta

is the miss rate of the first flow minus that of the second, in percentage points (negative favours the first), with a 95% paired bootstrap interval over screenshots, items repaired and broken by the first flow, and the exact McNemar

p

(Holm-adjusted over the two comparisons marked

\dagger

). Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction.

Source	Model	Constrained	Unrestricted	Cross-fit logistic	Cross-fit trees	In-sample logistic	In-sample trees	10 labels logistic	Gap [95% CI]
		Oracles		Selectors fitted with labels
DocLayNet	GPT-6.1	63.2	63.2	$-$ 8.4	$-$ 7.4	1.6	42.8	$-$ 7.7	71.6 [60.1, 87.4]
DocLayNet	Sonnet 5.5	59.8	59.8	17.3	14.8	22.6	50.1	16.6	42.5 [34.2, 54.0]
DocLayNet	Gemini 3.8	45.2	45.2	5.9	1.3	6.9	37.6	3.2	39.3 [30.1, 50.3]
DOTA	GPT-6.1	74.3	74.4	39.9	39.6	40.4	52.1	39.6	34.4 [25.1, 42.6]
DOTA	Sonnet 5.5	41.9	41.9	23.3	22.6	24.1	31.1	21.3	18.6 [13.7, 25.1]

Table 14: Selection among three calls of one model (detection sources, list interface, lowest setting, 60 images per cell; reduction of

E

relative to the first call, in %, higher is better). The constrained oracle optimizes over the same feasible family as the selectors (Proposition 12 ); the unrestricted value lets two members of one cluster match two targets and is not attainable. The two oracles differ on 25 of 720 images. Selectors are logistic or boosted-tree record gates on output features; they read no labels at inference but are fitted with labels: cross-fitted (5 folds by image, persisted), in-sample, or on 10 labelled images (median over 40 draws, each evaluated on the images it did not use). Gap: constrained oracle minus cross-fitted logistic, with a 95% image-bootstrap interval for the fitted selector; it mixes information absent from the features with selector and estimation shortfall (Proposition 7 ).

	Same model only						Any construction (incl. cross-model)
	$E$			F1			$E$			F1
Source	GPT	Son.	Gem.	GPT	Son.	Gem.	GPT	Son.	Gem.	GPT	Son.	Gem.
DocLayNet	$+$ 7 ⋆	$-$ 10	$+$ 3	$-$ 3 ⋆	$-$ 1	$+$ 19	$+$ 4 ⋆	$-$ 36	$-$ 13	$-$ 7 ⋆	$+$ 15	$+$ 75
DOTA	$-$ 22	$-$ 16 ⋆	$-$ 25	$-$ 1 ⋆	$-$ 1	$+$ 2	$-$ 30	$-$ 17 ⋆	$-$ 22	$-$ 6 ⋆	$+$ 11	$+$ 17
FSC-147	$+$ 3 ⋆	$+$ 7	$+$ 5	$-$ 2 ⋆	$-$ 6	$+$ 1	$+$ 58 ⋆	$-$ 1	$-$ 1	$-$ 35 ⋆	$+$ 1	$+$ 68
SKU-110K	$-$ 25	$-$ 32	$-$ 10 ⋆	$-$ 8 ⋆	$+$ 1	$+$ 13	$-$ 30	$-$ 36	$-$ 8 ⋆	$-$ 22 ⋆	$+$ 165	$+$ 21

Table 15: Defaults evaluated by leave-one-source-out transfer: for each source, the construction is the one that is best on the other five sources (same model only, or any construction including cross-model ones), and it is evaluated on the held-out source against a single call of the user’s model. Entries are relative changes of the loss in % (for

E

, negative is better; for F1, positive is better). Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction. ⋆ : the user’s model is the strongest single model on that source and loss; every significantly worse cross-model entry carries a

\star

. GPT, Son., Gem.: GPT-6.1, Sonnet 5.5, Gemini 3.8. Costs, recall and the chosen construction per cell are in Table B.4 .

Source	Model	Interface	Setting	Change	95% CI
		Chosen
DocLayNet	GPT-6.1	regions	lowest	$+$ 1.5	[ $-$ 2.6, 6.0]
DocLayNet	Sonnet 5.5	regions	default	$-$ 6.5	[ $-$ 10.4, $-$ 3.1]
DocLayNet	Gemini 3.8	list	default	$+$ 14.4	[6.6, 22.5]
DOTA	GPT-6.1	regions	lowest	$-$ 1.7	[ $-$ 5.3, 1.7]
DOTA	Sonnet 5.5	regions	default	$+$ 3.1	[ $-$ 1.8, 7.4]

Table 16: Single-call configuration chosen on the other three detection sources and applied to the fourth. F1 at the primary scoring of Table 2 (boxes: same class, IoU

\geq 0.75

; FSC-147: point tolerance), in %. Reference: list interface at the lowest setting. Change: chosen minus reference, in F1 points, with a 95% paired bootstrap interval over images. Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction. Macro F1, the mean over the 12 cells: 42.1 (reference) and 49.1 (chosen); the best configuration of each cell chosen with its own labels gives 51.7. The shipped configuration (Table 4 ) is chosen by the same rule on all four sources (Table B.5 ).

Component	Use when	Measured effect	No significant gain, or harm, when	Cost
List first, read the list count; stated count only if the list is flagged	counting, above all several classes in dense scenes	per-type count error $-85.2\%$ (GPT-6.1), $-98.8\%$ (Sonnet 5.5); same response, readout only: $-71\%$ to $-84\%$ (§ 7.2 )	one-class photographs, and aerial or document images whose lists miss objects: small, n.s. for GPT-6.1 and Sonnet 5.5 (§ 7.2 )	one call
Allowed names in the prompt, aliases mapped in code	the model uses names outside the vocabulary	Sonnet 5.5’s “diamond”: error 5.42 to 0.42 on the same lists (Table 8 )	names already in the vocabulary	no extra calls
Degeneracy flags; Gemini 3.8 at its default setting for counts	long lists that may break	flagged lists 90% to 12%, count error $-53.6\%$ at the same cap (Table 10 )	Gemini 3.8 at its lowest setting, even with the fallback: $+15.6\%$ , n.s.	more reasoning tokens
Two-stage zoom	clicks, small targets	miss rate $-65\%$ (Gemini 3.8), $-41\%$ (Sonnet 5.5) (§ 7.5 )	GPT-6.1: n.s.	about 2 calls
Views from a planner, merged; each view confirms only records it contains	small objects; views from a planner or crops	four crops (6% overlap) confirming a whole-image report: set error up to $-45.5\%$ , better in 4 of 6 cells (§ 7.5 )	where unconfirmed records are mostly true ( $p_{1}>1/2$ ): worse in 2 of 6 cells; tiles without a one-object overlap lose large objects	one call per view
Three-call unanimity, coordinates averaged	set error on over-reporting families (detection, layout, screens)	significantly better in 9 of 15 cells, significantly worse in none (§ 7.9 )	F1 (2 better, 4 worse, 12 n.s.); under-reporting dense points (worse in 2 of 3)	3 calls

Table 17: FSR at a glance: what each component is for, its measured effect, where it gives no significant gain or harms, and its cost. Effects are relative changes of the stated loss against one call of the same model unless stated; “n.s.”: not significant. The last row lists remedies with small or no significant gains.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	$n$	Stated	Error	Change (%)	95% CI	Error
				Raw list count			Mapped list count
PixMo-Count total, 6.0 obj.	GPT-6.1	100	0.16	0.15	$-$ 6.3	[ $-$ 23.1, 0.0]	not applicable (one class)
	Sonnet 5.5	100	0.17	0.14	$-$ 17.6	[ $-$ 38.5, 0.0]	not applicable (one class)
	Gemini 3.8	100	0.33	0.27	$-$ 18.2	[ $-$ 37.5, 0.0]	not applicable (one class)
FSC-147 total, 43.8 obj.	GPT-6.1	100	2.73	2.70	$-$ 1.1	[ $-$ 8.5, 8.8]	not applicable (one class)
	Sonnet 5.5	100	9.24	8.85	$-$ 4.2	[ $-$ 11.9, 2.1]	not applicable (one class)

Appendix

Table B.1: Same-response readout across datasets (count-first prompts, lowest setting, one call; data of Figure 4 ). Count error of the stated count and of the list count in the same response, without and with the declared alias map (lower is better; total error for one-class datasets, per-type error otherwise). Change: relative change from the stated count to the list count, with its paired image-bootstrap 95% interval; this is a separate run from Table 8 , so shared rows can differ in the last digit of an interval, and the two list-count columns use separate draws. Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction. Under each dataset: the error measure and the mean number of objects per image.

Source	Base	Partner	$\Delta c$	$\Delta r$	$p_{1}$	Selection	Geometry	Total
DocLayNet	GPT-6.1	same model	0.92	0.48	0.527	$-$ 0.05	$-$ 0.30	$-$ 0.35
DocLayNet	GPT-6.1	Sonnet 5.5	2.63	0.92	0.348	0.80	$-$ 1.10	$-$ 0.30
DocLayNet	GPT-6.1	Gemini 3.8	8.08	5.15	0.637	$-$ 2.22	0.43	$-$ 1.78
DocLayNet	Sonnet 5.5	same model	1.33	0.32	0.237	0.70	0.10	0.80
DocLayNet	Sonnet 5.5	GPT-6.1	3.03	0.45	0.148	2.13	2.77	4.90
DocLayNet	Sonnet 5.5	Gemini 3.8	8.30	3.97	0.478	0.37	3.07	3.43

Appendix

Table B.2: Selection and geometry terms for every base/partner pair (60 images per source, one call each; data of Table 11 ).

\Delta c

: deleted scored records per image;

\Delta r

: matches lost per image;

p_{1}=\Delta r/\Delta c

, the marginal matching rate of the deleted records. Selection, geometry and total are per-image reductions of

E

(positive is better); selection plus geometry equals the total on every row (Propositions 1 and 3 ).

Model	Points	Acceptance coverage (%)	Miss rate, same model (%)	Miss rate, cross-model (%)
GPT-6.1	unzoomed	93.1	7.0	5.8
	after zoom	90.4	3.8	2.9
		91.7	3.8	3.4
		93.1	3.7	3.7
Sonnet 5.5	unzoomed	87.7	10.5	10.1
		88.5	11.3	10.6

Appendix

Table B.3: Click alarms at matched acceptance coverage (260 items; data of Table 12 and Figure 6 ; point estimates without intervals). Rows are the acceptance coverages that the same-model and the cross-model alarm both reach on the grid of

\delta

; miss rates are those of the acted-on items, lower is better.

Source	Model	Default applied	Single	Default	95% CI	Recall	Cost
Same-model constructions only; set error $E$ per image (lower is better)
DocLayNet	GPT-6.1	unanimity of 3 calls, averaged	8.57	9.15	[ $-$ 0.35, 1.65]	67.7	0.0323
DocLayNet	Sonnet 5.5	unanimity of 3 calls, averaged	13.77	12.33	[ $-$ 2.25, $-$ 0.70]	57.1	0.0384
DocLayNet	Gemini 3.8	unanimity of 3 calls, averaged	11.83	12.18	[ $-$ 0.63, 1.42]	23.0	0.0132
DOTA	GPT-6.1	unanimity of 3 calls, averaged	45.00	34.98	[ $-$ 16.73, $-$ 4.97]	46.4	0.0785
DOTA	Sonnet 5.5	unanimity of 3 calls, averaged	37.87	31.95	[ $-$ 8.70, $-$ 3.77]	32.1	0.0472

Appendix

Table B.4: Leave-one-source-out transfer, every cell (data of Table 15 ). Default applied: the construction chosen on the other five sources; “averaged” means the coordinates of the kept records are averaged over the calls that report them. Single and Default: loss of one call of the model and of the default applied. 95% CI: paired image bootstrap of the change, default minus single (negative is better for

E

, positive for F1). Bold : the paired 95% interval excludes 0 in the favourable direction; underlined : it excludes 0 in the unfavourable direction. Recall (%) and cost per image ($, every call counted) are those of the default applied.

	GPT-6.1		Sonnet 5.5		Gemini 3.8
Configuration	F1	Cost	F1	Cost	F1	Cost
list, lowest	56.2	0.0295	37.1	0.0201	33.1	0.0084
regions, lowest	58.6	0.0327	31.6	0.0291	39.4	0.0101
list, default	54.0	0.0436	44.7	0.0237	45.0	0.0192
regions, default	56.7	0.0475	48.1	0.0477	44.8	0.0200

Appendix

Table B.5: The four fixed single-call configurations on the four detection sources: F1 (%, primary scoring of Table 2 ), the mean over sources of the per-source mean F1 of the first of the three stored calls, and cost per image ($, list prices). The shipped F1 configuration of each model is the highest F1 of its column (bold); for Gemini 3.8 the two default-setting configurations differ by less than 0.2 points.

Experiment	Data and arms	Models	Calls	Cost
Pilot v1	12 procedural scenes, 768 px; responsibility-split arms; cap 8,192	G5.6, G6.1, S5.5, O5.5, Q	3,540	$29.30
Pilot v2	24 scenes, 1024 px, 16 to 384 targets; dose response	same	4,080	$47.50
Subset execution	12 scenes, 96 or 192 targets; representation and drift arms	G5.6, G6.1, O5.5, S5.5	1,924	$53.84
Scan responsibility and grids	48 scenes; scan responsibility, grid on/off, granularity; repairs, smoke, supplement	above + Gem	2,752	$113.80
Detection	SKU-110K, DOTA, DocLayNet, FSC-147, 60 images each; list / regions $\times$ lowest / default; 16k control	G6.1, S5.5, Gem	6,523	$171.13
GUI	ScreenSpot-Pro 200, ScreenSpot-v2 60, GroundCUA 60, ScreenParse 60; click and parse arms; pilots 1,050	same	7,232	$67.77

Appendix

Table C.1: All experiments with model calls (67,364 calls, about $758).

Model	Lowest setting as sent	Observed
GPT-6.1	reasoning_effort=low	reasoning in most calls (SCP 100%, PixMo 61%, clicks 39%); default: 100%
GPT-5.6	effort none	no reasoning
Sonnet 5.5	thinking: between_tools	lowest: 0%; default (adaptive thinking): 0% to 23% on the dense, FSC and screen runs, 4 of 48 calls on controlled scenes, 0% to 90% on the detection sources by cell; pilots: not logged
Opus 5.5	effort low (adaptive thinking cannot be disabled)	under 1% of logged calls at effort low; pilots: not logged
Gemini 3.8	thinking_budget=0	little on list and box tasks (FSC list 6%, regions 12%); still reasons on clicks and points (GUI clicks about 99%, PixMo 23%, FSC points 32%); default: 51% to 100%
Qwen3-VL	instruct model, no reasoning parameter	no reasoning

Appendix

Table C.2: Reasoning settings as sent and as observed (share of calls with reasoning tokens).

#	Claim	Operation	Comparator	Metric	Evidence
H1	On SCP-dense image 28, GPT-6.1 lists 173 records that match all 173 shapes and states counts that sum to 188. Exact lists: GPT-6.1 20 of 50 scenes, with wrong stated counts in 19; Sonnet 5.5 39 of 50, wrong in all 39. List length misses the true total by 0.04 and 0.02; stated totals by 7.78 and 10.48	none (same response)	stated counts, same response	exact list (every record matched, every shape covered); total-count error; SCP-dense 50, lowest setting	Fig. 1 , Table 8
H2	Reading the list cuts per-type count error by 71% to 84%	list count	stated per-type counts, same response	per-type error; GPT $-75.6\%$ / $-84.2\%$ , Sonnet $-77.7\%$ / $-70.7\%$ (two protocols)	Table 8
H3	Complete reporting and decoding protocol $-85.2\%$ (GPT-6.1), $-98.8\%$ (Sonnet 5.5); Gemini 3.8 (lowest) $+15.6\%$ $[-5.3,39.4]$ , not significant	list-first request, scan, allowed names and alias map, readout of Step 6	count-first prompt, stated count	per-type error	Table 8
H4	Alias map takes Sonnet 5.5 from 5.42 to 0.42	alias map on the same list	raw list count of the same list	per-type error	Table 8
H5	Degeneracy check flags 45 of 50 Gemini 3.8 (lowest) lists; 40 fall back to the stated count, 5 keep the list count	four L3 flags	none	flag rate, readout branch	Table 10
H6	Default setting at the same 16k cap: flagged lists 90% to 12%, readout error $-54\%$ $[-72,-30]$ ; 32k cap: further $-37\%$ $[-62,8]$ , not significant	setting, then cap	lowest 16k, then default 16k	count readout error	Table 10

Appendix

Table E.1: Headline claims.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 734 papers

Feb 26, 2026

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

The lack of reasoning capabilities in Vision-Language Models (VLMs) has remained at the forefront of research discourse. We posit that this behavior stems from a reporting bias in their training data. That is, how people communicate about visual content by default omits tacit information needed to supervise some types of reasoning; e.g., "at the game today!" is a more likely caption than "a photo of 37 people standing behind a field". We investigate the data underlying the popular VLMs OpenCLIP, LLaVA-1.5 and Molmo through the lens of theories from pragmatics, and find that reporting bias results in insufficient representation of four reasoning skills (spatial, temporal, negation, and counting), despite the corpora being of web-scale, and/or synthetically generated. With a set of curated benchmarks, we demonstrate that: (i) VLMs perform poorly on the aforementioned types of reasoning suppressed in the training data by reporting bias; (ii) contrary to popular belief, scaling data size, model size, and to multiple languages does not result in emergence of these skills by default; but, promisingly, (iii) incorporating annotations specifically collected to obtain tacit information is effective. Our findings highlight the need for more intentional training data curation methods, rather than counting on scale for emergence of reasoning capabilities.

Amita Kamath, Jack Hessel, Khyathi Chandu +3

University of Washington · University of California, Los Angeles · Samaya AI +2

VLM Reasoning Reasoning for VLMs Pragmatic Reasoning in Language Models Training Data Curation Visual Spatial Reasoning

Ask PDF

Apr 10, 2026

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

When a Vision-Language Model (VLM) sees a blue banana and answers "yellow", is the problem of perception or arbitration? We explore the question in ten VLMs with various sizes and reveal an Encoding-Grounding Dissociation: models that fail to report what they see (and thus provide a wrong answer) still encode the visual evidence as strongly as models that provide the correct answer. Using Multimodal Arbitration Crossover (MAC) analysis with layer-by-layer Logit Lens probing, we track the competition between visual and prior signals across every layer of each model. We show that visual attributes can be linearly decodable from early layers (AUC > 0.86). The accuracy remains nearly identical for both successful and failed samples. However, the gap in the final-layer logit - not the strength of encoding - better predicts grounding outcomes with a correlation of

ρ=

0.847. After having studied when VLMs base their answers on image clues rather than prior knowledge, we want to understand the causal relationships. We establish causality through full-sequence activation patching. The standard last-token interventions in LLM interpretability do not affect VLMs. In contrast, replacing the full token sequence at layers identified by MAC alters 60 to 84% of outputs. Partial-token decomposition shows that image tokens carry almost all of the causal impact, while text tokens have none. Scaling addresses the remaining architectural differences to achieve perfect retention. Moving from diagnosis to intervention, we show that training-free activation steering - both linear and sparse autoencoder-guided - in early layers can improve visual grounding by up to +3.8% with degrading performance in some setups. Overall, these findings lead to a clear conclusion: VLMs already see well, but the challenge is acting on what they see. Targeted interventions can help to bridge this gap.

Farhad Nooralahzadeh, Omid Rohanian, Yi Zhang +2

Institute of Computer Science, Zurich University of Applied Sciences, Switzerland · University of Oxford, UK.

Vision-Language Grounding Visual Evidence Sensitivity in VLMs Faithfulness in VLMs VLM Interpretability

Ask PDF

Apr 11, 2026

Counting to Four is still a Chore for VLMs

Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks, yet they still fail on simple grounding skills such as object counting. Existing evaluations mostly assess only final outputs, offering limited insight into where these failures arise inside the model. In this work, we present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis. We introduce COUNTINGTRICKS, a controlled evaluation suite of simple shape-based counting cases designed to expose vulnerabilities under different patchification layouts and adversarial prompting conditions. Using attention analysis and component-wise probing, we show that count-relevant visual evidence is strongest in the modality projection stage but degrades substantially in later language layers, where models become more susceptible to text priors. Motivated by this finding, we further evaluate Modality Attention Share (MAS), a lightweight intervention that encourages a minimum budget of visual attention during answer generation. Our results suggest that counting failures in VLMs stem not only from visual perception limits, but also from the underuse of visual evidence during language-stage reasoning. Code and dataset will be released at https://github.com/leduy99/-CVPRW26-Modality-Attention-Share.

Duy Le Dinh Anh, Patrick Amadeus Irawan, Tuan Van Vo

MBZUAI

Attention Intervention in VLMs Vision-Language Models VLM Evaluation VLM Interpretability

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning

Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts

Counting to Four is still a Chore for VLMs