Leading GUI grounders score 93 to 96% on ScreenSpot-v2, and reported gaps between new and prior models are about a point. At this level the useful question is what the remaining errors are. Benchmarks count a click as correct only inside one annotated box, but interfaces often expose one action through several elements: a duplicated link, the label of a checkbox, the title next to a logo. The box is not wrong. The rule is incomplete. We call a click correct when executing it produces the same outcome as clicking the annotated target. A probe that executes and fingerprints about a thousand clicks per page turns this definition into outcome maps and an execution judge for live web applications, with no new annotation. We execute the predictions of three strong 7--8B grounders on a code forge and an online shop. About one in five and one in three of the clicks that box scoring rejects reach the intended outcome, and the shares remain one in seven and one in three when the instructions are rewritten as natural tasks. A blind audit of 11 models on ScreenSpot-v2 (three 7--8B grounders and eight 3B variants) finds the same effect on web screenshots, with two independent reviewers on a 60-unit subset. The share is highest for the strongest grounders. Training seeds and test-set sampling add noise that more runs and paired intervals control. Box scoring instead under-counts correct clicks, which repetition does not remove, and the audit-based correction of 0.2 to 1.8 points on ScreenSpot-v2 is as large as the gaps the field reports. The correction is needed in evaluation. In our 3B GRPO study, crediting equivalent clicks in the reward gives no stable gain over box rewards across three seeds and four test sets. We release the probe, the outcome maps, the audit and every prediction.
Figures & tables
Figure 2: The probe. Each screen point p is executed in the interface state s and fingerprinted by the first matching signature. Running it over a grid gives an outcome map, and running it on a model’s prediction gives an execution judgment.
Instrument
Check
Result
Probe
Same outcome after fresh reload
97.9 / 99.8%
Same outcome across probe versions
99.4%
Disagreements confirmed in self-check
28/30
Server database after probing
unchanged
Judge
Box centre reaches mapped outcome
98.1 / 98.7%
Same verdict on re-execution
98.6 / 98.7%
Table 1: Checks on each instrument. Pairs of values are Gitea / Oscar. Probe repeatability is the mean per page of a random 10% of points re-probed from a fresh load. Version agreement covers 60,995 points on 58 pages. Re-execution covers 712 predictions. Cohen’s κ [ 8 ] is computed over E, N and U on 60 units. Details are in the supplement.
Type
Targets
Duplicate link (navigation to the same URL)
566
Label and its control
86
Other page-change equivalent
60
At least 10% inert area in the box
64
Clickable area beyond the box
69
Any disagreement
774
Table 2: Disagreeing targets by type on the 85 pages with at least 95% repeatability (4,436 live targets). A target can have several types.
Model
Box
Outcome
Δ [95% CI]
Rescued
Gitea (3,586 items)
UI-Venus-1.5-8B
82.3
84.3
+2.0 [0.6, 3.3]
130 (20.5%)
GTA1-7B
81.1
83.2
+2.2 [0.9, 3.4]
135 (19.9%)
Qwen3-VL-8B
79.7
81.9
+2.2 [0.9, 3.5]
128 (17.6%)
Box centre
100
98.1
0
Oscar (1,147 items)
Table 3: Execution judging on two live applications. Accuracy (%) under box scoring and outcome scoring. “Rescued” counts box-rejected clicks that reach the target outcome, with their share of all box-rejected clicks. The box centre is a reference click; its agreement with the mapped outcome bounds environment drift. Intervals resample pages.
Figure 3: Outcome-correct clicks that ScreenSpot-v2 rejects. Zoomed audit panels: red box is the annotation, blue circle the model click, instruction in quotes. E-same clicks hit the target control outside its box, E-dup clicks hit another control with the same function. In the last panel the click follows the instruction and the box marks a different control. We release such cases as likely annotation errors and count them as N.
Split
Units
E-same
E-dup
N
U
E share [95% CI]
Web
145
8
12
119
6
14.4 [9.5, 21.2]
Mobile
104
1
0
97
6
1.0 [0.2, 5.6]
Desktop
111
0
1
105
5
0.9 [0.2, 5.2]
All
360
9
13
321
17
6.4 [4.3, 9.5]
Table 4: Screenshot audit of ScreenSpot-v2. Box-rejected clicks of 11 models, judged blind. E share is (E-same + E-dup)/(E + N) in % with Wilson intervals. Counting all U units as N or as E bounds the overall share between 6.1% and 10.8%.
Figure 4: The strongest grounders have the largest share of outcome-correct errors on ScreenSpot-v2. Share of each model’s audited box-rejected clicks judged equivalent, with Wilson intervals, against its box accuracy.
Figure 5: Seeds and sampling set the resolution of ScreenSpot-v2. (a) Outcome-reward minus box-reward accuracy for the same recipe trained with three seeds. Blue marks intervals that exclude zero (Gitea resamples pages, ScreenSpot-v2 resamples screenshots, Mind2Web resamples tasks). On ScreenSpot-v2 seed 42 shows a significant gain and seed 44 an equally significant loss. (b) Head-to-head gaps between a proposed model and the strongest prior model of the same size, as reported on ScreenSpot-v2 by 28 recent papers (blue: proposed model ahead). The band marks ± 1.42 points, the size of the single-seed effects in (a) that reverse across seeds.
Gitea held-out
Oscar
ScreenSpot-v2
Mind2Web
outcome
outcome
box
box
Base model
78.4
91.0
75.0
83.7
Box reward
85.2 [85.2, 85.3]
89.6 [89.4, 89.9]
80.5 [80.0, 81.3]
86.5 [86.4, 86.6]
Outcome reward
85.0 [84.6, 85.5]
89.7 [89.3, 90.2]
80.4 [78.8, 81.4]
86.7 [86.6, 86.9]
GUI-Libra
84.8 [84.5, 85.3]
90.0 [89.8, 90.2]
80.9 [80.5, 81.2]
86.7 [86.2, 87.0]
Outcome − box
− 0.18
+0.06
− 0.08
+0.23
Table 5: Reward definition and held-out accuracy (%). Each arm shows the mean over seeds 42, 43 and 44 with the range across seeds, and the last row gives the difference of the outcome and box means in points. Per-seed differences with confidence intervals are in Fig. 5 a.
Figure 6: Error budget of one-point comparisons on ScreenSpot-v2. Each row shows one source of variation on the same scale as the margins that recent papers report. Seed ranges come from the three box-reward and the three outcome-reward runs of Sec. 5 , interval half-widths from the three pairs of 7–8B grounders, and corrections from the audit (point estimates; intervals in the supplement). The rows have different statistical meanings, so the figure compares scales and is not an additive decomposition.
Figure S1 : Outcome maps of a self-hosted Gitea instance, made by executing about a thousand clicks per page. (a) Clicks that produce the same outcome share a colour. Each commit is opened by its hash and by its message, and the author name in every row opens the same profile. (b) Clicking the label of a text field (blue box) focuses the field (orange). (c) The annotated box covers the middle of an avatar, and clicks anywhere on the avatar open the same profile. Green dots reach the outcome of the annotated target and red dots reach a different one. Box scoring counts every green dot outside the blue box as an error.
Order
Signature
Recorded as
1
nav
main-frame navigation, aborted
2
popup
new window or tab
3
req
non-GET request, aborted
4
dialog
alert or confirm, dismissed
5
dom
visible change of the page
6
noop
none of the above
Table S1: Outcome signatures.
Same-outcome
Different-outcome
Unique
copies (193)
copies (301)
(3,092)
UI-Venus-1.5-8B
35.2 / 95.9
32.9 / 32.2
90.0 / 88.6
GTA1-7B
34.2 / 100
35.2 / 33.9
88.5 / 87.0
Qwen3-VL-8B
28.5 / 90.7
29.6 / 28.9
87.8 / 86.5
Box centre
100 / 100
100 / 98.0
100 / 98.0
Table S2 : Box / outcome accuracy (%) by item type on Gitea.
Model
Box
Outcome
Δ [95% CI]
Rescued
Gitea
UI-Venus-1.5-8B
71.9
75.0
+3.1 [1.7, 4.5]
16.9%
GTA1-7B
69.4
72.6
+3.2 [2.0, 4.5]
15.2%
Qwen3-VL-8B
64.9
67.4
+2.5 [1.1, 3.8]
11.5%
Oscar
UI-Venus-1.5-8B
79.6
85.9
+6.3 [4.3, 8.3]
35.9%
GTA1-7B
78.5
84.7
+6.2 [4.5, 7.8]
34.0%
Qwen3-VL-8B
73.9
82.3
+8.4 [6.7, 10.3]
34.7%
Table S3 : Execution judging with natural instructions (accuracy in %). “Rescued” is the share of box-rejected clicks that reach the target outcome. All 3,586 Gitea and 1,149 Oscar items were rewritten without exclusions. Pooled over models, the rescued share on items whose template text was unique is 6.8% on Gitea (2.8% with the template) and 30.6% on Oscar (16.4%).
Pair
Agreement
Cohen’s κ
Reviewer A vs reviewer B
98.3%
0.95
Ours vs reviewer A
96.7%
0.89
Ours vs reviewer B
95.0%
0.84
Table S4: Agreement on the 60 review units.
Model
Units
E/N/U
Box
Corrected [95% CI]
Rank
UI-Venus-1.5-8B
20
3/17/0
96.31
96.86 [96.31, 97.54]
1 → 1
GTA1-7B
33
8/22/3
93.40
95.16 [94.13, 96.29]
2 → 2
Qwen3-VL-8B
35
5/27/3
93.08
94.16 [93.34, 95.12]
3 → 3
3B, outcome s42
84
3/78/3
81.37
82.06 [81.37, 82.93]
4 → 4
3B, box s43
78
1/74/3
81.29
81.53 [81.29, 82.08]
5 → 7
3B, outcome s43
82
3/77/2
81.05
81.76 [81.05, 82.64]
6 → 5
Table S5 : ScreenSpot-v2 audit per model (accuracy in %). The correction only credits rejected clicks, since accepted clicks with a different outcome cannot be seen on screenshots. Ranks in bold change.
Pair (A vs B)
Box
Δ corr. [95% CI]
Corrected [95% CI]
box s44 vs box s42
+0.24
− 0.71 [ − 1.60, − 0.07]
− 0.48 [ − 1.36, 0.17]
box s43 vs outcome s43
+0.24
− 0.47 [ − 1.46, 0.41]
− 0.23 [ − 1.22, 0.65]
box s43 vs GUI-Libra
+0.79
− 0.80 [ − 1.77, − 0.01]
− 0.01 [ − 0.98, 0.78]
outcome s42 vs box s42
+1.42
− 0.71 [ − 1.59, − 0.06]
+0.70 [ − 0.17, 1.36]
box s43 vs box s42
+1.34
− 1.16 [ − 2.34, − 0.26]
+0.18 [ − 1.00, 1.08]
UI-Venus vs GTA1
+2.91
− 1.21 [ − 2.45, − 0.01]
+1.70 [0.46, 2.90]
Table S6 : Selected pairs on ScreenSpot-v2 (points). The corrected-gap interval adds the box gap to the interval of the difference in correction, so it reflects audit uncertainty and holds the test set fixed. All 55 pairs are released.
Pair
Diff
95% CI
Discord.
McNemar p
GTA1 − Venus
− 2.91
[ − 4.25, − 1.65]
5.7%
< 0.001
Qwen3 − Venus
− 3.22
[ − 4.48, − 1.97]
5.3%
< 0.001
GTA1 − Qwen3
+0.31
[ − 1.26, 1.81]
8.0%
0.77
Table S7 : Paired comparisons of 7–8B grounders on ScreenSpot-v2 (points). Discord. is the share of items that exactly one of the two models gets right.
GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents. However, existing grounding models remain error-prone, often causing downstream task failures. We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors. Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors. We construct ConClick-Data, a dataset containing 4,292 challenging target-distractor pairs, and train the model using complementary listwise ranking and pairwise margin objectives. ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones. We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks. Together, our analysis, dataset, and methods provide both empirical insights and practical tools for building more reliable GUI agents. Data and code are available at https://github.com/zhangruotian/ConClick.
We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.
Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot. Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations. Region-based patch scoring alleviates this ambiguity, but its spatial decisions remain tied to visual-token footprints that may span the target, neighboring controls, and background. To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction. Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule. This formulation captures the spatial extent of valid click locations and enables finer localization beyond the visual-token grid, making it well suited to small, densely arranged controls without requiring a verifier or zoom-in pass. Experiments across five backbone configurations from two model families demonstrate the effectiveness of Seg2Click. Seg2Click-9B achieves 92.06%, 94.10%, and 55.22% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding. Code will be available at https://github.com/jcwang0602/Seg2Click.
Jingchao Wang, Dingjiang Huang
School of Data Science and Engineering East China Normal University, Shanghai 200062, China