Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset (κ=0.94 well-formedness; κ=0.79 target selection). Nineteen vision-language models reach at most 32% strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on 60−92% of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman ρ=+0.74), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.
Figures & tables
Table 1
Figure 1: GUI-Primitives dataset construction pipeline. Two source corpora: real desktop screenshots from UI-Vision and controlled synthetic stimuli from our procedural renderer, feed into per-primitive minimal-pair miners, pass through automatic quality gates, are cross-checked by a five-annotator human verification pass (Fleiss κ=0.94 ), and released as the 994-item benchmark. Stage colors follow the primitive palette used throughout the paper.
Primitive
Relation words
Group
Ref. level
rel-pos-horizontal
left / right
cardinal
0.50
rel-pos-vertical
above / below
cardinal
0.50
containment
inside / outside
topology
0.50
alignment
same row / col.
layout
0.50
proximity
nearest / farthest
layout
0.50
list-ordinal
first / second
layout
0.50
Table 1: The seven elementary spatial primitives in GUI-Primitives , with their canonical relation words and two-candidate reference levels. All seven primitives, including list-ordinal , are realized as binary minimal-pair contrasts (71 items per side), so the reference level is 0.50 throughout; § 4 explains why this is a forced-choice reference rather than the chance level of unconstrained clicking.
Figure 2: A contrastive pair from GUI-Primitives (details in Appendix P ). The screenshot is fixed; one word flips, and the correct target shifts to the other candidate. A model exploiting an answer prior passes one twin and fails the other.
Figure 3: Shortcut controls (Qwen2.5-VL-7B, n=994 ). Text-only and shuffled drops are both highly significant; blur produces a smaller but significant drop, below the pre-registered ten-point threshold.
Model
Family
Full ( n=994 )
Core ( n=196 )
Clean ( n=185 )
Claude Opus 4.7
Anthropic
0.313
0.311
0.324
GPT-5 (reasoning)
OpenAI
0.265
0.291
0.297
Claude Sonnet 4.6
Anthropic
0.222
0.260
0.270
Claude Haiku 4.5
Anthropic
0.239
0.250
0.249
Qwen2.5-VL-7B
Qwen2.5-VL
0.220
0.245
0.249
Gemma 3-27B
Google open
0.134
—
0.157
Table 2: Overall accuracy on GUI-Primitives for 12 representative models, alongside accuracy on the human-clean 185-item subset. The strongest model (Claude Opus 4.7) reaches 32% on the clean core, a 65-point gap below human accuracy (96.9%). All 19 models are in Appendix A .
Figure 4: Human-clean core (185 items, 5 annotators, κ=0.94 ). Human accuracy is 96.9%. The strongest model is Claude Opus 4.7 at 32.4%.
Figure 5: Per-primitive accuracy across eight representative models, with bootstrap 95% confidence intervals. Dashed lines mark the two-candidate reference level (0.50 for all primitives; Table 1 ). On containment , occlusion , alignment , and proximity , every model falls below this reference, including the four strongest proprietary models; candidate-level analysis (Appendix R ) attributes this below-reference accuracy primarily to predictions outside both candidate regions.
Figure 6: Pair-consistency vs. raw accuracy, per model per primitive. Independent guessing would place each point near y=x2 (dashed). Almost all points lie below this curve, the What’s-Up signature of relational failure.
Figure 8: Intervention deltas vs. each model’s baseline, with Holm-adjusted significance (paired McNemar across primitives). SoM recovers 35–57 points on three of four tested models; the only marginal case is Claude Opus 4.7, which already has the highest baseline. CoT and activation steering yield no significant change.
Figure 9: Per-primitive SoM delta on Claude Opus 4.7 and GPT-5. Stars mark intervals excluding zero. On GPT-5, six of seven primitives gain at least 40 points.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Full
Core
Clean
Claude Opus 4.7
0.313
0.311
0.324
GPT-5 (reasoning)
0.265
0.291
0.297
Claude Haiku 4.5
0.239
0.250
0.249
Claude Sonnet 4.6
0.222
0.260
0.270
Qwen2.5-VL-7B-Instruct
0.220
0.245
0.249
Gemma 3-27B-it (HF)
0.134
—
0.157
Appendix
Table 3: Full benchmark accuracy for all 19 evaluated models on three splits: the full n=994 benchmark ( Full ), the human-verified n=196 subset ( Core ), and the cleaned n=185 subset after removing items judged invalid by a majority of the five annotators ( Clean ). Closed models were not run on the full split because of API cost; the Core column reports every model that was. Per-primitive accuracies with bootstrap intervals are released in the artifact.
Figure 10: Per-model per-primitive accuracy heatmap, all 19 models. The horizontal band at ∼ 0.80 on list-ordinal marks the only primitive any model solves reliably.
Model
Identifier
Family / Endpoint
Claude Opus 4.7
claude-opus-4-7
Anthropic API
Claude Sonnet 4.6
claude-sonnet-4-6
Anthropic API
Claude Haiku 4.5
claude-haiku-4-5
Anthropic API
GPT-5 (reasoning)
gpt-5
OpenAI API
GPT-4.1
gpt-4.1
OpenAI API
GPT-4o-mini
gpt-4o-mini
OpenAI API
Appendix
Table 4: Model identifiers used in this paper. Closed models were queried via official APIs in May 2026; open models were run from Hugging Face checkpoints with greedy decoding ( T=0 ) and a fixed seed.
Real (UI-Vision)
Synthetic
Model
strict
loose
strict
loose
Claude Haiku 4.5
0.000
0.230
0.363
0.763
Qwen2.5-VL-7B
0.028
0.234
0.300
0.714
OS-Atlas-Base-7B
0.000
0.162
0.142
0.724
Gemini 3.1 Flash Lite
0.000
0.115
0.111
0.741
GPT-4o-mini
0.000
0.033
0.111
0.733
Appendix
Table 5: Strict (point-in-box) vs. loose (within 2× target-diagonal) accuracy, split by image source. In real desktop screenshots, loose accuracy stays low even when the prediction is allowed to miss by a generous margin: failure is element misidentification, not sub-pixel imprecision.
Figure 12: Benchmark distribution analysis. (A) Contrastive-relation balance: every relation is represented by exactly 71 items per side, ruling out relation-prior exploitation. (B) Target–distractor spatial separation (normalized by image diagonal) per primitive; violin width encodes density. (C) Target size distribution (log-scale) by source arm; dashed lines are per-source medians. (D) Target centre positions (normalized to [0,1]2 , y=1 at top), coloured by primitive; targets span the full screen area.
Primitive
Real
Synthetic
Total
rel-pos-horizontal
68
74
142
rel-pos-vertical
48
94
142
alignment
66
76
142
proximity
84
58
142
list-ordinal
24
118
142
containment
0
142
142
Appendix
Table 6: Per-primitive composition of GUI-Primitives by source arm. Every primitive has 142 items, exactly 71 per relation word.
Figure 13: Model sensitivity analysis. (A) Left–right accuracy gap per model, sorted by magnitude. Red bars = left-biased (15/19 models; t=5.6 , p<.001 ). (B) Above–below accuracy gap. Weaker and not significant overall ( p=.10 ); OS-Atlas is an outlier with a strong above preference. (C) Mean accuracy stratified by target normalized area (tiny/medium/large) per primitive, pooled over all 19 models. (D) Parse-fail (null output) and default-coordinate rates per model; both are infrastructure errors, not spatial-reasoning failures.
Figure 14: rel-pos-horizontal (real screenshots). Three minimal pairs from UI-Vision. Within each row, both panels show the same screenshot; only the relation word ( right of vs. left of ), and the gold target moves accordingly. Across rows, the application and anchor change.
Figure 15: rel-pos-vertical (real screenshots). Three minimal pairs; above vs. below the anchor element across three different real GUIs.
Figure 16: alignment (real screenshots). Three minimal pairs in which the relation switches between same row as and same column as the anchor.
Figure 17: proximity (real screenshots). Three minimal pairs in which the two correct answers are the elements nearest to and farthest from the same anchor across three different real applications.
Figure 18: containment (synthetic). Three minimal pairs. Twin A is the contained within a parent; twin B is the parent containing that inner element. Synthetic stimuli are used because clean parent–child annotations are rare in real screenshot corpora.
Figure 19: list-ordinal (synthetic). Three minimal pairs of symmetric ordinal queries (the k -th item from the top vs. from the bottom). The only primitive every top-tier model reliably solves.
Figure 20: occlusion (synthetic). Three minimal pairs. Twin A is the element that is fully visible , twin B is the element that is partly hidden behind an overlay. A relational-shortcut model is drawn to the visually unusual (occluded) element on both twins, scoring zero at the pair level.
Figure 21: Case study 1: rel-pos-vertical , real screenshot. A: “Click the element above the replace.” B: “Click the element below the replace.” Opus and GPT-5 collapse to the same coordinate on both twins (both near (≈1180,130) ), getting B right by luck of which side they defaulted to and missing A. Qwen2.5-VL emits two completely unrelated clicks. Hollow markers = misses; filled markers = hits. Anchor-collapse failure.
Figure 22: Case study 2: proximity , real screenshot. A: “Select the button nearest to ‘current file directory’.” B: “Click the icon farthest from the current file directory.” All four models cluster their predictions in the top toolbar regardless of the instruction. The anchor (a sidebar element) and the two targets (sidebar items) are far from the predictions. This is the population-level pattern on proximity : 92% of predictions fall outside both candidate regions (Appendix R ), while the target is selected on 89% of within-region predictions, locating the deficit in candidate localization rather than relational interpretation.
Figure 23: Case study 3: alignment , synthetic. A: “Select the control aligned in the same row as ‘Paste’.” B: “Select the control aligned in the same column as ‘Paste’.” Models click the Paste anchor itself or a visually similar icon, instead of the aligned neighbor. Qwen2.5-VL emits an identical coordinate for both twins, the canonical “relation-ignored” signature.
Figure 24: Case study 4: alignment , real screenshot. A/B: “aligned in the same row/column as ‘Navigate Backward’.” The real-screenshot version of case study 3. GPT-5 wanders far from the anchor on both twins. Qwen2.5-VL again emits the same coordinate for both. alignment on real GUIs is the hardest cell in the benchmark.
Figure 25: Case study 5: occlusion , synthetic. A: the ‘Reset’ button that is fully visible . B: the ‘Reset’ button that is partly hidden behind a dialog. All four models point to the partly-hidden button on both twins (paired predictions differ by < 10 px). The models are drawn to the visually unusual element and never report the visible one. On this pair, the models select the contrastive element on both twins; pooled over all models and items, within-region selection on occlusion is 0.554, statistically indistinguishable from 0.50 (Appendix R ).
Figure 26: Case study 6: containment . A: target is the element contained within the parent. B: target is the parent containing the inner element. Models repeatedly click the inner element on both twins, missing the parent on twin B. containment carries the weakest relation-word signal in the benchmark: within-region selection splits 0.548/0.452 between target and distractor, and 92% of predictions fall outside both candidate regions (Appendix R ).
Figure 27: Case study 7: rel-pos-vertical , synthetic. The controlled-stimulus version of case study 1. Even on the clean, synthetic layout, the same anchor-collapse signature is visible: predictions cluster near the anchor instead of resolving above/below.
Figure 28: Case study 8: rel-pos-horizontal , real screenshot. A: “element to the right of X ”. B: “element to the left of X ”. The minimal-pair construction that opens the paper (Fig. 2 ) repeats here on a different application. Same shortcut, same failure mode.
Primitive
Tgt
Dis
Neither
Inv.
Sel.
n
p
containment
3.2%
2.6%
91.9%
2.3%
0.548
157
0.26
occlusion
6.1%
4.9%
88.1%
0.8%
0.554
298
0.07
alignment
4.7%
1.7%
92.5%
1.1%
0.728
173
<10−8
proximity
6.1%
0.7%
92.4%
0.7%
0.891
184
<10−28
rel-pos-horizontal
14.8%
1.6%
83.2%
0.4%
0.903
443
<10−72
rel-pos-vertical
16.1%
1.7%
79.2%
3.0%
0.904
481
<10−79
Appendix
Table 7: Candidate-level classification of all baseline predictions, pooled over 19 models (strict rule: a prediction is assigned to a candidate only if it lies inside that candidate’s box). Sel. = within-region selection rate, target / (target + distractor), reported with its denominator n . p is a two-sided exact binomial test against 0.50. Selection rates computed on a small n (e.g., alignment, n=173 ) are correspondingly less precise.
Figure 29: Error-type cheatsheet: one real example per error type, drawn from the actual prediction JSONLs. Right-of-marker labels sit in 50%-transparent boxes so the underlying screenshot remains visible. (A) Anchor-collapse : click lands on the anchor noun, not the relational neighbor. Opus 4.7’s prediction on ui_vision-199-0 (“click the element above the replace”) falls inside the dashed “anchor” zone (the “replace” button), not the green target above it. (B) Twin-identical prediction : same click for both twins → relation word ignored. Qwen2.5-VL emits the same click on the two alignment twins of pair 81; both targets are shown for context. (C) Constant default : model defaults to a canonical coordinate. Llama-3.2-V emits literal (0,0) on ∼ 46% of diagnostic items; the prediction marker sits in the upper-left corner regardless of where the target is. (D) Saturated-primitive regression : intervention hurts a primitive the model already solved. On list-ordinal the green-filled marker shows that Qwen2.5-VL’s baseline answer was correct, but the SoM adds an indirection and the same model misses (red × ). (E) Contrastive-option choice : on this pair, the models select the contrastive target on both twins. On the occlusion pair, both Opus (circles) and GPT-5 (squares) cluster on the partly-hidden target on both twins, ignoring the visible target. Pooled over all models and items, within-region selection on occlusion is statistically indistinguishable from 0.50 (Appendix R ). (F) Coordinate-frame error (now fixed) : API returns coords in a downsampled frame. GPT-5’s pre-fix ScreenSpot-Pro prediction (terra hollow circle) lands in a 1024-side frame far from the 4K target; after our pre-resize + scale-back wrapper, the post-fix prediction (indigo filled circle) sits near the gold box.
GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness, reasoning calibration-providing diagnostic signal that aggregate benchmarks cannot. We release the dataset, augmentation pipeline, and a fine-tuned model.
We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.
Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.
Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1