cs.CLAug 22, 2026
SaveGUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
Organizations: University of Southern California · USC Information Sciences Institute · Qatar Computing Research Institute (QCRI)
Abstract
Computer-use agents ground natural-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element. We introduce GUI-Primitives, a 994-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces (left/right, above/below, containment, alignment, proximity, list ordinal, occlusion). Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates. Five annotators validate a 196-item subset ( well-formedness; target selection). Nineteen vision-language models reach at most strict point-in-box accuracy. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within. Predictions fall outside both candidates on of items. Conditional on falling within a candidate region, target selection reaches 0.82-0.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding. Across ten models, benchmark accuracy correlates with ScreenSpot-Pro accuracy (Spearman ), an exploratory association at this sample size. Marking the two designated candidates raises selection accuracy by 35--57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method. We release the benchmark, predictions, and code.
Figures & tables
Figure 1: GUI-Primitives dataset construction pipeline. Two source corpora: real desktop screenshots from UI-Vision and controlled synthetic stimuli from our procedural renderer, feed into per-primitive minimal-pair miners, pass through automatic quality gates, are cross-checked by a five-annotator human verification pass (Fleiss ), and released as the 994-item benchmark. Stage colors follow the primitive palette used throughout the paper.
| Primitive | Relation words | Group | Ref. level |
|---|---|---|---|
| rel-pos-horizontal | left / right | cardinal | 0.50 |
| rel-pos-vertical | above / below | cardinal | 0.50 |
| containment | inside / outside | topology | 0.50 |
| alignment | same row / col. | layout | 0.50 |
| proximity | nearest / farthest | layout | 0.50 |
| list-ordinal | first / second | layout | 0.50 |
Table 1: The seven elementary spatial primitives in GUI-Primitives , with their canonical relation words and two-candidate reference levels. All seven primitives, including list-ordinal , are realized as binary minimal-pair contrasts (71 items per side), so the reference level is throughout; § 4 explains why this is a forced-choice reference rather than the chance level of unconstrained clicking.
Figure 2: A contrastive pair from GUI-Primitives (details in Appendix P ). The screenshot is fixed; one word flips, and the correct target shifts to the other candidate. A model exploiting an answer prior passes one twin and fails the other.
Figure 3: Shortcut controls (Qwen2.5-VL-7B, ). Text-only and shuffled drops are both highly significant; blur produces a smaller but significant drop, below the pre-registered ten-point threshold.
| Model | Family | Full ( ) | Core ( ) | Clean ( ) |
|---|---|---|---|---|
| Claude Opus 4.7 | Anthropic | 0.313 | 0.311 | 0.324 |
| GPT-5 (reasoning) | OpenAI | 0.265 | 0.291 | 0.297 |
| Claude Sonnet 4.6 | Anthropic | 0.222 | 0.260 | 0.270 |
| Claude Haiku 4.5 | Anthropic | 0.239 | 0.250 | 0.249 |
| Qwen2.5-VL-7B | Qwen2.5-VL | 0.220 | 0.245 | 0.249 |
| Gemma 3-27B | Google open | 0.134 | — | 0.157 |
Table 2: Overall accuracy on GUI-Primitives for 12 representative models, alongside accuracy on the human-clean 185-item subset. The strongest model (Claude Opus 4.7) reaches 32% on the clean core, a 65-point gap below human accuracy (96.9%). All 19 models are in Appendix A .
Figure 4: Human-clean core (185 items, 5 annotators, ). Human accuracy is 96.9%. The strongest model is Claude Opus 4.7 at 32.4%.
Figure 5: Per-primitive accuracy across eight representative models, with bootstrap 95% confidence intervals. Dashed lines mark the two-candidate reference level (0.50 for all primitives; Table 1 ). On containment , occlusion , alignment , and proximity , every model falls below this reference, including the four strongest proprietary models; candidate-level analysis (Appendix R ) attributes this below-reference accuracy primarily to predictions outside both candidate regions.
Figure 6: Pair-consistency vs. raw accuracy, per model per primitive. Independent guessing would place each point near (dashed). Almost all points lie below this curve, the What’s-Up signature of relational failure.
Figure 7: Per-model GUI-Primitives accuracy ( ) against ScreenSpot-Pro grounding accuracy. Spearman ( ).
Figure 8: Intervention deltas vs. each model’s baseline, with Holm-adjusted significance (paired McNemar across primitives). SoM recovers 35–57 points on three of four tested models; the only marginal case is Claude Opus 4.7, which already has the highest baseline. CoT and activation steering yield no significant change.
Figure 9: Per-primitive SoM delta on Claude Opus 4.7 and GPT-5. Stars mark intervals excluding zero. On GPT-5, six of seven primitives gain at least 40 points.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Full | Core | Clean |
|---|---|---|---|
| Claude Opus 4.7 | 0.313 | 0.311 | 0.324 |
| GPT-5 (reasoning) | 0.265 | 0.291 | 0.297 |
| Claude Haiku 4.5 | 0.239 | 0.250 | 0.249 |
| Claude Sonnet 4.6 | 0.222 | 0.260 | 0.270 |
| Qwen2.5-VL-7B-Instruct | 0.220 | 0.245 | 0.249 |
| Gemma 3-27B-it (HF) | 0.134 | — | 0.157 |
Table 3: Full benchmark accuracy for all 19 evaluated models on three splits: the full benchmark ( Full ), the human-verified subset ( Core ), and the cleaned subset after removing items judged invalid by a majority of the five annotators ( Clean ). Closed models were not run on the full split because of API cost; the Core column reports every model that was. Per-primitive accuracies with bootstrap intervals are released in the artifact.
Figure 10: Per-model per-primitive accuracy heatmap, all 19 models. The horizontal band at 0.80 on list-ordinal marks the only primitive any model solves reliably.
| Model | Identifier | Family / Endpoint |
| Claude Opus 4.7 | claude-opus-4-7 | Anthropic API |
| Claude Sonnet 4.6 | claude-sonnet-4-6 | Anthropic API |
| Claude Haiku 4.5 | claude-haiku-4-5 | Anthropic API |
| GPT-5 (reasoning) | gpt-5 | OpenAI API |
| GPT-4.1 | gpt-4.1 | OpenAI API |
| GPT-4o-mini | gpt-4o-mini | OpenAI API |
Table 4: Model identifiers used in this paper. Closed models were queried via official APIs in May 2026; open models were run from Hugging Face checkpoints with greedy decoding ( ) and a fixed seed.
| Real (UI-Vision) | Synthetic | |||
|---|---|---|---|---|
| Model | strict | loose | strict | loose |
| Claude Haiku 4.5 | 0.000 | 0.230 | 0.363 | 0.763 |
| Qwen2.5-VL-7B | 0.028 | 0.234 | 0.300 | 0.714 |
| OS-Atlas-Base-7B | 0.000 | 0.162 | 0.142 | 0.724 |
| Gemini 3.1 Flash Lite | 0.000 | 0.115 | 0.111 | 0.741 |
| GPT-4o-mini | 0.000 | 0.033 | 0.111 | 0.733 |
Table 5: Strict (point-in-box) vs. loose (within target-diagonal) accuracy, split by image source. In real desktop screenshots, loose accuracy stays low even when the prediction is allowed to miss by a generous margin: failure is element misidentification, not sub-pixel imprecision.
Figure 11: Item-level regression forest plot ( , pseudo- ).
Figure 12: Benchmark distribution analysis. (A) Contrastive-relation balance: every relation is represented by exactly 71 items per side, ruling out relation-prior exploitation. (B) Target–distractor spatial separation (normalized by image diagonal) per primitive; violin width encodes density. (C) Target size distribution (log-scale) by source arm; dashed lines are per-source medians. (D) Target centre positions (normalized to , at top), coloured by primitive; targets span the full screen area.
| Primitive | Real | Synthetic | Total |
|---|---|---|---|
| rel-pos-horizontal | 68 | 74 | 142 |
| rel-pos-vertical | 48 | 94 | 142 |
| alignment | 66 | 76 | 142 |
| proximity | 84 | 58 | 142 |
| list-ordinal | 24 | 118 | 142 |
| containment | 0 | 142 | 142 |
Table 6: Per-primitive composition of GUI-Primitives by source arm. Every primitive has 142 items, exactly 71 per relation word.
Figure 13: Model sensitivity analysis. (A) Left–right accuracy gap per model, sorted by magnitude. Red bars = left-biased (15/19 models; , ). (B) Above–below accuracy gap. Weaker and not significant overall ( ); OS-Atlas is an outlier with a strong above preference. (C) Mean accuracy stratified by target normalized area (tiny/medium/large) per primitive, pooled over all 19 models. (D) Parse-fail (null output) and default-coordinate rates per model; both are infrastructure errors, not spatial-reasoning failures.
Figure 14: rel-pos-horizontal (real screenshots). Three minimal pairs from UI-Vision. Within each row, both panels show the same screenshot; only the relation word ( right of vs. left of ), and the gold target moves accordingly. Across rows, the application and anchor change.
Figure 15: rel-pos-vertical (real screenshots). Three minimal pairs; above vs. below the anchor element across three different real GUIs.
Figure 16: alignment (real screenshots). Three minimal pairs in which the relation switches between same row as and same column as the anchor.
Figure 17: proximity (real screenshots). Three minimal pairs in which the two correct answers are the elements nearest to and farthest from the same anchor across three different real applications.
Figure 18: containment (synthetic). Three minimal pairs. Twin A is the contained within a parent; twin B is the parent containing that inner element. Synthetic stimuli are used because clean parent–child annotations are rare in real screenshot corpora.
Figure 19: list-ordinal (synthetic). Three minimal pairs of symmetric ordinal queries (the -th item from the top vs. from the bottom). The only primitive every top-tier model reliably solves.
Figure 20: occlusion (synthetic). Three minimal pairs. Twin A is the element that is fully visible , twin B is the element that is partly hidden behind an overlay. A relational-shortcut model is drawn to the visually unusual (occluded) element on both twins, scoring zero at the pair level.
Figure 21: Case study 1: rel-pos-vertical , real screenshot. A: “Click the element above the replace.” B: “Click the element below the replace.” Opus and GPT-5 collapse to the same coordinate on both twins (both near ), getting B right by luck of which side they defaulted to and missing A. Qwen2.5-VL emits two completely unrelated clicks. Hollow markers = misses; filled markers = hits. Anchor-collapse failure.
Figure 22: Case study 2: proximity , real screenshot. A: “Select the button nearest to ‘current file directory’.” B: “Click the icon farthest from the current file directory.” All four models cluster their predictions in the top toolbar regardless of the instruction. The anchor (a sidebar element) and the two targets (sidebar items) are far from the predictions. This is the population-level pattern on proximity : 92% of predictions fall outside both candidate regions (Appendix R ), while the target is selected on 89% of within-region predictions, locating the deficit in candidate localization rather than relational interpretation.
Figure 23: Case study 3: alignment , synthetic. A: “Select the control aligned in the same row as ‘Paste’.” B: “Select the control aligned in the same column as ‘Paste’.” Models click the Paste anchor itself or a visually similar icon, instead of the aligned neighbor. Qwen2.5-VL emits an identical coordinate for both twins, the canonical “relation-ignored” signature.
Figure 24: Case study 4: alignment , real screenshot. A/B: “aligned in the same row/column as ‘Navigate Backward’.” The real-screenshot version of case study 3. GPT-5 wanders far from the anchor on both twins. Qwen2.5-VL again emits the same coordinate for both. alignment on real GUIs is the hardest cell in the benchmark.
Figure 25: Case study 5: occlusion , synthetic. A: the ‘Reset’ button that is fully visible . B: the ‘Reset’ button that is partly hidden behind a dialog. All four models point to the partly-hidden button on both twins (paired predictions differ by 10 px). The models are drawn to the visually unusual element and never report the visible one. On this pair, the models select the contrastive element on both twins; pooled over all models and items, within-region selection on occlusion is 0.554, statistically indistinguishable from (Appendix R ).
Figure 26: Case study 6: containment . A: target is the element contained within the parent. B: target is the parent containing the inner element. Models repeatedly click the inner element on both twins, missing the parent on twin B. containment carries the weakest relation-word signal in the benchmark: within-region selection splits 0.548/0.452 between target and distractor, and 92% of predictions fall outside both candidate regions (Appendix R ).
Figure 27: Case study 7: rel-pos-vertical , synthetic. The controlled-stimulus version of case study 1. Even on the clean, synthetic layout, the same anchor-collapse signature is visible: predictions cluster near the anchor instead of resolving above/below.
Figure 28: Case study 8: rel-pos-horizontal , real screenshot. A: “element to the right of X ”. B: “element to the left of X ”. The minimal-pair construction that opens the paper (Fig. 2 ) repeats here on a different application. Same shortcut, same failure mode.
| Primitive | Tgt | Dis | Neither | Inv. | Sel. | ||
|---|---|---|---|---|---|---|---|
| containment | 3.2% | 2.6% | 91.9% | 2.3% | 0.548 | 157 | 0.26 |
| occlusion | 6.1% | 4.9% | 88.1% | 0.8% | 0.554 | 298 | 0.07 |
| alignment | 4.7% | 1.7% | 92.5% | 1.1% | 0.728 | 173 | |
| proximity | 6.1% | 0.7% | 92.4% | 0.7% | 0.891 | 184 | |
| rel-pos-horizontal | 14.8% | 1.6% | 83.2% | 0.4% | 0.903 | 443 | |
| rel-pos-vertical | 16.1% | 1.7% | 79.2% | 3.0% | 0.904 | 481 |
Table 7: Candidate-level classification of all baseline predictions, pooled over 19 models (strict rule: a prediction is assigned to a candidate only if it lies inside that candidate’s box). Sel. = within-region selection rate, target / (target + distractor), reported with its denominator . is a two-sided exact binomial test against 0.50. Selection rates computed on a small (e.g., alignment, ) are correspondingly less precise.
Figure 29: Error-type cheatsheet: one real example per error type, drawn from the actual prediction JSONLs. Right-of-marker labels sit in 50%-transparent boxes so the underlying screenshot remains visible. (A) Anchor-collapse : click lands on the anchor noun, not the relational neighbor. Opus 4.7’s prediction on ui_vision-199-0 (“click the element above the replace”) falls inside the dashed “anchor” zone (the “replace” button), not the green target above it. (B) Twin-identical prediction : same click for both twins → relation word ignored. Qwen2.5-VL emits the same click on the two alignment twins of pair 81; both targets are shown for context. (C) Constant default : model defaults to a canonical coordinate. Llama-3.2-V emits literal on 46% of diagnostic items; the prediction marker sits in the upper-left corner regardless of where the target is. (D) Saturated-primitive regression : intervention hurts a primitive the model already solved. On list-ordinal the green-filled marker shows that Qwen2.5-VL’s baseline answer was correct, but the SoM adds an indirection and the same model misses (red ). (E) Contrastive-option choice : on this pair, the models select the contrastive target on both twins. On the occlusion pair, both Opus (circles) and GPT-5 (squares) cluster on the partly-hidden target on both twins, ignoring the visible target. Pooled over all models and items, within-region selection on occlusion is statistically indistinguishable from (Appendix R ). (F) Coordinate-frame error (now fixed) : API returns coords in a downsampled frame. GPT-5’s pre-fix ScreenSpot-Pro prediction (terra hollow circle) lands in a 1024-side frame far from the 4K target; after our pre-resize + scale-back wrapper, the post-fix prediction (indigo filled circle) sits near the gold box.
More in this topic
GUI Grounding
Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.
See all 57 papers