We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.
Figures & tables
Figure 1: ScreenHaystack probing. Controlled probe icons are relocated across diverse GUI backgrounds to measure location-dependent grounding accuracy while reducing the influence of icon appearance and background content.
Figure 2: Overview of ScreenHaystack . Left: ScreenHaystack moves controlled target icons across diverse GUI backgrounds and aggregates model predictions into spatial accuracy maps to identify model-specific blind zones. These zones are validated on ScreenSpot-Pro, where targets inside them are harder to localize. Right: Blind-zone-oriented augmentation uses the identified blind zones to guide crop-and-padding transformations, moving training targets into weak regions to improve spatial coverage and grounding reliability.
Figure 3: Model-specific blind-zone maps from ScreenHaystack . Each heatmap shows grounding accuracy on the 40×40 probing grid after aggregating across backgrounds and probe icons. Persistent low-accuracy regions indicate model-specific blind zones used for ScreenSpot-Pro validation and controlled relocation.
Model
Blind-zone area ( ai<0.8 )
Blind-zone accuracy
Relative accuracy drop
Error concentration
R , % cells ↓
Ablind , % acc. ↑
Fblind , % ↓
Ge , Gini
Qwen3-VL-8B-Instruct
8.19
70.53
23.67
0.538
GTA1-7B
46.31
66.52
15.78
0.375
Qwen3-VL-32B-Instruct
45.63
19.46
68.27
0.583
UI-TARS-1.5-7B
62.81
64.84
11.74
0.306
UI-Venus-Ground-7B
22.00
47.39
45.14
0.759
Table 1: Blind-zone severity on ScreenHaystack . Blind-zone cells are defined by a fixed probing-accuracy threshold, B={i:ai<0.8} . R reports the fraction of such cells, Ablind reports their mean probing accuracy, and Fblind measures the relative accuracy drop from overall probing accuracy to blind-zone accuracy. Ge measures whether cell-wise errors are spatially concentrated or diffuse; larger values indicate more localized failures.
Figure 4: Real-task validation of discovered blind zones. We project the ScreenHaystack -derived blind-zone mask for each model onto unseen ScreenSpot-Pro samples and split targets by whether their centers fall inside or outside the mask. Across all evaluated models, targets inside blind zones have lower grounding accuracy than targets outside them. This shows that the probing-derived spatial maps predict failures on realistic GUI grounding examples, rather than only on synthetic probes.
Figure 5: Interventional validation via controlled spatial relocation. Wrap-around shifts relocate the same ScreenSpot-Pro targets across the blind-zone boundary, while within-region random shifts serve as controls. For targets originally inside blind zones, relocating them out raises accuracy above the In baseline; for targets originally outside, relocating them in lowers accuracy below the Out baseline. Random-shift controls stay close to their baselines, showing the effect is driven by the blind-zone boundary rather than the shift itself. Error bars denote standard error; Δ values are relocation effect sizes in percentage points.
Figure 6: Controlled synthetic experiment for testing whether uneven spatial coverage induces blind zones. We create synthetic GUI grounding data with intentionally imbalanced target-center coverage by omitting training examples from specific spatial regions. The models are fine-tuned on these under-represented training sets and evaluated on a comprehensive test set. This approach enables us to determine whether areas lacking supervision during training become blind spots during evaluation.
Model
Training
x -miss.
y -miss.
x – y -miss.
Phi-3.5-Vision Instruct
Original
76.86
79.30
53.90
Augmented
91.30
89.50
82.20
Gain
+14.44
+10.20
+28.30
Qwen3-VL-8B Instruct
Original
94.40
95.30
77.40
Augmented
97.30
97.90
94.10
Gain
+2.90
+2.60
+16.70
Table 2: Blind-zone-oriented augmentation in controlled synthetic settings. We fine-tune each model on synthetic training data with supervision removed from selected x , y , or joint x – y regions, and evaluate on a full-coverage test set. Accuracy (%) is reported for the original under-covered training data and for the augmented data, with gains measured as percentage-point improvements over the corresponding original setting.
Setting
FT
Acc.
Vanilla Qwen3-VL-8B
–
50.28
Original Click-100k
✓
59.45
Random augmentation
✓
60.38 ± 0.41
Blind-zone augmentation
✓
61.47 ± 0.29
Table 3: Blind-zone-oriented augmentation on real-world Click-100k fine-tuning. ScreenSpot-Pro accuracy is reported for the vanilla Qwen3-VL-8B model and for Qwen3-VL-8B models fine-tuned on Click-100k with different augmentation strategies.
Figure 7: Click-100k data distribution before and after augmentation. The heatmaps show target-center density in normalized screen coordinates. Blind-zone-oriented augmentation increases coverage in sparse regions, yielding a more balanced training distribution.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Illustration of the wrap-around shift transformation. The full screenshot is translated by a fixed offset, and pixels crossing one boundary reappear from the opposite boundary. The target bounding box is shifted by the same offset, so the target moves to a new spatial location while the image content is preserved.
Figure 10: Representative probe icon size calibration for UI-TARS-1.5-7B. We show example spatial probing accuracy maps from the icon-size calibration sweep. Small targets lead to broad low-accuracy regions, suggesting that failures are partly driven by target visibility and recognition difficulty. Overly large targets make the task close to saturated, leaving weak spatial contrast. The intermediate scale preserves clear location-dependent variation while keeping the target recognizable, and is therefore used for the main ScreenHaystack probing experiments.
Training setting
Original examples
Augmented examples
Total examples
Original Click-100k
∼ 101k
0
∼ 101k
Random augmentation
∼ 101k
23,392
∼ 124.4k
Blind-zone-oriented augmentation
∼ 101k
23,392
∼ 124.4k
Appendix
Table 4: Dataset sizes for Click-100k fine-tuning. Both augmentation settings add the same number of transformed examples, allowing random augmentation and blind-zone-oriented augmentation to be compared under the same data scale.
Mask source
Outside
Inside
Gap
Original 10 backgrounds
50.7
41.9
8.8
10 different backgrounds
51.5
42.9
8.6
Appendix
Table 5: Generalization across backgrounds for UI-Venus-Ground-7B. We report ScreenSpot-Pro accuracy (%) outside and inside each bottom- 20% blind-zone mask. Gap is Outside minus Inside in percentage points.
Grid
Cell size
Outside
Inside
Gap
40×40
84.0×48.1
52.9
36.8
16.1
50×50
67.2×38.5
52.4
40.2
12.2
60×60
56.0×32.1
52.1
39.7
12.4
Appendix
Table 6: Sensitivity to grid resolution for Qwen3-VL-8B-Instruct. Cell sizes are in pixels. Outside and Inside are ScreenSpot-Pro accuracies (%), and Gap is their difference in percentage points.
Mask
Outside
Inside
Gap
Bottom 10%
52.2
33.1
19.1
Bottom 20%
52.9
36.8
16.1
Bottom 30%
53.6
42.3
11.3
Appendix
Table 7: Sensitivity to blind-zone mask fraction for Qwen3-VL-8B-Instruct. Outside and Inside are ScreenSpot-Pro accuracies (%), and Gap is their difference in percentage points.
Stratum
Outside
Inside
Gap
Overall
52.90
36.80
16.10
Element type
Icon
22.98
13.04
9.94
Text
69.14
62.00
7.14
Platform
macOS
56.55
47.69
8.86
Appendix
Table 8: Stratified ScreenSpot-Pro analysis for Qwen3-VL-8B-Instruct. Outside and Inside are accuracies (%); Gap is Outside minus Inside in percentage points.
Figure 11: Background-wise ScreenHaystack accuracy maps for Qwen3-VL-32B-Instruct. For each background, accuracy is computed on the 40×40 probing grid by aggregating the three probe formats.
Figure 12: Background-wise ScreenHaystack accuracy maps for UI-Venus-Ground-7B. For each background, accuracy is computed on the 40×40 probing grid by aggregating the three probe formats.
Figure 13: Background-wise ScreenHaystack accuracy maps for GTA1-7B. For each background, accuracy is computed on the 40×40 probing grid by aggregating the three probe formats.
Figure 14: Background-wise ScreenHaystack accuracy maps for Qwen3-VL-8B-Instruct. For each background, accuracy is computed on the 40×40 probing grid by aggregating the three probe formats.
Figure 15: Background-wise ScreenHaystack accuracy maps for UI-TARS-1.5-7B. For each background, accuracy is computed on the 40×40 probing grid by aggregating the three probe formats.
GUI grounding models report over 85% accuracy on standard benchmarks, yet drop 27-56 percentage points when instructions require spatial reasoning rather than direct element naming. Current benchmarks miss this because they evaluate each screenshot once with a single fixed instruction. We introduce GUI-Perturbed, a controlled perturbation framework that independently varies visual scenes and instructions to measure grounding robustness. Evaluating three 7B models from the same architecture lineage, we find that relational instructions cause systematic accuracy collapse across all models, a 70% browser zoom produces statistically significant degradation, and rank-8 LoRA fine-tuning with augmented data degrades performance rather than improving it. By perturbing along independent axes, GUI-Perturbed isolates which specific capability axes are affected-spatial reasoning, visual robustness, reasoning calibration-providing diagnostic signal that aggregate benchmarks cannot. We release the dataset, augmentation pipeline, and a fine-tuned model.
GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.
GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.