Graphical user interface (GUI) grounding maps a screenshot and a natural-language instruction to the screen location where an agent should act. Zoomed views can reveal small targets but may lose the context needed to identify them. We introduce GroundCascade, a training-free controller that uses agreement between a grounder's full-screen and zoomed predictions to decide when to consult a complementary model. It selects between zoomed predictions while preserving corroborated full-screen locations. An optional Finalizer predicts from an unmarked joint crop when disagreement remains (Full); omitting it yields the two-stage Selective variant. GroundCascade reaches 85.33% on ScreenSpot-Pro with Indeed-UI-32B/8B/32B and 88.43% on the visible-target OSWorld-G-Refined subset with Qwen3.6-27B/3.8-27B/3.8-27B, gains of 2.66 and 2.35 percentage points over the stronger standalone two-pass grounder. On UI-Vision, GroundCascade reaches 72.20% category-balanced accuracy, with Selective slightly higher at 72.28%. Across these three evaluations, GroundCascade in Full mode uses 2.42-2.73 calls per request, 31-39% fewer than unconditionally acquiring both models' two-pass predictions. At matched second-model call budgets, selective acquisition gains 1.52-2.56 points over standalone Two-pass, compared with 0.22-0.69 points from stratified random acquisition. GroundCascade leads the official ScreenSpot-Pro leaderboard and exceeds reported GUI-specialist comparisons on OSWorld-G-Refined and UI-Vision, subject to differences in external evaluation protocols.
Figures & tables
Figure 1: GroundCascade versus reported GUI-focused systems (accuracy, %). I32/I8/I32 (max 32B) is used on SSP and Q36/Q38/Q38 (max 27B) elsewhere. ∗ Hatched ≤ bars conditionally bound OSG-R accuracy if published scores are micro-accuracies over 564 requests; they are not measured 510-item scores. UIV is category-balanced; external protocols may differ [ 7 , 43 , 10 , 36 , 6 , 28 , 21 , 29 , 4 ] .
Figure 2: GroundCascade’s two modes. Selective uses Stages 1–2: stop when the Grounder is stable; otherwise acquire the Regrounder and test Detail, then tighter Context agreement. Full adds Stage 3, an unmarked joint view, only if both comparisons fail. Markers and outlines illustrate predicted locations, not model inputs; call labels are cumulative upper bounds because invalid Context skips Detail.
Standalone accuracy (%)
GroundCascade accuracy (%)
Inference cost
G/R/F
Benchmark
Context
Two-pass
Selective
Full
Selective
Full
Saved (%)
I32/I8/I32
SSP
73.69
82.67
84.19
85.33
2.49
2.56
36.11
Q36/Q38/Q38
OSG-R
84.90
86.08
88.04
88.43
2.37
2.42
39.28
UIV
66.02
69.72
72.28
72.20
2.62
2.73
31.20
Table 1: Accuracy and inference cost on three benchmarks. Model allocations ( G/R/F ) are specified by row. Context and Two-pass show the stronger standalone model per benchmark. Selective uses Stages 1–2; Full additionally invokes the Finalizer on unresolved requests. Inference cost is calls/query for each mode, followed by Full’s savings relative to unconditional acquisition of both two-pass models. UIV accuracy is category-balanced; bold marks the highest accuracy in each row.
Figure 3: ScreenSpot-Pro recoveries. (a) The Finalizer repairs a zoom-induced error. (b) It finds a target missed by all four upstream predictions (the R points lie outside this excerpt). Prediction circles, Finalizer stars, and target boxes are evaluation overlays, not model inputs.
I32 first: I32/8B/I32
8B first: 8B/I32/I32
8B model
Acc.
Calls
I32
Acc.
Calls
I32
I8
85.33
2.56
2.06
83.93
2.54
0.54
K8
85.01
2.55
2.06
83.68
2.56
0.56
Table 2: Model identity and order on SSP (accuracy, %). Calls and I32 are mean total and 32B model invocations per request, respectively. Bold marks the highest accuracy and the fewest I32 calls.
Figure 4: Choosing which requests get a second opinion matters more than the extra calls. Diamonds: accuracy gain from the GroundCascade gate. Gray: the same number of executed calls given to randomly chosen requests (mean and central 95% range). Top: Regrounder acquisition (Selective vs . standalone Two-pass); random allocations are stratified by recorded call cost (and UIV category), and labels give the requests that received R . Bottom: Finalizer calls on SSP (Full vs . Selective), drawn uniformly among the 390 eligible requests. Both controls use 50,000 draws. SSP uses I32/I8/I32; OSG-R and UIV use Q36/Q38/Q38; UIV is category-balanced. Supplementary Tab. S1 lists the values.
Figure 5: Where the gain comes from. Stage contributions to the accuracy gain over the stronger standalone Two-pass model; annotations show net correct-answer counts. Allocations follow G/R/F as in Tab. 1 . Choosing between the two models’ Detail points gives most of the gain on every benchmark; preserving corroborated Context points adds little; the Finalizer helps on SSP (27 repairs, 9 regressions) and slightly hurts on UIV (70 repairs, 75 regressions). Black ticks mark Full. UIV is pooled over its 5479 requests here (category-balanced gain: + 2.48).
Figure 6: How requests move through GroundCascade on SSP (I32/I8/I32). Each row is rescaled to arriving requests; tinted wedges show escalation to the next stage. The first gate sends 390 requests on, 40.3% of which have a wrong Grounder Detail point, against 9.8% of the 1,191 that stop. Stop segments are split into correct (tinted by stage), wrong, and wrong although a correct prediction had already been acquired (hatched; 48 of the 232 errors). Supplementary Fig. S1 shows OSG-R and UIV.
Benchmark
R requests
Gate gain
Random mean
Random 95%
SSP
390
+1.52
+0.28
[-0.32, +0.89]
OSG-R
97
+1.96
+0.22
[-0.39, +0.78]
UIV
1728
+2.56
+0.69
[+0.33, +1.04]
Table S1 : Selective acquisition at fixed cost. Gains over standalone Two-pass (percentage points). Random columns show the mean and central 95% range across stratified random allocations; UIV uses category-balanced accuracy. Each row uses its flagship model allocation.
UIV category
Context
Two-pass
Selective
Full
F calls
Basic
71.16
72.63
75.45
75.68
138
Functional
66.48
68.74
71.67
71.78
149
Spatial
60.41
67.80
69.72
69.15
308
Mean
66.02
69.72
72.28
72.20
595
Table S2 : Q36/Q38/Q38 on UI-Vision. Category accuracy (%) and Finalizer invocations. Context and Two-pass use the standalone Q36 Grounder; Mean is the official equal-weight category score.
Benchmark
G/R/F
F req.
Rep.
Reg.
Δ pp
SSP
I32/I8/I32
99
27
9
+ 1.14
OSG-R
Q36/Q38/Q38
21
3
1
+0.39
UIV
Q36/Q38/Q38
595
70
75
-0.08
Table S3 : Finalizer effect relative to Selective. Repairs and regressions count corrected and spoiled Stage 2 answers. Δ is the whole-benchmark accuracy change in percentage points (category-balanced for UIV).
Metric
SSP
OSG-R
UIV
G→R precision
157/390
42/97
1038/1728
G→R recall
157/274
42/71
1038/1662
R→F precision
69/99
12/21
405/595
R→F recall
69/133
12/32
405/899
F requests
99
21
595
Before F : correct (%)
30.3
42.9
31.9
Table S4 : Gate detection and recovery. SSP uses I32/I8/I32; OSG-R and UIV use Q36/Q38/Q38. First-gate precision counts standalone Two-pass errors among escalations, and recall counts escalated standalone Two-pass errors among all such errors. Finalizer recall counts escalated Stage 2 errors among all Stage 2 errors; repair and regression rates use incorrect and correct Stage 2 outputs as denominators.
Figure S1 : Request flow on OSG-R (left) and UIV (right) , Q36/Q38/Q38, drawn as the SSP flow figure of the main paper. Regrounder exits combine Detail and Context agreement (11 and 150 Context exits). On UIV the Finalizer leaves 31.1% of its requests correct, against 31.9% for Selective.
GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround
Bingchao Wang, Yongcheng Jing, Chengyu Wang +4
Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing
We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.
GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on five GUI grounding benchmarks and six general-purpose base models show that GUI-Lens consistently improves grounding accuracy, achieving up to 9.3 times the accuracy of direct grounding on ScreenSpot-Pro.
Zichuan Fu, Shirong Wang, Wenlin Zhang +10
City University of Hong Kong · Westlake University · Tencent Jarvis Lab