GUI grounding maps a screenshot and a natural language instruction to the coordinates of the intended interface element. Existing approaches improve grounding through visual scaling, instruction scaling, and sampling scaling, yet it remains unclear which grounding bottlenecks these strategies address and where their gains saturate. To address this gap, we introduce ScaleDiag, a controlled diagnostic framework that isolates the effects of visual input, instruction information, and prediction sampling on GUI grounding by independently varying target scale and surrounding context, target-specific information and auxiliary descriptions, and candidate coverage and selection, respectively. We evaluate ScaleDiag across five benchmarks and up to eight vision-language models and find that each scaling strategy operates within a much narrower regime than commonly assumed. (1) Visual scaling is governed by target occupancy rather than a fixed crop ratio, with full-screen errors typically rescued once the target occupies only 0.32% of the cropped view, while further zooming eventually becomes harmful. (2) Instruction scaling is driven by information that identifies the target rather than description richness itself, as even ground-truth-derived attributes such as color, size, and shape are often neutral or harmful, while appearance and spatial descriptions account for over 90% of the achievable gain. (3) Repeated sampling exposes a different limitation, as correct coordinates are frequently generated but remain difficult to select, leaving 5.1--11.0 points of oracle headroom while geometric consensus provides almost no improvement. Together, these results show that test-time scaling is constrained by distinct operating regimes rather than benefiting monotonically from more visual focus, richer instructions, or additional samples. Our code is available at https://github.com/YenanLiu/ScaleDiag.
Figures & tables
Fig. 1: GUI grounding failures can arise from insufficient target visibility, missing target-identifying information, or unreliable candidate selection. These bottlenecks require different interventions, including visual scaling, instruction refinement, and verifier-guided sampling.
Fig. 2: Overview of our controlled diagnostic framework for GUI grounding. (a) Controlled visual interventions vary the visual evidence while keeping the instruction fixed. We construct target-centered, aspect-ratio-preserving crops at different area ratios, and separately add resized distractor screenshots while preserving the target screenshot at its native resolution. (b) Controlled instruction interventions keep the screenshot and target fixed while modifying the instruction. Rule-based attribute injection appends ground-truth-derived position, color, size, or shape cues, whereas an LLM generates natural descriptions from appearance, spatial, functional, and goal-oriented views. (c) Repeated sampling keeps both the screenshot and instruction fixed and generates multiple coordinate predictions, allowing us to compare greedy decoding, coordinate-medoid consensus, and oracle candidate coverage. Together, these interventions isolate the effects of visual evidence, instruction information, and candidate selection.
Fig. 3: Effects of target-centered cropping on GUI grounding performance. (a) Grounding accuracy across crop area ratios for each model and benchmark. (b) Accuracy gain of the best crop over the full screenshot.
Fig. 4: Difficulty-gated visual scaling under target-centered cropping. OCC denotes the ratio of the target bounding-box area to the visible image area. (a) Grounding accuracy across retained crop areas, grouped by native OCC measured on the full screenshot. Each point reports the accuracy of one native-OCC group at a given crop scale. (b) Within-item accuracy gain over the full-frame input as a function of post-crop OCC. Each crop prediction is compared with the full-frame prediction for the same item, and the resulting changes are averaged within OCC bins. Rescue OCC is the post-crop target occupancy at the least aggressive crop that converts a full-frame miss into a hit. The shaded region shows its interquartile range across rescued items, while the dashed line marks its median. Both panels pool approximately 34.9k model–benchmark–item instances.
Method
SS-Pro
UI-Vision
MMBench
OSW-G
SS-v2
All
T/I
Gap
All
T/I
Gap
All
T/I
Gap
All
All
T/I
Gap
Qwen3-VL-8B [ 35 ]
50.8
69.8/20.0
49.8
24.0
59.7/16.2
43.5
81.7
88.6/76.2
12.3
54.4
93.4
98.1/87.4
10.7
MAI-UI-8B [ 36 ]
62.9
78.3/37.9
40.4
39.6
76.5/31.6
44.9
88.5
94.7/83.5
11.2
62.1
95.4
98.2/91.7
6.5
Qwen3-VL-32B [ 35 ]
57.8
76.0/28.3
47.7
35.0
67.6/27.9
39.6
87.3
91.8/83.7
8.1
65.6
95.0
97.8/91.3
6.4
Qwen3.5-4B [ 37 ]
49.7
64.8/25.3
39.5
33.2
64.5/26.3
38.2
81.7
87.0/77.4
9.6
56.2
91.8
96.2/86.1
10.1
Qwen3.5-9B [ 37 ]
60.7
74.5/38.2
36.3
38.6
63.8/33.1
30.7
84.0
87.8/80.9
6.8
62.4
93.5
97.2/88.6
8.6
TABLE I: Overall grounding accuracy and target-type breakdown across GUI grounding benchmarks. All denotes overall accuracy, T/I denotes text and icon accuracy, and Gap denotes their difference. Best and second-best results for All and Gap are highlighted in green and purple , respectively.
Fig. 5: Instruction utility depends on target-identifying information. (a) Effects of appending annotation-derived coarse attributes, including position, color, size, shape, and their combination. Bars show the mean accuracy change relative to the original instruction over models and four benchmarks, while points show the corresponding per-model averages. (b) Grounding accuracy along an information ladder, from a generic instruction without target identity, through truncated and original instructions, to coarse and fine location clarifications. Qwen3.5 models are emphasized with darker curves. (c) Grounding accuracy under the original instruction and selected GPT-5.2-generated single-view and multi-view descriptions. Each bar is averaged over seven models and five benchmarks, and the dashed line marks the original-instruction baseline.
Setting
App
Spa
Fun
Goa
Qwen3-VL-8B
MAI-UI-8B
Qwen2.5-VL-7B
GTA1-7B
Qwen3.5-4B
Qwen3.5-9B
Qwen3.5-27B
Base
60.8
66.0
32.8
37.9
62.5
67.8
75.3
Single
✓
70.8
70.6
37.4
49.8
74.0
76.1
78.9
Single
✓
66.4
68.3
34.8
50.6
69.4
71.9
76.5
Single
✓
62.1
64.3
32.7
40.2
63.1
67.4
71.7
Single
✓
57.8
65.4
31.6
33.7
60.9
64.8
71.7
Pair
✓
✓
74.2
74.4
38.8
55.9
77.9
79.6
82.9
TABLE II: Grounding accuracy under different combinations of instruction views. Checkmarks indicate the views included in each instruction. Best and second-best results in each model column are highlighted in green and purple , respectively.
Fig. 6: Repeated sampling improves candidate coverage but leaves a selection gap. Sampling uses K=8 predictions at T=0.7 , while Greedy uses one prediction at T=0 . (a) Grounding accuracy under Greedy, coordinate-medoid selection, and Oracle@ 8 , averaged over five benchmarks with 500 items each. Medoid selects the coordinate with the smallest total distance to the other samples, whereas Oracle@ 8 counts an item as correct if any sample hits the target. (b) Distribution of the number of correct samples m , pooled over four models and five benchmarks ( N=10,000 ), with items categorized as all wrong ( m=0 ), partially correct ( 0<m<8 ), or all correct ( m=8 ). (c) Recoverable items as a function of m . Bars show the proportion of items missed by Greedy but covered by Oracle@ 8 ; curves show the normalized GRPO opportunity score, reward-variance factor, and Greedy hit rate.
GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on five GUI grounding benchmarks and six general-purpose base models show that GUI-Lens consistently improves grounding accuracy, achieving up to 9.3 times the accuracy of direct grounding on ScreenSpot-Pro.
Zichuan Fu, Shirong Wang, Wenlin Zhang +10
City University of Hong Kong · Westlake University · Tencent Jarvis Lab
GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround
Bingchao Wang, Yongcheng Jing, Chengyu Wang +4
Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing
GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.
Siqi Pei, Liang Tang, Tiaonan Duan +9
Lenovo Research, Beijing, China · Tsinghua University, Beijing, China