GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.
Figures & tables
Figure 1: Overview of Interactive GUI Grounding (IGG). Left : representative visibility constraints require observability , disambiguation , state control , or temporal control before the target can be clicked. Center : these capabilities and a compact action space define Viewport Constraint , Multi-State Constraint , and Temporal/Manipulation Constraint . Right : human solvability exceeds model grounding accuracy on all three constraint types, and the gap is largest on viewport constraints for Qwen3-VL and on temporal and manipulation constraints for GPT-5.5.
Figure 2: Examples of the three IGG constraint types: viewport, multi-state, and temporal or manipulation constraints. Additional examples appear in Appendix E.2 .
Action
Role
SCROLL (dir,Δpx)
Scroll to reveal off-screen regions.
MOVE (xpx,ypx)
Move the pointer to trigger hover states or expose disambiguating cues.
DRAG ( startX , startY , endX , endY )
Drag an element between the specified start and end coordinates.
WAIT (Δtms)
Pause for a specified duration to handle delayed or asynchronous UI updates.
CLICK (xpx,ypx)
Click the specified location; grounding is evaluated at this step.
Table 1: Minimal action space A for visibility recovery.
Figure 3: Distribution of IGG tasks. (a) Task distribution across the three constraint types and their sub-types. (b) Number of tasks per platform, broken down by sub-type.
Model
GA (%) ↑
AC ↓
RER ↑
TT (ms) ↓
Constraint Type 1: Viewport Constraint
Qwen2.5-VL-7B
10.8
7.37
0.70
14718
Qwen3-VL-8B
15.0
7.26
0.73
15246
GUI-Owl-1.5-8B
15.0
7.27
0.62
21985
GPT-5.5
86.2
4.29
0.65
56355
Constraint Type 2: Multi-State Constraint
Table 2: Results by constraint type and sub-type.
App
Qwen2.5-VL
Qwen3-VL
GUI-Owl-1.5
GPT-5.5
Social Feed
23.5
29.4
38.2
61.8
Planner
15.2
21.2
24.2
81.8
Developer Hub
11.4
25.7
25.7
57.1
Code Editor
25.0
21.9
28.1
87.5
Community Chat
22.4
26.5
34.7
61.2
Document Editor
32.5
37.5
42.5
85.0
Table 3: Grounding Accuracy (GA) (%) by application.
Figure 4: IGG exposes a large interaction gap. Human results summarize the solvability study, whereas model results span the complete benchmark.
Figure 5: Accuracy-efficiency trade-off on the validation set. GPT-5.5 offers substantially higher accuracy, while the trained Qwen3-VL is faster and runs without paid API inference.
Condition
Qwen3-VL
GPT-5.4 Mini (Low)
Full task
30.8
72.4
Interaction specified
33.9
–
Reveal performed
81.9
90.4
Table 4: Grounding accuracy (%) when a model solves the complete task, is told the required interaction, or receives the interface after the reveal interaction has been performed.
Figure 6: SFT expands action support, and GRPO selects among those actions. The best trained model improves Qwen3-VL from 40.0% to 59.2% GA on the 120-task validation split. Baseline VLMs solve some validation tasks without emitting every action in the ground-truth trajectory, but emit no drag actions. SFT introduces these actions into the policy’s support, after which GRPO improves task success.
Figure 7: GRPO ablation. GRPO is effective only when initialized from the frequency-balanced SFT seed: it reaches 59.2% GA, while GRPO from the standard SFT seed does not improve over its seed model. The frequency-balanced seed gives the policy access to rare reveal actions, after which online reward improves their selection and execution.
Sub-type
Trials
Success (%) ↑
Time (s) ↓
CT1-1
45
93.3
23.3
CT2-1
45
100.0
15.9
CT2-2
45
95.6
20.0
CT2-3
45
88.9
21.0
CT3-1
45
97.8
14.2
CT3-2
45
97.8
22.9
Table 5: Human task solvability on the 21-task suite (15 participants, 315 trials).
Environment
VR ↑
IR ↑
TE ↑
Music Player
4.93
4.80
5.00
Video Platform
4.73
4.87
4.80
E-commerce
4.67
4.80
4.80
Developer Hub
4.67
4.53
4.80
Code Editor
4.67
4.80
4.87
News Portal
3.93
4.60
4.60
Table 6: Human fidelity ratings averaged over 15 participants on a five-point Likert scale: visual realism (VR), interaction realism (IR), and task executability (TE).
Appendix figures & tables28 assets
Supplementary material from the paper’s appendix.
Appendix
Mockup
Reference
Social Feed
Twitter/X
Planner
Google Calendar
Developer Hub
GitHub
Code Editor
VS Code Web
Community Chat
Discord
Document Editor
Google Docs
Appendix
Table A1: The 16 mockup environments and their reference interfaces.
Figure A1: Representative states from productivity mockup applications.
Figure A2: Representative states from development and design mockup applications.
Figure A3: Representative states from social mockup applications.
Figure A4: Representative states from media mockup applications.
Figure A5: Representative states from lifestyle mockup applications.
Viewport
Base
Trained
Gain
1440×900
43.3
59.8
+16.5
390×844
43.3
60.6
+17.3
1024×768
42.5
55.9
+13.4
720×900
48.0
59.1
+11.1
1152×720
39.4
52.8
+13.4
Appendix
Table A2: Grounding accuracy (%) at five viewport sizes.
Pairing
Count
DINO ↑
SigLIP ↑
DreamSim ↓
Matched
16
0.7724
0.8094
0.2529
Non-matched
240
0.5442
0.6211
0.5229
Appendix
Table A3: Visual similarity between mockups and real-application references. DINO and SigLIP are cosine similarities, where higher is better. DreamSim is a perceptual distance, where lower is better.
Model
Env.
Succ.
GA ↑
AC ↓
RER ↑
Qwen2.5-VL
Mock
3/16
18.8
7.25
0.45
Real
3/16
18.8
7.69
0.42
Qwen3-VL
Mock
4/16
25.0
7.06
0.54
Real
5/16
31.3
7.06
0.35
GUI-Owl-1.5
Mock
6/16
37.5
6.13
0.55
Real
6/16
37.5
6.88
0.57
Appendix
Table A4: Task-level fidelity on 16 paired mock-real tasks. Succ. denotes the number of successful trials out of 16.
Figure A6: Paired examples in which the model fails in both the controlled and real-interface task.
Figure A7: Paired examples in which the model succeeds in both the controlled and real-interface task.
Figure A8: Excerpt of the system prompt used for IGG model evaluation. Ellipses indicate omitted formatting examples.
Figure A11: CT 2-2 : Hidden State Activation (Document Editor). “Click ‘Rename’ on ‘Product Research Weekly Brief’.”
Figure A12: CT 2-3 : Removable Occlusion (Video Platform). “Click the ‘Late-night Studio Vlog’ video card.”
Figure A13: CT 3-1 : Delayed State Activation (Community Chat). “Click ‘Send’ for the pptx upload.”
Figure A14: CT 3-2 : Dynamic Target Interaction (News Portal). “Click ‘Pre-Order Now’ in the ‘NexPhone’ ad.”
Figure A15: CT 3-3 : Movable Occlusion (Design Canvas). “Click the ‘QA Checklist’ note without deleting any.”
Failure mode
Sub-type
Task
Observed trace pattern
Spatial manipulation is not attempted
CT3-3 : Movable Occlusion
Design Canvas: find the QA Checklist note without changing zoom.
All three models exhausted the budget without using a drag operation. Qwen2.5-VL-7B and Qwen3-VL-8B produced eight scrolls; GUI-Owl-1.5-8B produced one wait followed by seven scrolls.
Dynamic target state is not tracked
CT3-2 : Dynamic Target Interaction
News Portal: find the Glass Harbor begins a 24-hour pilot headline in Breaking News.
The models interacted with the wrong state of the ticker. Qwen3-VL-8B clicked the ticker controls and ended on a different headline, while the other models scrolled or expanded unrelated page content.
Hover-based ambiguity is not resolved
CT2-1 : Visual Disambiguation
Community Chat: find the Creator Dock server.
The target server is identifiable only after hovering over ambiguous server icons. Qwen2.5-VL-7B repeatedly scrolled, while Qwen3-VL-8B and GUI-Owl-1.5-8B repeatedly clicked icons without first revealing their labels.
Appendix
Table A5: Representative failure cases selected from tasks failed by Qwen2.5-VL-7B, Qwen3-VL-8B, and GUI-Owl-1.5-8B.
Figure A16: Representative failed traces. The figure visualizes example failed traces from two representative cases, showing how models spend the interaction budget without performing the visibility-changing action needed to reveal the target.
Required setup action
Failed tasks
Omitted
Rate
Diagnostic interpretation
Drag
59
59
100.0%
Drag is absent from failed traces.
Scroll
247
207
83.8%
Scroll is often replaced by clicks or local moves.
Move
96
71
74.0%
Hover and disambiguation moves are often skipped.
Wait
3
1
33.3%
Few failed tasks require explicit waiting.
Appendix
Table A6: Trace-derived action-prior diagnostic for Qwen3-VL failed trajectories. Counts are non-exclusive because one task can require multiple setup actions.
Constraint sub-type
Interaction rule
Reveal success (%)
CT1-1 : Scroll-based Exploration
Scroll center by fixed steps
68.8
CT2-1 : Visual Disambiguation
Hover common icon areas
61.4
CT2-3 : Removable Occlusion
Click common dismiss areas
17.6
CT3-1 : Delayed State Activation
Wait in 1-s steps
87.1
CT3-3 : Movable Occlusion
Drag visible occluder aside
16.9
Appendix
Table A7: Reveal success of target-agnostic interaction rules on their corresponding constraint sub-types.
Model
Base
SFT+GRPO
Qwen3-VL-8B
40.0
59.2
InternVL3.5-8B
28.3
47.5
GUI-Owl-1.5-8B
35.0
52.5
Appendix
Table A8: Grounding accuracy before and after SFT+GRPO for three model architectures.
Component
Symbol
Value / rule
Target-click success
S
+2.2 if successful
Unsuccessful rollout
¬S
−1.0 base term
Step cost
T
−0.03 per action
Wrong non-empty click
W
−0.65 per event
Empty click
E
−1.00 per event
Invalid action
I
−1.00 per event
Appendix
Table A9: GRPO reward components used in the diagnostic training experiments. Positive progress bonuses are bounded per rollout and are applied only when the browser state or target evidence changes in the intended direction.
Qwen3-VL
GPT-5.5
App
GA
AC
RER
TT
GA
AC
RER
TT
Social Feed
29.4
6.47
0.93
17514
61.8
4.59
1.04
69757
Planner
21.2
6.88
0.79
13098
81.8
3.88
0.83
61011
Developer Hub
25.7
6.94
0.66
9763
57.1
5.57
0.69
138919
Code Editor
21.9
6.81
1.26
9575
87.5
3.88
0.97
59747
Community Chat
26.5
6.82
0.64
9357
61.2
4.82
1.03
108102
Appendix
Table A10: Per-application GA, AC, RER, and TT for Qwen3-VL and GPT-5.5 on all 601 tasks.
Figure A17: The web-based interface used in the human study. (a) Participants begin by entering an ID and reviewing the task format. (b) Each task is introduced with the instruction text. (c) Participants then interact with the live mockup; the interaction counter is shown in the top-right corner. (d) The task ends with success when the participant clicks the correct target, or (e) with failure when the eight click-or-drag attempt limit is reached.
Figure A18: The web-based interface used in the human fidelity test. (a) Participants enter the fidelity-study session. (b) For each application, participants compare the mock interface with the real-application reference and rate visual realism, task executability, and interaction realism.
We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.
GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents. However, existing grounding models remain error-prone, often causing downstream task failures. We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors. Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors. We construct ConClick-Data, a dataset containing 4,292 challenging target-distractor pairs, and train the model using complementary listwise ranking and pairwise margin objectives. ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones. We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks. Together, our analysis, dataset, and methods provide both empirical insights and practical tools for building more reliable GUI agents. Data and code are available at https://github.com/zhangruotian/ConClick.
GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.
Zichuan Fu, Shirong Wang, Wenlin Zhang +10
City University of Hong Kong · Westlake University · Tencent Jarvis Lab