IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Kyeongseon Kim; Jiyeon Son; Tae-Hyun Oh

cs.AIOct 23, 2026

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Authors: Kyeongseon Kim, Jiyeon Son, Tae-Hyun Oh

Organizations: KAIST

Abstract

GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG), a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT-5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.

Figures & tables

Figure 1: Overview of Interactive GUI Grounding (IGG). Left : representative visibility constraints require observability , disambiguation , state control , or temporal control before the target can be clicked. Center : these capabilities and a compact action space define Viewport Constraint , Multi-State Constraint , and Temporal/Manipulation Constraint . Right : human solvability exceeds model grounding accuracy on all three constraint types, and the gap is largest on viewport constraints for Qwen3-VL and on temporal and manipulation constraints for GPT-5.5.

Figure 2: Examples of the three IGG constraint types: viewport, multi-state, and temporal or manipulation constraints. Additional examples appear in Appendix E.2 .

Action	Role
SCROLL $(\mathrm{dir},\Delta_{\mathrm{px}})$	Scroll to reveal off-screen regions.
MOVE $(x_{\mathrm{px}},y_{\mathrm{px}})$	Move the pointer to trigger hover states or expose disambiguating cues.
DRAG ( startX , startY , endX , endY )	Drag an element between the specified start and end coordinates.
WAIT $(\Delta t_{\mathrm{ms}})$	Pause for a specified duration to handle delayed or asynchronous UI updates.
CLICK $(x_{\mathrm{px}},y_{\mathrm{px}})$	Click the specified location; grounding is evaluated at this step.

Table 1: Minimal action space

\mathcal{A}

for visibility recovery.

Figure 3: Distribution of IGG tasks. (a) Task distribution across the three constraint types and their sub-types. (b) Number of tasks per platform, broken down by sub-type.

Model	GA (%) $\uparrow$	AC $\downarrow$	RER $\uparrow$	TT (ms) $\downarrow$
Constraint Type 1: Viewport Constraint
Qwen2.5-VL-7B	10.8	7.37	0.70	14718
Qwen3-VL-8B	15.0	7.26	0.73	15246
GUI-Owl-1.5-8B	15.0	7.27	0.62	21985
GPT-5.5	86.2	4.29	0.65	56355
Constraint Type 2: Multi-State Constraint

Table 2: Results by constraint type and sub-type.

App	Qwen2.5-VL	Qwen3-VL	GUI-Owl-1.5	GPT-5.5
Social Feed	23.5	29.4	38.2	61.8
Planner	15.2	21.2	24.2	81.8
Developer Hub	11.4	25.7	25.7	57.1
Code Editor	25.0	21.9	28.1	87.5
Community Chat	22.4	26.5	34.7	61.2
Document Editor	32.5	37.5	42.5	85.0

Table 3: Grounding Accuracy (GA) (%) by application.

Panel of Figure 4 — Figure 4: IGG exposes a large interaction gap. Human results summarize the solvability study, whereas model results span the complete benchmark.

Figure 5: Accuracy-efficiency trade-off on the validation set. GPT-5.5 offers substantially higher accuracy, while the trained Qwen3-VL is faster and runs without paid API inference.

Condition	Qwen3-VL	GPT-5.4 Mini (Low)
Full task	30.8	72.4
Interaction specified	33.9	–
Reveal performed	81.9	90.4

Table 4: Grounding accuracy (%) when a model solves the complete task, is told the required interaction, or receives the interface after the reveal interaction has been performed.

Panel of Figure 6 — Figure 6: SFT expands action support, and GRPO selects among those actions. The best trained model improves Qwen3-VL from 40.0% to 59.2% GA on the 120-task validation split. Baseline VLMs solve some validation tasks without emitting every action in the ground-truth trajectory, but emit no drag actions. SFT introduces these actions into the policy’s support, after which GRPO improves task success.

Panel of Figure 7 — Figure 7: GRPO ablation. GRPO is effective only when initialized from the frequency-balanced SFT seed: it reaches 59.2% GA, while GRPO from the standard SFT seed does not improve over its seed model. The frequency-balanced seed gives the policy access to rare reveal actions, after which online reward improves their selection and execution.

Sub-type	Trials	Success (%) $\uparrow$	Time (s) $\downarrow$
CT1-1	45	93.3	23.3
CT2-1	45	100.0	15.9
CT2-2	45	95.6	20.0
CT2-3	45	88.9	21.0
CT3-1	45	97.8	14.2
CT3-2	45	97.8	22.9

Table 5: Human task solvability on the 21-task suite (15 participants, 315 trials).

Environment	VR $\uparrow$	IR $\uparrow$	TE $\uparrow$
Music Player	4.93	4.80	5.00
Video Platform	4.73	4.87	4.80
E-commerce	4.67	4.80	4.80
Developer Hub	4.67	4.53	4.80
Code Editor	4.67	4.80	4.87
News Portal	3.93	4.60	4.60

Table 6: Human fidelity ratings averaged over 15 participants on a five-point Likert scale: visual realism (VR), interaction realism (IR), and task executability (TE).

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Mockup	Reference
Social Feed	Twitter/X
Planner	Google Calendar
Developer Hub	GitHub
Code Editor	VS Code Web
Community Chat	Discord
Document Editor	Google Docs

Appendix

Table A1: The 16 mockup environments and their reference interfaces.

Panel of Figure 15 — Figure A1: Representative states from productivity mockup applications.

Panel of Figure 16 — Figure A2: Representative states from development and design mockup applications.

Panel of Figure 17 — Figure A3: Representative states from social mockup applications.

Panel of Figure 18 — Figure A4: Representative states from media mockup applications.

Panel of Figure 19 — Figure A5: Representative states from lifestyle mockup applications.

Viewport	Base	Trained	Gain
$1440\times 900$	43.3	59.8	+16.5
$390\times 844$	43.3	60.6	+17.3
$1024\times 768$	42.5	55.9	+13.4
$720\times 900$	48.0	59.1	+11.1
$1152\times 720$	39.4	52.8	+13.4

Appendix

Table A2: Grounding accuracy (%) at five viewport sizes.

Pairing	Count	DINO $\uparrow$	SigLIP $\uparrow$	DreamSim $\downarrow$
Matched	16	0.7724	0.8094	0.2529
Non-matched	240	0.5442	0.6211	0.5229

Appendix

Table A3: Visual similarity between mockups and real-application references. DINO and SigLIP are cosine similarities, where higher is better. DreamSim is a perceptual distance, where lower is better.

Model	Env.	Succ.	GA $\uparrow$	AC $\downarrow$	RER $\uparrow$
Qwen2.5-VL	Mock	3/16	18.8	7.25	0.45
Qwen2.5-VL	Real	3/16	18.8	7.69	0.42
Qwen3-VL	Mock	4/16	25.0	7.06	0.54
Qwen3-VL	Real	5/16	31.3	7.06	0.35
GUI-Owl-1.5	Mock	6/16	37.5	6.13	0.55
GUI-Owl-1.5	Real	6/16	37.5	6.88	0.57

Appendix

Table A4: Task-level fidelity on 16 paired mock-real tasks. Succ. denotes the number of successful trials out of 16.

Panel of Figure 23 — Figure A6: Paired examples in which the model fails in both the controlled and real-interface task.

Panel of Figure 24 — Figure A7: Paired examples in which the model succeeds in both the controlled and real-interface task.

Panel of Figure 25 — Figure A8: Excerpt of the system prompt used for IGG model evaluation. Ellipses indicate omitted formatting examples.

Panel of Figure 26 — Figure A9: CT 1-1 : Scroll-based Exploration (E-commerce). “Click ‘Cart’ for ‘Wireless Mechanical Keyboard’.”

Panel of Figure 27 — Figure A10: CT 2-1 : Visual Disambiguation (Community Chat). “Click ‘Reply’ on Minji’s message.”

Panel of Figure 28 — Figure A11: CT 2-2 : Hidden State Activation (Document Editor). “Click ‘Rename’ on ‘Product Research Weekly Brief’.”

Panel of Figure 29 — Figure A12: CT 2-3 : Removable Occlusion (Video Platform). “Click the ‘Late-night Studio Vlog’ video card.”

Panel of Figure 30 — Figure A13: CT 3-1 : Delayed State Activation (Community Chat). “Click ‘Send’ for the pptx upload.”

Panel of Figure 31 — Figure A14: CT 3-2 : Dynamic Target Interaction (News Portal). “Click ‘Pre-Order Now’ in the ‘NexPhone’ ad.”

Panel of Figure 32 — Figure A15: CT 3-3 : Movable Occlusion (Design Canvas). “Click the ‘QA Checklist’ note without deleting any.”

Failure mode	Sub-type	Task	Observed trace pattern
Spatial manipulation is not attempted	$\mathrm{CT}_{3\text{-}3}$ : Movable Occlusion	Design Canvas: find the QA Checklist note without changing zoom.	All three models exhausted the budget without using a drag operation. Qwen2.5-VL-7B and Qwen3-VL-8B produced eight scrolls; GUI-Owl-1.5-8B produced one wait followed by seven scrolls.
Dynamic target state is not tracked	$\mathrm{CT}_{3\text{-}2}$ : Dynamic Target Interaction	News Portal: find the Glass Harbor begins a 24-hour pilot headline in Breaking News.	The models interacted with the wrong state of the ticker. Qwen3-VL-8B clicked the ticker controls and ended on a different headline, while the other models scrolled or expanded unrelated page content.
Hover-based ambiguity is not resolved	$\mathrm{CT}_{2\text{-}1}$ : Visual Disambiguation	Community Chat: find the Creator Dock server.	The target server is identifiable only after hovering over ambiguous server icons. Qwen2.5-VL-7B repeatedly scrolled, while Qwen3-VL-8B and GUI-Owl-1.5-8B repeatedly clicked icons without first revealing their labels.

Appendix

Table A5: Representative failure cases selected from tasks failed by Qwen2.5-VL-7B, Qwen3-VL-8B, and GUI-Owl-1.5-8B.

Figure A16: Representative failed traces. The figure visualizes example failed traces from two representative cases, showing how models spend the interaction budget without performing the visibility-changing action needed to reveal the target.

Required setup action	Failed tasks	Omitted	Rate	Diagnostic interpretation
Drag	59	59	100.0%	Drag is absent from failed traces.
Scroll	247	207	83.8%	Scroll is often replaced by clicks or local moves.
Move	96	71	74.0%	Hover and disambiguation moves are often skipped.
Wait	3	1	33.3%	Few failed tasks require explicit waiting.

Appendix

Table A6: Trace-derived action-prior diagnostic for Qwen3-VL failed trajectories. Counts are non-exclusive because one task can require multiple setup actions.

Constraint sub-type	Interaction rule	Reveal success (%)
$\mathrm{CT}_{1\text{-}1}$ : Scroll-based Exploration	Scroll center by fixed steps	68.8
$\mathrm{CT}_{2\text{-}1}$ : Visual Disambiguation	Hover common icon areas	61.4
$\mathrm{CT}_{2\text{-}3}$ : Removable Occlusion	Click common dismiss areas	17.6
$\mathrm{CT}_{3\text{-}1}$ : Delayed State Activation	Wait in 1-s steps	87.1
$\mathrm{CT}_{3\text{-}3}$ : Movable Occlusion	Drag visible occluder aside	16.9

Appendix

Table A7: Reveal success of target-agnostic interaction rules on their corresponding constraint sub-types.

Model	Base	SFT+GRPO
Qwen3-VL-8B	40.0	59.2
InternVL3.5-8B	28.3	47.5
GUI-Owl-1.5-8B	35.0	52.5

Appendix

Table A8: Grounding accuracy before and after SFT+GRPO for three model architectures.

Component	Symbol	Value / rule
Target-click success	$S$	$+2.2$ if successful
Unsuccessful rollout	$\neg S$	$-1.0$ base term
Step cost	$T$	$-0.03$ per action
Wrong non-empty click	$W$	$-0.65$ per event
Empty click	$E$	$-1.00$ per event
Invalid action	$I$	$-1.00$ per event

Appendix

Table A9: GRPO reward components used in the diagnostic training experiments. Positive progress bonuses are bounded per rollout and are applied only when the browser state or target evidence changes in the intended direction.

App	GA	AC	RER	TT	GA	AC	RER	TT
	Qwen3-VL				GPT-5.5
Social Feed	29.4	6.47	0.93	17514	61.8	4.59	1.04	69757
Planner	21.2	6.88	0.79	13098	81.8	3.88	0.83	61011
Developer Hub	25.7	6.94	0.66	9763	57.1	5.57	0.69	138919
Code Editor	21.9	6.81	1.26	9575	87.5	3.88	0.97	59747
Community Chat	26.5	6.82	0.64	9357	61.2	4.82	1.03	108102

Appendix

Table A10: Per-application GA, AC, RER, and TT for Qwen3-VL and GPT-5.5 on all 601 tasks.

Panel of Figure 40 — Figure A17: The web-based interface used in the human study. (a) Participants begin by entering an ID and reviewing the task format. (b) Each task is introduced with the instruction text. (c) Participants then interact with the live mockup; the interaction counter is shown in the top-right corner. (d) The task ends with success when the participant clicks the correct target, or (e) with failure when the eight click-or-drag attempt limit is reached.

Panel of Figure 41 — Figure A18: The web-based interface used in the human fidelity test. (a) Participants enter the fidelity-study session. (b) For each application, participants compare the mock interface with the real-application reference and rate visual realism, task executability, and interaction realism.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

ScreenHaystack: Finding Blind Zones in GUI Grounding
Sep 25, 2026Chenyue Li, Xiaoxiao Sun, Yubo Deng +3GUI Grounding Data Augmentation
Stanford University
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI Grounding GUI Agents
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
Aug 4, 2026Zichuan Fu, Shirong Wang, Wenlin Zhang +10GUI Grounding Visual Grounding
City University of Hong Kong · Westlake University · Tencent Jarvis Lab

Sep 25, 2026J/K move · Enter open · S save

ScreenHaystack: Finding Blind Zones in GUI Grounding

Chenyue Li, Xiaoxiao Sun, Yubo Deng, Qinlin Zhao+2

Stanford University

GUI Grounding Data Augmentation

Open paper AskPDF arXiv

We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.

Abstract

GUI Grounding

Explore similar work

ScreenHaystack: Finding Blind Zones in GUI Grounding

ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs