GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Qingsong Xie; Jiahao Tian; Jiacheng Tang; Zhifeng Li; Madalina Ciobanu; Navan Preet Singh; Qingqing Mao; Ritankar Das

cs.AIOct 13, 2026

GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning

Authors: Qingsong Xie, Jiahao Tian, Jiacheng Tang, Zhifeng Li, Madalina Ciobanu, Navan Preet Singh, Qingqing Mao, Ritankar Das

Organizations: Prentis AI, San Francisco, CA · Westlake University, Hangzhou, China · Titan Holdings, San Francisco, CA

Abstract

Constructing robust GUI agents relies on multimodal visual grounding: the ability to precisely map natural language intents to actionable on-screen components. Despite abundant desktop, web, and mobile benchmarks, high-fidelity resources tailored for diverse environments are surprisingly scarce. To bridge this gap, we introduce the Screen Referring Grounding Dataset (ScreenRef), a comprehensive GUI grounding dataset consisting of GUI screenshots paired with natural language referring expressions and box or point annotations. Spanning desktop, web, and mobile domains, ScreenRef comprises 586K screenshots and over 633K referring expression instances, covering tasks such as text recognition, visual matching, spatial understanding, and refusal scenarios. To boost the model's grounding performance, we propose the Distillation-to-Reinforcement Learning (D2RL) method to address the limited generalization of supervised fine-tuning and the sparse reward problem in reinforcement learning. Utilizing ScreenRef, we train Qwen3-VL-8B-Instruct via sequential SFT and D2RL training, yielding GroundGUI-8B. GroundGUI-8B achieves 95.2% on ScreenSpot-V2, 63.9% on ScreenSpot-Pro, 65.4% on OSWorld-G, 86.8% on MMBench-GUI L2, and 38.3% on UI-Vision. Among 8B-scale models, it produces substantial gains over models trained on open-source datasets, with performance comparable or even superior to models trained on closed-source data. Code, models, and datasets are available at https://github.com/PrentisAI/GroundGUI.

Figures & tables

Dataset	Web	Mobile	Desktop	Images	Instr.	Res Range	Refusal
	Platform			Scale & Statistics
OS-ATLAS ( Wu et al., 2025b )	$\checkmark$	$\checkmark$	$\checkmark$	1.85M	—	(0.5, 5.2)	$\times$
FineWeb ( Penedo et al., 2024 )	$\checkmark$	$\times$	$\times$	1.4M	—	(2.1, 2.1)	$\times$
UGround ( Gou et al., 2025 )	$\checkmark$	$\times$	$\times$	773K	—	(0.4, 1.9)	$\times$
Aria-UI ( Yang et al., 2025 )	$\checkmark$	$\checkmark$	$\checkmark$	295K	—	(1.3, 1.9)	$\times$
SeeClick ( Cheng et al., 2024 )	$\checkmark$	$\times$	$\times$	270K	—	(2.1, 2.1)	$\times$

Table 1: Comparison with existing GUI grounding datasets. Instruction counts follow each dataset’s original reporting unit and are not directly comparable across datasets; details in Appendix B .

Figure 1: Overview of the ScreenRef data construction pipeline. Path 1 consists of screenshot crawling, metadata filtering, instruction generation, and verification stages. Path 2 contains tutorial video crawling, keyframe candidate extraction, joint instruction and action generation, and coordinate generation stages.

Figure 2: Overview of the D2RL method. The format check is a gate, not a reward term.

Open-Source Models
Model	Mobile		Desktop		Web		Avg.
Model	Text	Icon.	Text	Icon.	Text	Icon.	Avg.
Qwen3-VL-2B ( Bai et al., 2025 )	95.5	82.0	95.4	73.6	89.7	76.4	86.7
Phi-Ground-7B ( Zhang et al., 2025 )	90.2	76.4	93.6	75.9	96.5	62.0	83.8
OS-ATLAS-7B ( Wu et al., 2025b )	95.2	75.8	90.7	63.6	90.6	77.3	85.1
UGround-V1-7B ( Gou et al., 2025 )	83.6	90.5	85.8	86.3	95.5	83.2	87.7

Table 2: Performance on ScreenSpot-V2 . Bold values = best results.

Model	Windows		macOS		Linux		iOS		Android		Web		Avg.
Model	Bas.	Adv.	Bas.	Adv.	Bas.	Adv.	Bas.	Adv.	Bas.	Adv.	Bas.	Adv.	Avg.
Qwen3-VL-2B ( Bai et al., 2025 )	81.9	0.0	80.3	46.2	67.5	0.0	90.8	0.0	91.0	0.0	88.1	0.0	46.5
OS-ATLAS-7B ( Wu et al., 2025b )	36.9	18.8	44.4	21.7	31.4	13.3	74.8	48.8	69.6	46.8	61.3	35.4	41.4
Aguvis-7B ( Xu et al., 2024 )	37.3	21.7	48.1	33.3	33.5	25.0	67.5	65.2	61.0	51.0	61.6	45.5	45.7
UI-TARS-1.5-7B ( Qin et al., 2025 )	68.3	39.0	69.0	44.5	64.4	37.8	88.5	69.4	90.5	69.3	81.0	56.5	64.3
UGround-V1-7B ( Gou et al., 2025 )	66.8	39.0	71.3	48.6	56.5	31.1	92.7	70.9	93.5	71.0	88.7	64.6	65.7

Table 3: Performance comparison on MMBench-GUI L2 benchmark. Bold values = best results.

Model	CAD		Dev.		Creative		Scientific		Office		OS		Avg.
Model	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Text	Icon	Avg.
Qwen3-VL-2B ( Bai et al., 2025 )	31.0	15.6	55.2	11.7	59.1	16.1	64.6	22.7	72.3	34.0	59.8	23.6	41.9
InfiGUI-3B ( Liu et al., 2026a )	50.8	25.0	64.9	20.0	51.5	16.8	68.8	32.7	70.6	32.1	49.5	19.7	45.2
Ferret-UI Lite ( You et al., 2024 )	-	-	-	-	-	-	-	-	-	-	-	-	53.3
UI-TARS-7B ( Qin et al., 2025 )	20.8	9.4	58.4	12.4	50.0	9.1	63.9	31.8	63.3	20.8	30.8	16.9	35.7
Phi-Ground ( Zhang et al., 2025 )	26.9	17.2	70.8	16.7	56.6	13.3	58.0	29.1	76.4	44.0	55.1	25.8	43.2

Table 4: Performance comparison on ScreenSpot-Pro benchmark. Bold values = best results.

Model	Text Matching	Element Recognition	Layout Understanding	Fine-grained Manipulation	Refusal	Avg
JEDI-3B ( Xie et al., 2025 )	67.4	53.0	53.8	44.3	7.4	50.9
Qwen3-VL-2B ( Bai et al., 2025 )	61.7	45.8	54.2	39.6	-	45.9
OS-ATLAS-7B ( Wu et al., 2025b )	44.1	29.4	35.2	16.8	7.4	27.7
UGround-7B ( Gou et al., 2025 )	51.3	40.3	43.5	24.8	0.0	36.4
Aguvis-7B ( Xu et al., 2024 )	55.9	41.2	43.9	28.2	0.0	38.7
UI-TARS-7B ( Qin et al., 2025 )	60.2	51.8	54.9	35.6	0.0	47.5

Table 5: Performance comparison on OSWorld-G benchmark. Bold values = best results.

Model	Basic	Functional	Spatial	Avg
SeeClick ( Cheng et al., 2024 )	9.4	4.7	2.1	5.4
Qwen3-VL-2B ( Bai et al., 2025 )	0.0	19.2	0.1	6.2
OS-ATLAS-7B ( Wu et al., 2025b )	12.2	11.2	3.7	9.0
UGround-V1-7B ( Gou et al., 2025 )	15.4	17.1	6.3	12.9
Qwen3-VL-8B ( Bai et al., 2025 )	25.0	27.9	1.2	17.5
UI-TARS-7B ( Qin et al., 2025 )	20.1	24.3	8.4	17.6

Table 6: Performance comparison on UI-Vision benchmark. Bold values = best results.

Model	SS-Pro	OSW-G	MMB-GUI L2	SS-V2	Average
Qwen3-VL-8B (reported) ( Zhao et al., 2026b )	49.9	54.8	81.3	91.7	69.4
+ ScaleCUA SFT	52.8	61.7	83.6	93.6	72.9
+ GroundCUA SFT	58.5	59.6	84.9	91.9	73.7
+ ScreenRef SFT (ours)	62.4	64.9	84.5	93.9	76.4

Table 7: Qwen3-VL-8B fine-tuned on different grounding datasets under recommended settings for each. UI-Vision is omitted due to source overlap with GroundCUA. Bold values = best results.

Model	SS-V2	SS-Pro	MMB-GUI L2	UI-Vision	OSW-G	Average
SFT	93.9	62.4	84.5	34.3	64.9	68.0
+ OPSD	95.0	62.6	85.8	37.0	65.1	69.1
+ RLOO	95.0	63.4	85.8	36.5	64.0	68.9
+ RLOO †	95.0	62.8	86.4	37.5	63.8	69.1
+ D2RL	95.2	63.9	86.8	38.3	65.4	69.9

Table 8: Ablation study of Distillation-to-Reinforcement Learning on all GUI grounding benchmarks. RLOO † denotes a model first trained with SFT using 12K RL data and then optimized with RLOO. Bold values = best results.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Examples of the four ScreenRef task types in desktop environments.

Figure 4: Examples of the four ScreenRef task types in web environments.

Figure 5: Examples of the four ScreenRef task types in mobile environments.

Platform	Text	Visual	Spatial	Refusal	Total
Desktop	128,018	61,951	63,291	13,912	267,172
Web	167,115	62,294	67,799	6,582	303,790
Mobile	24,160	12,526	23,740	1,981	62,407
Total	319,293	136,771	154,830	22,475	633,369

Appendix

Table 9: Distribution of grounding capabilities across platforms.

Figure 6: Examples of element metadata from live screenshot crawling. Left: a real screenshot, cropped to the region containing metadata, with all element boxes in this region that survive filtering, colored by category (“(none)” marks elements without a category). Right: three numbered elements from the same screenshot, zoomed in, with their stored records (bounding box, text, and category).

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Nov 2, 2025Shijie Zhou, Viet Dac Lai, Hao Tan +4Multimodal Grounding GUI Grounding
University at Buffalo, USA · Adobe Research, USA
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University
DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
May 15, 2026Yichao Liu, Huawen Shen, Liu Yu +3GUI Agents GUI Grounding
1Nankai University · Institute of Information Engineering, Chinese Academy of Sciences

Nov 2, 2025J/K move · Enter open · S save

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil+3

University at Buffalo, USA · Adobe Research, USA

Multimodal Grounding GUI Grounding Large Vision-Language Models Cross-Modal Attention

Open paper AskPDF arXiv

Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .

Abstract

GUI Grounding

Explore similar work

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

GUI-C2^22: Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning

DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning