PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

Wenjun Hou; Kaishuai Xu

cs.CVOct 24, 2026

PreView: Enhancing GUI Grounding with Preprocessed Views and Cues

Authors: Wenjun Hou, Kaishuai Xu

Organizations: Independent Researcher

Abstract

GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.

Figures & tables

Figure 1: PreView prepares views and cues at the input level, without modifying the grounding model or accessing its internal states.

Figure 2: Overview of PreView . For localization, the auxiliary model refines the initial center $c_{1}$ into $c_{2}$ to prepare a task-relevant view through cropping and enlargement. For understanding, it provides a visual marker with a location hint and generates a candidate description. The frozen grounding model uses the prepared view, instruction, and available cues in a single call. The predicted click is then mapped back to the original screenshot.

$\leq\mathbf{3}$ B
Model	SSPro	OSW-G O	OSW-G R	MMB-GUI	SSv2	UI-V	Avg.
UI-R1-E-3B	$33.5$	–	$48.8$	$68.4$	$88.6$	$16.5$	–
SE-GUI-3B	$35.9$	–	$46.1$	$66.3$	$86.8$	$15.0$	$50.0$
InfiGUI-R1-3B	$35.7$	–	$42.9$	$70.6$	$89.5$	$17.8$	$51.3$
GUI-G 2 -3B	$36.4$	–	$53.5$	$66.3$	$87.6$	$18.7$	$52.5$
GUI-G1-3B	$37.1$	–	$49.5$	$71.0$	$89.5$	$20.3$	$53.5$

Table 1: Grounding accuracy (%) by model size. O and R denote original and refined OSW-G instructions, respectively. Bold marks PreView scores that are column-best within each size group.

\Delta

denotes percentage-point gains. ∗ marks our SSPro evaluations. – indicates unavailable or unverified scores. Reference averages follow GroundCUA. Our five-benchmark averages use OSW-G R .

Hyperparameter	Value
Epochs / precision	$2$ / BF16
Optimizer	AdamW
Learning rate	$5\times 10^{-5}$
Weight decay	$10^{-3}$
LR schedule / warmup	Linear / $3\%$
Micro / effective batch size	$4$ / $128$

Table 2: Auxiliary-model training configuration.

Method	Grounding model	Base	Enhanced	$\boldsymbol{\Delta}$
ReGround	OS-Atlas-7B	$18.9$	$40.2$	$+21.3$
ScreenSeekeR	OS-Atlas-7B	$18.9$	$48.1$	$+29.2$
RegionFocus	Qwen2.5-VL-7B	$26.8$	$32.1$	$+5.3$
ZoomClick	UI-Venus-7B	$50.3$	$65.7$	$+15.4$
UI-Zoomer	UI-Venus-7B	$50.0$	$61.8$	$+11.8$
AutoFocus	UI-Venus-7B	$50.3$	$67.8$	$+17.5$

Table 3: Comparison with training-free methods on SSPro. The Base and Enhanced columns show accuracy (%).

\Delta

is the gain in percentage points.

Figure 3: Auxiliary-model direct hit (a) and center containment (b). Initial, Refine 1, and Refine 2 correspond to

c_{1}

c_{2}

, and

c_{3}

, respectively.

Benchmark	Accept	Reject	$N_{\mathrm{Reject}}$
	Direct hit (%)
SSPro	$62.0$	$18.6$	$295$
OSW-G R	$77.2$	$25.4$	$71$
MMB-GUI	$82.0$	$41.3$	$625$
SSv2	$90.0$	$63.4$	$344$
UI-V	$49.6$	$12.8$	$1520$

Table 4: Auxiliary-model direct hit by verification response, on target-present examples. Weighted Avg. weights each benchmark by its accepted or rejected sample count for the corresponding column.

Model	Input	Visual Cue	Textual Cue	SSPro	OSW-G O	OSW-G R	MMB-GUI	SSv2	UI-V	Avg.
Qwen3-VL-2B	Full	–	–	$43.1$	$46.1$	$62.1$	$72.9$	$91.4$	$16.5$	$57.2$
	View	–	–	$56.9$	$52.7$	$66.3$	$75.7$	$84.8$	$26.0$	$61.9$
	View	$\checkmark$	–	$55.6$	$54.8$	$65.8$	$76.4$	$85.2$	$32.7$	$63.1$
	View	$\checkmark$	$\checkmark$	$59.3$	$55.0$	$66.8$	$77.0$	$86.0$	$33.7$	$64.6$
OS-Atlas-4B	Full	–	–	$2.8$	$10.5$	$8.5$	$26.6$	$62.7$	$7.3$	$21.6$
	View	–	–	$21.7$	$27.1$	$24.1$	$37.8$	$63.0$	$13.0$	$31.9$

Table 5: View and cue ablation (accuracy, %). Full uses the original screenshot. Avg. uses OSW-G R .

Figure 4: Effects of cropping area and enlargement on Qwen3-VL-2B at different visual budgets.

Figure 5: Cue recovery and regression under correct and incorrect cue locations for Qwen3-VL-2B.

Benchmark	Full	View
SSPro	$52.9$	$59.0$ $\uparrow$
OSW-G O	$56.2$	$57.6$ $\uparrow$
OSW-G R	$67.7$	$68.3$ $\uparrow$
MMB-GUI	$81.1$	$76.9$ $\downarrow$
SSv2	$90.4$	$81.8$ $\downarrow$
UI-V	$60.3$	$54.5$ $\downarrow$

Table 6: GroundNext-7B with View (accuracy, %). Arrows compare with Full.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Samples
SSPro	$1{,}581$
OSW-G R	$564$
MMB-GUI	$3{,}594$
SSv2	$1{,}272$
UI-V	$5{,}479$

Appendix

Table 7: Sample counts for the evaluation benchmarks.

		Qwen3-VL-2B		Qwen3-VL-4B
Category	$N$	Full	PreView	Full	PreView
Basic	$1772$	$23.8$	$43.8$	$33.5$	$49.6$
Functional	$1772$	$21.5$	$41.8$	$31.5$	$47.5$
Spatial	$1935$	$5.3$	$17.0$	$12.0$	$26.0$
Text	$980$	$42.6$	$61.2$	$54.1$	$69.7$
Icon	$4499$	$10.9$	$27.7$	$19.0$	$34.2$

Appendix

Table 8: UI-V results by instruction and element type.

		Qwen3-VL-2B		Qwen3-VL-4B
Category	$N$	Full	PreView	Full	PreView
CAD/Text	$197$	$32.0$	$61.4$	$59.9$	$68.0$
CAD/Icon	$64$	$18.8$	$28.1$	$20.3$	$32.8$
Dev/Text	$154$	$64.3$	$74.0$	$81.8$	$78.6$
Dev/Icon	$145$	$13.1$	$37.2$	$34.5$	$40.0$
Creative/Text	$198$	$57.6$	$67.7$	$71.2$	$71.7$

Appendix

Table 9: SSPro results by application domain and element type.

		Qwen3-VL-2B		Qwen3-VL-4B
Category	$N$	Full	PreView	Full	PreView
Text Matching	$261$	$74.7$	$73.2$	$81.2$	$78.5$
Element recognition	$330$	$66.1$	$72.7$	$73.3$	$75.5$
Layout understanding	$253$	$71.9$	$74.3$	$76.7$	$79.4$
Fine-grained manipulation	$152$	$48.7$	$52.0$	$62.5$	$59.9$
Refusal bucket	$9$	$0.0$	$0.0$	$22.2$	$0.0$

Appendix

Table 10: OSW-G R results by task type.

		Qwen3-VL-2B		Qwen3-VL-4B
Category	$N$	Full	PreView	Full	PreView
Windows/Basic	$271$	$80.4$	$87.8$	$86.3$	$90.4$
Windows/Adv	$272$	$45.6$	$52.6$	$68.0$	$59.9$
macOS/Basic	$345$	$78.0$	$87.8$	$90.4$	$89.9$
macOS/Adv	$346$	$50.3$	$61.6$	$73.1$	$68.5$
Linux/Basic	$191$	$66.0$	$75.4$	$79.6$	$77.0$

Appendix

Table 11: MMB-GUI results by platform and difficulty.

Panel of Figure 6 — Figure 6: Successful corrections and failure cases of view and cue preprocessing for Qwen3-VL-2B.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
Aug 4, 2026Zichuan Fu, Shirong Wang, Wenlin Zhang +10GUI Grounding Visual Grounding
City University of Hong Kong · Westlake University · Tencent Jarvis Lab
ScreenHaystack: Finding Blind Zones in GUI Grounding
Sep 25, 2026Chenyue Li, Xiaoxiao Sun, Yubo Deng +3GUI Grounding Data Augmentation
Stanford University
AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
Oct 27, 2026Bingchao Wang, Yongcheng Jing, Chengyu Wang +4GUI Grounding Visual Grounding
Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing

Aug 4, 2026J/K move · Enter open · S save

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

Zichuan Fu, Shirong Wang, Wenlin Zhang, Guojing Li+9

City University of Hong Kong · Westlake University · Tencent Jarvis Lab

GUI Grounding Visual Grounding Active Visual Perception

Open paper AskPDF arXiv

GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.

Abstract

GUI Grounding

Explore similar work

GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

ScreenHaystack: Finding Blind Zones in GUI Grounding

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding