UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

Fei Tang; Bofan Chen; Zhengxi Lu; Tongbo Chen; Songqin Nong; Tao Jiang; Wenhao Xu; Weiming Lu

cs.CVApr 15, 2026

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

Authors: Fei Tang, Bofan Chen, Zhengxi Lu, Tongbo Chen, Songqin Nong, Tao Jiang, Wenhao Xu, Weiming Lu,

Organizations: Zhejiang University · Ant Group

Abstract

GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose UI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4%, +10.3%, and +4.2% respectively, with no additional training required.

Figures & tables

Figure 1: Comparison of GUI grounding paradigms. (a) Direct grounding methods struggle with dense interfaces. (b) Iterative cropping methods incur large resource costs and use rigid cropping ratios. (c) Our UI-Zoomer applies Test-Time Scaling (TTS) with reliability gating, adaptively choosing between consensus voting and adaptive cropping, achieving stronger robustness with one-take time costs.

Method	Avg Acc	Time
w/o DiMo-GUI	81.84%	35:47
w/ DiMo-GUI	77.20%	6:43:07

Table 1: Accuracy and inference time of w/ and w/o iterative cropping on ScreenSpot-V2.

Figure 2: Overview of UI-Zoomer. (a) The model samples $N$ candidate predictions via Test-Time Scaling (TTS). (b) A reliability gate routes confident instances to consensus voting (Choice I) and uncertain ones to adaptive cropping (Choice II). (d) The crop window is derived from 2D Gaussian variance decomposition, enabling per-instance adaptive zoom-in.

Benchmark	Methods	Category 1			Category 2			Category 3			Overall
Benchmark	Methods	text	icon	avg	text	icon	avg	text	icon	avg	text	icon	avg
ScreenSpot-v2	UI-Venus-7B	99.0	90.1	95.2	97.9	89.3	94.3	94.9	89.7	92.5	97.4	89.7	94.0
	+ UI-Zoomer	98.6	90.5	95.2	99.0	92.9	96.4	95.7	90.6	93.4	97.8	91.2	94.9
	$\Delta$ Improvement	-0.4	+0.4	+0.0	+1.1	+3.6	+2.1	+0.8	+0.9	+0.9	+0.4	+1.5	+0.9
UI-Vision	UI-Venus-7B	68.5	23.8	33.3	65.5	19.9	29.7	38.7	7.8	11.4	60.6	16.5	24.4
	+ UI-Zoomer	80.5	32.3	42.5	74.7	30.6	40.1	55.9	15.1	19.7	72.7	25.2	33.7

Table 2: Performance of UI-Venus-7B with and without UI-Zoomer on ScreenSpot-v2 (Mobile / Desktop / Web) and UI-Vision (Basic / Functional / Spatial).

Methods	Development		Creative		CAD		Scientific		Office		OS		Overall
Methods	text	icon	text	icon	text	icon	text	icon	text	icon	text	icon	text	icon	avg
Proprietary Methods
GPT-4o Hurst et al. (2024)	1.3	0.0	1.0	0.0	2.0	0.0	2.1	0.0	1.1	0.0	0.0	0.0	1.3	0.0	0.8
Claude-3.7-Sonnet cla ()	-	-	-	-	-	-	-	-	-	-	-	-	-	-	27.7
Seed-1.5-VL Guo et al. (2025)	-	-	-	-	-	-	-	-	-	-	-	-	-	-	60.9
General Open-source Models

Table 3: Performance comparison on ScreenSpot-Pro across four models: Qwen2.5-VL-7B, GUI-G 2 -7B, UI-Venus-7B, and UI-Venus-72B. For a fair comparison, RegionFocus is evaluated using Qwen2.5-VL-7B as the backbone.

Figure 3: Ablation on sampling temperature

T

(left) and number of candidates

N

(right) on ScreenSpot-Pro.

Table 7

Table 8

Table 9

Figure 4: Ablation study of the gating threshold

\tau

and Gaussian spread

\sigma

on ScreenSpot-v2. Grey bars indicate the proportion of samples routed to the zoom-in cropping stage (CROP%), while blue curves show overall accuracy.

Figure 5: Histogram distributions of

C_{\mathrm{spatial}}

and

avg\_conf

on ScreenSpot-Pro (

N=1581

). The two signals exhibit complementary distributional characteristics, enabling a more discriminative gating mechanism when combined.

Figure 6: Here are four representative cases: the top two are successful examples, while the remaining two illustrate failure cases. Blue boxes denote the eight sampled bounding boxes, red indicates the cropped region, green represents the ground-truth box, and yellow marks the final prediction obtained after zooming on the cropped image.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Complete prompt template used in our experiments.

Proprietary Methods
Methods	Mobile			Desktop			Web			Overall
Methods	text	icon	avg	text	icon	avg	text	icon	avg	text	icon	avg
GPT-4o Hurst et al. (2024)	-	-	22.5	-	-	22.2	-	-	12.4	-	-	20.1
Claude-3.7-Sonnet cla ()	-	-	-	-	-	-	-	-	-	-	-	87.6
Seed-1.5-VL Guo et al. (2025)	-	-	-	-	-	-	-	-	-	-	-	95.2
General Open-source Models

Appendix

Table 10: Performance comparison on ScreenSpot-v2. We evaluate our UI-Zoomer strategy across three models: Qwen2.5-VL-7B, GUI-G 2 -7B, and UI-Venus-7B.

Proprietary Methods
Methods	Basic			Functional			Spatial			Overall
Methods	text	icon	avg	text	icon	avg	text	icon	avg	text	icon	avg
GPT-4o Hurst et al. (2024)	-	-	1.6	-	-	1.5	-	-	1.0	-	-	1.4
Claude-3.7-Sonnet cla ()	-	-	9.5	-	-	7.7	-	-	7.6	-	-	8.3
Seed-1.5-VL Guo et al. (2025)	-	-	-	-	-	-	-	-	-	-	-	-
General Open-source Models

Appendix

Table 11: Performance comparison on UI-Vision. We evaluate our UI-Zoomer strategy across four models: Qwen2.5-VL-7B, GUI-G 2 -7B, UI-Venus-7B, and UI-Venus-72B .

Threshold ( $\tau$ )	Overall Acc	Gating Pass Branch		Crop Branch
Threshold ( $\tau$ )	Overall Acc	Trigger Rate	Accuracy	Trigger Rate	Accuracy
0.6	57.56%	53.89%	70.66%	46.11%	42.24%
0.8	60.97%	22.77%	79.44%	77.23%	33.42%
0.9	61.48%	9.93%	82.80%	90.07%	59.13%
0.95	61.54%	5.57%	81.82%	94.43%	60.34%
1.0	61.80%	2.91%	93.48%	97.09%	60.85%

Appendix

Table 12: Ablation of the Gating Threshold (

\tau

). These are the results of UI-Venus-7B on ScreenSpot Pro, with the hyperparameter

\sigma

is set to 2.5.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 51 papers

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
Jun 29, 2026Chen Liu, Ling Chen, Hanzhang Zhou +5Efficient Multimodal Inference GUI Grounding
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
Mar 18, 2026Siqi Pei, Liang Tang, Tiaonan Duan +9GUI Grounding GUI Automation
Lenovo Research, Beijing, China · Tsinghua University, Beijing, China
AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
May 4, 2026Ruilin Yao, Shegnwu Xiong, Tianyu Zou +2Uncertainty Quantification GUI Grounding
Wuhan University of Technology

Jun 29, 2026J/K move · Enter open · S save

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

Chen Liu, Ling Chen, Hanzhang Zhou, Liangyu Chen+4

University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University

Efficient Multimodal Inference GUI Grounding Large Vision-Language Models Feature Fusion

Open paper AskPDF arXiv

MLLM-based GUI grounding methods commonly formulate target localization as autoregressive coordinate generation, enabling models to leverage the strong instruction-following and semantic understanding capabilities of MLLMs. However, this formulation requires the model to retain region-level target evidence while decoding coordinate tokens with the spatial precision demanded by GUI clicking. Our diagnostic analysis reveals that target-region awareness emerges in intermediate decoder layers but is neither retained nor translated into the final coordinate prediction. Existing ZoomIn-style methods address this issue through an external crop-and-rerun pass, which improves localization but increases end-to-end latency and computational cost. To retain the accuracy benefits of two-pass zooming without this extra cost, we propose InnerZoom, a single-forward framework for cross-layer evidence bridging. InnerZoom transforms target-related cues from the original forward pass into a compact cross-layer evidence state, then preserves, refines, and reinjects this state throughout later decoding layers to guide coordinate prediction. Extensive experimental results suggest that InnerZoom-4B achieves state-of-the-art performance on all six GUI grounding benchmarks, obtaining 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing the previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively. Under a controlled 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average and outperforms two-pass ZoomIn by 1.3 points on average, while reducing end-to-end latency by up to 31.8% and TFLOPs by about 29%. Code and models will be publicly available.

Abstract

GUI Grounding

Explore similar work

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding