AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Bingchao Wang; Yongcheng Jing; Chengyu Wang; Jun Huang; Jie Yang; Wei Liu; Dacheng Tao

cs.CVOct 27, 2026

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Authors: Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

Organizations: Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing

Abstract

GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround

Figures & tables

Figure 1: Accuracy of AdaGround, the single pass and the published results of MVP and ZoomClick, each measured from its own single pass. ZoomClick reports Qwen3-VL-32B only. UI-Vision is held out.

Figure 2: Overview of AdaGround. Compositional Structure Grounding (CSG) reads the instruction. Two zoom chains, Token-Normalized Adaptive Zooming (TNZ) and a fixed $4\times$ crop, run on every sample. Verifier-in-the-Loop Refinement (VLR) uses one Yes/No verifier to keep a zoom, pick a chain, retry with the rejected click crossed out, and abstain. Top : the pipeline on one ScreenSpot-Pro sample with Qwen3-VL-8B (real model outputs), where $v$ badges mark the model’s Yes/No checks. Bottom : (a) TNZ, (b) VLR and (c) CSG, of which (a) and (c) are schematic.

Method	Text	Element	Layout	Fine-gr.	Refusal	Overall
Closed-source / proprietary models
Operator †∘ ( OpenAI, 2025 )	51.3	42.4	46.6	31.5	0.0	40.6
Gemini-2.5-pro †∘ ( Comanici et al., 2025 )	59.8	45.5	49.0	33.6	38.9	45.2
Seed1.5-VL †∘ ( Guo et al., 2025 )	73.9	66.7	69.6	47.0	18.5	62.9
Open-source SFT / RL-trained specialists
Qwen2.5-VL-72B † ( Bai et al., 2025b )	52.6	74.6	74.7	55.3	0.0	62.2

Table 1: OSWorld-G accuracy (%). Refusal: abstention rate on the

54

absent-target instructions. † Quoted from the cited papers or the tables of Yang et al. (2026) and Zhang et al. (2026d) . ∘ Original instructions (Appendix D.2 ). –: not reported. Bold : best training-free result. Subscripts : gain over the method’s own single pass. ∗ FP8 weights.

Method	Dev.	Creative	CAD	Scientific	Office	OS	Overall
Closed-source / proprietary models
GPT-4o † ( Hurst et al., 2024 )	0.7	0.6	1.5	1.2	0.9	0.0	0.8
Claude Computer Use † ( Anthropic, 2024 )	12.6	16.8	11.9	25.8	26.9	8.1	17.1
UI-TARS-1.5 † ( Qin et al., 2025 )	63.9	50.4	58.2	69.3	79.6	51.0	61.6
Seed1.5-VL † ( Guo et al., 2025 )	53.8	59.2	59.0	61.4	74.8	60.2	60.9
Open-source SFT / RL-trained specialists

Table 2: ScreenSpot-Pro accuracy (%). † Quoted from the cited papers or as compiled by Yang et al. (2026) and Zhang et al. (2026d) . § Category averages computed from the reported text/icon accuracies. Bold : best training-free result. Subscripts : gain over the method’s own single pass. ∗ FP8 weights.

Method	Basic	Func.	Spatial	Avg.
Open-source SFT / RL-trained specialists †
UI-TARS-7B ( Qin et al., 2025 )	20.1	24.3	8.4	17.6
UI-TARS-72B ( Qin et al., 2025 )	31.4	30.5	14.7	25.5
UI-Venus-Ground-72B ( Gu et al., 2025 )	45.6	42.3	23.7	36.8
OS-Atlas-7B ( Wu et al., 2025b )	12.2	11.2	3.7	9.0
UGround-V1-7B ( Gou et al., 2025 )	15.4	17.1	6.3	12.9

Table 3: UI-Vision element grounding ( Nayak et al., 2025 ) (held out), accuracy (%). Avg.: mean of the three tasks (as published for † rows). † From Zhang et al. (2026d) (specialists from the UI-Vision leaderboard) and Jiang et al. (2025) . Subscripts : gain over the method’s own single pass (MVP: 27.2, 36.4; ZoomClick: 36.6). Bold : best training-free. ∗ FP8 weights.

	OSWorld-G		SSPro		SSv2
	8B	32B ∗	8B	32B ∗	8B	32B ∗
Single pass	64.7	71.8	51.3	55.3	92.1	94.7
Fixed $4\times$ crop	70.7	77.1	63.5	73.9	91.9	95.4
$c_{\mathrm{TNZ}}$	72.7	76.6	59.6	72.4	92.5	95.4
+ arbitration	73.6	76.8	65.6	75.6	92.5	96.0
+ re-grounding	73.2	77.0	66.2	76.0	91.9	95.4

Table 4: Where the gains come from : accuracy (%) as components are added (Algorithm 1 ). Shaded: AdaGround. Bold : best per column (oracle excluded). Underlined : the better chain. Oracle: either chain is right. SSPro/SSv2: ScreenSpot-Pro/-v2. ∗ FP8 weights.

Figure 3: Chain arbitration (counts: samples on which only that chain is right). The Yes/No check runs whenever the two clicks differ. With Qwen3-VL-32B the three bars read

72.4

56.9

and

50.0\%

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Constant	Value
Rendering	grid; pixels per view	$32$ px; $3{,}136$ to $8{,}847{,}360$
Coarse view	pixel budget $A_{0}$ ; floor $b_{\min}$ of $b_{1}$	$2.2$ MP; $0.25$
TNZ	token pitch $\pi$	$32$ (Qwen3-VL)
	target span $\tau$	$3$ tokens
	context multiple $\gamma$ ; error budget $\varepsilon$	$16$ ; $4$ tokens of the previous view
	maximum magnification $k_{\max}$ (linear)	$4$

Appendix

Table 5: Constants of AdaGround (Section 3.5 ). A single set serves both backbones and all benchmarks, and only

\pi

is read off the encoder.

d_{0}(W,H)

is the window side of Section 3.3 ,

L

is the longer side of the rendered view, and

n\times

area means

\sqrt{n}

times the linear size.

Qwen3-VL-8B	Acc.	Tok. (k)	Calls	Depth	Lat. (s)	p95
OSWorld-G
Single pass	64.7	1.7	1.0	1.0	0.9	1.3
Fixed $4\times$ crop	70.7	5.3	2.0	2.0	3.1	3.8
$c_{\mathrm{TNZ}}$	72.7	10.4	6.9	4.0	7.7	10.5
+ arbitration	73.6	16.2	9.3	4.2	11.1	14.6
+ re-grounding	73.2	17.7	10.9	5.8	12.0	16.5

Appendix

Table 6: Cost per sample (Qwen3-VL-8B, full benchmarks). Columns: accuracy (%), visual tokens (thousands), model calls, and calls in sequence (depth, with independent calls in parallel). Latency: mean and 95th-percentile seconds per sample on

150

random samples per benchmark, with sequential calls (Appendix A.4 ). The last row of each block is AdaGround.

Backbone	Rule	$\theta_{e}$	Overall	Present targets	Abstentions (correct / false)
Qwen3-VL-8B	no gate	–	73.6	81.4	0 / 0
	existence	0.01	74.6	77.8	24 / 31
	joint	0.03	76.1	81.2	15 / 4
Qwen3-VL-32B ∗	no gate	–	77.8	86.1	0 / 0
	existence	0.01	81.9	83.9	34 / 19
	joint	0.03	79.3	86.1	8 / 2

Appendix

Table 7: Abstention rules on OSWorld-G . Existence : abstain iff

e(S,I)<\theta_{e}

. Joint : abstain iff

\tfrac{1}{2}(v(p)+e(S,I))<\theta_{e}

(Eq. ( 8 )). Abstentions: correct (absent target) / false (present target). Present targets: accuracy where the target is present. The existence rule uses

\theta_{e}=0.01

, the most conservative value tried. Shaded: the configuration used elsewhere. ∗ FP8 weights.

	Accuracy (%)					Arbitration minus (points)		Verifier
Benchmark	$c_{\mathrm{TNZ}}$	$c_{4\times}$	random	arbitration	oracle	better chain	random	pick
Qwen3-VL-8B
OSWorld-G	72.7	70.7	71.7	73.6	76.1	$+0.9$ $[-0.3,+2.2]$ 8/3	$+1.9$ $[+0.6,+3.1]$	27/35
ScreenSpot-Pro	59.6	63.5	61.6	65.6	68.9	$+2.1$ $[+0.8,+3.3]$ 67/34	$+4.0$ $[+3.1,+4.9]$	177/227
ScreenSpot-v2	92.5	91.9	92.2	92.5	94.0	$0.0$ $[-0.6,+0.6]$ 8/8	$+0.3$ $[-0.2,+0.8]$	27/46
Qwen3-VL-32B ∗

Appendix

Table 8: Chain arbitration before re-grounding, CSG and abstention. Random: either chain with probability

\frac{1}{2}

(expected). Oracle: either chain is right. Underlined: the better chain, chosen after the fact. Differences:

95\%

CI and, against the better chain, samples won/lost. Pick: verifier picks the right chain / samples with exactly one chain right (verifier called). ∗ FP8 weights.

	Second chain	Chain	+ arbitration	AdaGround variant	$\Delta$ vs. AdaGround
Qwen3-VL-8B
OSWorld-G	$c_{\mathrm{TNZ}}$ (AdaGround)	72.7	73.6	76.1	–
	fixed $540{\times}540$ crop, $2.78\times$	71.1 ( $-1.6$ )	72.7 ( $-0.9$ $[-2.9,+1.2]$ )	75.7	$-0.4$ $[-2.3,+1.5]$
ScreenSpot-Pro	$c_{\mathrm{TNZ}}$ (AdaGround)	59.6	65.6	66.2	–
	fixed $540{\times}540$ crop, $2.78\times$	62.4 ( $+2.7$ )	65.7 ( $+0.1$ $[-1.1,+1.3]$ )	66.1	$-0.1$ $[-1.3,+1.1]$
ScreenSpot-v2	$c_{\mathrm{TNZ}}$ (AdaGround)	92.5	92.5	91.9	–

Appendix

Table 9: Fixed-geometry controls . Each control replaces

c_{\mathrm{TNZ}}

with another second chain and leaves everything else unchanged. Values are accuracy (%). In parentheses: the paired difference in points to the same column of the

c_{\mathrm{TNZ}}

(AdaGround) row, with

95\%

CI for arbitration.

2.78\times

: linear scale. h Held out and pooled over its instructions (Qwen3-VL-8B only). ∗ FP8 weights.

	OSWorld-G				ScreenSpot-Pro				ScreenSpot-v2
Backbone	Single	Fixed $4\times$	MVP	AdaGround	Single	Fixed $4\times$	MVP	AdaGround	Single	Fixed $4\times$	AdaGround
Qwen3-VL-8B	64.7	70.7	69.9/72.7 †	76.1	51.3	63.5	61.9/65.3 †	66.2	92.1	91.9	91.9
Qwen3-VL-32B ∗	71.8	77.1	72.0 †	79.3	55.3	73.9	74.1 ‡ /74.0 †	75.9	94.7	95.4	95.4

Appendix

Table 10: Overall accuracy (%) on the full benchmarks , with one configuration of AdaGround for both backbones and all benchmarks. Best value per backbone and benchmark in bold . MVP ( Zhang et al., 2026d ) values are our reproduction with its official code when unmarked, the authors’ released per-sample predictions when marked ‡ , and published results when marked † . ∗ FP8 weights.

OSWorld-G	Text	Element	Layout	Fine-gr.	Refusal	Overall
Qwen3-VL-8B single pass	78.2	68.5	71.9	53.7	0.0	64.7
+ fixed $4\times$ crop ( $c_{4\times}$ alone)	80.1	77.3	78.7	58.4	0.0	70.7 +6.0
+ MVP (official code ‡ , reproduced)	78.2	76.4	79.4	56.4	0.0	69.9 +4.8
+ AdaGround	85.1	80.0	81.8	65.1	27.8	76.1 +11.3
Qwen3-VL-32B ∗ single pass	81.6	75.8	77.5	63.1	0.0	71.8
+ fixed $4\times$ crop ( $c_{4\times}$ alone)	82.0	84.2	86.6	63.8	0.0	77.1 +5.3

Appendix

Table 11: Reproduced baselines , accuracy (%) per category, which the main tables omit. The fixed

4\times

crop is AdaGround’s chain

c_{4\times}

alone (an ablation, Section 4.3 ). MVP ( Zhang et al., 2026d ) is reproduced with its official code on Qwen3-VL-8B, or scored from the per-sample predictions released by its authors for Qwen3-VL-32B ScreenSpot-Pro. ‡ On OSWorld-G, MVP runs its ScreenSpot-Pro pipeline (Appendix D.1 ). Single pass and AdaGround as in Tables 1 , 2 and 3 . Subscripts : gain over the method’s own single pass. MVP’s single pass scores

65.1

and

51.5

in our reproduction, and the released predictions include none. UI-Vision Avg.: mean of the three tasks. ∗ FP8 weights.

	MVP ( Zhang et al., 2026d )	AdaGround (ours)
Target scale	not modelled; fixed-size attention crops	estimated from the model’s own box; $k$ and $d$ solved per sample (TNZ)
View selection	attention maps from a chosen layer	size estimate from the model’s own box output
Aggregation	spatial clustering of 3 predictions, largest group	verifier arbitration over a static chain portfolio; verification-gated acceptance
Failure handling	none	negative-feedback re-grounding; existence-gated abstention; compositional cell grounding
Model access	attention maps (white box)	black box: text and first-token probabilities
Decoding	greedy	greedy

Appendix

Table 12: AdaGround vs. MVP : design differences.

Benchmark	vs. single	vs. fixed $4\times$	vs. fixed $4\times^{a}$	vs. fixed $4\times^{b}$	vs. MVP	vs. MVP a	vs. MVP b
Qwen3-VL-8B
OSWorld-G	$+11.3$ $[+8.3,+14.4]$	$+5.3$ $[+2.3,+8.5]$	$+2.8$ $[+0.4,+5.4]$	$+2.3$ $[0.0,+4.8]$	$+6.2$ $[+3.3,+9.1]$	$+3.7$ $[+1.3,+6.1]$	$+3.0$ $[+0.6,+5.4]$
	71/7	47/17	32/16	32/19	53/18	38/17	38/21
ScreenSpot-Pro	$+14.9$ $[+12.8,+16.9]$	$+2.7$ $[+1.3,+4.0]$	–	–	$+4.2$ $[+2.6,+5.9]$	–	–
	276/41	81/39			125/58
ScreenSpot-v2	$-0.2$ $[-1.5,+1.0]$	$0.0$ $[-1.0,+1.0]$	–	–	–	–	–

Appendix

Table 13: Paired differences (points) of AdaGround on the full benchmarks, with

95\%

confidence intervals (Appendix D.2 ) and, below, samples won/lost. MVP: our reproduction (Qwen3-VL-8B) or the authors’ released predictions (Qwen3-VL-32B ScreenSpot-Pro). UI-Vision: held out, pooled over its instructions. a AdaGround without abstention. b The baseline with the abstention rule of Eq. ( 8 ) applied to its own answer (OSWorld-G). ∗ FP8 weights.

Figure 4: Accuracy versus token span per benchmark (bins with

\geq 30

views) and pooled (dashed), on views that contain the target (Table 14 ). Whiskers and bands:

95\%

screenshot-bootstrap intervals. Dotted line: TNZ’s target span

\tau=3

	ScreenSpot-Pro		OSWorld-G		ScreenSpot-v2		Pooled
$\sigma$ (tokens)	$n$	Acc.	$n$	Acc.	$n$	Acc.	$n$	Acc. [95% CI]
$0$ – $0.5$	342	24.0	366	42.6	0	–	708	33.6 [26.7, 40.4]
$0.5$ – $1$	4180	55.4	1109	78.5	185	88.1	5474	61.2 [58.9, 63.6]
$1$ – $1.5$	1396	63.1	532	78.2	743	90.3	2671	73.7 [70.6, 76.6]
$1.5$ – $2$	813	66.9	173	91.9	615	89.3	1601	78.2 [75.0, 81.3]
$2$ – $3$	317	75.4	174	87.4	1422	92.6	1913	89.3 [86.8, 91.6]

Appendix

Table 14: Accuracy versus token span (Qwen3-VL-8B, with the nine baseline passes of Appendix E ), on views that contain the target. Top: accuracy by the token span

\sigma

in the view the model saw, per benchmark and pooled (

n

= views). Pooled cells carry

95\%

screenshot-bootstrap intervals, with all views of a screenshot resampled together. Bottom: the same sample on its coarse view and on the crop of the same baseline, with mean span, accuracy before and after, and the paired change.

Figure 5: Within-sample change from the crop pass (Table 14 , bottom), by span before

\to

after (green gain, red loss). Above each bar: accuracy before the crop and

n

. Whiskers:

95\%

paired screenshot-bootstrap intervals.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
Mar 18, 2026Siqi Pei, Liang Tang, Tiaonan Duan +9GUI Grounding GUI Automation
Lenovo Research, Beijing, China · Tsinghua University, Beijing, China
One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
Jun 29, 2026Chen Liu, Ling Chen, Hanzhang Zhou +5Efficient Multimodal Inference GUI Grounding
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
Apr 15, 2026Fei Tang, Bofan Chen, Zhengxi Lu +8GUI Grounding Vision-Language Grounding
Zhejiang University · Ant Group

Mar 18, 2026J/K move · Enter open · S save

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

Siqi Pei, Liang Tang, Tiaonan Duan, Long Chen+8

Lenovo Research, Beijing, China · Tsinghua University, Beijing, China

GUI Grounding GUI Automation

Open paper AskPDF arXiv

GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.

Abstract

GUI Grounding

Explore similar work

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding