AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Bingchao Wang; Yongcheng Jing; Chengyu Wang; Jun Huang; Jie Yang; Wei Liu; Dacheng Tao

cs.CVOct 27, 2026

AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding

Authors: Bingchao Wang, Yongcheng Jing, Chengyu Wang, Jun Huang, Jie Yang, Wei Liu, Dacheng Tao

Organizations: Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing

Abstract

GUI grounding on high-resolution screens fails mostly on small targets, and zooming in is a common training-free remedy. However, most existing zooming pipelines do not match the zoom to the target's size, accept the zoomed answer unconditionally and never revisit a wrong answer. To address these issues, we propose AdaGround, a training-free framework with three components: (1) Token-Normalized Adaptive Zooming, which sets the magnification, field of view and zoom depth of each sample from the model's own estimate of the target size; (2) Verifier-in-the-Loop Refinement, which uses one marked-click Yes/No verifier to keep or reject each zoom, choose between zoom chains, trigger re-grounding and, with an existence query, decide when to abstain; and (3) Compositional Structure Grounding, which composes spreadsheet cells from their headers. On Qwen3-VL models, AdaGround surpasses state-of-the-art training-free zooming methods and many trained GUI models, improving accuracy over the backbone by 7.4 to 20.6 points on OSWorld-G and ScreenSpot-Pro, and its gains carry over to held-out UI-Vision. Code is available at https://github.com/bcwang-sjtu/AdaGround

Figures & tables

Figure 1: Accuracy of AdaGround, the single pass and the published results of MVP and ZoomClick, each measured from its own single pass. ZoomClick reports Qwen3-VL-32B only. UI-Vision is held out.

Figure 2: Overview of AdaGround. Compositional Structure Grounding (CSG) reads the instruction. Two zoom chains, Token-Normalized Adaptive Zooming (TNZ) and a fixed $4\times$ crop, run on every sample. Verifier-in-the-Loop Refinement (VLR) uses one Yes/No verifier to keep a zoom, pick a chain, retry with the rejected click crossed out, and abstain. Top : the pipeline on one ScreenSpot-Pro sample with Qwen3-VL-8B (real model outputs), where $v$ badges mark the model’s Yes/No checks. Bottom : (a) TNZ, (b) VLR and (c) CSG, of which (a) and (c) are schematic.

Method	Text	Element	Layout	Fine-gr.	Refusal	Overall
Closed-source / proprietary models
Operator †∘ ( OpenAI, 2025 )	51.3	42.4	46.6	31.5	0.0	40.6
Gemini-2.5-pro †∘ ( Comanici et al., 2025 )	59.8	45.5	49.0	33.6	38.9	45.2
Seed1.5-VL †∘ ( Guo et al., 2025 )	73.9	66.7	69.6	47.0	18.5	62.9
Open-source SFT / RL-trained specialists
Qwen2.5-VL-72B † ( Bai et al., 2025b )	52.6	74.6	74.7	55.3	0.0	62.2

Table 1: OSWorld-G accuracy (%). Refusal: abstention rate on the

54

absent-target instructions. † Quoted from the cited papers or the tables of Yang et al. (2026) and Zhang et al. (2026d) . ∘ Original instructions (Appendix D.2 ). –: not reported. Bold : best training-free result. Subscripts : gain over the method’s own single pass. ∗ FP8 weights.

Method	Dev.	Creative	CAD	Scientific	Office	OS	Overall
Closed-source / proprietary models
GPT-4o † ( Hurst et al., 2024 )	0.7	0.6	1.5	1.2	0.9	0.0	0.8
Claude Computer Use † ( Anthropic, 2024 )	12.6	16.8	11.9	25.8	26.9	8.1	17.1
UI-TARS-1.5 † ( Qin et al., 2025 )	63.9	50.4	58.2	69.3	79.6	51.0	61.6
Seed1.5-VL † ( Guo et al., 2025 )	53.8	59.2	59.0	61.4	74.8	60.2	60.9
Open-source SFT / RL-trained specialists

Table 2: ScreenSpot-Pro accuracy (%). † Quoted from the cited papers or as compiled by Yang et al. (2026) and Zhang et al. (2026d) . § Category averages computed from the reported text/icon accuracies. Bold : best training-free result. Subscripts : gain over the method’s own single pass. ∗ FP8 weights.

Method	Basic	Func.	Spatial	Avg.
Open-source SFT / RL-trained specialists †
UI-TARS-7B ( Qin et al., 2025 )	20.1	24.3	8.4	17.6
UI-TARS-72B ( Qin et al., 2025 )	31.4	30.5	14.7	25.5
UI-Venus-Ground-72B ( Gu et al., 2025 )	45.6	42.3	23.7	36.8
OS-Atlas-7B ( Wu et al., 2025b )	12.2	11.2	3.7	9.0
UGround-V1-7B ( Gou et al., 2025 )	15.4	17.1	6.3	12.9

Table 3: UI-Vision element grounding ( Nayak et al., 2025 ) (held out), accuracy (%). Avg.: mean of the three tasks (as published for † rows). † From Zhang et al. (2026d) (specialists from the UI-Vision leaderboard) and Jiang et al. (2025) . Subscripts : gain over the method’s own single pass (MVP: 27.2, 36.4; ZoomClick: 36.6). Bold : best training-free. ∗ FP8 weights.

	OSWorld-G		SSPro		SSv2
	8B	32B ∗	8B	32B ∗	8B	32B ∗
Single pass	64.7	71.8	51.3	55.3	92.1	94.7
Fixed $4\times$ crop	70.7	77.1	63.5	73.9	91.9	95.4
$c_{\mathrm{TNZ}}$	72.7	76.6	59.6	72.4	92.5	95.4
+ arbitration	73.6	76.8	65.6	75.6	92.5	96.0
+ re-grounding	73.2	77.0	66.2	76.0	91.9	95.4

Table 4: Where the gains come from : accuracy (%) as components are added (Algorithm 1 ). Shaded: AdaGround. Bold : best per column (oracle excluded). Underlined : the better chain. Oracle: either chain is right. SSPro/SSv2: ScreenSpot-Pro/-v2. ∗ FP8 weights.

Figure 3: Chain arbitration (counts: samples on which only that chain is right). The Yes/No check runs whenever the two clicks differ. With Qwen3-VL-32B the three bars read

72.4

56.9

and

50.0\%

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Constant	Value
Rendering	grid; pixels per view	$32$ px; $3{,}136$ to $8{,}847{,}360$
Coarse view	pixel budget $A_{0}$ ; floor $b_{\min}$ of $b_{1}$	$2.2$ MP; $0.25$
TNZ	token pitch $\pi$	$32$ (Qwen3-VL)
	target span $\tau$	$3$ tokens
	context multiple $\gamma$ ; error budget $\varepsilon$	$16$ ; $4$ tokens of the previous view
	maximum magnification $k_{\max}$ (linear)	$4$

Appendix

Table 5: Constants of AdaGround (Section 3.5 ). A single set serves both backbones and all benchmarks, and only

\pi

is read off the encoder.

d_{0}(W,H)

is the window side of Section 3.3 ,

L

is the longer side of the rendered view, and

n\times

area means

\sqrt{n}

times the linear size.

Qwen3-VL-8B	Acc.	Tok. (k)	Calls	Depth	Lat. (s)	p95
OSWorld-G
Single pass	64.7	1.7	1.0	1.0	0.9	1.3
Fixed $4\times$ crop	70.7	5.3	2.0	2.0	3.1	3.8
$c_{\mathrm{TNZ}}$	72.7	10.4	6.9	4.0	7.7	10.5
+ arbitration	73.6	16.2	9.3	4.2	11.1	14.6
+ re-grounding	73.2	17.7	10.9	5.8	12.0	16.5

Appendix

Table 6: Cost per sample (Qwen3-VL-8B, full benchmarks). Columns: accuracy (%), visual tokens (thousands), model calls, and calls in sequence (depth, with independent calls in parallel). Latency: mean and 95th-percentile seconds per sample on

150

random samples per benchmark, with sequential calls (Appendix A.4 ). The last row of each block is AdaGround.

Backbone	Rule	$\theta_{e}$	Overall	Present targets	Abstentions (correct / false)
Qwen3-VL-8B	no gate	–	73.6	81.4	0 / 0
	existence	0.01	74.6	77.8	24 / 31
	joint	0.03	76.1	81.2	15 / 4
Qwen3-VL-32B ∗	no gate	–	77.8	86.1	0 / 0
	existence	0.01	81.9	83.9	34 / 19
	joint	0.03	79.3	86.1	8 / 2

Appendix

Table 7: Abstention rules on OSWorld-G . Existence : abstain iff

e(S,I)<\theta_{e}

. Joint : abstain iff

\tfrac{1}{2}(v(p)+e(S,I))<\theta_{e}

(Eq. ( 8 )). Abstentions: correct (absent target) / false (present target). Present targets: accuracy where the target is present. The existence rule uses

\theta_{e}=0.01

, the most conservative value tried. Shaded: the configuration used elsewhere. ∗ FP8 weights.

	Accuracy (%)					Arbitration minus (points)		Verifier
Benchmark	$c_{\mathrm{TNZ}}$	$c_{4\times}$	random	arbitration	oracle	better chain	random	pick
Qwen3-VL-8B
OSWorld-G	72.7	70.7	71.7	73.6	76.1	$+0.9$ $[-0.3,+2.2]$ 8/3	$+1.9$ $[+0.6,+3.1]$	27/35
ScreenSpot-Pro	59.6	63.5	61.6	65.6	68.9	$+2.1$ $[+0.8,+3.3]$ 67/34	$+4.0$ $[+3.1,+4.9]$	177/227
ScreenSpot-v2	92.5	91.9	92.2	92.5	94.0	$0.0$ $[-0.6,+0.6]$ 8/8	$+0.3$ $[-0.2,+0.8]$	27/46
Qwen3-VL-32B ∗

Appendix

Table 8: Chain arbitration before re-grounding, CSG and abstention. Random: either chain with probability

\frac{1}{2}

(expected). Oracle: either chain is right. Underlined: the better chain, chosen after the fact. Differences:

95\%

CI and, against the better chain, samples won/lost. Pick: verifier picks the right chain / samples with exactly one chain right (verifier called). ∗ FP8 weights.

	Second chain	Chain	+ arbitration	AdaGround variant	$\Delta$ vs. AdaGround
Qwen3-VL-8B
OSWorld-G	$c_{\mathrm{TNZ}}$ (AdaGround)	72.7	73.6	76.1	–
	fixed $540{\times}540$ crop, $2.78\times$	71.1 ( $-1.6$ )	72.7 ( $-0.9$ $[-2.9,+1.2]$ )	75.7	$-0.4$ $[-2.3,+1.5]$
ScreenSpot-Pro	$c_{\mathrm{TNZ}}$ (AdaGround)	59.6	65.6	66.2	–
	fixed $540{\times}540$ crop, $2.78\times$	62.4 ( $+2.7$ )	65.7 ( $+0.1$ $[-1.1,+1.3]$ )	66.1	$-0.1$ $[-1.3,+1.1]$
ScreenSpot-v2	$c_{\mathrm{TNZ}}$ (AdaGround)	92.5	92.5	91.9	–

Appendix

Table 9: Fixed-geometry controls . Each control replaces

c_{\mathrm{TNZ}}

with another second chain and leaves everything else unchanged. Values are accuracy (%). In parentheses: the paired difference in points to the same column of the

c_{\mathrm{TNZ}}

(AdaGround) row, with

95\%

CI for arbitration.

2.78\times

: linear scale. h Held out and pooled over its instructions (Qwen3-VL-8B only). ∗ FP8 weights.

	OSWorld-G				ScreenSpot-Pro				ScreenSpot-v2
Backbone	Single	Fixed $4\times$	MVP	AdaGround	Single	Fixed $4\times$	MVP	AdaGround	Single	Fixed $4\times$	AdaGround
Qwen3-VL-8B	64.7	70.7	69.9/72.7 †	76.1	51.3	63.5	61.9/65.3 †	66.2	92.1	91.9	91.9
Qwen3-VL-32B ∗	71.8	77.1	72.0 †	79.3	55.3	73.9	74.1 ‡ /74.0 †	75.9	94.7	95.4	95.4

Appendix

Table 10: Overall accuracy (%) on the full benchmarks , with one configuration of AdaGround for both backbones and all benchmarks. Best value per backbone and benchmark in bold . MVP ( Zhang et al., 2026d ) values are our reproduction with its official code when unmarked, the authors’ released per-sample predictions when marked ‡ , and published results when marked † . ∗ FP8 weights.

OSWorld-G	Text	Element	Layout	Fine-gr.	Refusal	Overall
Qwen3-VL-8B single pass	78.2	68.5	71.9	53.7	0.0	64.7
+ fixed $4\times$ crop ( $c_{4\times}$ alone)	80.1	77.3	78.7	58.4	0.0	70.7 +6.0
+ MVP (official code ‡ , reproduced)	78.2	76.4	79.4	56.4	0.0	69.9 +4.8
+ AdaGround	85.1	80.0	81.8	65.1	27.8	76.1 +11.3
Qwen3-VL-32B ∗ single pass	81.6	75.8	77.5	63.1	0.0	71.8
+ fixed $4\times$ crop ( $c_{4\times}$ alone)	82.0	84.2	86.6	63.8	0.0	77.1 +5.3

Appendix

Table 11: Reproduced baselines , accuracy (%) per category, which the main tables omit. The fixed

4\times

crop is AdaGround’s chain

c_{4\times}

alone (an ablation, Section 4.3 ). MVP ( Zhang et al., 2026d ) is reproduced with its official code on Qwen3-VL-8B, or scored from the per-sample predictions released by its authors for Qwen3-VL-32B ScreenSpot-Pro. ‡ On OSWorld-G, MVP runs its ScreenSpot-Pro pipeline (Appendix D.1 ). Single pass and AdaGround as in Tables 1 , 2 and 3 . Subscripts : gain over the method’s own single pass. MVP’s single pass scores

65.1

and

51.5

in our reproduction, and the released predictions include none. UI-Vision Avg.: mean of the three tasks. ∗ FP8 weights.

	MVP ( Zhang et al., 2026d )	AdaGround (ours)
Target scale	not modelled; fixed-size attention crops	estimated from the model’s own box; $k$ and $d$ solved per sample (TNZ)
View selection	attention maps from a chosen layer	size estimate from the model’s own box output
Aggregation	spatial clustering of 3 predictions, largest group	verifier arbitration over a static chain portfolio; verification-gated acceptance
Failure handling	none	negative-feedback re-grounding; existence-gated abstention; compositional cell grounding
Model access	attention maps (white box)	black box: text and first-token probabilities
Decoding	greedy	greedy

Appendix

Table 12: AdaGround vs. MVP : design differences.

Benchmark	vs. single	vs. fixed $4\times$	vs. fixed $4\times^{a}$	vs. fixed $4\times^{b}$	vs. MVP	vs. MVP a	vs. MVP b
Qwen3-VL-8B
OSWorld-G	$+11.3$ $[+8.3,+14.4]$	$+5.3$ $[+2.3,+8.5]$	$+2.8$ $[+0.4,+5.4]$	$+2.3$ $[0.0,+4.8]$	$+6.2$ $[+3.3,+9.1]$	$+3.7$ $[+1.3,+6.1]$	$+3.0$ $[+0.6,+5.4]$
	71/7	47/17	32/16	32/19	53/18	38/17	38/21
ScreenSpot-Pro	$+14.9$ $[+12.8,+16.9]$	$+2.7$ $[+1.3,+4.0]$	–	–	$+4.2$ $[+2.6,+5.9]$	–	–
	276/41	81/39			125/58
ScreenSpot-v2	$-0.2$ $[-1.5,+1.0]$	$0.0$ $[-1.0,+1.0]$	–	–	–	–	–

Appendix

Table 13: Paired differences (points) of AdaGround on the full benchmarks, with

95\%

confidence intervals (Appendix D.2 ) and, below, samples won/lost. MVP: our reproduction (Qwen3-VL-8B) or the authors’ released predictions (Qwen3-VL-32B ScreenSpot-Pro). UI-Vision: held out, pooled over its instructions. a AdaGround without abstention. b The baseline with the abstention rule of Eq. ( 8 ) applied to its own answer (OSWorld-G). ∗ FP8 weights.

Figure 4: Accuracy versus token span per benchmark (bins with

\geq 30

views) and pooled (dashed), on views that contain the target (Table 14 ). Whiskers and bands:

95\%

screenshot-bootstrap intervals. Dotted line: TNZ’s target span

\tau=3

	ScreenSpot-Pro		OSWorld-G		ScreenSpot-v2		Pooled
$\sigma$ (tokens)	$n$	Acc.	$n$	Acc.	$n$	Acc.	$n$	Acc. [95% CI]
$0$ – $0.5$	342	24.0	366	42.6	0	–	708	33.6 [26.7, 40.4]
$0.5$ – $1$	4180	55.4	1109	78.5	185	88.1	5474	61.2 [58.9, 63.6]
$1$ – $1.5$	1396	63.1	532	78.2	743	90.3	2671	73.7 [70.6, 76.6]
$1.5$ – $2$	813	66.9	173	91.9	615	89.3	1601	78.2 [75.0, 81.3]
$2$ – $3$	317	75.4	174	87.4	1422	92.6	1913	89.3 [86.8, 91.6]

Appendix

Table 14: Accuracy versus token span (Qwen3-VL-8B, with the nine baseline passes of Appendix E ), on views that contain the target. Top: accuracy by the token span

\sigma

in the view the model saw, per benchmark and pooled (

n

= views). Pooled cells carry

95\%

screenshot-bootstrap intervals, with all views of a screenshot resampled together. Bottom: the same sample on its coarse view and on the crop of the same baseline, with mean span, accuracy before and after, and the paired change.

Figure 5: Within-sample change from the crop pass (Table 14 , bottom), by span before

\to

after (green gain, red loss). Above each bar: accuracy before the crop and

n

. Whiskers:

95\%

paired screenshot-bootstrap intervals.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

Mar 18, 2026

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment.

Siqi Pei, Liang Tang, Tiaonan Duan +9

Lenovo Research, Beijing, China · Tsinghua University, Beijing, China

GUI Grounding GUI Automation

Ask PDF

Jun 29, 2026

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

MLLM-based GUI grounding methods commonly formulate target localization as autoregressive coordinate generation, enabling models to leverage the strong instruction-following and semantic understanding capabilities of MLLMs. However, this formulation requires the model to retain region-level target evidence while decoding coordinate tokens with the spatial precision demanded by GUI clicking. Our diagnostic analysis reveals that target-region awareness emerges in intermediate decoder layers but is neither retained nor translated into the final coordinate prediction. Existing ZoomIn-style methods address this issue through an external crop-and-rerun pass, which improves localization but increases end-to-end latency and computational cost. To retain the accuracy benefits of two-pass zooming without this extra cost, we propose InnerZoom, a single-forward framework for cross-layer evidence bridging. InnerZoom transforms target-related cues from the original forward pass into a compact cross-layer evidence state, then preserves, refines, and reinjects this state throughout later decoding layers to guide coordinate prediction. Extensive experimental results suggest that InnerZoom-4B achieves state-of-the-art performance on all six GUI grounding benchmarks, obtaining 64.7 on OSWorld-G, 40.2 on UI-Vision, 73.1 on OSWorld-GR, and 87.6 on MMBench-GUI, surpassing the previous best results by 4.1, 3.2, 2.9, and 2.3 points, respectively. Under a controlled 4B setting, InnerZoom improves the same SFT+RL baseline by 5.3 points on average and outperforms two-pass ZoomIn by 1.3 points on average, while reducing end-to-end latency by up to 31.8% and TFLOPs by about 29%. Code and models will be publicly available.

Chen Liu, Ling Chen, Hanzhang Zhou +5

University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University

Efficient Multimodal Inference GUI Grounding Large Vision-Language Models Feature Fusion

Ask PDF

Apr 15, 2026

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose UI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4%, +10.3%, and +4.2% respectively, with no additional training required.

Fei Tang, Bofan Chen, Zhengxi Lu +8

Zhejiang University · Ant Group

GUI Grounding Vision-Language Grounding

Ask PDF

Abstract

GUI Grounding

Explore similar work

AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement

One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding

UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding