GUI grounding enables web agents to translate natural-language instructions into actionable interface locations. However, dense layouts and small, visually similar controls can lead models toward plausible but incorrect elements, while refinement around a single prediction may leave alternative targets unexplored. We propose the Multi-Hypothesis Verification Framework (MHVF) for training-free GUI grounding. It combines multi-hypothesis exploration with local visual verification using a single pretrained vision-language model. To explore alternative targets, MHVF constructs spatial hypotheses from multiple full-screen predictions and examines the corresponding regions through zoomed re-localization. To resolve ambiguity among the resulting candidates, it verifies their agreement with the instruction in marked local views. Verification scores then guide a second round of local search and rank candidates retained from both search levels for final selection. Experiments with five general-purpose and GUI-specific models across eight benchmark settings demonstrate improvements in grounding accuracy in most settings. MHVF improves UI-Venus-Ground-7B by 4.3 and 2.1 percentage points on Mind2Web and VisualWebBench, respectively, and raises Qwen2.5-VL-7B's accuracy on ScreenSpot-Pro from 23.6% to 53.6%.
Figures & tables
Figure 1. Web grounding with MHVF. (a) On a Mind2Web product-comparison task, UI-Venus-Ground-7B clicks the product image; adding MHVF locates its comparison checkbox. The dashed box marks the target. (b) Grounding accuracy of the same backbone on two web benchmarks; gains are in percentage points. A product-page example compares the UI-Venus-Ground-7B click on the product image with the MHVF click on its comparison checkbox. Two paired-dot plots show grounding accuracy on Mind2Web and VisualWebBench.
Figure 2. Overview of MHVF. The blue boundary encloses the inference procedure, which reuses a shared frozen VLM for hypothesis generation (P), zoomed re-localization (Z), and candidate verification (V). Reopening (R) refines the best-scoring region, and candidates from both levels remain eligible for final selection. The lower panels illustrate verification and local refinement for “add new tab,” where H4 is reopened and H4.1 yields the final click. A frozen VLM is shared by the MHVF inference procedure. The lower search level proposes and verifies global hypotheses. The upper search level refines the best region, while both candidate sets remain available for final selection. Two lower panels illustrate the local verification and refinement steps.
Web
Desktop and cross-platform
Model
Mind2Web
VisualWebBench
SS-Pro
OSW-G
OSW-GR
UI-V
WinDesk
MMB-GUI
General-purpose MLLMs
Qwen2.5-VL-3B
65.9
88.0
25.1
30.7
36.7
13.9
48.4
61.8
+ MHVF
71.2 +5.3
88.2 +0.2
50.0 +24.9
44.7 +14.0
55.1 +18.4
23.4 +9.5
57.3 +8.9
68.4 +6.6
Qwen2.5-VL-7B
69.0
85.7
23.6
33.9
38.7
15.6
50.3
67.6
+ MHVF
73.5 +4.5
87.8 +2.1
53.6 +30.0
52.0 +18.1
62.2 +23.6
31.2 +15.6
60.7 +10.4
76.3 +8.7
Table 1. Grounding accuracy (%) across eight benchmark settings. Rows with + MHVF apply our method to the model above. Bold marks the better result within each pair; superscripts show changes in percentage points. Mind2Web uses element-only instructions.
Model
Cross-task
Cross-website
Cross-domain
Overall
(a) Element only
Qwen2.5-VL-7B
66.1
62.9
71.4
69.0
+ MHVF
71.2
74.6
74.0
73.5
UI-Venus-Ground-7B
75.1
71.7
76.8
75.7
+ MHVF
77.5
80.6
80.6
80.0
(b) Task + element
Table 2. Mind2Web grounding accuracy (%) by test split. (a) Element descriptions only; (b) task instructions plus element descriptions. Bold marks the better result within each model pair. Overall aggregates all instances.
Method
3B
7B
Forward-inference methods
GUI-RC ( Du et al., 2026 )
23.6
24.0
RegionFocus ( Luo et al., 2025 )
–
32.1
VPSG ( Tao et al., 2025 )
13.3
19.1
UI-Zoomer ( Tang et al., 2026a )
–
41.0
ZoomClick ( Jiang et al., 2025 )
–
44.0
Table 3. ScreenSpot-Pro accuracy (%) of existing methods and MHVF, grouped by backbone. Shaded rows show original models and + MHVF; – denotes an unreported configuration.
Mind2Web
ScreenSpot-Pro
Variant
UI-Venus- Ground-7B
UI-Venus- Ground-7B
Qwen2.5- VL-7B
MHVF
80.0
66.6
53.9
Single initial hypothesis
79.2
64.1
47.5
w/o Magnification ( κ=1 )
78.2
61.6
41.4
w/o Reopening
79.7
65.5
50.9
w/o Verification and Reopening
78.9
61.9
45.7
Table 4. Component ablations (grounding accuracy, %). The full method is rerun for these comparisons.
Figure 3. Candidate coverage and final accuracy. Bars show the fraction of samples with a correct candidate at each stage; diamonds mark final accuracy and segments show the remaining selection gap. Grouped bars compare proposal coverage, coverage after level one, and coverage after both levels on Mind2Web and ScreenSpot-Pro. Diamonds mark final accuracy below the candidate-coverage upper bound.
Figure 4. Accuracy by target size; labels show gains in percentage points. Results pool 39,445 sample–model pairs across three benchmarks and five backbones. Grouped bars compare original-model and MHVF accuracy across five target-size bins. Gains are largest for targets smaller than one visual token, while the largest-target bin has a small decrease.
Figure 5. Accuracy versus visual input cost with UI-Venus-Ground-7B. Cost is the mean cumulative visual token estimate per sample, counting each input request once. The grid control and MHVF use comparable mean visual input on ScreenSpot-Pro. Two scatter plots compare the original UI-Venus-Ground-7B, Self-Zoom, Sample and Vote, and MHVF, with an additional fixed-grid crop and verification control on ScreenSpot-Pro. Mind2Web is above ScreenSpot-Pro. At approximately 36,000 mean cumulative visual tokens, the grid control reaches 54.7 percent and MHVF reaches 66.6 percent; a dotted segment marks their 11.9 percentage point accuracy gap.
Figure 6. Parameter sensitivity on ScreenSpot-Pro: (a) magnification κ and (b) the maximum number of first-level hypotheses k . Vertical dashed lines mark the adopted settings, κ=3 and k=4 . Two line plots show ScreenSpot-Pro accuracy versus magnification and the maximum number of first-level hypotheses for UI-Venus-Ground-7B and Qwen2.5-VL-7B.
Figure 7. Web grounding cases with UI-Venus-Ground-7B on Mind2Web (a, c) and VisualWebBench (b, d): successful localization, recovery through reopening, selection failure, and missing candidate coverage. Markers show logged predictions from both levels; dashed boxes mark targets. Screenshots are cropped for display. Four webpage crops show level-1 localization for Select University, recovery of the Posters link at level 2, an incorrect final selection despite a correct candidate for Select Model, and both search levels missing the target link for browse poetry categories.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
Model configuration
Acc. (%) ↑
Forward-inference methods
Iterative Focusing ( Li et al., 2025b )
OS-Atlas-7B
31.0
Iterative Narrowing ( Li et al., 2025b )
OS-Atlas-7B
31.9
ReGround ( Li et al., 2025b )
OS-Atlas-7B
40.2
ScreenSeekeR ( Li et al., 2025b )
OS-Atlas-7B + GPT-4o planner
48.1
ECP ( Lee et al., 2025b )
OS-Atlas-7B
40.4
Appendix
Table 5. Literature-reported ScreenSpot-Pro results (%) for additional backbones and multi-model systems. Each row gives one reported configuration.
College of Computer Science and Software Engineering Shenzhen University Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · Shenzhen University of Advanced Technology Shenzhen, China