UI grounding asks a model to find the screen element that a user instruction refers to. Most current systems use a vision-language model (VLM) that reads a screenshot and outputs pixel coordinates. We study this task on Urdu mobile web pages, a setting with no published UI grounding results. Instead of regressing coordinates, we take the candidate elements from the page DOM and train a 279M-parameter cross-encoder (mDeBERTa-v3-base) to pick one. The click point is the centre of the picked element's box, so it is exact whenever the right element is picked. We built a frozen test set of 304 instructions from six Urdu news sites; 149 of them (49%) are referential and never quote the full target text. Test domains were banned from training data before collection started. The ranker scores 73.4% point-in-box accuracy, against 29.6% for UI-TARS-2B and 23.7% for Qwen2.5-VL-7B, both zero-shot. On the referential slice the ranker scores 65.8%, against 30.2% for a string-matching baseline that needs no training. In a controlled comparison at small scale, with training data drawn from the same pages, a coordinate-regression model of similar size scores 1.6%. Adding element geometry as input tokens gives the best variant, at 75.7%. The ranker's errors concentrate in two places: elements that share identical text, and ordinal instructions such as "the fourth link".
Figures & tables
Generator
Type
Items
Share
Explicit
quotes text
155
51.0%
Ordinal
referential
101
33.2%
Partial
referential
31
10.2%
Section
referential
13
4.3%
Function
referential
4
1.3%
Total
304
100%
Table 1: Frozen test set by instruction generator. Referential items never quote the full target text; partial items quote a three-word prefix.
Variant
Candidate string
Max len
mindact_text
text [tag]
160
full
text [tag] + [ARIA]
224
[PH] [TITLE] [SEC]
full_noparent
full without [SEC]
224
full_geo
full + [BOX x y w h]
224
Table 2: Candidate representations. Attribute fields are included only when present and different from the element text. [SEC] is the parent section heading; [BOX] holds the normalised box coordinates.
System
Overall
Explicit
Referential
Heuristic (no training)
57.9
84.5
30.2
mindact_text ranker
74.0
80.6
67.1
full ranker
73.4
80.6
65.8
full_noparent
72.7
80.6
64.4
full_geo
75.7
86.5
64.4
UI-TARS-2B
29.6
40.0
18.8
Table 3: Point-in-box accuracy (%) on the frozen test set: 304 items, 155 explicit and 149 referential. All VLM rows are zero-shot. Hit counts: heuristic 176, mindact_text 225, full 223, full_noparent 221, full_geo 230, UI-TARS-2B 90, Qwen2.5-VL-7B 72, MAI-UI-2B 36, coordinate control 5. MAI-UI-2B parsed 220 of its 304 outputs; the rest contained no point and count as misses. Ferret-UI Lite has no row: it could not be evaluated, for the reason given in Section 5.
System
Explicit (155)
Ordinal (101)
Partial (31)
Section (13)
Function (4)
Heuristic
84.5
1.0
100
100
0.0
mindact_text
80.6
51.5
100
100
100
full
80.6
49.5
100
100
100
full_noparent
80.6
50.5
100
100
25.0
full_geo
86.5
47.5
100
100
100
UI-TARS-2B
40.0
4.0
48.4
46.2
75.0
Table 4: Accuracy (%) by instruction generator. The function slice has four items, so its percentages move in steps of 25. The ordinal slice is where the systems separate.
Figure 1: Two frozen test items: one the string-matching baseline can solve, one it cannot. The screenshots are cropped to the top half so the gold boxes stay legible; both targets sit in the page header, marked by the red boxes. (a) The instruction quotes the target text (“click ‘Latest”). (b) The instruction (“click the fourth link”) quotes no text, and the count runs over links in DOM order.
System
Params
Device
s/item
mindact_text
279M
CPU, 2-core VM
6.07
full
279M
CPU, 2-core VM
7.44
UI-TARS-2B
2B
GPU, Tesla T4
2.61
Qwen2.5-VL-7B
7B
GPU, Tesla T4
5.68
MAI-UI-2B
2B
GPU, Tesla T4
13.20
Coord. control
500M
GPU, Tesla T4
2.39
Table 5: Median seconds per item. Ranker times are medians over 49 frozen items with a median of 31 candidates per view, fp32, no batching or quantisation. VLM and control times are per screenshot on GPU. The devices differ, so read this table as a cost profile, not a race.
System
Siasat (75)
Dawn (75)
Express (75)
Khabrain (67)
Mashriq (11)
Ummat (1)
Heuristic
42.7
76.0
32.0
83.6
63.6
0.0
mindact_text
85.3
81.3
37.3
95.5
63.6
100
full
85.3
82.7
34.7
95.5
63.6
0.0
full_noparent
85.3
78.7
36.0
95.5
63.6
0.0
full_geo
90.7
82.7
36.0
95.5
81.8
0.0
UI-TARS-2B
29.3
49.3
4.0
34.3
45.5
0.0
Table 6: Accuracy (%) by test site, item counts in parentheses. Mashriq and Ummat are too small to read on their own; the four large sites carry the comparison. Express is the hardest site for every system, including both VLMs.
Graphical User Interface (GUI) grounding is a fundamental perception task for multimodal agents, enabling them to interpret natural language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive Vision-Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking dataset is available, we construct ranking supervision data from existing grounding datasets. A strict containment criterion and boundary-aware positive augmentation improve alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.
Liyang Fan, Xinping Bi, Yitai Li +3
College of Computer Science and Software Engineering Shenzhen University Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · Shenzhen University of Advanced Technology Shenzhen, China
Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.
Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1
GUI grounding requires locating small controls in cluttered screenshots and identifying which control a natural-language instruction refers to. Cropping and enlargement can expose local detail, but the resulting image may still leave ambiguity about the intended target. We propose PreView, an auxiliary preprocessor that improves the inputs to an existing grounding model without further training that model. PreView prepares two complementary types of input features: task-relevant views for localization and candidate-specific cues for target understanding. A trained lightweight auxiliary model predicts and refines a center, crops the screenshot into a local view, and enlarges that view for zoomed-in grounding. It also prepares a visual cue (i.e., a red-circle marker with a location hint) and a textual cue describing the candidate's identity or function. The frozen grounding model uses the prepared inputs to predict the final click. Across five benchmarks, PreView improves the average grounding accuracy of Qwen3-VL-2B, OS-Atlas-4B, and Qwen3-VL-4B without further training these grounding models. Extensive analyses examine the contributions of views and cues, center refinement, and visual input budgets.