PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Valentino Sacco; Saurabh Pandey; Alexander Florian Jercher; Maximilian Schilling

cs.AIOct 22, 2026

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Authors: Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling

Organizations: Sapienza University of Rome · Warmwind AG

Abstract

Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.

Figures & tables

Figure 1: PointerBench evaluates fine-grained grounding across spreadsheet geometry, character-level text targets, and professional interfaces. Results reveal substantial headroom and pronounced differences across model families.

	Claude			GPT			Grok	Kimi	Gemini	Qwen
	Sonnet 4.6	Fable 5	Opus 5	5.6 Luna	5.6 Terra	5.6 Sol	4.5	K2.5	3.7 Flash	3.8 Max
Sheets	88.8	97.6	98.8	92.0	91.6	97.8	15.2	10.8	4.0	4.8
Text	49.0	69.6	63.2	46.6	47.6	53.4	1.4	3.4	1.2	0.2
Pro	78.8	82.4	81.2	79.4	77.6	82.0	14.0	3.0	4.2	3.4
Avg.	72.2	83.2	81.1	72.7	72.3	77.7	10.2	5.7	3.1	2.8

Table 1: Success rate (%) on the three PointerBench subsets. Best and second-best results are bold and underlined , respectively.

	UI-TARS	GTA1		OpenCUA		UI-Venus			MAI-UI		MVP
	1.5-7B	7B	32B	7B	32B	Ground-7B	1.5-8B	2-9B	2B	8B	8B	32B
Sheets	46.2	60.2	65.0	59.8	66.0	47.8	56.4	64.2	36.6	59.2	60.4	68.4
Text	18.2	22.8	29.6	18.4	19.4	24.6	23.8	25.8	9.8	23.2	22.4	34.0
Pro	57.4	67.4	70.2	63.8	67.4	66.2	72.8	73.6	61.0	71.6	70.8	75.4
Avg.	40.6	50.1	54.9	47.3	50.9	46.2	51.0	54.5	35.8	51.3	51.2	59.3

Table 2: Success rate (%) of specialist GUI grounding configurations on the three PointerBench subsets (full 39-run evaluation in Appendix G ). Best and second-best results are bold and underlined , respectively.

Model	Word	Character	Punctuation	Caret	Interface text	Region
Claude Fable 5	90.2	70.4	78.3	64.7	82.4	56.0
Claude Opus 5	83.9	51.9	39.1	62.6	82.4	50.7
GPT-5.6 Sol	76.8	48.1	56.5	51.9	100.0	30.6
Qwen3-VL-32B + MVP	69.6	18.5	43.5	34.2	76.5	0.0
GTA1-32B	70.5	11.1	39.1	22.5	88.2	0.0
UI-Venus-2-9B	52.7	11.1	26.1	25.7	76.5	0.0

Table 3: PointerBench -Text accuracy (%) by target objective for the top frontier models (top) and representative UI specialist configurations (bottom). Region groups bounding-box targets; other columns score point-coordinate predictions. Best and second-best results are bold and underlined.

Figure 2: PointerBench -Text caret accuracy by placement objective.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Fine-grained targets	#
Cell	reference (82), reference with content (45), content (43)	170
Edge	column resize (43), row resize (32), right cell edge (22), bottom cell edge (19)	116
Relative	relative offset (33), relative row (27)	60
Header	column header (35), row header (24)	59
Color	color-specified cell or region (56)	56
Corner	bottom-left (13), bottom-right (12), top-left (9), top-right (5)	39

Appendix

Table 4: Task composition of PointerBench -Sheets. Counts sum to 500 examples.

Objective	Target type	#
Word	word-level locations	112
Character	individual characters	27
Punctuation	punctuation marks	23
Caret	insertion positions within and around text	187
Region	text spans and structured document fields	134
Interface text	text in document-interface controls	17

Appendix

Table 5: Task composition of PointerBench -Text. Region groups bounding-box targets; the remaining groups require point predictions.

Figure 3: Application-domain coverage of PointerBench -Pro. Each of the 100 applications contributes five examples; numbers in parentheses denote the number of applications in each domain.

Figure 4: Spatial and geometric distributions of PointerBench targets. Top: normalized target-center distributions and their marginals. Bottom: target area as a percentage of the $1024\!\times\!768$ screen; dashed lines mark suite medians.

Figure 5: Normalized target width and height by suite, shown as percentages of the corresponding screen dimension on a logarithmic scale.

Model	Disabled	Low	Medium
Claude Fable 5	–	Yes	Yes
Claude Sonnet 4.6	Yes	Yes	Yes
GPT-5.6 Luna	Yes	Yes	Yes
GPT-5.6 Sol	Yes	Yes	Yes
Grok 4.20	Yes	–	Yes
Qwen3.6 Flash	Yes	–	Yes

Appendix

Table 6: Reasoning settings available for the six models in the controlled reasoning study. A dash denotes an unavailable setting.

Model	Setting	Sheets	Text	Pro	Avg.
Claude Fable 5	Low	95.8	67.4	82.2	81.8
Claude Fable 5	Medium	97.6	69.6	82.4	83.2
GPT-5.6 Sol	Disabled	85.6	40.0	76.8	67.5
GPT-5.6 Sol	Low	96.4	54.6	80.8	77.3
GPT-5.6 Sol	Medium	97.8	53.4	82.0	77.7
GPT-5.6 Luna	Disabled	58.8	27.2	68.8	51.6

Appendix

Table 7: Success rate (%) across available reasoning settings.

Model	$\Delta$ (pp)	Corrected	Regressed	$p$
Claude Sonnet 4.6	$-1.80$	22	49	$0.00182$
GPT-5.6 Luna	$+19.20$	337	49	$5.59\!\times\!10^{-54}$
GPT-5.6 Sol	$+9.80$	199	52	$1.90\!\times\!10^{-21}$

Appendix

Table 8: Paired comparison of low reasoning against reasoning disabled. Corrected denotes an incorrect-to-correct transition; regressed denotes the reverse. Reported

p

-values use two-sided exact McNemar tests.

Model	Cell (170)	Header (59)	Color (56)	Relative (60)	Edge (116)	Corner (39)
Claude Fable 5	100.0	100.0	100.0	98.3	91.4	97.4
Claude Opus 5	100.0	100.0	98.2	98.3	98.3	94.9
GPT-5.6 Sol	98.8	100.0	100.0	96.7	95.7	94.9
GPT-5.6 Luna	95.3	98.3	96.4	95.0	81.0	89.7
Grok 4.5	15.9	33.9	28.6	16.7	2.6	0.0

Appendix

Table 9: PointerBench -Sheets accuracy (%) by target group. Counts are shown in the column headers.

Model	Text (154)	Other controls (156)	Icon (190)
Claude Fable 5	92.9	85.9	71.1
Claude Opus 5	92.2	83.3	70.5
GPT-5.6 Sol	90.9	81.4	75.3
GPT-5.6 Luna	89.0	81.4	70.0
Grok 4.5	20.8	16.0	6.8

Appendix

Table 10: PointerBench -Pro accuracy (%) by target type.

Model	English	Non-English
Claude Fable 5	69.6	69.6
Claude Opus 5	59.6	66.8
GPT-5.6 Sol	53.2	53.6
GPT-5.6 Luna	45.6	47.6
Grok 4.5	1.6	1.2

Appendix

Table 11: PointerBench -Text accuracy (%) by language group. Each group contains 250 examples.

Model	Correct	Clipped	Oversized	Both	Wrong region	Invalid
Claude Fable 5	56.0	6.0	26.1	10.4	1.5	0.0
Claude Opus 5	50.7	12.7	26.1	7.5	3.0	0.0
GPT-5.6 Sol	30.6	39.6	15.7	9.0	3.0	2.2
GPT-5.6 Luna	26.1	41.8	9.7	10.4	8.2	3.7
Grok 4.5	0.7	3.0	3.7	18.7	73.9	0.0

Appendix

Table 12: Outcome distribution (%) on the 134 PointerBench -Text bounding-box targets. Clipped and oversized boxes fail only the coverage or precision requirement, respectively; “both” fails both requirements.

Model	Official	Converted	$\Delta$
Gemini 3.7 Flash	3.1	78.1	+74.9
Qwen3.8 Max	2.8	55.4	+52.6
Qwen3.6 Flash (Medium)	1.5	31.9	+30.3

Appendix

Table 13: Coordinate-frame diagnostic. Official scores use unmodified model outputs; converted scores apply a fixed

0

–

1000

1024\times 768

transformation and are not leaderboard results.

Rank	Provider	Model	Sheets	Text	Pro	Avg.
1	Anthropic	Claude Fable 5	97.6	69.6	82.4	83.2
2	Anthropic	Claude Opus 5	98.8	63.2	81.2	81.1
3	OpenAI	GPT-5.6 Sol	97.8	53.4	82.0	77.7
4	OpenAI	GPT-5.6 Luna	92.0	46.6	79.4	72.7
5	OpenAI	GPT-5.6 Terra	91.6	47.6	77.6	72.3
6	Anthropic	Claude Sonnet 4.6	88.8	49.0	78.8	72.2

Appendix

Table 14: Full PointerBench evaluation. Success rate (%) on the three benchmark suites. Newly evaluated Gemini and Qwen entries use official raw outputs without coordinate-frame post-processing.

Rank	Family	Model	Sheets	Text	Pro	Avg.
1	MVP	Qwen3-VL-32B	68.4	34.0	75.4	59.3
2	GTA1	32B	65.0	29.6	70.2	54.9
3	UI-Venus	2-9B	64.2	25.8	73.6	54.5
4	MAI-UI	8B	59.2	23.2	71.6	51.3
5	MVP	Qwen3-VL-8B	60.4	22.4	70.8	51.2
6	UI-Venus	1.5-8B	56.4	23.8	72.8	51.0

Appendix

Table 15: Specialist GUI grounding results on PointerBench . Success rate (%) on the three suites. Best and second-best results in each column are bold and underlined.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Nov 2, 2025Shijie Zhou, Viet Dac Lai, Hao Tan +4Multimodal Grounding GUI Grounding
University at Buffalo, USA · Adobe Research, USA
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3VLM Evaluation Domain Randomization
1Fig · 2Manifold Research
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI Grounding GUI Agents

Nov 2, 2025J/K move · Enter open · S save

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

Shijie Zhou, Viet Dac Lai, Hao Tan, Jihyung Kil+3

University at Buffalo, USA · Adobe Research, USA

Multimodal Grounding GUI Grounding Large Vision-Language Models Cross-Modal Attention

Open paper AskPDF arXiv

Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .

Abstract

GUI Grounding

Explore similar work

GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding

GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models

ConClick: Element-Wise Contrastive Learning Improves GUI Grounding