How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

Yunxiang Li; Xixin Wu; Helen Meng

cs.AISep 21, 2026

How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

Authors: Yunxiang Li, Xixin Wu, Helen Meng

Organizations: The Chinese University of Hong Kong, Hong Kong SAR, China

Abstract

GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.

Figures & tables

Figure 1: Place-value asymmetry on two real EvoCUA- $8$ B predictions (Stata, ScreenSpot-Pro). (a) A correct click whose entropy lives in the units digit ( $H{=}2.29$ , near-uniform over ten candidates $1$ pixel apart) while the hundreds digit is locked ( $H{=}0.19$ ). (b) A wrong click whose entropy lives in the hundreds digit ( $H{=}2.06$ , five candidates $100$ pixels apart) while the units digit is locked ( $H{=}0.27$ ). Both cases share P(True) ${=}0.98$ . Coordinate-Entropy averages uniformly and ranks the correct click as more uncertain ( $1.41{>}0.92$ ). PACE weights by place value and recovers the correct ranking ( $0.35{<}1.90$ ).

	EvoCUA- $8$ B		MAI-UI- $8$ B		UGround-V $1$ - $7$ B
Method	AUROC $\uparrow$	AUARC $\uparrow$	AUROC $\uparrow$	AUARC $\uparrow$	AUROC $\uparrow$	AUARC $\uparrow$	Cost
Verbalized signals
Self-Refine	0.547	0.407	0.513	0.495	0.404	0.111	2 $\times$
Just Ask $1{-}10$	0.395	0.337	0.423	0.489	0.463	0.143	1 $\times$
P(True)	0.435	0.365	0.760	0.731	0.736	0.293	2 $\times$
Full-sequence signals:

Table 1: Confidence estimation on ScreenSpot-Pro across three primary agents. Bold and underlining mark the best and second-best values in each column. All methods, including sampling-based methods, score the same greedy-decoded click. Paired results on ScreenSpot-v

2

are reported in Appendix.

	ScreenSpot-Pro			ScreenSpot-v $2$
Model	Acc	Conf	Gap	Acc	Conf	Gap
MAI-UI- $8$ B	$0.494$	$0.939$	$0.445$	$0.945$	$0.952$	$0.007$
EvoCUA- $8$ B	$0.352$	$0.833$	$0.481$	$0.910$	$0.875$	$-0.035$

Table 2: Response-level token confidence and click accuracy. Conf is the logit-mean, computed as the per-token top-

1

softmax probability averaged across the response; Gap

{=}

Conf

{-}

Acc.

	ScreenSpot-Pro		ScreenSpot-v $2$
Model	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$
UI-TARS- $7$ B-DPO	$0.777$	$\mathbf{0.785}$	$0.654$	$\mathbf{0.762}$
MAI-UI- $8$ B	$0.759$	$\mathbf{0.790}$	$0.580$	$\mathbf{0.596}$
UI-TARS- $7$ B-SFT	$0.807$	$\mathbf{0.822}$	$0.678$	$\mathbf{0.770}$
UI-TARS- $72$ B-DPO	$0.808$	$\mathbf{0.833}$	$0.637$	$\mathbf{0.738}$
Aguvis- $7$ B- $720$ P	$0.836$	$\mathbf{0.837}$	$0.699$	$\mathbf{0.728}$

Table 3: Cross-agent comparison of PACE α=1 and Coordinate-Entropy.

Figure 2: Per-place informativeness justifies place-weighting. (a) Single-digit AUROC at each digit place; family mean rises monotonically units to hundreds. (b) Per-place entropy gap between wrong and correct predictions tracks the same gradient.

Scheme	Normalized weights $(v_{h},v_{t},v_{u})$	Mean AUROC
Hundreds only	$(1.000,0.000,0.000)$	$0.847$
LR-learned (pooled, $\ell_{1}$ )	$(0.710,0.280,0.010)$	$\mathbf{0.851}$
PACE α=1	$(0.901,0.090,0.009)$	$\mathbf{0.851}$

Table 4: Hundreds-only and learned weighting compared with PACE α=1 . LR weights are learned on pooled data with

\ell_{1}

regularization and

5

-fold cross-validation.

Digit	Standalone AUROC
Hundreds	$.746$
Tens	$\mathbf{.812}$
Units	$.801$

Table 5: Standalone digit discrimination within the same hundreds cell. Results are for EvoCUA-

8

B on ScreenSpot-Pro examples where the predicted click and target center share the same hundreds cell (

n{=}619

Figure 3: Confidence distributions on EvoCUA-

8

B ScreenSpot-Pro. PACE α=1 shows the clearest separation between correct (green) and incorrect (red) clicks. Full-sequence probability provides partial separation, while verbalized P(True) produces substantially overlapping distributions. AUROC values are shown within each panel.

	5%		10%		20%		50%
Agent	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$
EvoCUA- $8$ B	$97.7$	$97.7$	$97.7$	$97.7$	$93.8$	$93.2$	$80.3$	$\mathbf{81.0}$
MAI-UI- $8$ B	$78.0$	$\mathbf{90.0}$	$81.0$	$\mathbf{88.0}$	$81.0$	$\mathbf{88.5}$	$80.0$	$\mathbf{82.8}$
UGround-V $1$ - $7$ B	$87.1$	$\mathbf{91.9}$	$83.7$	$82.1$	$71.5$	$71.5$	$44.7$	$\mathbf{46.7}$
UI-TARS- $7$ B-SFT	$32.4$	$\mathbf{46.0}$	$50.7$	$\mathbf{56.2}$	$48.6$	$\mathbf{50.0}$	$34.4$	$\mathbf{35.3}$
UI-TARS- $7$ B-DPO	$23.8$	$\mathbf{40.5}$	$51.8$	$\mathbf{55.4}$	$63.5$	$63.5$	$49.5$	$\mathbf{49.8}$

Table 6: Selective accuracy at fixed coverage levels. PACE α=1 improves the five-agent mean over Coordinate-Entropy (

\alpha{=}0

) at every reported level. Bold marks individual settings in which PACE obtains higher selective accuracy; ties and lower PACE values are retained.

Benchmark	Abstention rate	PACE	CE
ScreenSpot-Pro	$30\%$	$\mathbf{56.1\%}$	$55.9\%$
ScreenSpot-Pro	$50\%$	$\mathbf{78.6\%}$	$77.9\%$
ScreenSpot-v $2$	$10\%$	$\mathbf{36.7\%}$	$29.1\%$
ScreenSpot-v $2$	$30\%$	$\mathbf{72.2\%}$	$59.5\%$

Table 7: Wrong-click prevention for EvoCUA-

8

B. Values report the percentage of incorrect clicks not executed at each abstention rate. PACE denotes PACE α=1 , and CE denotes Coordinate-Entropy.

Selection rule	Accuracy
PACE α=1 routing	$\mathbf{.717}$
Always-best single agent	$.711$
Coordinate-Entropy routing	$.693$

Table 8: Cross-agent routing on ScreenSpot-Pro. Each method selects one click from a common six-agent pool of

512

examples. PACE improves over Coordinate-Entropy routing, while its difference from the strongest fixed agent is small.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Breakdown	Total	Citation	License
ScreenSpot-Pro	$23$ application categories across $3$ operating systems, covering professional desktop interfaces	$1{,}581$	Li et al. (2025)	MIT
ScreenSpot-v $2$	$8$ platforms covering consumer mobile, web, and desktop interfaces	$1{,}272$	Wu et al. (2024)	Apache $2.0$

Appendix

Table 9: Evaluation benchmarks. Both datasets contain an instruction, a screenshot, and a target bounding box. A predicted click is correct if it falls inside the target box. The primary click is greedily decoded; additional stochastic generations are used only by sampling-based confidence baselines.

Signal family	Decoding	Confidence computation
PACE, CE, and full-sequence	Greedy, $T{=}0$	Token logits from one decode
SafeGround	$K{=}10$ , $T{=}1.0$	Uncertainty from sampled clicks
Verbalized	Greedy meta-response	Confidence extracted from the meta-prompt response

Appendix

Table 10: Inference configuration by confidence-signal family. Sampling and verbalized methods estimate confidence for the same greedily decoded primary click used by the single-pass methods. Models are evaluated in bfloat16 on NVIDIA H

100

or RTX

3090

GPUs.

Figure 5: Sensitivity to the place-weight exponent. AUROC across three fixed-scale agents as

\alpha

varies from

0

2

, with

95\%

paired-bootstrap confidence intervals.

Agent	ScreenSpot-Pro	ScreenSpot-v $2$
EvoCUA- $8$ B	$+0.010\;[-0.007,\,+0.026]$	$\mathbf{+0.065\;[+0.018,\,+0.111]}$
MAI-UI- $8$ B	$\mathbf{+0.031\;[+0.009,\,+0.052]}$	$+0.015\;[-0.041,\,+0.070]$
UGround-V $1$ - $7$ B	$\mathbf{+0.025\;[+0.013,\,+0.037]}$	$\mathbf{+0.054\;[+0.027,\,+0.080]}$

Appendix

Table 11: Paired AUROC differences between PACE and Coordinate-Entropy. Each entry reports

\Delta\mathrm{AUROC}=\mathrm{PACE}-\mathrm{CE}

with a

95\%

paired-bootstrap confidence interval. Bold indicates that the interval excludes zero.

	PACE / Coordinate-Entropy AUROC
Agent	Near ( $<50$ )	Intermediate ( $50$ – $150$ )	Far ( $\geq 150$ )
EvoCUA- $8$ B	$0.789/\mathbf{0.832}$	$\mathbf{0.872}/0.854$	$\mathbf{0.920}/0.858$
MAI-UI- $8$ B	$\mathbf{0.706}/0.694$	$\mathbf{0.886}/0.865$	$\mathbf{0.876}/0.811$
UGround-V $1$ - $7$ B	$0.779/\mathbf{0.802}$	$0.892/0.892$	$\mathbf{0.950}/0.900$

Appendix

Table 12: Discrimination across localization-error distances on ScreenSpot-Pro. Incorrect clicks are grouped by their pixel-distance error, and each group is compared against all correct clicks. Bold marks the higher AUROC within each pair.

	ScreenSpot-Pro		ScreenSpot-v $2$
Agent	CE	PACE	CE	PACE
EvoCUA- $8$ B	$0.835$	$\mathbf{0.841}$	$0.214$	$\mathbf{0.309}$
MAI-UI- $8$ B	$0.722$	$\mathbf{0.733}$	$0.078$	$\mathbf{0.107}$
UGround-V $1$ - $7$ B	$0.952$	$\mathbf{0.963}$	$0.416$	$\mathbf{0.502}$

Appendix

Table 13: AUPRC with incorrect clicks as the positive class. PACE and Coordinate-Entropy are evaluated on the same coordinate-output subset within each cell. Because AUPRC depends on error prevalence, values should be compared within, rather than across, agent–benchmark cells.

Weighting scheme	Relative weights	$\Delta$ AUROC [95% CI]
Coordinate-Entropy	$1{:}1{:}1{:}1$	$0.000$
Literal four-place PACE	$1000{:}100{:}10{:}1$	$+0.020\;[+0.001,\,+0.039]$
Format-aware PACE	$0{:}100{:}10{:}1$	$\mathbf{+0.061\;[+0.048,\,+0.076]}$

Appendix

Table 14: Four-digit coordinate weighting for UI-TARS-1.5 on AndroidControl. Results use

4{,}916

click steps collected on a fixed

924{\times}2100

screen. AUROC differences are measured relative to uniform Coordinate-Entropy. The format-aware variant omits the geometry-constrained thousands place.

Emission format	Example	Observed behavior	Recommended signal
Fixed-scale, three-digit integer	UI-TARS- $72$ B-DPO	PACE improves over Coordinate-Entropy on both benchmarks.	PACE with standard place-value weights.
Fixed-screen, four-digit integer	UI-TARS-1.5 on AndroidControl	Literal four-place weighting gives $+0.020$ AUROC; omitting the geometry-pinned leading digit gives $+0.061$ .	PACE with a format-aligned digit window.
Variable-resolution pixel coordinates	Qwen2.5-VL- $32$ B	Fixed place weighting gives $-0.012/-0.025$ AUROC relative to Coordinate-Entropy.	Coordinate-Entropy unless a stable geometric mapping is available.
Short coordinate emission	Magma- $8$ B	PACE approximately matches Coordinate-Entropy ( $-0.005/-0.010$ ).	Coordinate-Entropy.
Fractional coordinates in $[0,1]$	Aguvis- $7$ B	PACE remains close to Coordinate-Entropy ( $-0.006/+0.018$ ).	Coordinate-Entropy unless the informative precision is known.
Merged coordinate tokens	CogAgent- $9$ B	Individual digit positions cannot be identified because multiple digits form one token.	Coordinate-token entropy without digit-level weighting.

Appendix

Table 15: Applicability across coordinate-emission formats. Slash-separated differences report ScreenSpot-Pro and ScreenSpot-v

2

, respectively. PACE is used when digit positions have a stable geometric meaning; otherwise the method falls back to uniform Coordinate-Entropy.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

Jun 24, 2026

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.

Divake Kumar, Sina Tayebati, Devashri Naik +5

University of Illinois Chicago · Intel Labs · Capital One AI Labs

Computer-Use Agent Benchmarks Uncertainty Quantification Computer-Use Agents GUI Grounding Conformal Prediction AI Agent Benchmarks+1

Ask PDF

Oct 12, 2026

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot. Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations. Region-based patch scoring alleviates this ambiguity, but its spatial decisions remain tied to visual-token footprints that may span the target, neighboring controls, and background. To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction. Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule. This formulation captures the spatial extent of valid click locations and enables finer localization beyond the visual-token grid, making it well suited to small, densely arranged controls without requiring a verifier or zoom-in pass. Experiments across five backbone configurations from two model families demonstrate the effectiveness of Seg2Click. Seg2Click-9B achieves 92.06%, 94.10%, and 55.22% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding. Code will be available at https://github.com/jcwang0602/Seg2Click.

Jingchao Wang, Dingjiang Huang

School of Data Science and Engineering East China Normal University, Shanghai 200062, China

GUI Grounding GUI Agents

Ask PDF

Aug 4, 2026

MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models

Recent GUI visual grounding models generate screen coordinates as digit-token sequences that are parsed into numerical values and mapped to executable clicks. This generation-to-execution interface creates an attack surface that existing objectives over visual representations or coordinate-token sequences do not explicitly model. Although each coordinate digit is predicted as a token, its spatial effect after parsing depends on decimal position: changing a hundreds-place digit by one shifts the coordinate by 100 units, whereas the same change at the ones place shifts it by one. This mismatch motivates attack objectives that account for both numerical coordinate structure and click execution. Moreover, untargeted and targeted attacks require different objectives because they aim to move the click outside the correct region and into an attacker-specified region, respectively. We propose MissClick, an execution-aware white-box attack that aligns optimization with click-level success conditions. MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted redirection. On OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07% and 72.93% (+16.62 and +30.72 pp), while MissClick-T achieves targeted success rates of 44.86% and 62.67% (+31.73 and +47.06 pp). Among the evaluated objectives, soft-coordinate displacement performs best for untargeted attacks, whereas place-weighted target-digit optimization performs best for targeted attacks, supporting goal-specific execution-aware objective design.

Yu Ran, Wentao Zhao

National University of Defense Technology

Adversarial Attacks on VLMs Adversarial Attacks GUI Grounding

Ask PDF

Abstract

GUI Agents

Explore similar work

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models