How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

Yunxiang Li; Xixin Wu; Helen Meng

cs.AISep 21, 2026

How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation

Authors: Yunxiang Li, Xixin Wu, Helen Meng

Organizations: The Chinese University of Hong Kong, Hong Kong SAR, China

Abstract

GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.

Figures & tables

Figure 1: Place-value asymmetry on two real EvoCUA- $8$ B predictions (Stata, ScreenSpot-Pro). (a) A correct click whose entropy lives in the units digit ( $H{=}2.29$ , near-uniform over ten candidates $1$ pixel apart) while the hundreds digit is locked ( $H{=}0.19$ ). (b) A wrong click whose entropy lives in the hundreds digit ( $H{=}2.06$ , five candidates $100$ pixels apart) while the units digit is locked ( $H{=}0.27$ ). Both cases share P(True) ${=}0.98$ . Coordinate-Entropy averages uniformly and ranks the correct click as more uncertain ( $1.41{>}0.92$ ). PACE weights by place value and recovers the correct ranking ( $0.35{<}1.90$ ).

	EvoCUA- $8$ B		MAI-UI- $8$ B		UGround-V $1$ - $7$ B
Method	AUROC $\uparrow$	AUARC $\uparrow$	AUROC $\uparrow$	AUARC $\uparrow$	AUROC $\uparrow$	AUARC $\uparrow$	Cost
Verbalized signals
Self-Refine	0.547	0.407	0.513	0.495	0.404	0.111	2 $\times$
Just Ask $1{-}10$	0.395	0.337	0.423	0.489	0.463	0.143	1 $\times$
P(True)	0.435	0.365	0.760	0.731	0.736	0.293	2 $\times$
Full-sequence signals:

Table 1: Confidence estimation on ScreenSpot-Pro across three primary agents. Bold and underlining mark the best and second-best values in each column. All methods, including sampling-based methods, score the same greedy-decoded click. Paired results on ScreenSpot-v

2

are reported in Appendix.

	ScreenSpot-Pro			ScreenSpot-v $2$
Model	Acc	Conf	Gap	Acc	Conf	Gap
MAI-UI- $8$ B	$0.494$	$0.939$	$0.445$	$0.945$	$0.952$	$0.007$
EvoCUA- $8$ B	$0.352$	$0.833$	$0.481$	$0.910$	$0.875$	$-0.035$

Table 2: Response-level token confidence and click accuracy. Conf is the logit-mean, computed as the per-token top-

1

softmax probability averaged across the response; Gap

{=}

Conf

{-}

Acc.

	ScreenSpot-Pro		ScreenSpot-v $2$
Model	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$
UI-TARS- $7$ B-DPO	$0.777$	$\mathbf{0.785}$	$0.654$	$\mathbf{0.762}$
MAI-UI- $8$ B	$0.759$	$\mathbf{0.790}$	$0.580$	$\mathbf{0.596}$
UI-TARS- $7$ B-SFT	$0.807$	$\mathbf{0.822}$	$0.678$	$\mathbf{0.770}$
UI-TARS- $72$ B-DPO	$0.808$	$\mathbf{0.833}$	$0.637$	$\mathbf{0.738}$
Aguvis- $7$ B- $720$ P	$0.836$	$\mathbf{0.837}$	$0.699$	$\mathbf{0.728}$

Table 3: Cross-agent comparison of PACE α=1 and Coordinate-Entropy.

Figure 2: Per-place informativeness justifies place-weighting. (a) Single-digit AUROC at each digit place; family mean rises monotonically units to hundreds. (b) Per-place entropy gap between wrong and correct predictions tracks the same gradient.

Scheme	Normalized weights $(v_{h},v_{t},v_{u})$	Mean AUROC
Hundreds only	$(1.000,0.000,0.000)$	$0.847$
LR-learned (pooled, $\ell_{1}$ )	$(0.710,0.280,0.010)$	$\mathbf{0.851}$
PACE α=1	$(0.901,0.090,0.009)$	$\mathbf{0.851}$

Table 4: Hundreds-only and learned weighting compared with PACE α=1 . LR weights are learned on pooled data with

\ell_{1}

regularization and

5

-fold cross-validation.

Digit	Standalone AUROC
Hundreds	$.746$
Tens	$\mathbf{.812}$
Units	$.801$

Table 5: Standalone digit discrimination within the same hundreds cell. Results are for EvoCUA-

8

B on ScreenSpot-Pro examples where the predicted click and target center share the same hundreds cell (

n{=}619

Figure 3: Confidence distributions on EvoCUA-

8

B ScreenSpot-Pro. PACE α=1 shows the clearest separation between correct (green) and incorrect (red) clicks. Full-sequence probability provides partial separation, while verbalized P(True) produces substantially overlapping distributions. AUROC values are shown within each panel.

	5%		10%		20%		50%
Agent	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$	$\alpha{=}0$	$\alpha{=}1$
EvoCUA- $8$ B	$97.7$	$97.7$	$97.7$	$97.7$	$93.8$	$93.2$	$80.3$	$\mathbf{81.0}$
MAI-UI- $8$ B	$78.0$	$\mathbf{90.0}$	$81.0$	$\mathbf{88.0}$	$81.0$	$\mathbf{88.5}$	$80.0$	$\mathbf{82.8}$
UGround-V $1$ - $7$ B	$87.1$	$\mathbf{91.9}$	$83.7$	$82.1$	$71.5$	$71.5$	$44.7$	$\mathbf{46.7}$
UI-TARS- $7$ B-SFT	$32.4$	$\mathbf{46.0}$	$50.7$	$\mathbf{56.2}$	$48.6$	$\mathbf{50.0}$	$34.4$	$\mathbf{35.3}$
UI-TARS- $7$ B-DPO	$23.8$	$\mathbf{40.5}$	$51.8$	$\mathbf{55.4}$	$63.5$	$63.5$	$49.5$	$\mathbf{49.8}$

Table 6: Selective accuracy at fixed coverage levels. PACE α=1 improves the five-agent mean over Coordinate-Entropy (

\alpha{=}0

) at every reported level. Bold marks individual settings in which PACE obtains higher selective accuracy; ties and lower PACE values are retained.

Benchmark	Abstention rate	PACE	CE
ScreenSpot-Pro	$30\%$	$\mathbf{56.1\%}$	$55.9\%$
ScreenSpot-Pro	$50\%$	$\mathbf{78.6\%}$	$77.9\%$
ScreenSpot-v $2$	$10\%$	$\mathbf{36.7\%}$	$29.1\%$
ScreenSpot-v $2$	$30\%$	$\mathbf{72.2\%}$	$59.5\%$

Table 7: Wrong-click prevention for EvoCUA-

8

B. Values report the percentage of incorrect clicks not executed at each abstention rate. PACE denotes PACE α=1 , and CE denotes Coordinate-Entropy.

Selection rule	Accuracy
PACE α=1 routing	$\mathbf{.717}$
Always-best single agent	$.711$
Coordinate-Entropy routing	$.693$

Table 8: Cross-agent routing on ScreenSpot-Pro. Each method selects one click from a common six-agent pool of

512

examples. PACE improves over Coordinate-Entropy routing, while its difference from the strongest fixed agent is small.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Breakdown	Total	Citation	License
ScreenSpot-Pro	$23$ application categories across $3$ operating systems, covering professional desktop interfaces	$1{,}581$	Li et al. (2025)	MIT
ScreenSpot-v $2$	$8$ platforms covering consumer mobile, web, and desktop interfaces	$1{,}272$	Wu et al. (2024)	Apache $2.0$

Appendix

Table 9: Evaluation benchmarks. Both datasets contain an instruction, a screenshot, and a target bounding box. A predicted click is correct if it falls inside the target box. The primary click is greedily decoded; additional stochastic generations are used only by sampling-based confidence baselines.

Signal family	Decoding	Confidence computation
PACE, CE, and full-sequence	Greedy, $T{=}0$	Token logits from one decode
SafeGround	$K{=}10$ , $T{=}1.0$	Uncertainty from sampled clicks
Verbalized	Greedy meta-response	Confidence extracted from the meta-prompt response

Appendix

Table 10: Inference configuration by confidence-signal family. Sampling and verbalized methods estimate confidence for the same greedily decoded primary click used by the single-pass methods. Models are evaluated in bfloat16 on NVIDIA H

100

or RTX

3090

GPUs.

Figure 5: Sensitivity to the place-weight exponent. AUROC across three fixed-scale agents as

\alpha

varies from

0

2

, with

95\%

paired-bootstrap confidence intervals.

Agent	ScreenSpot-Pro	ScreenSpot-v $2$
EvoCUA- $8$ B	$+0.010\;[-0.007,\,+0.026]$	$\mathbf{+0.065\;[+0.018,\,+0.111]}$
MAI-UI- $8$ B	$\mathbf{+0.031\;[+0.009,\,+0.052]}$	$+0.015\;[-0.041,\,+0.070]$
UGround-V $1$ - $7$ B	$\mathbf{+0.025\;[+0.013,\,+0.037]}$	$\mathbf{+0.054\;[+0.027,\,+0.080]}$

Appendix

Table 11: Paired AUROC differences between PACE and Coordinate-Entropy. Each entry reports

\Delta\mathrm{AUROC}=\mathrm{PACE}-\mathrm{CE}

with a

95\%

paired-bootstrap confidence interval. Bold indicates that the interval excludes zero.

	PACE / Coordinate-Entropy AUROC
Agent	Near ( $<50$ )	Intermediate ( $50$ – $150$ )	Far ( $\geq 150$ )
EvoCUA- $8$ B	$0.789/\mathbf{0.832}$	$\mathbf{0.872}/0.854$	$\mathbf{0.920}/0.858$
MAI-UI- $8$ B	$\mathbf{0.706}/0.694$	$\mathbf{0.886}/0.865$	$\mathbf{0.876}/0.811$
UGround-V $1$ - $7$ B	$0.779/\mathbf{0.802}$	$0.892/0.892$	$\mathbf{0.950}/0.900$

Appendix

Table 12: Discrimination across localization-error distances on ScreenSpot-Pro. Incorrect clicks are grouped by their pixel-distance error, and each group is compared against all correct clicks. Bold marks the higher AUROC within each pair.

	ScreenSpot-Pro		ScreenSpot-v $2$
Agent	CE	PACE	CE	PACE
EvoCUA- $8$ B	$0.835$	$\mathbf{0.841}$	$0.214$	$\mathbf{0.309}$
MAI-UI- $8$ B	$0.722$	$\mathbf{0.733}$	$0.078$	$\mathbf{0.107}$
UGround-V $1$ - $7$ B	$0.952$	$\mathbf{0.963}$	$0.416$	$\mathbf{0.502}$

Appendix

Table 13: AUPRC with incorrect clicks as the positive class. PACE and Coordinate-Entropy are evaluated on the same coordinate-output subset within each cell. Because AUPRC depends on error prevalence, values should be compared within, rather than across, agent–benchmark cells.

Weighting scheme	Relative weights	$\Delta$ AUROC [95% CI]
Coordinate-Entropy	$1{:}1{:}1{:}1$	$0.000$
Literal four-place PACE	$1000{:}100{:}10{:}1$	$+0.020\;[+0.001,\,+0.039]$
Format-aware PACE	$0{:}100{:}10{:}1$	$\mathbf{+0.061\;[+0.048,\,+0.076]}$

Appendix

Table 14: Four-digit coordinate weighting for UI-TARS-1.5 on AndroidControl. Results use

4{,}916

click steps collected on a fixed

924{\times}2100

screen. AUROC differences are measured relative to uniform Coordinate-Entropy. The format-aware variant omits the geometry-constrained thousands place.

Emission format	Example	Observed behavior	Recommended signal
Fixed-scale, three-digit integer	UI-TARS- $72$ B-DPO	PACE improves over Coordinate-Entropy on both benchmarks.	PACE with standard place-value weights.
Fixed-screen, four-digit integer	UI-TARS-1.5 on AndroidControl	Literal four-place weighting gives $+0.020$ AUROC; omitting the geometry-pinned leading digit gives $+0.061$ .	PACE with a format-aligned digit window.
Variable-resolution pixel coordinates	Qwen2.5-VL- $32$ B	Fixed place weighting gives $-0.012/-0.025$ AUROC relative to Coordinate-Entropy.	Coordinate-Entropy unless a stable geometric mapping is available.
Short coordinate emission	Magma- $8$ B	PACE approximately matches Coordinate-Entropy ( $-0.005/-0.010$ ).	Coordinate-Entropy.
Fractional coordinates in $[0,1]$	Aguvis- $7$ B	PACE remains close to Coordinate-Entropy ( $-0.006/+0.018$ ).	Coordinate-Entropy unless the informative precision is known.
Merged coordinate tokens	CogAgent- $9$ B	Individual digit positions cannot be identified because multiple digits form one token.	Coordinate-token entropy without digit-level weighting.

Appendix

Table 15: Applicability across coordinate-emission formats. Slash-separated differences report ScreenSpot-Pro and ScreenSpot-v

2

, respectively. PACE is used when digit positions have a stable geometric meaning; otherwise the method falls back to uniform Coordinate-Entropy.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Jun 24, 2026Divake Kumar, Sina Tayebati, Devashri Naik +5Computer-Use Agent Benchmarks Uncertainty Quantification
University of Illinois Chicago · Intel Labs · Capital One AI Labs
Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
Oct 12, 2026Jingchao Wang, Dingjiang HuangGUI Grounding GUI Agents
School of Data Science and Engineering East China Normal University, Shanghai 200062, China
MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models
Aug 4, 2026Yu Ran, Wentao ZhaoAdversarial Attacks on VLMs Adversarial Attacks
National University of Defense Technology

Jun 24, 2026J/K move · Enter open · S save

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Divake Kumar, Sina Tayebati, Devashri Naik, Amanda Sofie Rios+4

University of Illinois Chicago · Intel Labs · Capital One AI Labs

Computer-Use Agent Benchmarks Uncertainty Quantification Computer-Use Agents GUI Grounding Conformal Prediction AI Agent Benchmarks+1

Open paper AskPDF arXiv

Computer-use agents turn vision-language model (VLM) predictions into executable GUI clicks, so reliable uncertainty estimates are essential for rejection, calibration, miss-severity ranking, and spatial safety regions. Yet evidence on post-hoc uncertainty quantification (UQ) for these agents is fragmented across isolated model and dataset pairs, leaving it unclear whether UQ rankings stay stable when the agent, benchmark, or observable interface changes. We present Argus, a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding: a 27-method open-weight matrix over 4 VLM agents and 4 datasets, plus an 8-method closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. Evaluated methods span logit-based scores, sampling and consistency measures, hidden-state and density estimators (Mahalanobis, SAPLMA), attention-based scores, P(True) and verbalised-confidence prompting, and split-conformal prediction. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods are the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Within-model ranking transfer is strong (Spearman rho up to 0.969), but cross-tier transfer to closed-source vendors averages only +0.08, so closed-source UQ should be reranked on the target rather than extrapolated. Conformal click regions show score-level discrimination is not enough for deployment: locally weighted disks shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage degrades under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts for regime-aware UQ selection in GUI agents.

Abstract

GUI Agents

Explore similar work

Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models