Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

Qijia Chen; Giulio Jacucci

cs.CLAug 22, 2026

Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

Authors: Qijia Chen, Giulio Jacucci

Organizations: Department of Computer Science University of Helsinki Helsinki, Finland

Abstract

GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.

Figures & tables

Figure 1: Overview of the measurement audit. (A) Text-metadata GUI grounding is treated as same-screen candidate ranking given an instruction. (B) High instruction–element embedding similarity may reflect visible-label overlap rather than semantic grounding. (C) We compare off-the-shelf single-vector encoders against lexical controls and diagnose the results with label-type stratification, target-text dependence perturbation, LCC, and ESH.

Figure 2: Paired

\Delta R@1

of each sentence-embedding retriever versus BM25 on the three corpora. Error bars are

95\%

cluster-bootstrap CIs by episode or annotation_id .

Retriever	$R@1$	$R@5$	$R@10$	MRR
Random	0.022	0.107	0.212	0.089
Jaccard	0.547	0.703	0.741	0.624
BM25	0.557	0.707	0.742	0.632
MiniLM	0.489	0.709	0.768	0.592
MPNet	0.467	0.716	0.783	0.583
BGE-large	0.520	0.724	0.785	0.616

Table 1: Target-ranking metrics on the clean-main subset (

n{=}32{,}646

type	$n$	Rand.	Jacc.	BM25	Mini	MPNet	BGE	E5	Qwen3
text-only	21,539	0.023	0.612	0.624	0.520	0.504	0.563	0.552	0.507
both	2,668	0.016	0.531	0.530	0.548	0.398	0.549	0.484	0.504
cd-only	8,441	0.023	0.386	0.396	0.392	0.397	0.402	0.401	0.407
resource-only	9,669	0.018	0.063	0.064	0.085	0.085	0.099	0.095	0.124
pseudo	2,629	0.009	0.010	0.011	0.008	0.008	0.007	0.006	0.005
none	13,132	0.027	0.013	0.013	0.016	0.011	0.006	0.006	0.020

Table 2:

R@1

by target label-type on all DB rows. Best non-random per row in bold.

	MoTIF ( $n{=}4{,}724$ )				Mind2Web ( $n{=}5{,}941$ )
Retriever	$R@1$	$R@5$	$R@10$	MRR	$R@1$	$R@5$	$R@10$	MRR
Random	0.022	0.131	0.262	0.101	0.005	0.020	0.036	0.022
Jaccard	0.217	0.386	0.483	0.314	0.039	0.115	0.177	0.088
BM25	0.224	0.391	0.503	0.320	0.041	0.125	0.195	0.094
MiniLM	0.216	0.458	0.631	0.342	0.045	0.116	0.165	0.090
MPNet	0.180	0.459	0.633	0.316	0.035	0.091	0.135	0.073

Table 3: Cross-corpus target-ranking metrics on MoTIF and Mind2Web. Bold marks the best non-random value in each metric column.

Corpus	$n$	BM25	BGE	E5	Qwen3	MPNet	Mini	ESH or
			LCC
AC main	32,646	0.557	0.880	0.882	0.860	0.834	0.856	$+0.070$
MoTIF	4,724	0.224	0.845	0.874	0.851	0.825	0.868	$+0.064$
M2W all	5,941	0.041	0.867	0.861	0.845	0.844	0.849	$+0.027$
M2W rich	5,467	0.044	0.855	0.849	0.835	0.839	0.836	$+0.029$

Table 4: Cross-corpus LCC and oracle ESH. BM25 is

R@1

; ESH or is

\textrm{ESH}^{\textrm{oracle}}

(BM25

+

BGE).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	$\Delta R@1$ [95% CI]	$\Delta$ MRR [95% CI]
MiniLM	$+0.068$ [ $+0.063,+0.073$ ]	$+0.040$ [ $+0.037,+0.044$ ]
MPNet	$+0.090$ [ $+0.085,+0.096$ ]	$+0.049$ [ $+0.044,+0.053$ ]
BGE-large	$+0.037$ [ $+0.033,+0.042$ ]	$+0.016$ [ $+0.012,+0.019$ ]
E5-large	$+0.050$ [ $+0.045,+0.055$ ]	$+0.029$ [ $+0.025,+0.032$ ]
Qwen3-Emb.	$+0.076$ [ $+0.071,+0.082$ ]	$+0.055$ [ $+0.051,+0.059$ ]

Appendix

Table 5: Paired cluster-bootstrap deltas of BM25 over each encoder on clean-main (

n{=}32{,}646

;

B{=}2000

by episode_id ). Positive favours BM25; all CIs exclude zero.

Figure 3: Per-cell $R@1$ on AndroidControl (all DB rows, $n{=}58{,}078$ ; six-way label-type partition), MoTIF (all steps, $n{=}22{,}427$ ; four-way partition: text-only, resource-only, pseudo, none), and Mind2Web (all test splits, $n{=}5{,}941$ ; five-way label-type partition).

Method	Orig.	Masked	$\Delta$ [95% CI]
BM25	0.508	0.125	$-0.383$ [ $-0.402,-0.364$ ]
MiniLM	0.487	0.165	$-0.321$ [ $-0.343,-0.300$ ]
MPNet	0.434	0.139	$-0.296$ [ $-0.316,-0.275$ ]
BGE-large	0.503	0.188	$-0.315$ [ $-0.336,-0.294$ ]
E5-large	0.470	0.159	$-0.311$ [ $-0.330,-0.291$ ]
Qwen3-Emb.	0.466	0.251	$-0.216$ [ $-0.237,-0.196$ ]

Appendix

Table 6: Target-text dependence on an equal-allocation stratified AndroidControl clean-main sample (

n{=}3{,}000

;

1{,}000

per label stratum). Negative deltas measure the loss in top-1 recovery when the gold candidate’s exposed label is removed while the query and non-target candidates stay fixed.

Set	Enc.	Full AUC	CV AUC
AC main	MiniLM	0.855	0.855
	MPNet	0.834	0.834
	BGE-large	0.880	0.880
	E5-large	0.882	0.882
	Qwen3-Emb.	0.860	0.860
MoTIF	MiniLM	0.868	0.867

Appendix

Table 7: LCC full-data AUC vs. 5-fold episode-grouped CV AUC. The two diverge by at most

0.013

(Qwen3-Embedding-0.6B, M2W all), confirming that the lexical coupling signal is not an in-sample artefact.

Figure 4: Oracle min-rank (

\textrm{ESH}^{\textrm{oracle}}

, grey) vs. deployable reciprocal-rank fusion (

\textrm{ESH}^{\textrm{real}}

, red)

\Delta R@1

over BM25 on AndroidControl clean-main. Error bars show

B{=}2000

cluster-bootstrap

95\%

CIs for deployable RRF deltas; oracle bars are target-aware upper bounds.

Combiner	$R@1$	$\Delta$ vs BM25	$R@1$	$\Delta$ vs BM25	MRR (RRF)
	Oracle min-rank		True RRF
BM25	0.555	–	0.555	–	0.631
BM25 $+$ Jaccard	0.588	$+0.033$	0.578	$+0.023$ [ $+0.021,+0.025$ ]	0.642
BM25 $+$ BGE	0.626	$+0.071$	0.568	$+0.012$ [ $+0.009,+0.016$ ]	0.640
BM25 $+$ E5	0.615	$+0.060$	0.566	$+0.011$ [ $+0.008,+0.014$ ]	0.638
BM25 $+$ Qwen3	0.617	$+0.062$	0.560	$+0.005$ [ $+0.002,+0.008$ ]	0.630

Appendix

Table 8: Oracle min-rank vs. true RRF on AndroidControl clean-main (

n{=}32{,}646

). True-RRF

\Delta

vs BM25 95% CIs are paired cluster bootstrap by episode_id with

B{=}2000

. “5 emb.” denotes MiniLM, MPNet, BGE-large, E5-large, and Qwen3-Embedding-0.6B; “lex” adds Jaccard to BM25.

Method	Tuning	$R@1$	$R@5$	$R@10$	MRR	$\Delta R@1$ vs BM25 (pp)
BM25	–	0.558	0.694	0.733	0.628	–
MiniLM	–	0.518	0.717	0.775	0.611	$-3.93$ [ $-5.60,-2.44$ ]
BM25+MiniLM RRF	fixed $k{=}60$	0.567	0.702	0.758	0.635	$+0.95$ [ $-0.03,+1.94$ ]
BM25+MiniLM calibrated	val $\alpha{=}0.60$	0.580	0.747	0.794	0.659	$+2.27$ [ $+1.35,+3.23$ ]
Oracle BM25+MiniLM upper	target-aware	0.625	0.779	0.825	0.696	$+6.71$ [ $+5.76,+7.65$ ]
MPNet	–	0.492	0.729	0.789	0.603	$-6.54$ [ $-8.29,-4.80$ ]

Appendix

Table 9: Validation-tuned calibrated score fusion on AndroidControl clean-main test (

n{=}2{,}949

). Scores are z-normalized within each candidate pool;

\alpha

is selected on validation by

R@1

with MRR as a tie-breaker. Oracle rows are target-aware upper bounds, not deployable systems. CIs use paired cluster bootstrap by episode with

B{=}2000

Split	$n$	BM25	$M_{0}$	$M_{\mathrm{ft}}$	BGE 0
Overall	2,949	0.558	0.518	0.694	0.513
IDD	1,536	0.548	0.490	0.703	0.501
app_unseen	36	0.778	0.722	0.833	0.694
task_unseen	1,377	0.562	0.544	0.680	0.523
text-only	1,784	0.607	0.523	0.678	0.534
cd-only	903	0.458	0.485	0.711	0.452

Appendix

Table 10: Full

R@1

breakdown of fine-tuning evaluation on the AndroidControl test clean-main subset.

M_{0}

= zero-shot MiniLM-L6-v2;

M_{\mathrm{ft}}

= same model after one epoch; BGE 0 = zero-shot BGE-large.

Method	$R@1$	LCC	BM25-miss rec.
$M_{0}$	0.518	0.865	0.151
$M_{\mathrm{ft}}$	0.694	0.877	0.374
BGE 0	0.513	0.868	0.156

Appendix

Table 11: Fine-tuned MiniLM coupling audit on the same held-out AndroidControl clean-main test rows (

n{=}2{,}949

). LCC is 5-fold episode-grouped CV AUC using BM25 rank, Jaccard rank, candidate-pool size, and label type. BM25-miss recovery is the fraction of rows with BM25 rank

>1

that the method ranks first.

Method	Orig.	Masked	$\Delta$ [95% CI]
BM25	0.558	0.134	$-0.423$ [ $-0.444,-0.402$ ]
$M_{0}$	0.518	0.178	$-0.340$ [ $-0.363,-0.316$ ]
$M_{\mathrm{ft}}$	0.694	0.151	$-0.542$ [ $-0.562,-0.520$ ]

Appendix

Table 12: Held-out target-text dependence perturbation for the fine-tuning control (

R@1

). The perturbation blanks the gold candidate’s visible text and content description while retaining the query, non-target candidates, resource IDs, class names, and candidate pools. Negative deltas measure sensitivity to target-exposed text.

Model	current	label-only	action-aware
MiniLM	+0.750	+0.767	+0.800
MPNet	+0.684	+0.704	+0.718
BGE-large	+0.749	+0.772	+0.803
E5-large	+0.821	+0.798	+0.814
Qwen3-Emb.	+0.714	+0.727	+0.797

Appendix

Table 13: Step-level M3–Jaccard Spearman (

n{=}32{,}636

). All cells

p<10^{-300}

Stress test	Corpus	$n$	BM25	Method	$\Delta$ pp	Coupling AUC
Qwen3-Emb.	AC main	32,646	0.557	0.481	$-7.64$ [ $-8.22,-7.10$ ]	0.860
Qwen3-Emb.	MoTIF	4,724	0.224	0.167	$-5.72$ [ $-6.97,-4.44$ ]	0.851
Qwen3-Emb.	M2W all	5,941	0.041	0.020	$-2.04$ [ $-2.62,-1.45$ ]	0.845
SPLADE	AC main	32,646	0.557	0.535	$-2.21$ [ $-2.65,-1.77$ ]	0.911
SPLADE	MoTIF	4,724	0.224	0.213	$-1.12$ [ $-2.22,-0.04$ ]	0.893
SPLADE	M2W all	5,941	0.041	0.042	$+0.12$ [ $-0.45,+0.67$ ]	0.888

Appendix

Table 16: Qwen3 paired intervals and sparse-retrieval stress test. Deltas are method minus BM25 at

R@1

; negative deltas mean the method is below BM25. For SPLADE, Coupling AUC is the same hit@1 predictability diagnostic, not the encoder-specific LCC definition.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 57 papers

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI Grounding Multimodal Model Evaluation
Sapienza University of Rome · Warmwind AG
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI Grounding GUI Agents
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3VLM Evaluation Domain Randomization
1Fig · 2Manifold Research

Oct 22, 2026J/K move · Enter open · S save

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher, Maximilian Schilling

Sapienza University of Rome · Warmwind AG

GUI Grounding Multimodal Model Evaluation

Open paper AskPDF arXiv

Multimodal Large Language Models have shown promising performance in computer-use tasks, from planning user workflows to grounding natural-language actions to precise UI targets. Existing end-to-end benchmarks primarily emphasize long-horizon task completion, while GUI grounding benchmarks remain focused on conventional controls and a limited set of professional interfaces. We introduce POINTERBENCH, a diagnostic suite of 1,500 screenshot-instruction pairs probing three complementary capabilities: thin sub-element identification and geometric grounding in spreadsheets (POINTERBENCH-Sheets), character-level grounding in long-text documents (POINTERBENCH-Text), and mixed GUI grounding across 100 professional applications (POINTERBENCH-Pro). We carefully curate each task with pixel-exact target geometry and evaluate both general-purpose multimodal models and 39 specialist GUI grounding configurations. We find text grounding to be the most challenging setting, with Claude Fable 5, GPT-5.6 Sol, and the leading native specialist UI-Venus-2-9B achieving 69.6%, 53.4%, and 25.8% success on POINTERBENCH-Text despite respective 83.2%, 77.7%, and 54.5% average success across POINTERBENCH. These results reveal substantial headroom in fine-grained GUI grounding across frontier and specialist models. We publicly release the benchmark and evaluation code at https://github.com/warmwindOS/pointerbench.

Abstract

GUI Grounding

Explore similar work

PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models