ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

Ruotian Zhang; Yuhui Zhang; Junzhe Huang; Yu Xin; Hehe Fan; Xiaoxiao Sun

cs.CVOct 9, 2026

ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

Authors: Ruotian Zhang, Yuhui Zhang, Junzhe Huang, Yu Xin, Hehe Fan, Xiaoxiao Sun

Abstract

GUI grounding, the task of locating the interface element specified by a user instruction, is a fundamental capability of GUI agents. However, existing grounding models remain error-prone, often causing downstream task failures. We systematically evaluate 10 grounding models across five datasets and identify a major error pattern: most incorrect clicks hit non-target interface elements rather than the background, with different models repeatedly selecting the same distractors. Motivated by these findings, we propose ConClick, a contrastive learning framework that teaches models to distinguish the target element from plausible distractors. We construct ConClick-Data, a dataset containing 4,292 challenging target-distractor pairs, and train the model using complementary listwise ranking and pairwise margin objectives. ConClick improves the grounding accuracy of Qwen3-VL-8B from 54.71% to 66.03% on ScreenSpot-Pro, from 83.72% to 86.28% on MMBench-GUI L2, and from 63.33% to 70.20% on OSWorld-G, outperforming prior methods built on the same or similarly sized backbones. We further introduce ReClick, an inference-time strategy that revisits promising regions and compares predictions across competing candidates, yielding an additional 2.96 percentage-point improvement in average accuracy across the three benchmarks. Together, our analysis, dataset, and methods provide both empirical insights and practical tools for building more reliable GUI agents. Data and code are available at https://github.com/zhangruotian/ConClick.

Figures & tables

Figure 1: Characterizing incorrect clicks. (a) GUI grounding predicts a click within the target region given an instruction and a screenshot. (b) Incorrect clicks are categorized by whether they fall on an element or background and inside or outside the target-centered $3w\times 3h$ neighborhood. (c) Mean accuracy and four error rates across 10 models on five datasets, each using all predictions as the denominator. (d) Spatial distribution of incorrect clicks pooled across 10 models on Click100K Val. Offsets are normalized by target width and height; colors show the percentage of all predictions per $0.25\times 0.25$ bin on a logarithmic scale. Red indicates more errors.

Figure 2: Further analysis of incorrect clicks. (a) Observed versus random element-hit proportions across 10 models and five datasets. Points represent models; random click expectations vary with each model’s error set. Box lines and labels show medians. Bottom percentages show dataset-wide mean element coverage. (b) Agreement of at least $k$ models on the same distractor, among examples with at least $k$ incorrect predictions. (c) Element-error examples. Green boxes mark targets, red boxes mark distractors, and orange crosses indicate incorrect element clicks from different models.

Figure 3: Element-wise contrastive training with ConClick . In the example on the left, the target is “Review details” and the distractor is “Pay now.” Learned weights combine spatial attention maps from the shared grounding model into a selection score map for GUI element regions. The listwise ranking loss contrasts the target with candidate distractors, while the pairwise margin loss encourages a higher score for the target than for the annotated distractor.

Figure 4: ReClick at inference time. Given a screenshot and an instruction, ConClick produces an initial click $p_{0}$ and a selection score map. ReClick selects two high-scoring regions (Crop1 and Crop3) and a crop centered on $p_{0}$ (Crop*), then predicts clicks $p_{1},p_{2},p_{3}$ from the enlarged crops. It compares all four candidates using cross-view support and spatial agreement. Cross-view support measures coordinate likelihood in other views containing the candidate, while spatial agreement favors candidates with smaller mean distances to the others. The final click $\hat{p}$ maximizes the equally weighted sum of the two standardized scores.

	ScreenSpot-Pro		OSWorld-G		MMB-GUI L2		Avg.
Method	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$
GUI-Actor-7B ( Wu et al., 2026 )	44.09 $\dagger$	43.83	43.92	42.35	69.59	19.37	52.53	35.18
SE-GUI-7B ( Yuan et al., 2025 )	46.74 $\dagger$	37.51	49.02	42.16	76.74	19.89	57.50	33.19
GUI-G2-7B ( Tang et al., 2026b )	48.39 $\dagger$	33.14	40.59	41.37	79.27	17.84	56.08	30.78
OpenCUA-7B ( Wang et al., 2026b )	49.46 $\dagger$	34.98	60.98	31.96	81.02	16.17	63.82	27.70
UI-TARS-1.5-7B ( Seed, 2025 )	49.65 $\dagger$	33.02	60.78	31.76	75.43	20.67	61.95	28.48

Table 1: Performance of different models. Acc. is point accuracy, and E-err. is the element error rate over all predictions. ConClick -8B and ConClick -8B + ReClick use the same trained checkpoint. For other models, ∗ denotes published accuracies reported in the original publications; all other results are re-evaluated using released weights and implementations.

\dagger

marks ScreenSpot-Pro accuracy within one percentage point of the published result, supporting the fidelity of the reproduction used to compute E-err. Avg. is the mean across the three benchmarks. All values are percentages. Best and second-best results are in blue bold and black bold , respectively.

Figure 5: Spatial errors and attention-score gaps. (a) Incorrect clicks on ScreenSpot-Pro, with target-centered offsets normalized by target width and height. Red indicates more errors per bin as a percentage of all predictions. (b) Standardized target–distractor score gaps on Click100K Val examples where Qwen3-VL-8B fails and ConClick -8B succeeds.

Data		Loss			ScreenSpot-Pro		OSWorld-G		MMB-GUI L2		Avg.
Random	Our	$\mathcal{L}_{\mathrm{coord}}$	$\mathcal{L}_{\mathrm{list}}$	$\mathcal{L}_{\mathrm{pair}}$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$
					54.71	33.84	63.33	29.22	83.72	14.66	67.25	25.91
$\checkmark$		$\checkmark$			59.90	32.64	67.65	26.08	84.47	14.16	70.67	24.29
	$\checkmark$	$\checkmark$			62.43	32.26	68.63	25.49	84.92	13.72	71.99	23.82
	$\checkmark$	$\checkmark$	$\checkmark$		64.26	31.82	70.20	25.10	85.81	13.41	73.42	23.44
	$\checkmark$	$\checkmark$		$\checkmark$	64.20	32.07	69.41	25.29	85.92	13.36	73.18	23.57

Table 2: Ablations of training data and loss components on Qwen3-VL-8B. Colors are normalized within each column, with darker green indicating better performance; bold marks the best results.

Figure 6: Examples, comparisons, and ablations of ReClick . (a) Examples where ReClick corrects initial clicks near and far from the target. Orange crosses mark initial incorrect clicks, and blue circles mark corrected predictions. (b) Accuracy comparison of ZoomClick, MVP, and ReClick across three backbones. (c) Ablations of crop proposal and candidate scoring components using the same ConClick -8B checkpoint. Results in (b,c) are on ScreenSpot-Pro.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Prediction distributions for ten baseline models across five datasets. Each bar reports accuracy and four error rates over all predictions, with a

3w\times 3h

neighborhood.

Figure 8: Neighborhood-scale sensitivity across five datasets (left) and individual-model density ratios at

r=3

(right). Small points represent ten models; diamonds represent pooled errors. The dashed line marks equal near and far density.

Figure 9: Pooled spatial distributions of incorrect clicks across five datasets, extending the main-text heatmap. Colors give the percentage of all predictions per bin on a shared logarithmic scale; red denotes more errors.

Setting	Initial training	Refinement
Learning rate	$3.45\times 10^{-5}$	$1\times 10^{-6}$
Schedule length (updates)	320	30
Warmup updates	10	10
Selected checkpoint (update)	110	10
Selection loss weight $\lambda_{\mathrm{sel}}$	0.1	0.05

Appendix

Table A1: Training settings for ConClick -8B. Both stages use cosine learning-rate decay.

Figure 10: Sensitivity to training and inference parameters. (a) Pairwise margin on Click100K Val. (b) ReClick crop fraction with two score-guided crops and (c) number of score-guided crops at crop fraction 0.6, evaluated on Click100K Val. Labels show accuracy (%); bold marks the highest value in each panel.

InternVL3.5-8B
Data		Training objective	ScreenSpot-Pro	OSWorld-G	MMB-GUI L2	Avg.
Random	Our		Acc. $\uparrow$	Acc. $\uparrow$	Acc. $\uparrow$	Acc. $\uparrow$
$\checkmark$		Coordinate-only	25.62	46.67	71.45	47.91
	$\checkmark$	Coordinate-only	25.81	47.65	72.65	48.70
	$\checkmark$	Full objective	26.69	49.22	72.95	49.62
Qwen3-VL-30B-A3B-Instruct

Appendix

Table A2: Training ablations on additional backbones. All results are accuracy (%). Darker green indicates better performance; bold marks the best results within each backbone.

		ScreenSpot-Pro		OSWorld-G		MMB-GUI L2
Backbone	Method	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$
UI-TARS-1.5-7B	MVP	59.84	32.51	60.78	30.98	76.27	20.17
	ZoomClick	57.24	35.48	57.25	37.06	73.57	21.87
	ReClick	61.23	30.55	65.88	30.59	77.74	19.59
GTA1-7B	MVP	64.39	32.38	69.02	26.47	83.70	15.41
	ZoomClick	59.33	37.38	65.49	28.43	80.75	17.58

Appendix

Table A3: Comparison of inference methods across three backbones and benchmarks. Acc. denotes accuracy and E-err. denotes the element error rate, both in percentages. Within each backbone, blue bold and black bold mark the best and second-best results, respectively.

	ScreenSpot-Pro		OSWorld-G		MMB-GUI L2		Avg.
Variant	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$
ReClick	72.36	25.87	71.76	23.73	87.28	12.10	77.13	20.57
w/o score-guided crops	66.29	30.42	69.80	25.10	86.28	13.05	74.12	22.86
w/o learned crop selection	72.17	25.68	70.59	24.90	87.26	12.27	76.67	20.95
w/o $S_{\mathrm{cross}}$	70.97	26.31	71.57	23.92	87.15	12.19	76.56	20.81
w/o $S_{\mathrm{agree}}$	72.23	26.12	71.18	24.51	87.48	12.08	76.96	20.90

Appendix

Table A4: Component ablations of ReClick . Acc. and E-err. are percentages over all predictions. Avg. is the unweighted mean across the three benchmarks. All variants use the same ConClick -8B checkpoint. Bold marks the best value in each column.

Model	Layers	Heads	Attention scores	Aggregation
ConClick -8B	18–23	All 32	QK	Learned weighted sum
GTA1-7B	12–25	All	Probabilities	Maximum
UI-TARS-1.5-7B	24	25	QK	Single head

Appendix

Table A5: Model-specific attention readouts for ReClick . QK denotes pre-softmax query–key scores. Layer and head indices are zero-based. External readouts use temperature 1. Each configuration is shared across all three evaluation benchmarks.

Method	CAD		Dev		Creative		Scientific		Office		OS		Overall
	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$	Acc. $\uparrow$	E-err. $\downarrow$
GUI-Actor-7B	31.03	52.11	39.13	46.49	43.70	43.99	48.03	46.46	60.87	28.26	44.90	43.37	44.09 $\dagger$	43.83
SE-GUI-7B	41.76	41.38	43.81	42.81	37.83	39.88	54.72	34.65	70.43	22.17	35.20	41.84	46.74 $\dagger$	37.51
GUI-G2-7B	48.28	37.55	46.49	38.13	40.47	36.36	53.54	31.10	68.70	20.00	34.69	32.14	48.39 $\dagger$	33.14
OpenCUA-7B	32.95	39.85	44.82	41.14	45.16	42.23	58.66	29.53	71.74	22.17	47.96	28.57	49.46 $\dagger$	34.98
UI-TARS-1.5-7B	45.98	32.57	46.15	40.47	44.28	36.95	55.12	32.68	69.57	20.00	38.78	31.12	49.65 $\dagger$	33.02

Appendix

Table A6: Performance by domain on ScreenSpot-Pro (%). Acc. is point accuracy; E-err. is the percentage of all predictions in each domain that fall outside the target region but inside a parsed GUI element. Each domain pools Text and Icon targets. Overall pools all examples and matches Table 1 ; all results use the same locally evaluated models and predictions.

\dagger

retains the Table 1 marker on overall accuracy. Blue bold and black bold mark the best and second-best values in each column, respectively; ties receive the same marking.

Method	Windows		macOS		Linux		iOS		Android		Web		Overall
	Basic	Adv.	Basic	Adv.	Basic	Adv.	Basic	Adv.	Basic	Adv.	Basic	Adv.	Basic	Adv.	All
GUI-Actor-7B	61.99	55.51	59.71	59.83	49.74	41.84	90.45	82.73	70.79	79.72	89.68	72.08	71.80	67.40	69.59
SE-GUI-7B	79.34	56.25	75.07	58.67	65.45	44.39	95.22	83.64	94.66	85.35	90.97	71.10	84.89	68.68	76.74
GUI-G2-7B	80.44	56.99	81.45	66.76	68.06	48.47	95.86	82.42	95.79	84.51	92.90	76.95	87.24	71.39	79.27
OpenCUA-7B	90.77	62.13	84.06	65.03	81.68	58.16	95.22	80.00	95.22	79.72	94.19	76.30	90.77	71.39	81.02
UI-TARS-1.5-7B	76.75	56.62	73.62	60.98	70.16	51.53	90.45	80.00	92.42	80.00	85.48	72.40	82.48	68.46	75.43

Appendix

Table A7: Subset accuracy on MMBench-GUI L2 (%). All results use the locally evaluated models and predictions from Table 1 . Each platform is divided into Basic and Advanced (Adv.) instructions. Overall pools examples across all subsets; All matches the corresponding accuracy in Table 1 . Blue bold and black bold mark the best and second-best values in each column, respectively; ties receive the same marking.

GUI Grounding

Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.

See all 54 papers

Apr 27, 2026

GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

Graphical User Interface (GUI) element grounding (precisely locating elements on screenshots based on natural language instructions) is fundamental for agents interacting with GUIs. Deploying this capability directly on resource-constrained devices like mobile phones is increasingly critical for GUI agents requiring low latency. However, this goal faces a significant challenge, as current visual grounding methods typically employ large vision-language model (VLM) (more than 2.5B parameters), making them impractical for on-device execution due to memory and computational constraints. To address this, this paper introduces GoClick, a lightweight GUI element grounding VLM with only 230M parameters that achieves excellent visual grounding accuracy, even on par with significantly larger models. Simply downsizing existing decoder-only VLMs is a straightforward way to design a lightweight model, but our experiments reveal that this approach yields suboptimal results. Instead, we select an encoder-decoder architecture, which outperforms decoder-only alternatives at small parameter scales for GUI grounding tasks. Additionally, the limited capacity of small VLMs encourages us to develop a Progressive Data Refinement pipeline that utilizes task type filtering and data ratio adjustment to extract a high-quality 3.8M-sample core set from a 10.8M raw dataset. Training GoClick using this core set brings notable grounding accuracy gains. Our experiments show that GoClick excels on multiple GUI element grounding benchmarks while maintaining a small size and high inference speed. GoClick also enhances GUI agent performance when integrated into a device-cloud collaboration framework, where GoClick helps cloud-based task planners perform precise element localization and achieve higher success rates. We hope our method serves as a meaningful exploration within the GUI agent community.

Hongxin Li, Yuntao Chen, Zhaoxiang Zhang

University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.

Efficient VLM Inference Training Data Curation GUI Agents GUI Grounding On-Device Inference

Ask PDF

Oct 12, 2026

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

Visual grounding is a central challenge for GUI agents, requiring an instruction to be mapped to a target within a crowded screenshot. Coordinate generation with single-point supervision reduces a spatially extended target to one position, despite the presence of many valid click locations. Region-based patch scoring alleviates this ambiguity, but its spatial decisions remain tied to visual-token footprints that may span the target, neighboring controls, and background. To address these limitations, we propose Seg2Click, which reformulates GUI grounding as executable-region prediction: identifying where a click can validly execute the given instruction. Using bounding-box supervision, Seg2Click predicts an executable-region mask and computes the final click point through a deterministic rule. This formulation captures the spatial extent of valid click locations and enables finer localization beyond the visual-token grid, making it well suited to small, densely arranged controls without requiring a verifier or zoom-in pass. Experiments across five backbone configurations from two model families demonstrate the effectiveness of Seg2Click. Seg2Click-9B achieves 92.06%, 94.10%, and 55.22% on ScreenSpot, ScreenSpot-v2, and ScreenSpot-Pro, respectively, while matched-backbone comparisons show improvements over both coordinate generation and patch-based grounding. Code will be available at https://github.com/jcwang0602/Seg2Click.

Jingchao Wang, Dingjiang Huang

School of Data Science and Engineering East China Normal University, Shanghai 200062, China

GUI Grounding GUI Agents

Ask PDF

Sep 25, 2026

ScreenHaystack: Finding Blind Zones in GUI Grounding

We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each target at a fixed position, ScreenHaystack systematically relocates controlled target icons across high-resolution GUI backgrounds and measures whether models can localize them consistently. Using this benchmark, we find that leading GUI grounding models, including Qwen3-VL, UI-TARS, GTA, and UI-Venus, exhibit blind zones: spatial regions where grounding accuracy drops sharply despite fixed target appearance and instruction. These blind zones transfer to unseen ScreenSpot-Pro examples: targets inside blind zones are consistently harder to ground, with Qwen3-VL-8B dropping by 16.1 percentage points, and controlled relocation shows that moving targets into blind zones decreases accuracy while moving them out improves accuracy. We further show through controlled synthetic experiments that uneven spatial coverage in training data can induce such blind zones. Therefore, we propose a simple strategy, blind-zone-oriented augmentation, which adds supervision in blind zones and improves ScreenSpot-Pro accuracy over both original and randomly augmented Click-100k fine-tuning.

Chenyue Li, Xiaoxiao Sun, Yubo Deng +3

Stanford University

GUI Grounding Data Augmentation

Ask PDF

Abstract

GUI Grounding

Explore similar work

GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction

Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

ScreenHaystack: Finding Blind Zones in GUI Grounding