GUI Grounding

GUI: Graphical User Interface

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 30

All topics

Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
Oct 5, 2026Aadi Chauhan, Arthur IlyasovGUI Grounding Mobile GUI Agents
Stanford University
Recurrent Latent Visual Search for GUI Grounding
Oct 4, 2026Kaiyu Wu, Beichen Zheng, Weiyao Huang +1GUI Grounding Vision-Language Grounding
Sun Yat-sen University
SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents
Sep 29, 2026Shengtian Yang, Ziyu Xiong, Kaibing Yang +6Reinforcement Learning RL for GUI Agents
Southeast University, Nanjing, China · Kuaishou Technology, Beijing, China · Unaffiliated
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Sep 16, 2026Liyang Fan, Xinping Bi, Yitai Li +3Efficient Multimodal Inference Multimodal Reranking
College of Computer Science and Software Engineering Shenzhen University Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · Shenzhen University of Advanced Technology Shenzhen, China
Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
Sep 14, 2026Yizhou Liu, Fei Tang, Yuchen Yan +8Test-Time Adaptation of VLMs GUI Grounding
Zhejiang University · Ant Group
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
Aug 11, 2026Shiyu Xuan, Zechao LiTest-Time Adaptation On-Policy Self-Distillation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China, 210094
LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
Aug 10, 2026Renshan Zhang, Haoyang Meng, Yixiao He +3GUI Grounding Visually Grounded Reasoning
Harbin Institute of Technology, Shenzhen · Beijing University of Posts and Telecommunications · Shenzhen Loop Area Institute +1
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
Aug 10, 2026Yuke Li, Xuehan HouGUI Grounding Multimodal Large Language Models
School of Electronic and Computer Engineering, Peking University, Shenzhen, China
MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models
Aug 4, 2026Yu Ran, Wentao ZhaoAdversarial Attacks on VLMs Adversarial Attacks
National University of Defense Technology
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
Aug 4, 2026Zichuan Fu, Shirong Wang, Wenlin Zhang +10GUI Grounding Visual Grounding
City University of Hong Kong · Westlake University · Tencent Jarvis Lab
Prior Directions: Why GUI Grounding Gets Locked in the Past
Jul 29, 2026Weile Gong, Zijian Lu, Mingcai Chen +3Vision-Language Models GUI Grounding
Nanjing University of Posts and Telecommunications, Nanjing, China
One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
Jun 29, 2026Chen Liu, Ling Chen, Hanzhang Zhou +5Efficient Multimodal Inference GUI Grounding
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
Jun 29, 2026Sunqi Fan, Lingshan Chen, Runqi Yin +4RL for GUI Agents GUI Agents
Tsinghua University · Tencent Hunyuan
WinDOM: Self-Family Distillation for Small-Model GUI Grounding
Jun 24, 2026Chengheng Li-Chen, Zhiqian Zhou, Hao Chen +1Synthetic Data Generation GUI Grounding
Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Logitech · Universitat Politècnica de Catalunya (UPC), Barcelona, Spain.
Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasets
Jun 24, 2026Divake Kumar, Sina Tayebati, Devashri Naik +5Computer-Use Agent Benchmarks Uncertainty Quantification
University of Illinois Chicago · Intel Labs · Capital One AI Labs
Trust the Right Teacher: Quality-Aware Self-Distillation for GUI Grounding
Jun 16, 2026Jingyuan Huang, Zuming Huang, Yucheng Shi +4On-Policy Self-Distillation GUI Grounding
University of Georgia · 2INFLY Tech · *This work was done during an internship at INFLY Tech. +2
VISTA: View-Consistent Self-Verified Training for GUI Grounding
Jun 12, 2026Xinyu Qiu, Yunzhu Zhang, Heng Jia +3Multi-View Consistency VLM Robustness
Zhejiang University · Venus Team, Ant Group
DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions
Jun 4, 2026Nathan Bout, Maxime Langevin, Ronan RiochetGUI Agents GUI Grounding
1H
GUI-C $^2$ : Coarse-to-Fine GUI Grounding via Difficulty-Aware Reinforcement Learning
May 29, 2026Junlong Li, Chao Hao, Lap-Pui Chau +1Multimodal Grounding Agentic RL
The Hong Kong Polytechnic University
DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding
May 15, 2026Yichao Liu, Huawen Shen, Liu Yu +3GUI Agents GUI Grounding
1Nankai University · Institute of Information Engineering, Chinese Academy of Sciences
WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
May 13, 2026Haoren Zhao, Tianyi Chen, Zhen WangMultimodal Robustness GUI Agents
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · 2Microsoft
What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs
May 10, 2026Jiaping Lin, Fei Shen, Junzhe Li +4GUI Grounding Vision-Language Grounding
1Guangming Laboratory · 2National University of Singapore · 3Peking University +2
BAMI: Training-Free Bias Mitigation in GUI Grounding
May 7, 2026Borui Zhang, Bo Zhang, Bo Wang +6Perturbation-Based Feature Attribution GUI Grounding
Tsinghua University, China · Lenovo Research, China
AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding
May 4, 2026Ruilin Yao, Shegnwu Xiong, Tianyu Zou +2Uncertainty Quantification GUI Grounding
Wuhan University of Technology
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
May 1, 2026Yan Zhang, Daiqing Wu, Huawen Shen +2GUI Grounding On-Policy Distillation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · VCIP & TMCC & DISSec, College of Computer Science, Nankai University
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM Inference Training Data Curation
University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.
Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
Apr 23, 2026Wenkai Wang, Xiyun Li, Hongcan Guo +5Reinforcement Learning GUI Grounding
Zhejiang University · Tencent AI Lab · The University of Hong Kong
GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models
Apr 15, 2026Yangyue Wang, Harshvardhan Sikka, Yash Mathur +3VLM Evaluation Domain Randomization
1Fig · 2Manifold Research
GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
Mar 27, 2026Rui Xie, Zhi Gao, Chenrui Shi +3Multimodal RAG Domain Adaptation
Shanghai Jiao Tong University, Shanghai, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China · Beijing Institute of Technology, Beijing, China
GUI-AIMA: Aligning Intrinsic Multimodal Attention with a Context Anchor for GUI Grounding
Nov 2, 2025Shijie Zhou, Viet Dac Lai, Hao Tan +4Multimodal Grounding GUI Grounding
University at Buffalo, USA · Adobe Research, USA

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?

Aadi Chauhan, Arthur Ilyasov

Stanford University

GUI Grounding Mobile GUI Agents Vision-Language Grounding

Open paper AskPDF arXiv

A GUI agent decides which action to take and where to take it; we ask how a small grounding model should receive the action type. Fine-tuning Qwen2-VL-2B with LoRA on Android in the Wild, we compare a flat baseline with five ways of supplying the type under matched data, compute, and decoding: an auxiliary loss, a hard-routed action word, an additive learned embedding, a prepended learned token, and the type written into the prompt. With five seeds, an episode-clustered bootstrap, and seed-level paired tests, the ranking on a mixed stream is clear: the auxiliary loss, the additive embedding, and the prompt word each gain five to seven [email protected] points over the baseline, while hard routing and the prepended token are not distinguishable from it. Much of that gain is protection from a preprocessing choice of ours rather than a spatial prior. Our serializer clamps the off-screen touch point AITW records for type events to the origin; that class degrades the baseline's click grounding, and removing it lifts the baseline by nearly seven points, after which no mechanism's hit rate beats it and the intervals exclude a two-point effect, though the auxiliary loss still shortens the average miss; on a stream of taps and swipes none helps. Whether this generalizes beyond one serialization is open. For deployment, the pipeline's margin over the baseline with predicted rather than gold types is not established (+0.016, 95% interval [-0.017, +0.052]), and a wrong type collapses every model conditioned at inference. The prepended token does not help at the shared learning rate, where its rows barely move from initialization; trained ten times faster it reaches the level of the other three, with a margin three seeds do not establish. We also document a silent failure: injecting conditioning through inputs_embeds makes Qwen2-VL fall back to 1-D positions for image tokens, costing nine points.

Momentum

Related topics

Latest papers 30