GUI Grounding

GUI: Graphical User Interface

GUI grounding studies how models identify the buttons, fields, and other screen regions that match a user’s instruction, enabling agents to operate websites and desktop or mobile applications. Common approaches train vision-language models to predict coordinates or boxes, or use reinforcement learning, distillation, and element-matching objectives; inference-time methods add zooming, visual search, or verification to refine predictions. Benchmarks such as ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, and UI-Vision test grounding across interface types and levels of complexity. A central challenge is making predictions reliable under clutter, occlusion, and changes in layout while controlling the extra computation used for refinement.

Latest papers

57 papers12 papers in the last four weeks, up 140% on the four weeks before. 0.1% of all new papers.

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding
Oct 27, 2026Ruilin Yao, Xinyu Qiu, Tianyu Zou +4Active Visual Perception Transfer Learning
Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences (CAS) · Zhejiang University +1
AdaGround: Token-Normalized Adaptive Zooming with Verifier-in-the-Loop Refinement for GUI Grounding
Oct 27, 2026Bingchao Wang, Yongcheng Jing, Chengyu Wang +4GUI Grounding Visual Grounding
Shanghai Jiao Tong University · Nanyang Technological University · Alibaba Cloud Computing
Right Click, Wrong Box: Grading GUI Grounding by What a Click Does
Oct 27, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2GUI Grounding Automated Evaluation
Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China
PreView: Enhancing GUI Grounding with Preprocessed Views and Cues
Oct 24, 2026Wenjun Hou, Kaishuai XuGUI Grounding Vision-Language Grounding
Independent Researcher
IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints
Oct 23, 2026Kyeongseon Kim, Jiyeon Son, Tae-Hyun OhGUI Grounding GUI Agent Benchmarks
KAIST
PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding
Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI Grounding Multimodal Model Evaluation
Sapienza University of Rome · Warmwind AG
From Seeing to Reporting: A Theory of Reliable Visual Set Generation in Vision-Language Models
Oct 18, 2026Junzhe Shi, Yuan Gan, Shida JiangVision-Language Models Visual Grounding
Systems Engineering, University of California, Berkeley, Berkeley, CA, USA · Department of AI, Quotr AI, Emeryville, CA, USA
GroundGUI: GUI Grounding by High-Quality Data and Distillation-to-Reinforcement Learning
Oct 13, 2026Qingsong Xie, Jiahao Tian, Jiacheng Tang +5GUI Grounding Multimodal Grounding
Prentis AI, San Francisco, CA · Westlake University, Hangzhou, China · Titan Holdings, San Francisco, CA
MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search
Oct 13, 2026Minghua Chen, Xingming Liao, Weiqiang Zhong +1GUI Grounding Vision-Language Grounding
School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510090, China
Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding
Oct 12, 2026Jingchao Wang, Dingjiang HuangGUI Grounding GUI Agents
School of Data Science and Engineering East China Normal University, Shanghai 200062, China
ConClick: Element-Wise Contrastive Learning Improves GUI Grounding
Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI Grounding GUI Agents
Decoupling What from Where: How Should a Small GUI Grounding Model Receive the Action Type?
Oct 5, 2026Aadi Chauhan, Arthur IlyasovGUI Grounding Mobile GUI Agents
Stanford University
Recurrent Latent Visual Search for GUI Grounding
Oct 4, 2026Kaiyu Wu, Beichen Zheng, Weiyao Huang +1GUI Grounding Vision-Language Grounding
Sun Yat-sen University
DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents
Oct 1, 2026A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong +2Computer-Use Agents GUI Grounding
ETH Zurich · IBM Research Zurich · Microsoft
SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents
Sep 29, 2026Shengtian Yang, Ziyu Xiong, Kaibing Yang +6Reinforcement Learning RL for GUI Agents
Southeast University, Nanjing, China · Kuaishou Technology, Beijing, China · Unaffiliated
ScreenHaystack: Finding Blind Zones in GUI Grounding
Sep 25, 2026Chenyue Li, Xiaoxiao Sun, Yubo Deng +3GUI Grounding Data Augmentation
Stanford University
MintAct: A Unified Visual Agent for Digital Environments
Sep 18, 2026Mingfei Gao, Rui Tian, Haiming Gang +11GUI Automation Tool Use in VLMs
Apple
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
Sep 16, 2026Liyang Fan, Xinping Bi, Yitai Li +3Efficient Multimodal Inference Multimodal Reranking
College of Computer Science and Software Engineering Shenzhen University Shenzhen, China · Shenzhen Key Laboratory for High Performance Data Mining Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Shenzhen, China · Shenzhen University of Advanced Technology Shenzhen, China
Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
Sep 14, 2026Yizhou Liu, Fei Tang, Yuchen Yan +8Test-Time Adaptation of VLMs GUI Grounding
Zhejiang University · Ant Group
LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents
Sep 9, 2026Zhangxuan Gu, Haoxing Chen, Qi Qin +17GUI Agents Multimodal Agents
AGI Research Center, Inclusion AI · Venus Team · Westlake University
GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
Aug 22, 2026Md Abrar Jahin, Md Rizwan ParvezGUI Grounding Spatial Reasoning Benchmarks
University of Southern California · USC Information Sciences Institute · Qatar Computing Research Institute (QCRI)
Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web
Aug 22, 2026Qijia Chen, Giulio JacucciGUI Grounding Text Embedding Evaluation
Department of Computer Science University of Helsinki Helsinki, Finland
Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation
Aug 11, 2026Shiyu Xuan, Zechao LiTest-Time Adaptation On-Policy Self-Distillation
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China, 210094
LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
Aug 10, 2026Renshan Zhang, Haoyang Meng, Yixiao He +3GUI Grounding Visually Grounded Reasoning
Harbin Institute of Technology, Shenzhen · Beijing University of Posts and Telecommunications · Shenzhen Loop Area Institute +1
Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching
Aug 10, 2026Yuke Li, Xuehan HouGUI Grounding Multimodal Large Language Models
School of Electronic and Computer Engineering, Peking University, Shenzhen, China
MissClick: Execution-Aware Adversarial Attacks on Coordinate Generation in GUI Grounding Models
Aug 4, 2026Yu Ran, Wentao ZhaoAdversarial Attacks on VLMs Adversarial Attacks
National University of Defense Technology
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
Aug 4, 2026Zichuan Fu, Shirong Wang, Wenlin Zhang +10GUI Grounding Visual Grounding
City University of Hong Kong · Westlake University · Tencent Jarvis Lab
Prior Directions: Why GUI Grounding Gets Locked in the Past
Jul 29, 2026Weile Gong, Zijian Lu, Mingcai Chen +3Vision-Language Models Representation Geometry in VLMs
Nanjing University of Posts and Telecommunications, Nanjing, China
One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Grounding
Jun 29, 2026Chen Liu, Ling Chen, Hanzhang Zhou +5Efficient Multimodal Inference GUI Grounding
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots
Jun 29, 2026Sunqi Fan, Lingshan Chen, Runqi Yin +4RL for GUI Agents GUI Agents
Tsinghua University · Tencent Hunyuan

Oct 27, 2026J/K move · Enter open · S save

Learning to Look Beyond Screens: Transferable Visual Observation from GUI Grounding

Ruilin Yao, Xinyu Qiu, Tianyu Zou, Yi Rong+3

Wuhan University of Technology · Institute of Automation, Chinese Academy of Sciences (CAS) · Zhejiang University+1

Active Visual Perception Transfer Learning GUI Grounding

Open paper AskPDF arXiv

Visual perception is evolving from passive recognition toward active, task-oriented observation. We investigate GUI grounding as a scalable source of spatial supervision for learning observation behaviors that transfer beyond interfaces. We introduce LBS (Look Beyond Screens), a vision-language model trained through supervised initialization on teacher-generated GUI grounding trajectories, followed by reinforcement learning for multi-turn crop-and-inspect decisions. Under fixed inference settings, LBS models based on Qwen3.5-35B-A3B and Qwen3.8-27B consistently improve over their corresponding baselines. Gains beyond supervised fine-tuning transfer to PerceptionBench for both backbones and to HR-Bench and ChartQAPro for Qwen3.5-35B-A3B, supporting observation strategies that generalize beyond GUI-specific imitation. Further analysis suggests complementary roles for the two stages: SFT establishes visual inspection primitives, while RL improves their downstream effectiveness. These findings support GUI grounding as a practical supervision paradigm for transferable active visual perception and broader multimodal reasoning. Our code and model are publicly available at https://github.com/inclusionAI/UI-Venus/tree/LBS and https://huggingface.co/inclusionAI/LBS-27B, respectively.