GUI Agents

GUI: Graphical User Interface

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 283% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 124

All topics

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining
May 14, 2026Weimin Xiong, Shuhao Gu, Bowen Ye +5GUI Agents LLM Agent Training
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University
Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment
May 14, 2026Yuchen Sun, Pei Fu, Shaojie Zhang +6GUI Agents AI Agent Evaluation
1Shanghai Jiao Tong University · 2Xiaomi Inc.
MMSkills: Towards Multimodal Skills for General Visual Agents
May 13, 2026Kangning Zhang, Shuai Shao, Qingyao Li +8GUI Agents LLM Agent Skill Learning
1Shanghai Jiao Tong University · 2Xiaohongshu Inc. · 3Southeast University
WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments
May 13, 2026Haoren Zhao, Tianyi Chen, Zhen WangMultimodal Robustness GUI Agents
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China · 2Microsoft
Executable Agentic Memory for GUI Agent
May 12, 2026Zerui Qin, Sheng Yue, Xingyuan Hua +2GUI Agents LLM Agent Memory
1Tsinghua University, China · 2Sun Yat-sen University, China
How Mobile World Model Guides GUI Agents?
May 11, 2026Weikai Xu, Kun Huang, Yunren Feng +10World Model Learning GUI Agents
1Nanyang Technological University · 2MiLM Plus, Xiaomi Inc. · 3Independent Researchers +3
Mem-W: Latent Memory-Native GUI Agents
May 10, 2026Guibin Zhang, Yaohui Ling, Fanci Meng +2Agent Memory GUI Agents
LV-NUS Lab, NUS & NTU
LiteGUI: Lightweight GUI Agents via Multi-Solution Guided Distillation and Dual-Level Reinforcement Learning
May 8, 2026Yubin Wu, Zicheng Cai, Liping Ning +4RL for GUI Agents GUI Agents
Moore Threads AI
X-OmniClaw Technical Report: A Unified Mobile Agent for Multimodal Understanding and Interaction
May 7, 2026Xiaoming Ren, Ru Zhen, Chao Li +11Multimodal Grounding GUI Agents
Multi-X Team, OPPO AI Center
Faithful Mobile GUI Agents with Guided Advantage Estimator
May 2, 2026Haowen Hu, Pengzhou Cheng, Zheng Wu +3Reinforcement Learning RL for GUI Agents
1Shanghai Jiao Tong University.
A11y-Compressor: A Framework for Enhancing the Efficiency of GUI Agent Observations through Visual Context Reconstruction and Redundancy Reduction
May 1, 2026Michito Takeshita, Takuro Kawada, Takumi Ohashi +2GUI Agents Context Compression
Hosei University, Tokyo, Japan
FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting
Apr 30, 2026Fengxian Ji, Jingpu Yang, Zirui Song +5GUI Agents Vision-Language Grounding
MBZUAI · Northeastern University, China
GUI Agents with Reinforcement Learning: Toward Digital Inhabitants
Apr 30, 2026Junan Hu, Jian Liu, Jingxiang Lai +6Reinforcement Learning RL for GUI Agents
1Shandong University · 2The Hong Kong University of Science and Technology · 3The Hong Kong University +2
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Apr 30, 2026Jinchao Li, Yunxin Li, Chenrui Zhao +3Computer-Use Agent Benchmarks Computer-Use Agents
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
Apr 28, 2026Alice Gao, Weixi Tong, Rishab Vempati +4Computer-Use Agents GUI Agents
University of Washington Seattle, WA, USA · Purdue University West Lafayette, IN, USA
Benchmarking and Improving GUI Agents in High-Dynamic Environments
Apr 28, 2026Enqi Liu, Liyuan Pan, Zhi Gao +5GUI Agents GUI Automation
Beijing Institute of Technology, Beijing, China · Beijing Institute for General Artificial Intelligence, Beijing, China · Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing, China +1
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
GoClick: Lightweight Element Grounding Model for Autonomous GUI Interaction
Apr 27, 2026Hongxin Li, Yuntao Chen, Zhaoxiang ZhangEfficient VLM Inference Training Data Curation
University of Chinese Academy of Sciences, Beijing, China. · New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China. · Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China.
VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
Apr 23, 2026Qijun Han, Haoqin Tu, Zijun Wang +11GUI Agents GUI Automation
UC Santa Cruz · CMU · UNC-Chapel Hill +2
AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents
Apr 22, 2026Jeonghyeon Kim, Byeongjun Joung, Junwon Lee +3Mobile GUI Automation Computer-Use Agents
Department of Computer Science and Engineering Sungkyunkwan University (SKKU) Suwon, Republic of Korea
HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
Apr 19, 2026Chao Jin, Wenkui Yang, Hao Sun +6GUI Agents Hallucination in Language Models
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences
FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems
Apr 16, 2026Wenhao Wang, Haoting Shi, Mengying Yuan +7Benchmark Design GUI Agents
Zhejiang University · Shanghai Jiao Tong University · Multi-Agent Governance & Intelligence Crew (MAGIC) +2
Are GUI Agents Focused Enough? Automated Distraction via Semantic-level UI Element Injection
Apr 9, 2026Wenkui Yang, Chao Jin, Haisu Zhu +7GUI Agents AI Agent Security
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · 2MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences · 4ShanghaiTech University +2
OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Apr 3, 2026Felix Henry, Lihan Lin, Jiangyou Zhu +4Mobile GUI Automation GUI Agents
XPeng Motors
GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
Mar 27, 2026Rui Xie, Zhi Gao, Chenrui Shi +3Multimodal RAG Domain Adaptation
Shanghai Jiao Tong University, Shanghai, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China · Beijing Institute of Technology, Beijing, China
CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training
Mar 23, 2026Yuxi Chen, Haoyu Zhai, Chenkai Wang +4Synthetic Data Generation GUI Agents
University of Illinois Urbana-Champaign.
SlowBA: An efficiency backdoor attack towards VLM-based GUI agents
Mar 9, 2026Junxian Li, Tu Lan, Haozhen Tan +2Backdoor Attacks GUI Agents
School of Computer Science, Shanghai Jiao Tong University
GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Models
Jan 26, 2026Shaokang Wang, Pei Fu, Ruoceng Zhang +7GUI Agents Test-Time Scaling
MiLM Plus, Xiaomi Inc, Beijing, China · School of Computer Science, Peking University, Beijing, China
Mind the Gap: Action Rebinding Attacks against Android GUI Agents
Jan 18, 2026Yi Qian, Kunwei Qian, Xingbang He +7Mobile GUI Automation Adversarial Attacks
State Key Laboratory for Novel Software Technology, Nanjing University · Hornor Device Co., Ltd · Institute of Dataspace, Hefei Comprehensive National Science Center China
InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training
Jan 7, 2026Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang +4Web Application Generation GUI Agents
1Peking University · 2Nanjing University · 3Microsoft Research Asia

May 14, 2026, cs.CLJ/K move · Enter open · S save

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

Weimin Xiong, Shuhao Gu, Bowen Ye, Zihao Yue+4

National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University

GUI Agents LLM Agent Training

Open paper AskPDF arXiv

Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization remains constrained by the scarcity of large-scale training data spanning diverse real-world applications. Existing datasets rely heavily on costly manual annotations and are typically confined to narrow domains. To address this challenge, we propose Video2GUI, a fully automated framework that extracts grounded GUI interaction trajectories directly from unlabeled Internet videos. Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, we construct WildGUI, a large-scale dataset containing 12 million interaction trajectories spanning over 1,500 applications and websites. Pre-training Qwen2.5-VL and Mimo-VL on WildGUI yields consistent improvements of 5-20% across multiple GUI grounding and action benchmarks, matching or surpassing state-of-the-art performance. We will release both the WildGUI dataset and the Video2GUI pipeline to support future research of GUI agents.

Momentum

Related topics

Latest papers 124