GUI Agents

GUI: Graphical User Interface

Ask about this topic

Overview

GUI agents use multimodal models to interpret screens and carry out tasks through clicks, typing, dragging, and sometimes structured accessibility or command-line tools. Researchers improve them with supervised demonstrations, reinforcement learning in interactive environments, symbolic planning, memory, and verification of action outcomes. Benchmarks such as OSWorld, AndroidWorld, and WebArena test agents across desktop, mobile, and web tasks, with newer evaluations emphasizing real apps and long professional workflows. Agents can handle many routine tasks, but precise grounding, cross-app generalization, error recovery, safety, and efficient long-horizon execution remain difficult.

Latest papers

CardsList

178 papers30 papers in the last four weeks, up 67% on the four weeks before. 0.2% of all new papers.

  1. From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

    Oct 27, 2026Ziyang Chen, Qian Li, Chengwei QinGUI AgentsProcedural Memory for LLM Agents

    The Hong Kong University of Science and Technology (Guangzhou) · Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University

  2. GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

    Oct 26, 2026Ye Li, Haoran Ding, Huanan Liu +10Efficient InferenceGUI Agents

    Tsinghua University · The Chinese University of Hong Kong · Joy Future Academy, JD Group, Beijing, China

  3. IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

    Oct 23, 2026Kyeongseon Kim, Jiyeon Son, Tae-Hyun OhGUI GroundingGUI Agent Benchmarks

    KAIST

  4. Correct Answers, Closed Gates: SolveBind for GUI Agents at the Verification Barrier

    Oct 23, 2026Xinhao Song, Haoxuan Peng, Su Su +5GUI AgentsRuntime Enforcement for AI Agents

    Shanghai Jiao Tong University · The University of Sydney · Shandong University

  5. Subgoal-aware Memory Graph for GUI Agent

    Oct 22, 2026Dezhao Luo, Daniel Beechey, Derek Yuen +1Agent MemoryGUI Agents

    Huawei Noah’s Ark Lab

  6. Diagnosing Personalization Failures in Mobile GUI Agents

    Oct 22, 2026Zhixin Lin, Dongliang Xu, Jungang Li +6GUI AgentsMobile GUI Agents

    Shandong University · VIVO AI Lab · The Hong Kong University of Science and Technology (GZ) +5

  7. ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents

    Oct 19, 2026Zhicheng Zheng, Cheng Zhang, Shengping Yu +1Memory-Augmented VLMsMemory-Augmented Language Model Agents

    The Hong Kong University of Science and Technology (Guangzhou) · Imperial College London

  8. Focused Omni-Modal GUI Agents

    Oct 19, 2026Fuyao Zhang, Wenwen He, Jiaming Zhang +6GUI AgentsMultimodal Agents

    Nanyang Technological University · Alibaba Group

  9. Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution

    Oct 14, 2026Wenkui Yang, Yiming Hao, Kai Zhou +6GUI AgentsAI Agent Governance

    SAIS, University of Chinese Academy of Sciences, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China +2

  10. How GUI Agents Fail: GraceBench for Execution Stability Diagnosis

    Oct 14, 2026Ridong Zhang, Shuxian Liang, Chaoqun Wan +2Computer-Use Agent BenchmarksGUI Agents

    Institute of AI for Engineering, Tongji University

  11. Learning How to Remember: Memory Modality Selection for GUI Agents

    Oct 13, 2026Taeyun Roh, Suhyeong Park, Gijin Kim +2Memory-Augmented AgentsGUI Agents

    Korea University · Myongji University · AIGEN Sciences

  12. PrivacyGuard: Source-Linked Sanitization and Locally Resolvable PII References for Cloud GUI Agents

    Oct 12, 2026Zian Yuan, Wei LiGUI AgentsPII Detection

  13. Learn from Move: the Next Step for GUI Agents

    Oct 12, 2026Jiahao Lu, Yize Shen, Chuanyuan Tan +6GUI Agent BenchmarksGUI Agents

    Fudan University · Shanghai Innovation Institute · Soochow University

  14. Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding

    Oct 12, 2026Jingchao Wang, Dingjiang HuangGUI GroundingGUI Agents

    School of Data Science and Engineering East China Normal University, Shanghai 200062, China

  15. OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools

    Oct 9, 2026Chenyang Yan, Yingying Zhang, Kanzhi Cheng +21Mobile Agent BenchmarksMobile GUI Agents

    Nanjing University · SenseTime · The University of Hong Kong +3

  16. ConClick: Element-Wise Contrastive Learning Improves GUI Grounding

    Oct 9, 2026Ruotian Zhang, Yuhui Zhang, Junzhe Huang +3GUI GroundingGUI Agents

  17. Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

    Oct 9, 2026Yuqi Huang, Hui Li, Yifei Cao +9GUI AgentsAgent Skill Learning

    Shanghai Jiao Tong University · Fudan University · Nanyang Technological University +2

  18. Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training

    Oct 5, 2026Yongxin Ning, Runliang Niu, Qianli Xing +4World ModelsSynthetic Data Generation

    School of Artificial Intelligence, Jilin University · College of Computer Science, Jilin University · OPPO

  19. MedImageOSWorld: Benchmarking GUI Agents for Medical Image Consoles

    Oct 3, 2026Ziyang Long, Xinqi Li, Lujing Xing +1Medical AI AgentsGUI Agents

    Biomedical Imaging Research Institute, Cedars-Sinai Medical Center, Los Angeles, CA, USA · University of California, Los Angeles, Los Angeles, CA, USA · Technical University of Munich +1

  20. Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents

    Oct 1, 2026Beining Wu, Zihao Ding, Jun HuangContinual Learning for LLM AgentsGUI Agents

    Department of Electrical Engineering and Computer Science, South Dakota State University

  21. AutoGUIWorld: Image Generators as Visual World Models for GUI Agent

    Oct 1, 2026Cheng Yang, Yifan Wu, Yutao Huang +18World Model LearningSynthetic Data Generation

    Hunyuan AI Data Team

  22. GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution

    Oct 1, 2026Geyi Yang, Zikun Qu, Xiang Li +4GUI AgentsAgent Harness Optimization

    The Chinese University of Hong Kong, Shenzhen · Tianjin University · Harbin Institute of Technology (Shenzhen) +2

  23. Learning Reliable GUI Agents under Imperfect Priors

    Sep 30, 2026Bo Han, Qianyi Wang, Shuai Liu +9GUI AgentsLLM Agent Training

    Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.

  24. Action Conditioned Bisimulation For GUI Agent Memory

    Sep 30, 2026Hongbo Zhang, Liuyang Song, Quanquan Li +3Agent MemoryGUI Agents

    Peking University · East China Normal University · University of Southern California

  25. HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents

    Sep 29, 2026Tongbo Chen, Junbo Niu, Zhengxi Lu +8Computer-Use AgentsGUI Agents

    Zhejiang University · Peking University · Tsinghua University

  26. MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows

    Sep 29, 2026Mei Wu, Rui Xie, Runyu Zhang +6GUI AgentsAI Agent Benchmarks

    X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Suzhou Laboratory, Suzhou, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China +3

  27. PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval

    Sep 29, 2026Bin Kang, Jiarui Ouyang, Li Jiang +2GUI AgentsLong-Horizon Agent Evaluation

    University of Chinese Academy of Sciences · Harbin Institute of Technology · Shenzhen Loop Area Institute +2

  28. WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

    Sep 28, 2026Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2Web Agent BenchmarksGUI Agents

    DAIMLD · HSE University · NUST MISIS

  29. LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

    Sep 28, 2026Bingo Zhang, Haochuan Lu, Zongjie Li +3GUI AgentsLong-Horizon Agent Evaluation

    Vera Praxis · Tencent · The Hong Kong University of Science and Technology +2

  30. One Rollout Is All You Get: Fully Test-Time Adaptation for GUI Agents

    Sep 28, 2026Ziqiang Wang, Li Gu, Zhixiang Chi +6Test-Time AdaptationContinual Test-Time Adaptation

    Concordia University · Mila – Qu´ebec AI Institute · University of Toronto +1