Graphical User Interface Agents

Ask about this topic

Momentum

21 papers in the last four weeks, up 425% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 108

All topics
CardsList

  1. Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents

    Oct 1, 2026Beining Wu, Zihao Ding, Jun HuangSelf-Improving AgentsGraphical User Interface Agents

    Department of Electrical Engineering and Computer Science, South Dakota State University

  2. Learning Reliable GUI Agents under Imperfect Priors

    Sep 30, 2026Bo Han, Qianyi Wang, Shuai Liu +9Graphical User Interface AgentsReliable

    Xi’an Jiaotong University · MiLM Plus, Xiaomi Inc.

  3. Action Conditioned Bisimulation For GUI Agent Memory

    Sep 30, 2026Hongbo Zhang, Liuyang Song, Quanquan Li +3Graphical User Interface AgentsMerge

    Peking University · East China Normal University · University of Southern California

  4. MatToolBench: Benchmarking Multimodal Agents in Real-World Materials Science Workflows

    Sep 29, 2026Mei Wu, Rui Xie, Runyu Zhang +6Materials ScienceMultimodal Agents

    X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Suzhou Laboratory, Suzhou, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China +3

  5. PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval

    Sep 29, 2026Bin Kang, Jiarui Ouyang, Li Jiang +2Graphical User Interface AgentsGraphical User Interface

    University of Chinese Academy of Sciences · Harbin Institute of Technology · Shenzhen Loop Area Institute +2

  6. PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

    Sep 28, 2026Yangqin Jiang, Lingrui Xu, Chao HuangDeviceGraphical User Interface Agents

    The University of Hong Kong

  7. WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agents

    Sep 28, 2026Anton Emelyanov, Maria Tikhonova, Zaven Martirosian +2Web AgentsWeb

    DAIMLD · HSE University · NUST MISIS

  8. LongPuzzleBench: Evaluating GUI Agents on Long-Horizon Visual Puzzles

    Sep 28, 2026Bingo Zhang, Haochuan Lu, Zongjie Li +3PuzzleGraphical User Interface Agents

    Vera Praxis · Tencent · The Hong Kong University of Science and Technology +2

  9. One Rollout Is All You Get: Fully Test-Time Adaptation for GUI Agents

    Sep 28, 2026Ziqiang Wang, Li Gu, Zhixiang Chi +6Graphical User Interface AgentsTest-Time Adaptation

    Concordia University · Mila – Qu´ebec AI Institute · University of Toronto +1

  10. Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize?

    Sep 28, 2026Tien Tran, Namho Koh, Daiki E. Matsunaga +2Graphical User Interface AgentsAgentic Benchmarks

    KAIST · CMU

  11. GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

    Sep 28, 2026Shaoqing Zhang, Kehai Chen, Xuefeng Bai +4Graphical User Interface AgentsGraphical User Interface

    Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China

  12. Relevance Does Not Imply Applicability: Experience Activation for Personal GUI Agents

    Sep 27, 2026Fuyao Zhang, Xuan Wang, Zherui Li +3Graphical User Interface AgentsGraphical User Interface

    NTU & Alibaba

  13. Jev-Mobile: Jev as an Executor for Mobile GUI Agents

    Sep 24, 2026Linghua ZhangGraphical User Interface AgentsDevice

  14. Invisible in Space, Visible in Time: Motion Vision CAPTCHA against GUI Agents

    Sep 23, 2026Zeyu Zhang, Dingyi Rong, Zijian Chen +3CaptchaMotion-Based Perception

    Shanghai Jiao Tong University Shanghai, China · Shanghai Artificial Intelligence Laboratory Shanghai, China

  15. Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction

    Sep 22, 2026Yan Zhang, Pei Fu, Daiqing Wu +8Graphical User Interface AgentsGraphical User Interface

    Institute of Information Engineering, Chinese Academy of Sciences · MiLM Plus, Xiaomi Inc. · School of Cyber Security, University of Chinese Academy of Sciences +1

  16. A Dual-Process Perspective on Nudge Susceptibility in LLM-Based GUI Agents

    Sep 17, 2026Haya Halimeh, Sascha Kaltenpoth, Kevin Bösch +1NudgesGraphical User Interface Agents

    Information Systems, Paderborn University, Paderborn, Germany.

  17. Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents

    Sep 15, 2026Bofan Chen, Boxuan Zhang, Fei Tang +7Graphical User Interface AgentsGraphical User Interface

    Zhejiang University · UESTC

  18. BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents

    Sep 11, 2026Tong Ye, Kunyang Han, Guozhi Wang +40Graphical User Interface AgentsDevice

  19. JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition

    Sep 9, 2026Zixiang Chen, Yuheng Lu, Zihao Cheng +8Graphical User Interface AgentsGraphical User Interface

    School of Computer Science and Engineering, Beihang University, Beijing, China · Beijing Institute of Technology, Beijing, China · Baidu Inc., Beijing, China

  20. APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

    Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Graphical User Interface AgentsDevice

    Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

  21. ElderBench: Benchmarking Autonomous Mobile Agents for Older Adults

    Sep 7, 2026Weide Zhan, Qumu Shaqu, Yuanqing Liu +6Graphical User Interface AgentsAgentic Benchmarks

    Fudan University Shanghai, China

  22. Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

    Sep 3, 2026Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng +7Graphical User Interface AgentsGraphical User Interface

    School of Computer Science, Shanghai Jiao Tong University · Ant Group

  23. CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications

    Aug 12, 2026Linqiang Guo, Li Gu, Zihuan Jiang +8Graphical User Interface AgentsText-Only Adaptation

    Peter · Concordia University · Mila – Québec AI Institute +2

  24. Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation

    Aug 11, 2026Shiyu Xuan, Zechao LiGraphical User InterfaceGraphical User Interface Agents

    School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China, 210094

  25. Software Engineering for and with GUI Agent

    Aug 10, 2026Shengcheng Yu, Yuchen Ling, Junyang Xing +3Graphical User Interface AgentsSoftware Engineering

    State Key Laboratory for Novel Software Technology, Nanjing University, China · Technical University of Munich, Germany

  26. The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

    Aug 6, 2026Weiwei Li, Junzhuo Liu, Tong Chu +2Graphical User Interface AgentsHindsight

    University of Electronic Science and Technology of China

  27. StepReflect: Structured UI Transition Reflection for Mobile GUI Agents

    Aug 6, 2026Linqiang Guo, Wei Liu, Li Gu +3Graphical User Interface AgentsReflections

    Peter · Concordia University Montreal, Quebec, Canada

  28. MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation

    Jul 31, 2026Hang Yan, Zhangxuan GU, Beitong Zhou +5Mixture-Of-AgentsGraphical User Interface Agents

    1Xi’an Jiaotong University · 2Ant Group · 3Shanghai Jiao Tong University

  29. Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees

    Jul 30, 2026Zihan Dong, Rui Qian, Qishi Zhan +3Graphical User Interface AgentsGraphical User Interface

    Georgia Institute of Technology · Fudan University · Marquette University +3

  30. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

    Jul 30, 2026Hanzhang Zhou, Panrong Tong, Xu Zhang +13Graphical User Interface AgentsDevice

    MAI-UI Team Alibaba Token Hub, Alibaba Group

  31. AgentGUI: An Interface for Observing and Steering Long-Running AI Agents

    Jul 28, 2026Xuan Zhao, Jiwoong Sohn, Qinyue Zheng +1Graphical User Interface AgentsArtificial Intelligence Agents

    ETH Zürich · ETH AI Center

  32. Interactive Reward Agent: GUI Task Evaluation via Environment-State Verification

    Jul 28, 2026Chenrui Shi, Yuwei Wu, Yang Liu +5Graphical User Interface AgentsInstruction

    Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology, Beijing, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University, Shenzhen, China +1

  33. Scaling GUI Agents with Visual State Transitions

    Jul 27, 2026Xiangyan Liu, Kaixin Li, Haonan Wang +6Graphical User Interface AgentsGraphical User Interface

    NUS · UTS · University of Liverpool +1

  34. StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

    Jul 24, 2026Yan Yang, Xiangru Jian, Ziyang Luo +7Computer-Use AgentsState-Tracking

    Salesforce AI Research

  35. SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis

    Jul 20, 2026Zhuohang Fan, Beichen Zhang, Yuanfa Li +4Graphical User Interface AgentsGraphical User Interface

    Harbin Institute of Technology, Weihai, China · Harbin Institute of Technology (Weihai) Qingdao Research Institute, Qingdao, China

  36. Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents

    Jul 17, 2026Zedong Yu, Qianxing Li, Zhi Gao +8Graphical User Interface AgentsGraphical User Interface

    Beijing University of Posts and Telecommunications, Beijing, China · State Key Laboratory for General Artificial Intelligence, BIGAI, Beijing, China · China University of Geosciences (Beijing), Beijing, China +1

  37. SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

    Jul 17, 2026Xue Yu, Bo Yuan, Kailin Zhao +3Graphical User Interface AgentsProbabilistic Safety

    JIUTIAN Research

  38. Plover: Steering GUI Agents through Plan-Centric Interaction

    Jul 16, 2026Madhumitha Venkatesan, Shicheng Wen, Jiajing Guo +3Graphical User Interface AgentsGraphical User Interface

    University of California, Davis · Davis, California, USA · Bosch Research North America +1

  39. HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

    Jul 16, 2026Hy Vision Team, Huawen Shen, Zhengyang Tang +20Graphical User Interface AgentsMultimodal Agents

    Hy Vision Team

  40. KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

    Jul 14, 2026Yunxin Li, Jinchao Li, Shibo Su +7Graphical User Interface AgentsAgentic Framework

    Lychee Team∗, Harbin Institute of Technology, Shenzhen

  41. VIA: Visual Interface Agent for Robot Control

    Jul 13, 2026Hengyuan Hu, Priya Sundaresan, Jensen Gao +1Robot SystemsReal-World Manipulation Tasks

    Stanford University

  42. UI-MOPD: Multi-Platform On-Policy Distillation for Continual GUI Agent Learning

    Jul 5, 2026Niu Lian, Alan Chen, Zhehao Yu +8Graphical User Interface AgentsMultimodal Agents

    Tsinghua Shenzhen International Graduate School, Tsinghua University · Harbin Institute of Technology, Shenzhen · Zhejiang University +2

  43. Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

    Jul 5, 2026Guijia Zhang, Harry YangGraphical User Interface AgentsPerception Gap

    1Shenzhen University · 2The Hong Kong University of Science and Technology

  44. A Task-State Representation for Long-Horizon Mobile GUI Agents

    Jul 1, 2026Yujie Zheng, Zikang Liu, Xin Zhao +1Graphical User Interface AgentsState-Tracking

    School of Software, Beihang University · Gaoling School of Artificial Intelligence, Renmin University of China

  45. What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

    Jun 30, 2026Chen Liu, Ling Chen, Hanzhang Zhou +7Graphical User Interface AgentsDevice

    University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University

  46. Xiaomi-GUI-0 Technical Report

    Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Graphical User Interface AgentsMultimodal Agents

  47. GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

    Jun 29, 2026Sunqi Fan, Lingshan Chen, Runqi Yin +4Graphical User Interface AgentsCraft

    Tsinghua University · Tencent Hunyuan

  48. Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning

    Jun 25, 2026Tianyi Men, Zhuoran Jin, Pengfei Cao +3Large Language Model PlanningMultimodal Agents

    The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

  49. GUI agent: Guided Exploration of User-Sensitive Screens

    Jun 24, 2026Aradhana Nayak, Mussadiq Nazeer, Wang Peng +1Graphical User Interface AgentsMulti-Llm Agents

    1Huawei Heisenberg Research Center (Munich) · 2Huawei Technologies Co., Ltd

  50. VisCritic: Visual State Comparison as Process Reward for GUI Agents

    Jun 23, 2026Jiachen QianGraphical User Interface AgentsSoft Actor-Critic

    City University of Hong Kong

  51. Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation

    Jun 23, 2026Marta Sumyk, Oleksandr KosovanGraphical User Interface AgentsComputer-Use Agents

    Ukrainian Catholic University, Lviv, Ukraine

  52. GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents

    Jun 22, 2026Xiao Zhou, Siyue Zhang, Yilun Zhao +4Graphical User Interface AgentsGraphical User Interface

    NNYU Shanghai · Y Yale NLP Lab

  53. ENVS: Environment-Native Verified Search for Long-Horizon GUI Agents

    Jun 22, 2026Yincheng Zhou, Athena Zhuoming Zhong, Shijie Zhang +3Graphical User Interface AgentsSynthetic Environments

    University of Pennsylvania · 2Peking University

  54. Skill-Guided Continuation Distillation for GUI Agents

    Jun 17, 2026Zhimin Fan, Hongwei Yu, Yeqing Shen +9Unsupervised Skill DiscoveryGraphical User Interface Agents

    1StepFun · University of Science and Technology Beijing · 4Nanyang Technological University +1

  55. Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?

    Jun 16, 2026Li Gu, Zihuan Jiang, Linqiang Guo +6Graphical User Interface AgentsDevice

    1Mila – Québec AI Institute · 2Concordia University · University of Toronto +1

  56. Naive Visual Memory is Not Enough: A Failure-Mode Study of GUI Agents

    Jun 12, 2026Seoyoung Choi, Minseok Ko, Hyunseok Lee +4Graphical User Interface AgentsGraphical User Interface

    1Seoul National University · 2POSTECH · 3KAIST +1

  57. ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm

    Jun 11, 2026Jiaxin Ai, Tao Hu, Xuemeng Yang +11Computer-Use AgentsSolidworks

    Wuhan University · Shanghai Artificial Intelligence Laboratory · Shanghai Innovation Institute +4

  58. CAPED: Context-Aware Privacy Exposure Defense for Mobile GUI Agents

    Jun 10, 2026Siyu Shen, Fenghao Xu, Wenrui Diao +1Graphical User Interface AgentsPrivacy

    The Chinese University of Hong Kong · Southeast University · Shandong University

  59. VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluation

    Jun 9, 2026Yunan Lu, Ryan Shea, Yusen Zhang +1Agentic EvaluationsUser Simulation

    Department of Computer Science, Columbia University, 2Arklex.ai

  60. Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

    Jun 9, 2026Liya Zhu, Jingzhe Ding, Jian Zhang +63Graphical User Interface AgentsAgentic Workflow Design