Mobile GUI Agents

GUI: Graphical User Interface

Overview

Mobile GUI agents interpret screenshots and accessibility trees, then tap, type, and navigate apps to complete user goals, often across multiple steps and applications. Common approaches combine vision-language models with structured planning, memory and reusable skills, reinforcement learning, or tools that speed up routine actions; simulation and real-device training provide interaction data and feedback. Benchmarks increasingly test long workflows, unseen apps, personalization, robustness, and real-world execution, not just task completion in familiar interfaces. Agents are becoming more capable, but generalization and reliability remain uneven, while prompt injection, privacy, and consequential actions make safe behavior a central open challenge.

Latest papers

91 papers8 papers in the last four weeks, down 33% on the four weeks before. 0.1% of all new papers.

Oct 7, 2025

Agent+P: Guiding UI Agents via Symbolic Planning

Large Language Model (LLM)-based UI agents show great promise for UI automation but often hallucinate in long-horizon tasks due to their lack of understanding of the global UI transition structure. To address this, we introduce AGENT+P, a novel framework that leverages symbolic planning to guide LLM-based UI agents. Specifically, we model an app's UI transition structure as a UI Transition Graph (UTG), which allows us to reformulate the UI automation task as a pathfinding problem on the UTG. This further enables an off-the-shelf symbolic planner to generate a provably correct and optimal high-level plan, preventing the agent from redundant exploration and guiding the agent to achieve the automation goals. AGENT+P is designed as a plug-and-play framework to enhance existing UI agents. Evaluation on the AndroidWorld benchmark demonstrates that AGENT+P improves the success rates of state-of-the-art UI agents by up to 14.32% and reduces the action steps by 37.70%. Our code is available at: https://github.com/KiteFlyKid/agentp.