Mobile GUI Agents
GUI: Graphical User Interface
Overview
Mobile GUI agents interpret screenshots and accessibility trees, then tap, type, and navigate apps to complete user goals, often across multiple steps and applications. Common approaches combine vision-language models with structured planning, memory and reusable skills, reinforcement learning, or tools that speed up routine actions; simulation and real-device training provide interaction data and feedback. Benchmarks increasingly test long workflows, unseen apps, personalization, robustness, and real-world execution, not just task completion in familiar interfaces. Agents are becoming more capable, but generalization and reliability remain uneven, while prompt injection, privacy, and consequential actions make safe behavior a central open challenge.
Latest papers
84 papers12 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.