Coati: Host-Grounded Execution for Installable Mobile GUI Agents

Yudou Ying; Pu Shi; Hanlei Shi; Yuhua Zheng

cs.MANov 2, 2026

Coati: Host-Grounded Execution for Installable Mobile GUI Agents

Authors: Yudou Ying, Pu Shi, Hanlei Shi, Yuhua Zheng

Organizations: Independent Researcher, China

Abstract

Mobile GUI agents can carry out tasks across apps, yet most run on a separate computer that drives a phone over the Android Debug Bridge (ADB), out of reach of ordinary users and of the apps through which most people reach Web services. We present Coati, a source-available agent that users install as an Android app. Its runtime on the phone, the host, records what a model alone cannot see: the effect of each action at its target, result screens across apps, and the progress of repeated steps. VerifyLoop checks action outcomes and completion claims against host evidence and reports a completion it cannot audit as unverified. AppMemory builds per-app procedural memory from observations that the interface supports and reuses it on later visits. A routed fast executor lets a lightweight selection model take simple, confident steps and lets the host repeat loops of simple steps that the large model plans once. On all 116 AndroidWorld tasks, with Gemini 3.8 Flash as every agent's backbone, Coati succeeds on 86.5% of the tasks over three trials, ahead of the strongest open-source agent Artemis (77.6% with the same accessibility-only access, 76.5% even when allowed ADB) and far ahead of FinalRun and Midscene (about 59%), and VerifyLoop cuts false completion claims by about a third at comparable success. In everyday use, the fast executor cuts the median time of phone tasks by 39% at comparable success and runs three of four loop tasks 2.8-5.4x faster with 6-9x fewer model tokens, while AppMemory shortens a repeated deep route in an unfamiliar app from 18.5 steps to 6-7. On an Android 16 phone, routed execution completes all 27 benchmark runs with 76% fewer Reasoning calls on the critical path. Host evidence, not model self-assessment, makes an installable agent reliable and fast.

Figures & tables

Figure 1. Overview of Coati. Hybrid perception produces a parsed screen from the accessibility tree, vision, or both. The router selects fast or Reasoning execution. VerifyLoop checks action outcomes using host evidence, while AppMemory provides advisory entries and updates the manual after accepted tasks.

Outcome	Interpretation	Control Decision
SUCCESS	The expected transition is observed.	Continue from the new state.
FAILED	The action is ineffective or the expected state is absent.	Return failure feedback and replan.
DRIFTED	Execution enters a task-incompatible state.	Recover from the observed state or terminate.
UNEXPECTED	The interface changes differently from the prediction.	Continue reasoning from the observed state.
UNKNOWN	Available evidence is insufficient.	Re-observe without assuming success.

Table 1. Verification outcomes and corresponding control decisions in VerifyLoop.

Decision	Condition	Action
MATCHED	Existing fact	Keep $M_{i}$ unchanged.
NEW	New/changed fact and $E_{t}=1$	Sanitize and add to $\mathcal{C}_{i}$ .
UNKNOWN	Insufficient evidence	Discard the observation.
NO_CHANGE	No valid difference	Preserve $M_{i}$ .
EDIT	Bounded additions	Apply a constrained local edit.
SUMMARIZE	Redundancy or capacity limit	Reorganize under validation.

Table 2. AppMemory admission and update decisions.

Agent	Trials	SR	pass 3	False done	Time	Tokens
Midscene.js	1	59.1	–	7.5	179	222
FinalRun	1	59.9	–	14.5	224	177
Artemis (ADB)	1	76.5	–	–	185	–
Artemis (GUI)	3	77.6	69.8	–	209	–
Coati	3	86.5	81.9	5.4	172	154

Table 3. AndroidWorld, all 116 tasks, Gemini 3.8 Flash as every agent’s backbone, same emulator, task instances, budgets and evaluators. SR: success rate (%), mean over trials. False done: share of declared completions that the evaluator judged failed (%). Time: mean seconds per episode; tokens: mean thousands per episode, as reported by each agent’s model API. Midscene, FinalRun and Artemis (ADB) drive the phone over ADB; Coati and Artemis (GUI) use the accessibility API only. A dash is a value the agent does not report through the harness.

Configuration	Easy	Medium	Hard	SR	False done
C1: no VerifyLoop	96.7	86.1	73.7	89.7	8.0
C2: Coati	96.2	79.6	68.4	86.5	5.4

Table 4. VerifyLoop on AndroidWorld (116 tasks, same build). C1: one trial; C2: mean of three. SR by difficulty and overall (%). False done: share of declared completions that the evaluator judged failed (%).

Figure 2. Median execution time for seven tasks under Classic and Routed execution. Lower values indicate faster execution; annotations report successful runs and relative change.

Figure 3. Mean token usage (thousands per run) for seven tasks under Classic and Routed execution. Lower values indicate lower cost; annotations report successful runs and relative change.

	Classic				Routed + loops
Task	OK	Time	Steps	Tok.	OK	Time	Steps	Tok.
R1 first 10 to favorites	5/5	140	13	145	5/5	50	12 (11)	21
R2 six given IDs to compare	5/5	309	25	304	5/5	57	14 (13)	33
R3 hide every Sold listing	4/5	210	22	245	5/5	63	14 (13)	39
R4 refresh until price drops	5/5	100	9	101	3/3	92	8 (7)	90

Table 5. Loops of simple steps in Car Market’s repeat mode, five runs per configuration (R4 routed: three runs after the stop rule of Section 3.4 ). Time: mean seconds; steps: mean (fast-route steps in parentheses); tokens: mean thousands.

App	Memory	Run 1	Run 2	Run 3	Run 4
Manual Lab	off	14 / 171	19 / 198	18 / 183	19 / 198
	session 1	19 / 286	6 / 61	6 / 61	6 / 61
	session 2	14 / 138	7 / 80	7 / 71	7 / 85
	session 3	15 / 163	7 / 77	7 / 77	7 / 117
Messages	off	5 / 46	5 / 49	5 / 52	5 / 46
	session 1	5 / 43	5 / 46	6 / 88	5 / 49

Table 6. AppMemory on a deep route in an unfamiliar app (Manual Lab, open the “Archive” page four menus deep) and on a familiar app (Messages, Advanced settings). Steps and seconds per run; each learning session starts from an empty manual.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Vision parse	+ OCR repair	$\Delta$
Elements found	51.7%	63.7%	+12.0
Exact text	49.2%	57.6%	+8.4
Tap hit	26.4%	32.0%	+5.6
Elements made worse	–	0

Appendix

Table 7. Parse-level evaluation on 14 device screens (28 vision parses). Worse counts elements whose text or tap position a repair made worse.

Method	Model	Screen Representation	SR
(a) Published methods (paper-reported)
UI-TARS ( Qin et al., 2025 )	UI-TARS-72B	Screenshot	46.6
Agent S2 ( Agashe et al., 2025 )	Agent S2	Screenshot	54.3
V-Droid ( Dai et al., 2026 )	V-Droid (Llama-8B)	A11y tree	59.5
Mobile-Agent-v3 ( Ye et al., 2025 )	GUI-Owl-32B	Screenshot	73.3
MobileRL ( Xu et al., 2026 )	MobileRL-9B (GLM-4.1V-9B)	Screenshot + A11y tree	80.2

Appendix

Table 8. Success on AndroidWorld. SR is the task success rate (%). (a) Published methods, with numbers from their papers. (b) Submissions to the public AndroidWorld leaderboard ( Google Research, 2026 ) (accessed October 2026), which are self-reported, single-trial and not independently verified; we summarize them rather than rank against them. (c) Coati under the protocol of Section 4.1 , mean over three trials. A11y tree denotes the Android accessibility UI tree.

		Per-step Reasoning				Routed (Coati)
Task	Interface	Succ.	Time (s)	Steps	Reas.	Succ.	Time (s)	Steps	Reas.
L1	native, several menus deep	2/3	410.9	31.0	31.0	3/3	295.2	34.3	13.7
L2	native, search and text entry	3/3	78.7	6.0	6.0	3/3	32.4	6.3	0.0
L3	native, dependent setting	3/3	60.9	5.0	5.0	3/3	21.2	5.0	0.0
W1	custom canvas, no tree	3/3	99.1	7.0	7.0	3/3	82.7	9.0	0.7
W2	canvas and text entry	3/3	100.5	7.7	7.7	3/3	120.8	16.3	1.0

Appendix

Table 9. On-device benchmark (Android 16 phone, Gemini 3.8 Flash for Reasoning and Fast). Each cell reports verified successes over three runs, mean task time of the verified runs, mean steps, and mean Reasoning calls on the critical path per run (the routed configuration’s background planning calls are not counted).

Figure 4. AppMemory on the learning benchmark (Manual Lab, task L1). (A) The manual is bound to the application identity and retrieved as bounded, advisory entries; on the fast route it is the selector’s app guide. (B) Page observations pass the evidence gate, and the task-local buffer keeps the latest candidates, where the route that worked (Me $\rightarrow$ Settings $\rightarrow$ Advanced settings $\rightarrow$ Archive) appears after exploration. (C) After the audit accepts the task, a curator applies a bounded edit that is committed only if validation succeeds; warm runs then take five steps.

Mobile GUI Agents

Mobile GUI agents use visual and structured interface information to carry out tasks across smartphone apps.

See all 87 papers

Sep 28, 2026

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

Mobile GUI agents operate through a perception--action loop: at each step they screenshot the device, invoke a vision--language model (VLM), and emit an action. It is slow, costly, and brittle, yet most of what it does is navigation---and everyday navigation is static, ordered, and endlessly repeated. We present PhoneCLI, which compiles an app's GUI navigation into callable commands, without any app-internal API, runtime instrumentation, or model training. Offline, PhoneCLI explores a target app from the outside and distills its screens, interactive elements, and navigation edges into a semantically annotated map; each screen yields one deterministic command: a replay sequence that reaches it. Online, the agent selects a command, verifies it before execution, and then executes it deterministically in sub-second time at zero VLM cost; open-ended interaction and every failure of the compiled path fall back to the embedded VLM interpreter, exactly the pure VLM agent, so compilation can only help. On AndroidLab, PhoneCLI improves the task success rate while reducing steps and token consumption, and it transfers to AndroidWorld's official M3A agent with consistent efficiency gains. What PhoneCLI compiles is the app's navigation rather than one run, so it serves new tasks, not only repeated ones.

Yangqin Jiang, Lingrui Xu, Chao Huang

The University of Hong Kong

Mobile GUI Automation Mobile GUI Agents

Ask PDF

Sep 7, 2026

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible. Existing benchmarks typically trade off these goals: simplified apps lack real-world mobile complexity, whereas live commercial apps introduce uncontrolled variation from recommendations, advertisements, accounts, and changing content. We propose AppSim-Bench, which addresses this trade-off through controllable simulated apps that preserve task-relevant interaction logic while supporting deterministic evaluation. Built through a coding-agent-assisted and human-verified workflow, it contains 557 tasks across 17 high-frequency Chinese and English apps. Its controllable backend data and outcome-based verification remove major sources of environmental stochasticity, enabling reproducible cross-model comparison. Evaluating 19 GUI agents, spanning general-purpose and GUI-specialized systems, we find that autonomous mobile execution remains far from solved. The best model completes only 50.27% of tasks, and 28.55% of tasks are not solved by any agent. Further analysis shows that failures concentrate in longer workflows, numerical reasoning tasks, and inefficient trajectories marked by high action overhead and budget exhaustion. Our project is available at https://github.com/Acrab-Agentic-Labs/AppSim.

Jintian Feng, Long Chen, Xiao Yu +7

Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

Mobile GUI Automation GUI Agents Agent Evaluation Mobile GUI Agents Mobile Agent Benchmarks

Ask PDF

Jun 16, 2026

Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?

Recent advances in mobile agents are dominated by the GUI paradigm, in which agents perceive UI information and emit screen interactions. However, mobile platforms also expose a command-line interface (CLI) that provides direct access to device services and data. We argue CLI deserves first-class consideration alongside GUI. We evaluate three coding agents (Claude Code, Terminus-2, mini-swe-agent) across four model APIs on AndroidWorld and MobileWorld without any mobile-specific post-training, comparing against three reproducible GUI baselines (GUI-Owl-1.5-32B, MAI-UI, Qwen3-VL-32B). Claude Code (Opus 4.7) reaches 71.8% and 51.9%, outperforming every reproducible GUI baseline (69.3/68.1/57.8% on AndroidWorld; 43.2/26.3/13.3% on MobileWorld), while every other CLI configuration remains competitive. To establish the paradigm's ceiling, we provide oracle CLI solutions that reach 88.8% on AndroidWorld (103/116 tasks CLI-solvable) and 86.3% on MobileWorld (101/117 tasks CLI-solvable), indicating substantial room for future improvement. To cover everyday user intents beyond the GUI scope, we introduce the CLI-Advantage Task Suite, comprising 45 templates across five categories: bulk operations, multi-condition filtering, aggregation, cross-app workflows, and hidden device state. Every CLI agent outperforms every GUI baseline in all five categories, with substantially fewer steps per task (10.7 vs.\ 18.6). To support future research on mobile CLI agents, we will open-source agent implementations, oracle solutions, the CLI-Advantage suite, and evaluation infrastructure.

Li Gu, Zihuan Jiang, Linqiang Guo +6

1Mila – Québec AI Institute · 2Concordia University · University of Toronto +1

Mobile GUI Automation AI Coding Agents AI Agent Benchmarks LLM Agents Mobile Agent Benchmarks

Ask PDF

Abstract

Mobile GUI Agents

Explore similar work

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?