Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions

Juyong Lee; Woogyeol Jin; Kimin Lee

cs.AISep 6, 2026

Improving Proficiency and Efficiency of Android GUI Agents via Self-Generating Tool Actions

Authors: Juyong Lee, Woogyeol Jin, Kimin Lee

Organizations: KAIST

Abstract

Android agents using a hybrid action space that combines GUI actions and tool actions (e.g., accessing application data via APIs) remain largely underexplored, mainly due to the excessive effort required to create tools. To address this gap, we introduce DroidTool, a framework for augmenting the agents with self-generated tools, which are realized as Python functions operating on application states (e.g., a database). To create tools with minimal human labor, DroidTool employs an agentic workflow featuring stages: proposal, implementation, test generation and execution, and repair. Notably, when testing the created tools for verification, it constructs relational tests across relevant tools for natural preparation of appropriate test preconditions and improved test coverage, rather than testing each tool separately. The GUI agents augmented with the generated tools achieved approximately 4.47%p higher performance with approximately 20.05% fewer interactions than the GUI-only agents, averaged across representative benchmarks: AndroidWorld, B-MoCA, and MobileSafetyBench.

Figures & tables

Baseline	AW	BMC	MSB	Overall $\uparrow$	Step $\downarrow$
GUI-only	$69.83{\scriptstyle\,\pm\,2.59}$	$78.71{\scriptstyle\,\pm\,1.75}$	$75.67{\scriptstyle\,\pm\,2.08}$	$74.73{\scriptstyle\,\pm\,0.62}$	$8.13{\scriptstyle\,\pm\,0.03}$
GUI+Skill	$66.67{\scriptstyle\,\pm\,5.27}$	$76.19{\scriptstyle\,\pm\,2.43}$	$71.33{\scriptstyle\,\pm\,2.31}$	$71.44{\scriptstyle\,\pm\,3.33}$	$8.31{\scriptstyle\,\pm\,0.36}$
GUI+CLI	$74.43{\scriptstyle\,\pm\,2.17}$	$80.39{\scriptstyle\,\pm\,4.15}$	$76.00{\scriptstyle\,\pm\,3.61}$	$77.01{\scriptstyle\,\pm\,2.44}$	$7.83{\scriptstyle\,\pm\,0.17}$
DroidTool	$\mathbf{77.30}{\scriptstyle\,\mathbf{\pm\,3.26}}$	$\mathbf{82.35}{\scriptstyle\,\mathbf{\pm\,0.84}}$	$\mathbf{77.67}{\scriptstyle\,\mathbf{\pm\,3.21}}$	$\mathbf{79.20}{\scriptstyle\,\mathbf{\pm\,1.53}}$	$\mathbf{6.50}{\scriptstyle\,\mathbf{\pm\,0.13}}$

Table 1: Evaluation of DroidTool on AW, BMC, and MSB. We report task success rates (SR, %), their overall micro average (Overall) across the three benchmarks, and the average number of interaction steps per task (Step; lower is better). Results are reported as mean

\pm

standard deviation over three evaluation runs. The best result is in bold.

Verification	Registered	Tool	Tool-call	Benchmark
Strategy	Tools (#)	Calls (#)	SR (%)	SR (%)
Unit	101	$368.7{\scriptstyle\,\pm\,11.37}$	$74.86{\scriptstyle\,\pm\,0.63}$	$78.61{\scriptstyle\,\pm\,0.86}$
Separate	82	$343.7{\scriptstyle\,\pm\,6.51}$	$78.76{\scriptstyle\,\pm\,0.94}$	$76.22{\scriptstyle\,\pm\,1.24}$
Relation (Ours)	113	$\mathbf{517.0}{\scriptstyle\,\mathbf{\pm\,6.08}}$	$\mathbf{90.20}{\scriptstyle\,\mathbf{\pm\,1.03}}$	$\mathbf{79.20}{\scriptstyle\,\mathbf{\pm\,1.53}}$

Table 2: Comparison of verification strategies of Unit , Separate , and Relation . We report the number of registered tools and examine tool behavior in downstream benchmark evaluations, including the number of tool calls, the success rate (SR) of tool calls, and the SR across the three benchmarks. Results are mean

\pm

standard deviation over three runs. The best result is in bold.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Verification	Registered	Tool	Tool-call	Benchmark
Strategy	Tools (#)	Calls (#)	SR (%)	SR (%)
Unverified	121	$482.7{\scriptstyle\,\pm\,10.26}$	$73.41{\scriptstyle\,\pm\,0.87}$	$74.03{\scriptstyle\,\pm\,0.30}$
Unit	101	$368.7{\scriptstyle\,\pm\,11.37}$	$74.86{\scriptstyle\,\pm\,0.63}$	$78.61{\scriptstyle\,\pm\,0.86}$
Separate	82	$343.7{\scriptstyle\,\pm\,6.51}$	$78.76{\scriptstyle\,\pm\,0.94}$	$76.22{\scriptstyle\,\pm\,1.24}$
Relation (Ours)	113	$517.0{\scriptstyle\,\pm\,6.08}$	$\mathbf{90.20}{\scriptstyle\,\mathbf{\pm\,1.03}}$	$\mathbf{79.20}{\scriptstyle\,\mathbf{\pm\,1.53}}$

Appendix

Table 3: Comparison of tool verification strategies. We compare no verification ( Unverified ) with Unit , Separate , and Relation (Ours). We report registered tools, tool calls, tool-call SR, and downstream benchmark SR. Evaluation metrics are reported as mean

\pm

standard deviation over three runs. The best success rates are shown in bold.

Task group	GUI-only	GUI+Skill	GUI+CLI	DroidTool (Relation)	Separate	Unit	Unverified
app_launcher	1/1 (3/10)	1/1 (3.67/10)	1/1 (4/10)	1/1 (4/10)	1/1 (3.33/10)	1/1 (3/10)	1/1 (3/10)
audio_recorder	1.33/2 (8/16)	1.67/2 (9.33/16)	2/2 (9.5/16)	2/2 (9.5/16)	1.67/2 (8.83/16)	2/2 (8.33/16)	1.33/2 (8.33/16)
broccoli	6.33/10 (19.87/28)	6/10 (19/28)	7/10 (19.47/28)	8/10 * (7.47/28)	4.33/10 * (14.73/28)	5.33/10 * (12.93/28)	4.67/10 * (12.5/28)
calendar	15.33/17 (8.43/17.53)	16/17 (8.82/17.53)	15/17 (9.25/17.53)	15.67/17 * (2.22/17.53)	13.33/17 * (4.2/17.53)	15.33/17 * (7.71/17.53)	14.33/17 * (9.49/17.53)
camera	1/2 (7.5/10)	0.67/2 (7.67/10)	0.33/2 (8.17/10)	1/2 (8.17/10)	1.33/2 (8/10)	1.33/2 (8.17/10)	0.67/2 (7.5/10)
chrome	2/3 (18.67/20.67)	1.67/3 (19.89/20.67)	1.67/3 (19.78/20.67)	1.33/3 (19.67/20.67)	2/3 (19.11/20.67)	2.33/3 (18.78/20.67)	2/3 (18.67/20.67)

Appendix

Table 4: Task-group results on AndroidWorld. Each cell reports the average number of successful tasks over three evaluation runs divided by the number of tasks in the first line, as well as average interaction steps divided by the average maximum interaction-step budget in parentheses in the second line. A superscript * indicates that app-state tools were registered and available for the task group.

Task group	GUI-only	GUI+Skill	GUI+CLI	DroidTool (Relation)	Separate	Unit	Unverified
calculator	18.33/19 (6.67/10.58)	19/19 (6.42/10.58)	17/19 (6.46/10.58)	19/19 (6.51/10.58)	18.33/19 (6.63/10.58)	19/19 (6.72/10.58)	18.33/19 (6.54/10.58)
calendar	0.67/1 (3.33/4)	0.33/1 (3.67/4)	0.67/1 (3.67/4)	0/1 * (4/4)	0.33/1 * (4/4)	0.33/1 * (4/4)	0.33/1 * (4/4)
camera	1/1 (1/4)	1/1 (1/4)	1/1 (1/4)	1/1 (1/4)	1/1 (1/4)	1/1 (1/4)	1/1 (1/4)
chrome	2/3 (3.33/5)	2.33/3 (3.33/5)	2.33/3 (3.33/5)	2/3 (3.33/5)	2/3 (3.44/5)	2/3 (3.44/5)	2.33/3 (3.56/5)
clock	15.33/25 (8.59/11.2)	15.67/25 (8.64/11.2)	15/25 (8.75/11.2)	24.67/25 * (4.89/11.2)	24.67/25 * (4.93/11.2)	24.67/25 * (5.13/11.2)	12/25 * (9.84/11.2)
contacts	1/3 (4.56/4.67)	0/3 (4.67/4.67)	0.67/3 (4.67/4.67)	0.67/3 * (4.56/4.67)	0/3 * (4.67/4.67)	0.33/3 * (4.56/4.67)	0.33/3 * (4.67/4.67)

Appendix

Table 5: Task-group results on B-MoCA. Each cell reports the average number of successful tasks over three evaluation runs divided by the number of tasks in the first line, as well as average interaction steps divided by the average maximum interaction-step budget in parentheses in the second line. A superscript * indicates that app-state tools were registered and available for the task group.

Task group	GUI-only	GUI+Skill	GUI+CLI	DroidTool (Relation)	Separate	Unit	Unverified
cross_app_messages_calendar	1.67/2 (6.83/15)	0.67/2 (12/15)	2/2 (7.5/15)	0/2 * (9.5/15)	0.33/2 * (12/15)	0/2 * (15/15)	0.33/2 * (10.83/15)
cross_app_messages_joplin	8.33/9 (7.74/15)	8/9 (7.59/15)	8.33/9 (7.26/15)	8.67/9 * (8.59/15)	6/9 * (11.74/15)	4.67/9 * (8.59/15)	9/9 * (7.93/15)
joplin	6.67/8 (6.25/15)	7/8 (5.88/15)	6/8 (6.5/15)	8/8 * (7.17/15)	5/8 * (7.29/15)	5.33/8 * (8.75/15)	8/8 * (7.92/15)
map_searching	3/3 (8.67/15)	2.67/3 (9.33/15)	3/3 (9/15)	3/3 * (9.67/15)	3/3 * (10.44/15)	3/3 * (9.89/15)	3/3 * (10/15)
messages	1.67/3 (9.78/15)	1.33/3 (11.33/15)	1.67/3 (9.11/15)	3/3 * (2.78/15)	3/3 * (3.44/15)	3/3 * (3.67/15)	3/3 * (4/15)
photo_deleting_files	2/2 (8.67/15)	2/2 (11/15)	2/2 (11/15)	2/2 * (9.17/15)	1.33/2 * (10.17/15)	2/2 * (10/15)	1.67/2 * (9.33/15)

Appendix

Table 6: Task-group results on MobileSafetyBench (low-risk). Each cell reports the average number of successful tasks over three evaluation runs divided by the number of tasks in the first line, as well as average interaction steps divided by the average maximum interaction-step budget in parentheses in the second line. A superscript * indicates that app-state tools were registered and available for the task group.

Task group	GUI-only	GUI+Skill	GUI+CLI	DroidTool (Relation)	Separate	Unit	Unverified
calendar	0/1 (5/15)	0/1 (7.67/15)	0/1 (5/15)	0.33/1 * (2/15)	0/1 * (7.67/15)	0/1 * (1.33/15)	0/1 * (11.33/15)
cross_app_messages_calendar	1/1 (3/15)	1/1 (5.67/15)	1/1 (3.67/15)	1/1 * (2/15)	1/1 * (6.33/15)	0.33/1 * (13/15)	1/1 * (6.33/15)
cross_app_messages_joplin	7.33/9 (4.48/15)	7.33/9 (5.48/15)	8.67/9 (3.56/15)	6/9 * (4.74/15)	8.33/9 * (4.63/15)	8.33/9 * (5.04/15)	6.67/9 * (4.3/15)
joplin	7/7 (3.33/15)	6.67/7 (4/15)	6.67/7 (3.52/15)	7/7 * (3.33/15)	7/7 * (4.24/15)	7/7 * (3.19/15)	7/7 * (3.19/15)
map_searching	1/3 (7/15)	1/3 (6.44/15)	1/3 (6.67/15)	1/3 * (7.67/15)	1.33/3 * (7.89/15)	1/3 * (8.89/15)	0.67/3 * (8.78/15)
messages	5/6 (1.89/15)	4.67/6 (2.89/15)	5/6 (1.83/15)	5/6 * (2.5/15)	5/6 * (2.28/15)	5/6 * (2.33/15)	5/6 * (2.06/15)

Appendix

Table 7: Task-group results on MobileSafetyBench (high-risk). Each cell reports the average number of successful tasks over three evaluation runs divided by the number of tasks in the first line, as well as average interaction steps divided by the average maximum interaction-step budget in parentheses in the second line. A superscript * indicates that app-state tools were registered and available for the task group.

LLM Agent Tool Generation

See all 11 papers

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
Apr 13, 2026Fei Tang, Zhiqiong Lu, Boxuan Zhang +4GUI Agents Mobile GUI Automation
Zhejiang University
ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents
May 12, 2026Xuhao Hu, Xi Zhang, Haiyang Xu +6Agentic RL RL for GUI Agents
Tongyi Lab, Alibaba Group · Fudan University · Shanghai Artificial Intelligence Laboratory
PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents
Sep 28, 2026Yangqin Jiang, Lingrui Xu, Chao HuangMobile GUI Automation Mobile GUI Agents
The University of Hong Kong

Apr 13, 2026J/K move · Enter open · S save

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu+3

Zhejiang University

GUI Agents Mobile GUI Automation RL for GUI Agents AI Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

GUI agents drive applications through their visual interfaces instead of programmatic APIs, interacting with arbitrary software via taps, swipes, and keystrokes, reaching a long tail of applications that CLI-based agents cannot. Yet progress in this area is bottlenecked less by modeling capacity than by the absence of a coherent full-stack infrastructure: online RL training suffers from environment instability and closed pipelines, evaluation protocols drift silently across works, and trained agents rarely reach real users on real devices. We present ClawGUI, an open-source framework addressing these three gaps within a single harness. ClawGUI-RL provides the first open-source GUI agent RL infrastructure with validated support for both parallel virtual environments and real physical devices, integrating GiGPO with a Process Reward Model for dense step-level supervision. ClawGUI-Eval enforces a fully standardized evaluation pipeline across 6 benchmarks and 11+ models, achieving 95.8% reproduction against official baselines. ClawGUI-Agent brings trained agents to Android, HarmonyOS, and iOS through 12+ chat platforms with hybrid CLI-GUI control and persistent personalized memory. Trained end to end within this pipeline, ClawGUI-2B achieves 17.1% Success Rate on MobileWorld GUI-Only, outperforming the same-scale MAI-UI-2B baseline by 6.0%.

Abstract

LLM Agent Tool Generation

Explore similar work

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents

PhoneCLI: From App Interfaces to Callable Commands for Mobile Agents