How GUI Agents Fail: GraceBench for Execution Stability Diagnosis

Ridong Zhang; Shuxian Liang; Chaoqun Wan; Zhen Huang; Xian-Sheng Hua

cs.AIOct 14, 2026

How GUI Agents Fail: GraceBench for Execution Stability Diagnosis

Authors: Ridong Zhang, Shuxian Liang, Chaoqun Wan, Zhen Huang, Xian-Sheng Hua

Organizations: Institute of AI for Engineering, Tongji University

Abstract

As GUI agents are increasingly deployed for mobile task automation, their reliability depends not only on final task outcomes, but also on whether they reach these outcomes through stable interaction trajectories. In automation workflows, unnecessary actions, detours, and execution errors can add noise to collected data, introduce irreversible side effects, or undermine trust, even when the agent eventually completes the task. It is therefore important to systematically diagnose the execution stability, including whether an agent deviates from a feasible path, enters a loop, becomes stagnant, or recovers from unstable actions. Existing benchmarks, however, primarily assess final task success or a small set of intermediate milestones, leaving step-level execution behavior unclear. To bridge this gap, we introduce GraceBench, an online evaluation framework that represents the reachable interaction space of mobile applications as UI Transition Graphs (UTGs), providing a structural reference for step-level execution diagnosis. To support reliable alignment between dynamic UI observations and graph states, GraceBench uses video-grounded trajectory evaluation over this graph-aligned representation, assessing process stability beyond final success with metrics such as deviation rate, loop burden rate, and recovery ability. GraceBench contains 169 tasks spanning 10 applications and constructs 10 UTGs with 2,909 canonical GUI states and 3,116 observed transitions. Across 3,718 trajectories, GraceBench shows that outcome scores hide execution instability: GPT-5.6 Sol achieves the highest success rate (SR) but the lowest recovery ability, 51.24% of successful runs still contain a deviation or loop, and trajectories with the same success rate (SR) and completion rate (CR) can differ substantially in recovery behavior. Code and data will be released on https://iaie-physicalai.github.io/GraceBench.

Figures & tables

Figure 1: Limitation of existing milestone-based process evaluation. A clean execution and an unstable execution with an intermediate loop can satisfy the same semantic milestones and therefore receive the same outcome. Observing only milestone attainment hides the loop, recovery behavior, and noise introduced along the execution trajectory.

Benchmark	Venue	Process Eval	Online Env.	Multi-Path	Graph Ref.	Stability Metrics	Video Ev.
AndroidWorld ( Rawles et al., 2025 )	ICLR 2025	✗	✓	✗	✗	✗	✗
SPA-Bench ( Chen et al., 2025 )	ICLR 2025	✓	✓	✗	✗	✗	✗
A3 ( Chai et al., 2026a )	Findings ACL 2026	✓	✓	✓	✗	✗	✗
AndroidDaily ( Sui et al., 2026 )	arXiv 2026	✓	✓	✓	✗	✗	✗
ProBench ( Yang et al., 2025 )	AAAI 2026	✓	✓	✗	✗	✗	✗
ColorBench ( Song et al., 2025 )	WWW 2026	✓	✗	✓	✓	✗	✗

Table 1: Comparison with representative GUI-agent benchmarks. Venue denotes the representative publication venue or preprint year; other columns summarize support for process evaluation, online execution, multi-path tasks, graph references, stability metrics, and video evidence.

Figure 2: Overview of GraceBench task curation and evaluation. GraceBench first builds application-level UTGs through agent-driven exploration, curates task-specific subgraphs and instructions, and then evaluates online agent trajectories by mapping step-level video evidence to graph-aligned execution labels and stability metrics.

Figure 3: Distribution of the 169 GraceBench tasks. (a) Functional domains across 10 applications. (b) Reference-path length distribution.

Figure 4: Trajectory-diagnosis examples on task-specific UTGs. (a) Steps 2–3 leave the task subgraph and step 4 recovers, providing the labels used to compute DR, ARL, RA, and RSR. (b) Repeated transitions between the same local state pair form a loop event and determine LBR. Dashed edges denote feasible but unexecuted task-subgraph transitions; stagnation is omitted for clarity.

Detailed instruction
Type	Model	SR $\uparrow$	CR $\uparrow$	DR $\downarrow$	LBR $\downarrow$	RA $\uparrow$	RSR $\uparrow$	ARL $\downarrow$
GUI-specific	GUI-Owl-1.5-8B-Instruct	0.1657	0.4403	0.3306	0.5559	0.6603	0.8212	1.6877
GUI-specific	UI-TARS-1.5-7B	0.1006	0.2956	0.5149	0.5133	0.6017	0.7888	1.9501
General open-source	DeepSeek-V4.1-Flash	0.5148	0.6099	0.1348	0.5934	0.5602	0.6987	1.8716
	GLM-5.3-Flash	0.5976	0.7304	0.0743	0.4470	0.5605	0.6857	1.4583
	Qwen3.8-Flash	0.3254	0.4474	0.1591	0.6606	0.5834	0.7819	1.7755

Table 2: Results for all 3,718 trajectories: eleven models, 169 tasks, and two instruction conditions. All rows use the supplied per-trajectory evaluation summaries, preserving the reported SR labels and process scores. Data provenance is detailed in Appendix C . Rates are decimals; ARL is measured in steps. Within each instruction block, best values are bolded and second-best values are underlined.

Figure 5: Task outcomes hide execution instability. (a) Mutually exclusive process-pattern proportions among successful trajectories. (b) Exact-CR matched pairs with process gap at least 0.2000; bars distinguish successful and failed pairs, and whiskers show task-cluster 95% intervals.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Task domain	Application/scenario assignment	Tasks	Share
Social & Video Entertainment	TikTok; Xiaohongshu; Hongguo Short Drama	33	19.53%
Local Services	Douyin (Group Buying); Dianping	26	15.38%
E-commerce	Douyin (Mall); Douyin Mall	22	13.02%
Account & General Services	Douyin (Home Sidebar; Profile)	16	9.47%
Content Creation	Douyin (Content Creation)	15	8.88%
Music	Qishui Music	14	8.28%

Appendix

Table 3: Task-to-domain mapping used in Figure 3 . Tasks are grouped by their primary user goal. Parenthesized names denote scenario-specific subgraphs within an application; the six Douyin subgraphs remain components of one Douyin application-level UTG.

	Detailed instruction			Concise instruction
Metric	Success	Failure	Gap	Success	Failure	Gap
DR	0.0889	0.3067	+0.2178	0.1412	0.3618	+0.2206
Loop incidence	0.2066	0.6525	+0.4459	0.2748	0.6520	+0.3772
LBR	0.4257	0.6390	+0.2133	0.4840	0.6142	+0.1302
RA	0.7285	0.5668	-0.1618	0.6394	0.5918	-0.0476

Appendix

Table 4: Process instability by final task outcome across all 3,718 trajectories. Gaps are unsuccessful minus successful. DR is averaged over all trajectories; loop incidence is the fraction of trajectories containing a loop; LBR is computed only over trajectories in which a loop occurs; RA is computed only over trajectories with recovery opportunities.

Figure 6: Instruction effects across all eleven models. Concise-minus-detailed SR and CR, with task-cluster 95% intervals. Each model has 169 pairs; the pooled estimate has 1,859 pairs. All 88 replacement runs are selected before pairing.

Model	$\Delta$ SR	$\Delta$ CR	$\Delta$ DR	$\Delta$ LBR	$\Delta$ RA
UI-TARS-1.5-7B	+0.0532	+0.0687	-0.0061	-0.0561	-0.0004
GUI-Owl-1.5-8B-Instruct	+0.0710	-0.0010	+0.0501	-0.0518	-0.0163
DeepSeek-V4.1-Flash	-0.1006	-0.0936	+0.0780	+0.0173	+0.0418
Qwen3.8-Max	+0.0355	-0.0354	+0.0043	+0.0158	+0.0850
Qwen3.8-27B	-0.0474	-0.0684	+0.0621	+0.0120	-0.0293
GLM-5.3-Flash	+0.0296	-0.0628	+0.0560	-0.0181	-0.0045

Appendix

Table 5: Concise-minus-detailed differences from the displayed four-decimal aggregates in Table 2 . Paired intervals in Figure 6 use the complete per-trajectory records for all eleven models.

Measure	Pairs	Mean change	95% task-cluster interval
DR	426	0.0467	[0.0236, 0.0713]
Loop incidence	426	0.0423	[-0.0072, 0.0922]
LBR	34	0.0339	[-0.0653, 0.1232]
RA	96	-0.0911	[-0.1717, -0.0133]
CR	426	-0.0591	[-0.0879, -0.0325]

Appendix

Table 6: Paired process changes when both descriptions succeed. Opportunity-dependent comparisons use their own common support.

Excluded unit	SR range	CR range	DR range
One model	[0.0024, 0.0195]	[-0.0427, -0.0218]	[0.0485, 0.0581]
One scenario	[-0.0057, 0.0183]	[-0.0365, -0.0153]	[0.0345, 0.0598]

Appendix

Table 7: Leave-one-group-out sensitivity of concise-minus-detailed effects.

Figure 7: Scenario-specific paired instruction effects. All eleven models are included. Intervals resample task clusters; small scenarios have correspondingly less precise estimates. These comparisons are exploratory and unadjusted for multiplicity.

Figure 8: Outcome-controlled process diagnostics on all 3,718 trajectories. (a) Exact-CR matched pairs with process gap at least 0.2000; bars distinguish successful and failed pairs and whiskers are task-cluster 95% intervals. (b) Upper triangle: common-support Spearman correlation; lower triangle: residual correlation controlling task, model, condition, SR, and CR. (c–d) Threshold sensitivity for exact-CR matched pairs; shaded bands show task-cluster 95% bootstrap intervals.

Outcome	Metric	Large / eligible	Fraction	95% interval
Success	DR	410/1779	0.2305	[0.1815, 0.2806]
Success	LBR	42/101	0.4158	[0.2778, 0.5333]
Success	RA	153/287	0.5331	[0.4492, 0.6227]
Failure	DR	926/1939	0.4776	[0.4417, 0.5128]
Failure	LBR	425/856	0.4965	[0.4567, 0.5348]
Failure	RA	860/1444	0.5956	[0.5613, 0.6298]

Appendix

Table 8: Exact-CR matched pairs with process gap at least 0.2000. Pair counts are not independent sample sizes; intervals cluster by task.

Metric	Detailed $\rho$	Permutation $p$	Concise $\rho$	Permutation $p$
DR	0.9516 [0.8636, 0.9909]	$<0.0001$	0.9228 [0.8182, 0.9727]	$<0.0001$
LBR	0.9284 [0.8182, 0.9909]	$<0.0001$	0.8381 [0.6727, 0.9636]	0.0001
RSR	0.8100 [0.6182, 0.9364]	0.0002	0.8269 [0.7091, 0.9364]	0.0002
ARL	0.5816 [0.2455, 0.8727]	0.0042	0.5880 [0.1998, 0.8636]	0.0039
RA	0.5983 [0.3000, 0.8818]	0.0029	0.8508 [0.6361, 0.9636]	$<0.0001$

Appendix

Table 9: Cross-task reproducibility of model profiles. Mean Spearman

\rho

between eleven-model rankings on disjoint task subsets over 1,000 stratified 84/85-task partitions (higher is more reproducible); brackets show the central 95% of partitions, not confidence intervals. One-sided

p

-values use 200,000 model-identity permutations; all remain significant after Bonferroni correction over ten tests.

Figure 9: Timely path return is associated with task success. (a) Overlap-weighted SR after matching task, instruction condition, deviation-onset bin, and prior-progress bin. (b) Adjusted SR differences under onset and observation-window restrictions; points control task–condition and model fixed effects, onset bin, and prior progress, and whiskers are 95% task-cluster intervals.

Figure 10: Grouped predictive ablation on all 3,718 records. Points show baseline MSE minus augmented-model MSE; positive values favor the augmentation. Whiskers are 95% task-cluster intervals. Labels give relative MSE reduction as decimals. Blue rows add one metric to the availability-aware baseline; green rows add an omitted metric back to the other two.

Target	Added metric	Relative gain	MSE reduction	95% interval
Log duration	DR	0.0336	0.0040	[0.0015, 0.0065]
Log duration	LBR	0.0431	0.0052	[0.0033, 0.0071]
Log duration	RA	0.0642	0.0079	[0.0057, 0.0103]
Steps	DR	0.0571	0.4599	[0.2865, 0.6385]
Steps	LBR	0.0416	0.3296	[0.2165, 0.4413]
Steps	RA	0.0929	0.7780	[0.5955, 0.9621]

Appendix

Table 10: Conditional leave-one-out gains. Relative gains are decimals; absolute MSE units differ between targets.

Figure 11: All eleven models in the full dataset. Detailed and concise SR, CR, DR, and LBR recomputed from the same 3,718 per-trajectory records used throughout the experiments.

Application	Tasks	States	Transitions	Branch states	Mean / max steps
Douyin	63	827	763	91	6.60 / 11
Dianping	12	206	237	29	9.25 / 10
Douyin Mall	12	268	311	41	6.42 / 12
Fanqie Novel	11	312	361	38	8.00 / 15
Amap	14	171	189	33	7.14 / 9
Hongguo	14	370	399	47	8.93 / 11

Appendix

Table 11: Structural coverage of the ten application-level UTGs. States are canonical GUI states; transitions are distinct observed action-conditioned edges; branch states expose at least two distinct outgoing action–target pairs. Task steps are computed from the 169 curated reference paths and include the cold start as one action.

Measure	Mean	95% CI	Median	IQR	Range
Core task states	5.91	[5.62, 6.18]	6	[5, 7]	[1, 11]
Additional one-hop states	35.19	[32.88, 37.66]	34	[24, 43]	[6, 102]
One-hop closure	41.09	[38.62, 43.69]	40	[30, 49]	[7, 112]
Per-task expansion factor	7.20 $\times$	[6.82, 7.59]	7.14 $\times$	[5.33, 8.80]	[2.80, 15.50]

Appendix

Table 12: Canonical states supporting each task subgraph. Statistics cover all 169 tasks. Confidence intervals are nonparametric task-bootstrap intervals with 10,000 resamples and seed 20260925.

Field	Cohen’s $\kappa$
State match	0.7980
Deviation	0.7952
Recovery	0.7663
Loop	0.8656

Appendix

Table 13: Agreement between generated and human expert labels. Cohen’s

\kappa

compares expert-agent generated binary labels with human-adjudicated reference labels over the 3,487 sampled transitions.

Computer-Use Agent Benchmarks

Computer-use agent benchmarks test whether AI systems can operate graphical interfaces, terminals, and applications reliably, safely, and efficiently.

See all 119 papers

GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions
Sep 28, 2026Shaoqing Zhang, Kehai Chen, Xuefeng Bai +4Agent Failure Analysis GUI Agents
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China
Xiaomi-GUI-0 Technical Report
Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Mobile GUI Automation RL for GUI Agents
APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
Sep 7, 2026Jintian Feng, Long Chen, Xiao Yu +7Mobile GUI Automation GUI Agents
Laboratory for Artificial Intelligence and New Forms of Education, Central China Normal University · Faculty of Artificial Intelligence in Education, Central China Normal University · Agentic Labs, Acrab AI

Sep 28, 2026J/K move · Enter open · S save

GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

Shaoqing Zhang, Kehai Chen, Xuefeng Bai, Zhuosheng Zhang+3

Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China

Agent Failure Analysis GUI Agents AI Agent Evaluation

Open paper AskPDF arXiv

Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) functionally equivalent screens are evaluated in isolation, obscuring systematic failure patterns across shared screens; and (2) the long-tailed GUI distribution renders failures on rare but critical screens invisible under standard metrics. To address these issues, we propose GUITAR, a state-centric diagnostic framework that performs structured failure analysis over both states and transitions, using a State Transition Graph (STG) by mapping visually diverse screens to shared functional states. Across 8 agents and 6 tasks from AndroidControl and Mind2Web, GUITAR reveals that 60.4% of failures occur in 20% of states, localizing errors to a small set of bottlenecks. Bottleneck-targeted guidance improves SR by 2.8% and retains a 1.88% average gain across 7 agents under three-fold trajectory-held-out evaluation with fully automatic STGs. These findings demonstrate the diagnostic and actionable value of structure-aware evaluation within the evaluated mobile and web tasks. Code is available at https://github.com/sqzhang-lazy/GUITAR

Abstract

Computer-Use Agent Benchmarks

Explore similar work

GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

Xiaomi-GUI-0 Technical Report

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents