Evaluating GUI agents presents a distinct challenge: trajectories are long, visually grounded, and open-ended, yet evaluation must be both accurate and interpretable. Existing approaches typically apply a single holistic judgment over the entire action-observation sequence-a strategy that proves unreliable on long-horizon tasks and yields binary verdicts offering no insight into where or why an agent fails. This opacity limits the utility of evaluation as a diagnostic tool for agent development. We introduce GUIDE (GUI Understanding and Interpretable Diagnostic Evaluation), a framework that decomposes trajectory assessment into three sequential stages mirroring the compositional structure of GUI tasks. Trajectory Segmentation partitions the full trace into semantically coherent subtask units. Subtask Diagnosis evaluates each unit in context, assigning a completion verdict and generating a structured error analysis with corrective recommendations. Overall Summary aggregates per-subtask diagnoses into a task-level judgment. By operating on bounded subtask segments rather than full trajectories, GUIDE mitigates the context overload that degrades existing evaluators as task complexity grows. We validate GUIDE on three benchmarks: an industrial e-commerce dataset of 932 trajectories, AGENTREWARDBENCH spanning five web agent tasks with 1302 trajectories, and AndroidBench for mobile device control. Across all settings, GUIDE substantially outperforms existing evaluators-achieving up to 5.35 percentage points higher accuracy than the strongest baseline-while producing structured diagnostic reports that directly inform agent improvement.
Figures & tables
Figure 1. Comparison of GUI agent evaluation paradigms. From left to right: (1) text-only LLM evaluators that ignore visual state; (2) lightweight multimodal methods (AgentTrek) that append only the final screenshot; (3) WebJudge, which retrieves key frames from the trajectory and evaluates them holistically in a single context; and (4) GUIDE (ours), which decomposes the trajectory into subtask segments and applies structured diagnosis to each, combining full visual coverage with bounded per-call context.
Figure 2. Overview of the GUIDE framework. Given a task description and a full agent trajectory, Module 1 ( Trajectory Segmentation ) partitions the action-observation trace into semantically coherent subtask units. Module 2 ( Subtask Diagnosis ) evaluates each unit independently, producing a structured diagnosis with a completion verdict, error analysis, and corrective recommendations. Module 3 ( Overall Summary ) aggregates the per-subtask diagnoses into a final task-level judgment. A pipeline diagram showing the GUIDE framework with three sequential modules: Module 1 segments a full GUI agent trajectory into subtask units, Module 2 diagnoses each subtask independently producing structured verdicts and error analyses, and Module 3 aggregates the diagnoses into a final task-level success or failure judgment.
Figure 3. Statistics of the industrial e-commerce dataset. (a) Trajectory length distribution across six groups. (b) Overall success/failure ratio (37.0% vs. 63.0%). (c) Per-group success rate; success decreases monotonically from 39.4% ( < 10 steps) to 18.2% (50–80 steps), confirming that longer trajectories are inherently more challenging.
Method
Acc ↑
P ↑
R ↑
F1 ↑
AgentTrek
65.02
51.42
100.00
67.91
Plan&Solve-like
82.94
72.91
85.80
78.83
Autonomous Eval
84.44
77.93
80.87
79.37
WebJudge
90.45
84.04
91.59
87.66
GUIDE (Ours)
95.80
95.00
93.62
94.31
Table 1. Results on the industrial e-commerce dataset. Best per column in bold ; ↑ = higher is better.
Overall
Per-Benchmark Precision ↑
Method
P ↑
Asst.
VWA
WA
W.Arena
W.Arena ++
Rule-based †
83.8
25.0
85.2
79.0
100.0
83.3
NNetNav †
52.5
20.8
54.5
54.3
77.3
43.2
Autonomous Eval †
67.6
83.3
61.2
67.6
96.4
59.3
GPT-4o (A11y Tree) †
69.8
77.8
63.0
70.2
94.6
63.0
Claude 3.7 Sonnet †
69.4
71.4
64.8
69.3
85.3
66.7
Table 2. Results on AGENTREWARDBENCH (1,302 trajectories, 5 web benchmarks). Methods marked † are taken from ( Lù et al., 2025 ) . Per-benchmark columns report Precision (%).
Method
Overall
AitW
AutoUI-b
AutoUI-l
CogAgent
QWen-VL †
70.2
—
—
—
—
GPT-4V †
90.6
—
—
—
—
Cap.+GPT-4 †
89.8
—
—
—
—
Cap.+Mixtral †
92.9
—
—
—
—
AgentTrek
92.4±0.1
86.1±0.5
93.9±0.5
99.2±0.0
90.3±0.5
WebJudge
92.6±0.5
83.6±1.0
93.3±1.7
100.0±0.0
93.3±0.8
Table 3. Accuracy (%) on AndroidBench ( Pan et al., 2024 ) (480 trajectories, 4 agent policies). Methods marked † are from Pan et al. ( Pan et al., 2024 ) and use different backbones. Remaining methods use gemini-3.0-flash ; all values show mean ±std over 3 independent runs.
Method
< 10 steps
10–20
20–30
30–40
40–50
50–80
AgentTrek
81.4
62.2
55.9
54.0
48.0
34.1
Plan&Solve-like
94.1
92.1
88.3
67.1
40.0
27.3
Autonomous Eval
87.9
85.0
75.2
85.5
85.3
81.8
WebJudge
92.9
92.5
89.0
89.5
85.3
75.0
GUIDE (Ours)
97.6
94.9
93.8
96.1
95.9
93.2
Table 4. Accuracy (%) by trajectory length on the industrial e-commerce dataset. GUIDE maintains stable performance across all groups while baselines degrade as trajectories grow longer.
Variant
Acc ↑
P ↑
R ↑
F1 ↑
Naive (single LLM call)
85.61
78.59
84.06
81.23
w/o Seg
74.14
61.40
81.16
69.91
w/o Diag
93.22
88.32
94.20
91.16
w/o Sum
87.41
95.60
69.28
80.34
GUIDE (full)
95.80
95.00
93.62
94.31
Table 5. Ablation study on the industrial e-commerce dataset.
Score
Quality Level
%
5
Highly Usable
96.9
4
Usable
2.5
3
Minor Issues
0.3
2
Risky
0.2
1
Unusable
0.1
Usable ( ≥4 )
99.4
Table 6. Segmentation quality score distribution across 3.3k subtasks on the industrial e-commerce dataset, as assessed by the MLLM evaluator.
GUI agents must reason about how actions transform interface states, but end-to-end success rates entangle this ability with perception, grounding, planning, and recovery. We introduce EvoGUI, a diagnostic framework that converts normalized GUI trajectories into three complementary visual question answering probes: temporal ordering, inverse action/value prediction, and contrastive one-step successor discrimination. Their labels are derived from trajectory order and logged actions, requiring no additional task-label annotation after trajectory normalization. We instantiate EvoGUI-Bench from Mind2Web and WebLINX, yielding 3,000 instances across 120 domains, and evaluate 28 vision-language model configurations zero-shot. The strongest model reaches only 60.4 EvoGain, while model scale and GUI specialization do not reliably predict performance. These results establish EvoGUI-Bench as a scalable diagnostic complement to end-to-end GUI-agent evaluation while exposing substantial headroom in state-transition understanding. The source code is publicly available at https://github.com/Yyhhh6/EvoGUI.
Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, a progressive diagnostic benchmark that evaluates the same annotated dashboard interactions under systematically varied contextual and spatial support. DashAct contains 348 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its three stages evaluate end-to-end execution, verified-context action prediction, and target-conditioned grounding under four visual conditions. Aligned action and spatial annotations connect outcomes across these settings, providing diagnostic evidence for interpreting online failures. Experiments reveal persistent difficulties in action prediction and fine-grained grounding, alongside model-dependent responses to contextual and spatial support. A Qwen-based adaptation study further improves target localization and online Milestone Coverage, illustrating how DashAct can guide practical agent improvements.
Chuhan Zhang, Qi Xie, Ziyue Wang +4
State Key Laboratory of CAD&CG, Zhejiang University · College of Computer Science and Technology, Zhejiang University
Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) functionally equivalent screens are evaluated in isolation, obscuring systematic failure patterns across shared screens; and (2) the long-tailed GUI distribution renders failures on rare but critical screens invisible under standard metrics. To address these issues, we propose GUITAR, a state-centric diagnostic framework that performs structured failure analysis over both states and transitions, using a State Transition Graph (STG) by mapping visually diverse screens to shared functional states. Across 8 agents and 6 tasks from AndroidControl and Mind2Web, GUITAR reveals that 60.4% of failures occur in 20% of states, localizing errors to a small set of bottlenecks. Bottleneck-targeted guidance improves SR by 2.8% and retains a 1.88% average gain across 7 agents under three-fold trajectory-held-out evaluation with fully automatic STGs. These findings demonstrate the diagnostic and actionable value of structure-aware evaluation within the evaluated mobile and web tasks. Code is available at https://github.com/sqzhang-lazy/GUITAR
Shaoqing Zhang, Kehai Chen, Xuefeng Bai +4
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China