Evaluating GUI agents presents a distinct challenge: trajectories are long, visually grounded, and open-ended, yet evaluation must be both accurate and interpretable. Existing approaches typically apply a single holistic judgment over the entire action-observation sequence-a strategy that proves unreliable on long-horizon tasks and yields binary verdicts offering no insight into where or why an agent fails. This opacity limits the utility of evaluation as a diagnostic tool for agent development. We introduce GUIDE (GUI Understanding and Interpretable Diagnostic Evaluation), a framework that decomposes trajectory assessment into three sequential stages mirroring the compositional structure of GUI tasks. Trajectory Segmentation partitions the full trace into semantically coherent subtask units. Subtask Diagnosis evaluates each unit in context, assigning a completion verdict and generating a structured error analysis with corrective recommendations. Overall Summary aggregates per-subtask diagnoses into a task-level judgment. By operating on bounded subtask segments rather than full trajectories, GUIDE mitigates the context overload that degrades existing evaluators as task complexity grows. We validate GUIDE on three benchmarks: an industrial e-commerce dataset of 932 trajectories, AGENTREWARDBENCH spanning five web agent tasks with 1302 trajectories, and AndroidBench for mobile device control. Across all settings, GUIDE substantially outperforms existing evaluators-achieving up to 5.35 percentage points higher accuracy than the strongest baseline-while producing structured diagnostic reports that directly inform agent improvement.
Figures & tables
Figure 1. Comparison of GUI agent evaluation paradigms. From left to right: (1) text-only LLM evaluators that ignore visual state; (2) lightweight multimodal methods (AgentTrek) that append only the final screenshot; (3) WebJudge, which retrieves key frames from the trajectory and evaluates them holistically in a single context; and (4) GUIDE (ours), which decomposes the trajectory into subtask segments and applies structured diagnosis to each, combining full visual coverage with bounded per-call context.
Figure 2. Overview of the GUIDE framework. Given a task description and a full agent trajectory, Module 1 ( Trajectory Segmentation ) partitions the action-observation trace into semantically coherent subtask units. Module 2 ( Subtask Diagnosis ) evaluates each unit independently, producing a structured diagnosis with a completion verdict, error analysis, and corrective recommendations. Module 3 ( Overall Summary ) aggregates the per-subtask diagnoses into a final task-level judgment. A pipeline diagram showing the GUIDE framework with three sequential modules: Module 1 segments a full GUI agent trajectory into subtask units, Module 2 diagnoses each subtask independently producing structured verdicts and error analyses, and Module 3 aggregates the diagnoses into a final task-level success or failure judgment.
Figure 3. Statistics of the industrial e-commerce dataset. (a) Trajectory length distribution across six groups. (b) Overall success/failure ratio (37.0% vs. 63.0%). (c) Per-group success rate; success decreases monotonically from 39.4% ( < 10 steps) to 18.2% (50–80 steps), confirming that longer trajectories are inherently more challenging.
Method
Acc ↑
P ↑
R ↑
F1 ↑
AgentTrek
65.02
51.42
100.00
67.91
Plan&Solve-like
82.94
72.91
85.80
78.83
Autonomous Eval
84.44
77.93
80.87
79.37
WebJudge
90.45
84.04
91.59
87.66
GUIDE (Ours)
95.80
95.00
93.62
94.31
Table 1. Results on the industrial e-commerce dataset. Best per column in bold ; ↑ = higher is better.
Overall
Per-Benchmark Precision ↑
Method
P ↑
Asst.
VWA
WA
W.Arena
W.Arena ++
Rule-based †
83.8
25.0
85.2
79.0
100.0
83.3
NNetNav †
52.5
20.8
54.5
54.3
77.3
43.2
Autonomous Eval †
67.6
83.3
61.2
67.6
96.4
59.3
GPT-4o (A11y Tree) †
69.8
77.8
63.0
70.2
94.6
63.0
Claude 3.7 Sonnet †
69.4
71.4
64.8
69.3
85.3
66.7
Table 2. Results on AGENTREWARDBENCH (1,302 trajectories, 5 web benchmarks). Methods marked † are taken from ( Lù et al., 2025 ) . Per-benchmark columns report Precision (%).
Method
Overall
AitW
AutoUI-b
AutoUI-l
CogAgent
QWen-VL †
70.2
—
—
—
—
GPT-4V †
90.6
—
—
—
—
Cap.+GPT-4 †
89.8
—
—
—
—
Cap.+Mixtral †
92.9
—
—
—
—
AgentTrek
92.4±0.1
86.1±0.5
93.9±0.5
99.2±0.0
90.3±0.5
WebJudge
92.6±0.5
83.6±1.0
93.3±1.7
100.0±0.0
93.3±0.8
Table 3. Accuracy (%) on AndroidBench ( Pan et al., 2024 ) (480 trajectories, 4 agent policies). Methods marked † are from Pan et al. ( Pan et al., 2024 ) and use different backbones. Remaining methods use gemini-3.0-flash ; all values show mean ±std over 3 independent runs.
Method
< 10 steps
10–20
20–30
30–40
40–50
50–80
AgentTrek
81.4
62.2
55.9
54.0
48.0
34.1
Plan&Solve-like
94.1
92.1
88.3
67.1
40.0
27.3
Autonomous Eval
87.9
85.0
75.2
85.5
85.3
81.8
WebJudge
92.9
92.5
89.0
89.5
85.3
75.0
GUIDE (Ours)
97.6
94.9
93.8
96.1
95.9
93.2
Table 4. Accuracy (%) by trajectory length on the industrial e-commerce dataset. GUIDE maintains stable performance across all groups while baselines degrade as trajectories grow longer.
Variant
Acc ↑
P ↑
R ↑
F1 ↑
Naive (single LLM call)
85.61
78.59
84.06
81.23
w/o Seg
74.14
61.40
81.16
69.91
w/o Diag
93.22
88.32
94.20
91.16
w/o Sum
87.41
95.60
69.28
80.34
GUIDE (full)
95.80
95.00
93.62
94.31
Table 5. Ablation study on the industrial e-commerce dataset.
Score
Quality Level
%
5
Highly Usable
96.9
4
Usable
2.5
3
Minor Issues
0.3
2
Risky
0.2
1
Unusable
0.1
Usable ( ≥4 )
99.4
Table 6. Segmentation quality score distribution across 3.3k subtasks on the industrial e-commerce dataset, as assessed by the MLLM evaluator.