Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

Penny Chong; Harshavardhan Abichandani; Jiyuan Shen; Atin Ghosh; Min Pyae Moe; Yifan Mai; Daniel Dahlmeier

cs.AIMar 16, 2026

Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

Authors: Penny Chong, Harshavardhan Abichandani, Jiyuan Shen, Atin Ghosh, Min Pyae Moe, Yifan Mai, Daniel Dahlmeier

Organizations: SAP · Stanford University

Abstract

Agent applications are increasingly adopted to automate workflows across diverse tasks. However, due to the heterogeneous domains they operate in, it is challenging to create a scalable evaluation framework. Prior works each employ their own methods to determine task success, such as database lookups, regex match, etc., adding complexity to the development of a unified agent evaluation approach. Moreover, they do not systematically account for the user's role nor expertise in the interaction, providing incomplete insights into the agent's performance. We argue that effective agent evaluation goes beyond correctness alone, incorporating conversation quality, efficiency and systematic diagnosis of agent errors. To address this, we introduce the TED framework (Talk, Evaluate, Diagnose). (1) Talk: We leverage reusable, generic expert and non-expert user persona templates for user-agent interaction. (2) Evaluate: We adapt existing datasets by representing subgoals-such as tool signatures, and responses-as natural language grading notes, evaluated automatically with LLM-as-a-judge. We propose new metrics that capture both turn efficiency and intermediate progress of the agent complementing the user-aware setup. (3) Diagnose: We introduce an automated error analysis tool that analyzes the inconsistencies of the judge and agents, uncovering common errors, and providing actionable feedback for agent improvement. We show that our TED framework reveals new insights regarding agent performance across models and user expertise levels. We also demonstrate potential gains in agent performance with peaks of 8-10% on our proposed metrics after incorporating the identified error remedies into the agent's design.

Figures & tables

Figure 1: Our proposed two-step automated error discovery approach that automatically identifies common errors of the agent based on judge and agent inconsistencies. Identical error colors indicate that similar low-level errors are clustered into the same high-level category.

$\tau^{2}$ -bench Airline Domain (Easy)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$pass@k$
gpt-4.1	0.95 \| 0.82	1.00 \| 1.00	0.99 \| 0.81	0.80 \| 0.50	1.00 \| 1.00
gpt-4o	0.79 \| 0.86	1.00 \| 1.00	0.96 \| 0.86	0.70 \| 0.53	1.00 \| 1.00
gpt-4o-mini	0.70 \| 0.61	0.90 \| 0.90	0.85 \| 0.73	0.60 \| 0.37	0.80 \| 0.80
gpt-5	0.92 \| 0.92	1.00 \| 1.00	0.97 \| 0.88	0.67 \| 0.54	1.00 \| 1.00
mistral-nemo	0.87 \| 0.49	1.00 \| 0.80	0.97 \| 0.67	0.67 \| 0.48	1.00 \| 0.60

Table 1: Overall performance of different agent models on

\tau^{2}

-bench airline and ToolSandbox, using gpt-4.1 as user proxy and LLM-as-a judge. Results are displayed with scores for Expert Persona | Non-expert Persona. For metrics with

@k

, the number of trials is

n=k=20

for

\tau^{2}

-bench and

n=k=8

for ToolSandbox.

MaxProgressRate@k

is abbreviated as

MaxProg@k

Panel of Figure 2 — Figure 2: Progress curves for selected ToolSandbox samples. (a) search_reminder_with_recency_upcoming : mistral-nemo (non-expert, purple; AUC =0.88, PPT =0.20) vs. gpt-5 (non-expert, blue; AUC =0.61, PPT =0.20). (b) find_current_city_low_battery_mode : mistral-nemo (expert, purple; AUC =0.77) vs. gpt-5 (non-expert, blue; AUC =0.64). (c) add_reminder_content_and_date_and_time : gpt-4.1 (non-expert, green; AUC =0.50) vs. mistral-large (non-expert, red; AUC =0.34).

Figure 3:

\mathbb{E}[progress(i,G_{i},\tau_{i}^{l})]

(dot) and

\mathrm{Var}[progress(i,G_{i},\tau_{i}^{l})]

(error bar) on

\tau^{2}

-bench gpt-4o-mini agent using non-expert user proxy gpt-4.1. Each dot is an agent trajectory from a single trial and each task sample from the airline domain is evaluated using

n=k=20

agent trials. We display only three example samples here. Sample 14 belongs to the hard split and others in the easy split.

Agent Model	$\tau^{2}$ -bench Airline Domain (Easy)			$\tau^{2}$ -bench Airline Domain (Easy + Hard)
	$pass@k$	$pass$ ^ $4$	$pass$ ^ $k$	$pass@k$	$pass$ ^ $4$	$pass$ ^ $k$
gpt-4.1	1.00	0.60	0.20	0.76	0.29	0.10
gpt-4o	0.80	0.40	0.20	0.57	0.13	0.05
gpt-4o-mini	1.00	0.28	0.20	0.67	0.10	0.05
mistral-nemo	0.80	0.06	0.00	0.43	0.01	0.00
mistral-large	1.00	0.16	0.00	0.71	0.07	0.00

Table 2: Comparison of different agent models using the evaluation paradigm from the

\tau^{2}

-bench ( Barres et al., 2025 ) original paper and their performances on the

pass@k

pass

4

, and

pass

k

metrics in the airline domain. The number of trials used is

n=k=20

Agent Model	$\tau^{2}$ -bench Retail Domain
	$pass@k$	$pass$ ^ $4$	$pass$ ^ $k$
gpt-4.1	1.00	0.54	0.20
gpt-4o	1.00	0.43	0.04
gpt-4o-mini	0.92	0.23	0.04
mistral-nemo	0.76	0.08	0.00
mistral-large	0.96	0.33	0.08

Table 3: Comparison of different agent models using the evaluation paradigm from the

\tau^{2}

-bench ( Barres et al., 2025 ) original paper and their performances on the

pass@k

pass

4

, and

pass

k

metrics in the retail domain. The number of trials used is

n=k=20

Panel of Figure 4 — Figure 4: (a) $\tau^{2}$ -bench airline sample_8 : mistral-nemo expert ( AUC =1.0, PPT =1.0) vs. gpt-4o-mini expert ( AUC =0.982, PPT =0.5) and mistral-nemo non-expert ( AUC =0.482, PPT =0.25) vs. gpt-4o-mini non-expert ( AUC =0.714, PPT =0.125) (b) $\tau^{2}$ -bench airline sample_8 : Barres et al. (2025) ’s original evaluation approach with $n=k=20$ trials.

Agent Model	ToolSandbox Dataset
	(Base Scenarios)	(Base Scenarios + Variants)
	$MaxScore_{M+}@k$	$MaxScore_{M+}@k$
gpt-4.1	0.92	0.90
gpt-4o	0.94	0.91
gpt-4o-mini	0.90	0.88
mistral-nemo	0.87	0.85

Table 4: Comparison of different agent models using the evaluation paradigm from the ToolSandbox ( Lu et al., 2024 ) original paper and their performances on the milestone similarity

Score_{M+}

metric. The number of trials used is

n=k=8

Figure 5: $\tau^{2}$ -bench airline sample 14. The blue box shows a truncated task instruction $i\in I$ for the non-expert user proxy gpt-4.1 model. The green boxes contain the truncated dialogue for trajectory 6 (left) and agent’s trajectory 18 (right). The agent model is gpt-4o-mini. The top-right box shows the errors identified. Zoom in for a larger view.

$\tau^{2}$ -bench Airline Domain (Special Split: Samples 7, 14, 21, 23, and 29)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$
gpt-4o-mini + EI	0.37 $\uparrow$ 0.09 \| 0.31 $\downarrow$ 0.01	0.66 $\uparrow$ 0.06 \| 0.61 $\pm$ 0.00	0.59 $\uparrow$ 0.05 \| 0.39 $\downarrow$ 0.02	0.27 $\uparrow$ 0.03 \| 0.11 $\downarrow$ 0.01
gpt-4o-mini + HN	0.30 $\uparrow$ 0.02 \| 0.33 $\uparrow$ 0.01	0.63 $\uparrow$ 0.02 \| 0.61 $\pm$ 0.00	0.53 $\downarrow$ 0.01 \| 0.45 $\uparrow$ 0.04	0.24 $\pm$ 0.00 \| 0.14 $\uparrow$ 0.02
gpt-4o-mini + HTC	0.29 $\uparrow$ 0.01 \| 0.26 $\downarrow$ 0.06	0.54 $\downarrow$ 0.06 \| 0.51 $\downarrow$ 0.10	0.50 $\downarrow$ 0.04 \| 0.41 $\pm$ 0.00	0.32 $\uparrow$ 0.08 \| 0.13 $\uparrow$ 0.01
gpt-4o-mini + HTC †	0.33 $\uparrow$ 0.05 \| 0.33 $\uparrow$ 0.01	0.61 $\uparrow$ 0.01 \| 0.65 $\uparrow$ 0.04	0.57 $\uparrow$ 0.03 \| 0.43 $\uparrow$ 0.02	0.29 $\uparrow$ 0.05 \| 0.13 $\uparrow$ 0.01
gpt-4.1 + EI	0.52 $\pm$ 0.00 \| 0.38 $\pm$ 0.00	0.78 $\uparrow$ 0.04 \| 0.77 $\downarrow$ 0.08	0.66 $\uparrow$ 0.02 \| 0.41 $\downarrow$ 0.06	0.26 $\downarrow$ 0.01 \| 0.10 $\downarrow$ 0.02

Table 5: Agent improvement results over the baseline agent for Expert Persona | Non-expert Persona, using gpt-4.1 as user proxy and LLM-as-a judge. Blue

\uparrow

shows an improvement over the baseline agent, and red

\downarrow

shows a decrease. We compare the gain in performance for several agent models using the Error Insertion (EI), Human Notes (HN), HiTEC-ICL (HTC) ( Cui et al., 2025 ) , and HTC † variant methods.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

User Persona	Errors
User Persona	Role confusion	Missing or violate instructions	Erroneous responses
Expert	0.0	0.06	0.0
Non-expert	0.0	0.125	0.0

Appendix

Table 6: Correctness of user proxy. Both agent and user proxy use gpt4.1 model.

Dataset	Cohen’s Kappa
$\tau^{2}$ -bench airline (42 subgoals from 10 samples)	0.84
ToolSandbox (31 subgoals from 10 samples)	0.92

Appendix

Table 7: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-5 model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.

Dataset	Cohen’s Kappa
$\tau^{2}$ -bench airline (37 subgoals from 9 samples)	0.71
ToolSandbox (40 subgoals from 10 samples)	0.60

Appendix

Table 8: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent, LLM-as-judge, and user proxy use gpt-4.1 model.

Dataset	Cohen’s Kappa
$\tau^{2}$ -bench airline (27 subgoals from 10 samples)	0.90
ToolSandbox (32 subgoals from 10 samples)	0.80

Appendix

Table 9: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-4o model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.

Dataset	Cohen’s Kappa
$\tau^{2}$ -bench airline (30 subgoals from 10 samples)	0.93
ToolSandbox (35 subgoals from 10 samples)	0.94

Appendix

Table 10: Agreement between human and LLM-as-a-judge measured using Cohen’s Kappa. The agent uses gpt-4o-mini model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.

Dataset	Per-subgoal error
Dataset	Disagreement with human
$\tau^{2}$ -bench airline (37 subgoals from 9 samples)	0.22
ToolSandbox (40 subgoals from 10 samples)	0.23

Appendix

Table 11: Correctness of the identified TED errors. The agent, LLM-as-a-judge, and user proxy use the gpt-4.1 model.

Dataset	Per-subgoal error
Dataset	Disagreement with human
$\tau^{2}$ -bench airline (27 subgoals from 10 samples)	0.19
ToolSandbox (32 subgoals from 10 samples)	0.09

Appendix

Table 12: Correctness of the identified TED errors. The agent uses gpt-4o model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.

Dataset	Per-subgoal error
Dataset	Disagreement with human
$\tau^{2}$ -bench airline (30 subgoals from 10 samples)	0.23
ToolSandbox (35 subgoals from 10 samples)	0.06

Appendix

Table 13: Correctness of the identified TED errors. The agent uses gpt-4o-mini model while the LLM-as-a-judge and the user proxy use gpt-4.1 model.

$\tau^{2}$ -bench Airline Domain (Easy + Hard)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$Pass@k$
gpt-4.1	0.75 \| 0.67	0.94 \| 0.96	0.85 \| 0.68	0.44 \| 0.25	0.81 \| 0.86
gpt-4o	0.63 \| 0.55	0.91 \| 0.88	0.84 \| 0.65	0.44 \| 0.23	0.81 \| 0.71
gpt-4o-mini	0.53 \| 0.53	0.86 \| 0.88	0.79 \| 0.66	0.43 \| 0.26	0.62 \| 0.76
gpt-5	0.80 \| 0.77	0.96 \| 0.97	0.89 \| 0.77	0.47 \| 0.30	0.91 \| 0.91
mistral-nemo	0.67 \| 0.36	0.96 \| 0.71	0.87 \| 0.56	0.44 \| 0.25	0.86 \| 0.38

Appendix

Table 14: Overall performance of different agent models on

\tau^{2}

-bench airline domain, using gpt-4.1 as user proxy and LLM-as-a-judge. Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona | Non-expert Persona. For metrics with

@k

, the number of trials is

n=k=20

Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$Pass@k$
$\tau^{2}$ -bench Retail Domain
gpt-4.1	0.92 \| 0.88	0.99 \| 1.00	0.94 \| 0.83	0.51 \| 0.24	0.92 \| 0.96
gpt-4o	0.87 \| 0.75	1.00 \| 0.99	0.95 \| 0.83	0.54 \| 0.25	1.00 \| 0.96
gpt-4o-mini	0.81 \| 0.76	1.00 \| 0.98	0.95 \| 0.82	0.54 \| 0.25	1.00 \| 0.88
mistral-nemo	0.57 \| 0.58	0.99 \| 0.97	0.86 \| 0.79	0.29 \| 0.23	0.88 \| 0.84
mistral-large	0.80 \| 0.81	1.00 \| 1.00	0.94 \| 0.83	0.49 \| 0.23	0.96 \| 1.00

Appendix

Table 15: Overall performance of different agent models on

\tau^{2}

-bench retail domain, using gpt-4.1 as user proxy and LLM-as-a-judge. Results are displayed with scores for Expert Persona | Non-expert Persona. For metrics with

@k

, the number of trials is

n=k=20

Figure 6: Scatter plot results on the

\tau^{2}

-bench airline domain (easy + hard) illustrating the relationship between

MaxAUC@k

and

MaxProgressRate@k

(left subplot), and between

MaxPPT@k

and

MaxProgressRate@k

(right subplot) using the setting

n=k=20

trials.

Figure 7: Dataset level performance of different agent models on the

\tau^{2}

-bench airline domain (easy), with error bars representing 95% confidence intervals. The top graph shows the

MaxProgressRate@k

, middle graph shows

MaxAUC@k

, bottom graph shows

MaxPPT@k

Figure 8: Dataset level performance of different agent models on the ToolSandbox dataset, with error bars representing 95% confidence intervals. The top graph shows the

MaxProgressRate@k

, middle graph shows

MaxAUC@k

, bottom graph shows

MaxPPT@k

Figure 9: Dataset level performance of different agent models on the

\tau^{2}

-bench airline domain (easy+hard), with error bars representing 95% confidence intervals. The top graph shows the

MaxProgressRate@k

, middle graph shows

MaxAUC@k

, bottom graph shows

MaxPPT@k

Figure 10: Dataset level performance of different agent models on the

\tau^{2}

-bench retail domain, with error bars representing 95% confidence intervals. The top graph shows the

MaxProgressRate@k

, middle graph shows

MaxAUC@k

, bottom graph shows

MaxPPT@k

$\tau^{2}$ -bench Airline Domain (Easy + Hard)
User Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$pass@k$
gpt-4.1	0.75 \| 0.67	0.94 \| 0.96	0.85 \| 0.68	0.44 \| 0.25	0.81 \| 0.86
gpt-4o	0.72 \| 0.61	0.95 \| 0.95	0.82 \| 0.66	0.36 \| 0.22	0.81 \| 0.81
gpt-4o-mini	0.73 \| 0.64	0.95 \| 0.95	0.80 \| 0.65	0.35 \| 0.22	0.86 \| 0.86
gpt-5	0.71 \| 0.73	0.92 \| 0.95	0.85 \| 0.83	0.46 \| 0.37	0.76 \| 0.86

Appendix

Table 16: Overall performance of a gpt-4.1 agent with different user proxy models on the

\tau^{2}

-bench airline domain, using gpt-4.1 model as LLM-as-a-judge. Dataset contains easy and hard samples. Results are displayed with scores for Expert Persona | Non-expert Persona. For metrics with

@k

, the number of trials is

n=k=20

Panel of Figure 11 — Figure 11: Example of generated grading notes for the ToolSandbox sample modify_contact_with_message_recency .

Panel of Figure 12 — Figure 12: Example of generated grading notes for the ToolSandbox sample update_contact_relationship_with_relationship_twice_multiple_user_turn .

Panel of Figure 13 — Figure 13: Example of generated grading notes for the ToolSandbox sample find_current_city_low_battery_mode .

Panel of Figure 14 — Figure 14: Example of generated grading notes for the ToolSandbox sample convert_currency .

Panel of Figure 15 — Figure 15: Example of generated grading notes for the ToolSandbox sample search_message_with_recency_oldest .

Panel of Figure 16 — Figure 16: Example of generated grading notes for the ToolSandbox sample search_reminder_with_recency_yesterday .

Panel of Figure 17 — Figure 17: Example of user task instructions for the ToolSandbox samples

Panel of Figure 18 — Figure 18: Example of user task instructions for the $\tau^{2}$ -bench airline samples

Figure 19: Agent dialogue excerpt for gpt-5 and Mistral-Nemo, illustrating how Mistral-Nemo makes tool calls early and quickly satisfies the grading notes, while gpt-5 asks clarifying questions and progresses more gradually. Despite these differing strategies, both agents complete the task in the same number of turns.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

May 21, 2026

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.

Asaf Yehudai, Lilach Eden, Michal Shmueli-Scheuer

IIBM Research

LLM Agent Evaluation AI Agent Benchmarks Automated Evaluation Automated Agent Evaluation

Ask PDF

Apr 6, 2026

GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis

Evaluating GUI agents presents a distinct challenge: trajectories are long, visually grounded, and open-ended, yet evaluation must be both accurate and interpretable. Existing approaches typically apply a single holistic judgment over the entire action-observation sequence-a strategy that proves unreliable on long-horizon tasks and yields binary verdicts offering no insight into where or why an agent fails. This opacity limits the utility of evaluation as a diagnostic tool for agent development. We introduce GUIDE (GUI Understanding and Interpretable Diagnostic Evaluation), a framework that decomposes trajectory assessment into three sequential stages mirroring the compositional structure of GUI tasks. Trajectory Segmentation partitions the full trace into semantically coherent subtask units. Subtask Diagnosis evaluates each unit in context, assigning a completion verdict and generating a structured error analysis with corrective recommendations. Overall Summary aggregates per-subtask diagnoses into a task-level judgment. By operating on bounded subtask segments rather than full trajectories, GUIDE mitigates the context overload that degrades existing evaluators as task complexity grows. We validate GUIDE on three benchmarks: an industrial e-commerce dataset of 932 trajectories, AGENTREWARDBENCH spanning five web agent tasks with 1302 trajectories, and AndroidBench for mobile device control. Across all settings, GUIDE substantially outperforms existing evaluators-achieving up to 5.35 percentage points higher accuracy than the strongest baseline-while producing structured diagnostic reports that directly inform agent improvement.

Yuwen Zhai, Runze Li, Liang Wang +6

Alibaba Group · School of Computer Science and Technology, East China Normal University · Shanghai Innovation Institute

AI Agent Evaluation GUI Agents Long-Horizon Agent Evaluation AI Agent Benchmarks

Ask PDF

May 12, 2026

An Empirical Study of Automating Agent Evaluation

Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.

Kang Zhou, Sangmin Woo, Haibo Ding +15

AWS AI Labs

Agent Evaluation AI Agent Benchmarks Automated Agent Evaluation Automated Evaluation

Ask PDF

$\tau^{2}$ -bench Airline Domain (Easy)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$pass@k$
gpt-4.1	0.95 \| 0.82	1.00 \| 1.00	0.99 \| 0.81	0.80 \| 0.50	1.00 \| 1.00
gpt-4o	0.79 \| 0.86	1.00 \| 1.00	0.96 \| 0.86	0.70 \| 0.53	1.00 \| 1.00
gpt-4o-mini	0.70 \| 0.61	0.90 \| 0.90	0.85 \| 0.73	0.60 \| 0.37	0.80 \| 0.80
gpt-5	0.92 \| 0.92	1.00 \| 1.00	0.97 \| 0.88	0.67 \| 0.54	1.00 \| 1.00
mistral-nemo	0.87 \| 0.49	1.00 \| 0.80	0.97 \| 0.67	0.67 \| 0.48	1.00 \| 0.60

$\tau^{2}$ -bench Airline Domain (Special Split: Samples 7, 14, 21, 23, and 29)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$
gpt-4o-mini + EI	0.37 $\uparrow$ 0.09 \| 0.31 $\downarrow$ 0.01	0.66 $\uparrow$ 0.06 \| 0.61 $\pm$ 0.00	0.59 $\uparrow$ 0.05 \| 0.39 $\downarrow$ 0.02	0.27 $\uparrow$ 0.03 \| 0.11 $\downarrow$ 0.01
gpt-4o-mini + HN	0.30 $\uparrow$ 0.02 \| 0.33 $\uparrow$ 0.01	0.63 $\uparrow$ 0.02 \| 0.61 $\pm$ 0.00	0.53 $\downarrow$ 0.01 \| 0.45 $\uparrow$ 0.04	0.24 $\pm$ 0.00 \| 0.14 $\uparrow$ 0.02
gpt-4o-mini + HTC	0.29 $\uparrow$ 0.01 \| 0.26 $\downarrow$ 0.06	0.54 $\downarrow$ 0.06 \| 0.51 $\downarrow$ 0.10	0.50 $\downarrow$ 0.04 \| 0.41 $\pm$ 0.00	0.32 $\uparrow$ 0.08 \| 0.13 $\uparrow$ 0.01
gpt-4o-mini + HTC †	0.33 $\uparrow$ 0.05 \| 0.33 $\uparrow$ 0.01	0.61 $\uparrow$ 0.01 \| 0.65 $\uparrow$ 0.04	0.57 $\uparrow$ 0.03 \| 0.43 $\uparrow$ 0.02	0.29 $\uparrow$ 0.05 \| 0.13 $\uparrow$ 0.01
gpt-4.1 + EI	0.52 $\pm$ 0.00 \| 0.38 $\pm$ 0.00	0.78 $\uparrow$ 0.04 \| 0.77 $\downarrow$ 0.08	0.66 $\uparrow$ 0.02 \| 0.41 $\downarrow$ 0.06	0.26 $\downarrow$ 0.01 \| 0.10 $\downarrow$ 0.02

$\tau^{2}$ -bench Airline Domain (Easy + Hard)
Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$Pass@k$
gpt-4.1	0.75 \| 0.67	0.94 \| 0.96	0.85 \| 0.68	0.44 \| 0.25	0.81 \| 0.86
gpt-4o	0.63 \| 0.55	0.91 \| 0.88	0.84 \| 0.65	0.44 \| 0.23	0.81 \| 0.71
gpt-4o-mini	0.53 \| 0.53	0.86 \| 0.88	0.79 \| 0.66	0.43 \| 0.26	0.62 \| 0.76
gpt-5	0.80 \| 0.77	0.96 \| 0.97	0.89 \| 0.77	0.47 \| 0.30	0.91 \| 0.91
mistral-nemo	0.67 \| 0.36	0.96 \| 0.71	0.87 \| 0.56	0.44 \| 0.25	0.86 \| 0.38

Agent Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$Pass@k$
$\tau^{2}$ -bench Retail Domain
gpt-4.1	0.92 \| 0.88	0.99 \| 1.00	0.94 \| 0.83	0.51 \| 0.24	0.92 \| 0.96
gpt-4o	0.87 \| 0.75	1.00 \| 0.99	0.95 \| 0.83	0.54 \| 0.25	1.00 \| 0.96
gpt-4o-mini	0.81 \| 0.76	1.00 \| 0.98	0.95 \| 0.82	0.54 \| 0.25	1.00 \| 0.88
mistral-nemo	0.57 \| 0.58	0.99 \| 0.97	0.86 \| 0.79	0.29 \| 0.23	0.88 \| 0.84
mistral-large	0.80 \| 0.81	1.00 \| 1.00	0.94 \| 0.83	0.49 \| 0.23	0.96 \| 1.00

$\tau^{2}$ -bench Airline Domain (Easy + Hard)
User Model	$MeanProg@k$	$MaxProg@k$	$MaxAUC@k$	$MaxPPT@k$	$pass@k$
gpt-4.1	0.75 \| 0.67	0.94 \| 0.96	0.85 \| 0.68	0.44 \| 0.25	0.81 \| 0.86
gpt-4o	0.72 \| 0.61	0.95 \| 0.95	0.82 \| 0.66	0.36 \| 0.22	0.81 \| 0.81
gpt-4o-mini	0.73 \| 0.64	0.95 \| 0.95	0.80 \| 0.65	0.35 \| 0.22	0.86 \| 0.86
gpt-5	0.71 \| 0.73	0.92 \| 0.95	0.85 \| 0.83	0.46 \| 0.37	0.76 \| 0.86

Abstract

LLM Agent Evaluation

Explore similar work

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis

An Empirical Study of Automating Agent Evaluation