When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench
Organizations: Seoul School of Integrated Sciences and Technologies (aSSIST), Seoul, Republic of Korea. · Present address: AIM Intelligence, Seoul, Republic of Korea.
Abstract
Agentic workflows, in which a large language model (LLM) calls tools to read and write data, are evaluated in two ways. Benchmarks compare the final state against a gold answer. Deployed services have none; an LLM judge reads the transcript and decides. Practice treats the two as interchangeable. This paper measures whether they are. We ran 2,160 episodes in tau2-bench's airline and retail domains and recorded every database write. We scored each episode four ways: native scoring against gold state, an LLM judge, that judge given the policy document, and the boundary evaluator, a minimal rule-based check of whether a write was expected against whether one occurred in the trace. Two annotators independently labeled 120 episodes for task correctness, policy compliance, and commitment correctness. The judge repeated its verdict 98.2% of the time but agreed with native scoring on 56.2%. In 11 episodes the agent executed a forbidden write. The judge passed 10 to 11, with or without the policy or transcript. A stronger judge still passed 6. The boundary evaluator, with no model call, caught all 11; no judge significantly exceeded it on human labels. Forbidden writes occurred in 11.1% of episodes without gates. Verification, correction, and safe-termination gates did not reduce them significantly. The traces show a candidate cause: the gates' evidence digest dropped the order status and customer authentication, so valid requests were rejected. The choice of evaluator decides the conclusion; the minimal check exposes failures the judges miss. Traces, code, prompts, and labels are released.
Figures & tables
| Write committed | No write committed | |
| Task expects a write | correct_action | missed_action |
| Task expects no write | premature_commitment | correct_refusal |
| Evaluator | Information access | Input |
| Native scoring | Full gold | Gold final state and required action list |
| LLM judge | No gold, no policy | Transcript and tool-call log |
| Policy-aware judge (judge-P) | No gold, policy | Transcript, tool-call log, policy document |
| Boundary evaluator | Partial gold | Commit outcome from the trace and per-task write expectation |
| Human three-axis labels | Policy, transcript, tool actions, commit facts | No evaluator output, no gold |
| Item | Value |
| Benchmark | sierra-research/tau2-bench @ 363133ada1936491fb5bcec33cd62c3518a99f65 |
| Domains / tasks | airline 50 (tasks.json 0–49) retail 40 (test split of split_tasks.json) |
| Agent models | gpt-4.1-mini ( gpt-4.1-mini-2025-04-14 ) gpt-4.1 ( gpt-4.1-2025-04-14 ) |
| Simulated customer | gpt-4.1-mini ( gpt-4.1-mini-2025-04-14 ) |
| Gate verifier (C1–C3) | gpt-4.1-mini ( gpt-4.1-mini-2025-04-14 ), temperature 0.0, up to 2 corrections |
| LLM judge | gpt-4.1-mini, temperature 0.4, 2 calls per episode |
| Evaluator pair | Agreement | 95% CI |
| native scoring – judge | 0.562 | [0.496, 0.627] |
| native scoring – boundary evaluator | 0.761 | [0.700, 0.820] |
| judge – boundary evaluator | 0.782 | [0.732, 0.829] |
| judge self-agreement (2 calls) | 0.982 | [0.975, 0.988] |
| Evaluator | A1 task | A2 policy | A3 commitment | Overall | False pass (A1) | False fail (A1) |
| native scoring | 0.818 [.750, .882] | 0.653 [.558, .748] | 0.758 [.677, .834] | 0.766 [.689, .839] | 0.070 | 0.112 |
| judge | 0.584 [.479, .687] | 0.672 [.570, .764] | 0.575 [.470, .678] | 0.496 [.399, .595] | 0.399 | 0.018 |
| judge-P | 0.622 [.525, .721] | 0.671 [.577, .758] | 0.600 [.499, .699] | 0.521 [.424, .615] | 0.378 | 0.000 |
| boundary evaluator | 0.766 [.679, .846] | 0.595 [.493, .692] | 0.708 [.615, .795] | 0.664 [.572, .753] | 0.216 | 0.019 |
| Judge (untruncated input) | A1 | A3 | Premature commitments passed (of 11) | Fully correct episodes failed (of 55) |
| gpt-4.1-mini, no policy | 0.604 | 0.581 | 10 | 1 |
| gpt-4.1-mini, policy | 0.615 | 0.598 | 11 | 0 |
| gpt-4.1, no policy | 0.617 | 0.603 | 9 | 4 |
| gpt-4.1, policy | 0.615 | 0.593 | 10 | 3 |
| gpt-5.6 Sol, no policy | 0.737 | 0.673 | 6 | 11 |
| gpt-5.6 Sol, policy | 0.580 | 0.557 | 0 | 40 |
| Domain | Model | Metric | Discordant pairs: event in C0 only / in C3 only | Permutation | Holm- adjusted |
| airline | gpt-4.1 | premature commitment | 9 / 9 | 1.000 | 1.000 |
| airline | gpt-4.1 | boundary success | 21 / 14 | 0.353 | 1.000 |
| airline | gpt-4.1-mini | premature commitment | 11 / 4 | 0.121 | 0.846 |
| airline | gpt-4.1-mini | boundary success | 21 / 20 | 1.000 | 1.000 |
| retail | gpt-4.1 | premature commitment | 1 / 0 | 1.000 | 1.000 |
| retail | gpt-4.1 | boundary success | 17 / 4 | 0.030 | 0.240 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Role | Model identifier |
| Agent (gpt-4.1-mini) | gpt-4.1-mini-2025-04-14 |
| Agent (gpt-4.1) | gpt-4.1-2025-04-14 |
| Simulated customer | gpt-4.1-mini-2025-04-14 |
| Gate verifier / correction (C1–C3) | gpt-4.1-mini-2025-04-14 |
| Judge and policy-aware judge (main experiment, 2,160 episodes) | gpt-4.1-mini-2025-04-14 , temperature 0.4, 2 calls per episode |
| Additional judges (labeled sample, A.3) | gpt-4.1-2025-04-14 (temperature 0.4); gpt-5.6-sol (reasoning model; temperature parameter unsupported and omitted; provider-default reasoning effort); 2 calls per episode each |
| Evaluator | A1 | A2 | A3 | Overall | False pass (A1) | False fail (A1) | Premature commitments passed | Human A3 fails caught (of 53) |
| Judge (main text, length-limited input) | 0.584 | 0.672 | 0.575 | 0.496 | 0.399 | 0.018 | 11/11 | 8 |
| Judge, untruncated | 0.604 [.508, .700] | 0.680 [.582, .768] | 0.581 [.485, .677] | 0.502 | 0.385 | 0.011 | 10/11 | 6 |
| Judge-P (main text, length-limited input) | 0.622 | 0.671 | 0.600 | 0.521 | 0.378 | 0.000 | 11/11 | 9 |
| Judge-P, untruncated | 0.615 [.521, .709] | 0.671 [.573, .763] | 0.598 [.501, .694] | 0.513 | 0.385 | 0.000 | 11/11 | 7 |
| Judge (untruncated input) | A1 | A2 | A3 | Overall | False pass (A1) | False fail (A1) | Premature commitments passed | Human A3 fails caught (of 53) |
| gpt-4.1-mini, no policy | 0.604 [.508, .700] | 0.680 [.582, .768] | 0.581 [.485, .677] | 0.502 | 0.385 | 0.011 | 10/11 | 6 |
| gpt-4.1-mini, policy | 0.615 [.521, .709] | 0.671 [.573, .763] | 0.598 [.501, .694] | 0.513 | 0.385 | 0.000 | 11/11 | 7 |
| gpt-4.1, no policy | 0.617 [.512, .719] | 0.667 [.567, .759] | 0.603 [.499, .705] | 0.543 | 0.331 | 0.053 | 9/11 | 14 |
| gpt-4.1, policy | 0.615 [.513, .715] | 0.666 [.570, .758] | 0.593 [.490, .694] | 0.542 | 0.340 | 0.044 | 10/11 | 10 |
| gpt-5.6 Sol, no policy | 0.737 [.650, .819] | 0.599 [.506, .694] | 0.673 [.583, .759] | 0.685 | 0.131 | 0.132 | 6/11 | 34 |
| gpt-5.6 Sol, policy | 0.580 [.482, .671] | 0.460 [.360, .563] | 0.557 [.466, .647] | 0.647 | 0.000 | 0.420 | 0/11 | 51 |
| Axis | Round | Raw agreement | Cohen’s | Gwet’s AC 1 |
| A1 | 1 | 0.842 | 0.686 | 0.685 |
| A2 | 1 | 0.650 | 0.317 | 0.315 |
| A2 | 2 (after recalibration) | 0.742 | 0.306 | 0.595 |
| A3 | 1 | 0.750 | 0.477 | 0.526 |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 562 papers