cs.AINov 3, 2026

Silent Instrument Errors Reorder Agent Leaderboards

Authors: Yangze Liu

Organizations: Shandong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

On a public agent leaderboard, a general-purpose agent trails a scaffold built for an airline customer-service benchmark by 13 points on the tasks that touch its database, and a reader could conclude that generality costs accuracy. The gap comes from the grader. The general agent often ends its conversations in a way that never triggers grading, so only about half of its episodes were scored and the rest were stored as failures. Grading the ungraded half under the benchmark's own rule closes the pooled gap, without rerunning any agent. We audit two carefully maintained public leaderboards, HAL and tau2-bench, from their published artifacts alone and find three more instrument errors. A cost column counts nested model calls twice, unevenly across rows. Archived trajectories record grader-written actions as agent actions. Re-grading saved trajectories under a newer evaluator is itself a new measurement: a correct and announced fix reorders a leaderboard, and a mode check that misreads four files collapses their scores by 59 to 82 points while printing no error. Each produced a plausible number, and the HAL maintainers confirmed their three within five weeks. Together the cases show that the infrastructure of agent evaluation, meaning the code that grades, logs, replays and versions an evaluation, is part of the measurement and fails silently, so scores and orderings can move with no visible sign. These failure modes follow from how agent evaluation pipelines are built, and those pipelines change as fast as the agents they measure, so we expect more of them elsewhere. We propose cheap checks that expose the instrument: report the graded fraction, aggregate cost from root spans, tag the provenance of every recorded action, and pin evaluator versions with a published re-grade diff.

Figures & tables

CaseWhat the score claimedWhat the instrument didHow much it moved
HAL costDollars spent by the agent in a runSummed usage over a wrapper span and its child span, which carry the same responseExactly 2×2\times in 70 of 134 runs, 1×1\times in 46
HAL Generalist gradingAccuracy of the Generalist scaffoldGraded only episodes the agent transferred or the simulated user closed, stored the rest as failures0.30 to 0.43, closing a 13-point gap to Tool-Calling
HAL trajectoriesActions taken by the agentWrote the grader’s gold replay into the agent’s action listPresent in 68% of 2,334 airline records
τ2\tau^{2} -bench re-gradeThe same trajectory scored againApplied new criteria or rebuilt the environment without the user’s tools−7-7 to +2.5+2.5 points on airline, up to −82-82 on telecom
Table 1: The four cases. In none of them did the pipeline raise an error, and each produced a number in the range a reader would expect.
Figure 1: Generalist against Tool-Calling on HAL TAU-bench Airline, for the nine models run under both, on records whose gold contains at least one non-terminal action, pooled over each model’s runs. Arrows run from the Generalist’s published accuracy to its accuracy when every episode is graded. Triangles are Tool-Calling after the same re-grading, which moves it by at most 2.5 points. The top row pools all nine models, where the published gap closes to a tie. The asterisk marks the one model on which the two scaffolds change order. Each per-model row rests on 40 to 82 Generalist records. Few-Shot, which HAL withdrew, is in Table 2 .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

 GeneralistTool-Generalist,Generalist,Few-Shot
ModelrecordsCallingpublishedall graded(withdrawn)
claude-3.7-sonnet810.490.520.600.46
claude-opus-482n/a0.410.500.57
claude-opus-4.1820.510.370.500.55
deepseek-r1410.310.100.240.24
deepseek-v3410.380.170.370.22
Appendix
Table 2: Per-model TAU-bench Airline accuracy on HAL, records whose gold contains at least one non-terminal action, pooled over each model’s runs. Tool-Calling and Few-Shot are shown after re-grading, which leaves Few-Shot unchanged and moves Tool-Calling only for claude-3.7-sonnet (published 0.48) and gpt-4.1 (published 0.27). HAL withdrew Few-Shot for test-set leakage, and its archived traces are shown for reference. claude-opus-4 has no Tool-Calling run.
More in this topic

AI Agent Evaluation

AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.

See all 544 papers

Explore similar work

CardsList
Sep 7, 2026

What Does an LLM-Agent Leaderboard Rank Actually Compare?

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under known finite-sample conditions and show why uncertainty must be included when judging sensitivity to target reweighting. Across SWE-bench, AgentRewardBench, and tau2-bench, close rank differences are often unresolved; proxy labels and utility rules can also change which system is selected. DataAgentBench and Open Agent show what remains estimable from coarser public records. A leaderboard score summarizes a released evaluation, whereas a fine-grained superiority claim additionally depends on the estimand and uncertainty rule used to interpret the difference.

Wei-Jung Huang

Independent Researcher United States

LLM Agent EvaluationAI Agent BenchmarksAutomated Evaluation
AskPDF
Jun 22, 2026

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.

Jeffrey Flynt

Independent Researcher

LLM Agent VerificationAI Agent BenchmarksTool-Use EvaluationLLM GroundingTool-Augmented Language Model AgentsLLM-as-a-Judge+8
AskPDF
Oct 8, 2026

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation. We introduce TRACE, a protocol that turns a score change from a verdict into a testable diagnosis: it applies a targeted change to one part of an evaluation, compares paired runs, checks whether the agent's behavior changed, and rescores unchanged trajectories to test whether the scoring rule is responsible. In a controlled suite of 25 synthetic tasks, renaming tools lowers a scripted agent's score by 0.250 even though it performs exactly the same operations; restoring the original names at scoring time closes the entire gap, while the same mutation exposes a genuine behavioral failure in a second agent. On public τ2τ^2-bench tasks with four LLM agents, an initial 30-task study finds mixed reward changes whose one clear effect does not replicate. In a larger follow-up on 88 new tasks with repeated runs per condition, renaming tools or reformatting tool outputs leaves reward unchanged to within ±\pm0.10 for seven of eight agent-change pairs, whereas tool names that deliberately mislead lower every agent's reward by 0.20-0.44, showing that the setup can detect real effects. Identical reruns flip 15-36% of task outcomes, so single-run comparisons cannot separate presentation effects from run-to-run variation. Two frontier LLM judges give consistent verdicts when a fixed trajectory is presented differently, yet disagree with each other on 57% of the same records, largely because one grades procedure rather than outcome. TRACE thus separates what a score change says about the agent from what it says about the measurement.

Radhika Gaonkar

Prime Intellect

LLM Agent EvaluationAI Agent EvaluationAI Agent Benchmarks
AskPDF