What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
Authors: Idunnuoluwa A. Adeniji, Shilpika, Michael E. Papka
Organizations: Department of Computer Science, University of Illinois Chicago, Chicago, IL, USA · Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA
LLM agents are often evaluated by reducing execution traces to labels such as final-answer correctness, tool verification, and fault propagation. These labels are measurements produced by parsers, behavioral detectors, and harness rules, and they may not fully capture what the trace shows. We study this in a tool-use setting built on GSM8K, where agents call a calculator whose outputs are occasionally corrupted, giving clear ground truth for what the tool returned, what the model later used, and what the detectors reported. We audit three trace-level detectors across six production models and 30 problems per model. Final correctness, visible re-verification, and fault propagation can disagree in systematic ways. A key failure mode is silent tool-output override: the calculator returns a corrupted value, but the model later uses a different, often correct, value without calling the tool again, so the bad value stops propagating while a verification detector records no re-check. Human review of 90 traces agrees perfectly with the automated labels for final correctness (67/67), fault impact (19/19), and fault relevance (18/18), but only 67/81 for verification; all 14 disagreements fall into four recurring patterns. Execution limits also change the reported result: in an ablation across two models, one ranges from 21/30 to 27/30 correct as step and token budgets change, while the other stays between 27/30 and 29/30. These findings motivate measurement transparency: an automated label should come with the trace evidence behind it, whether the question the detector asks applied to that trace, whether the run was cut off by an execution limit, and any human corrections. Our trace viewer makes this information inspectable alongside the reported results.
Figures & tables
Detector
What it reports
Why the label may need qualification
D-correct
Whether the parsed final answer matches the gold answer.
A correct answer can be missed if the response format defeats the parser.
D-verify
Whether the trace contains an explicit calculator re-check or only a claim of checking.
Recovery without a second tool call is not visible to this detector.
D-impact
Whether a corrupted value appears to propagate into later arithmetic.
If neither the corrupted nor corrected value reappears, the propagation question may be inconclusive.
Table 1: Automatic measurements audited in this work.
Table 2: System workflow. Each layer consumes the output of the layer above it. GSM8K questions and seeded calculator faults produce agent execution traces; the measurement layer applies D-correct, D-verify, and D-impact to generate automatic labels and risk flags; the review layer exposes these outputs through a terminal summary and interactive viewer for trace inspection and human comparison.
Class
Issue
Description
F1
Detector blind spot
The behavior occurs in the trace, but the detector’s rule cannot represent it.
F2
Parser loss
The model gives a usable answer, but extraction fails.
F3
Inapplicable question
The detector asks a question that the trace does not answer.
F4
Formatting sensitivity
Small formatting differences change the detector verdict. Example: 184-145 and 184 -145 can yield different detector behavior.
F5
Instrument assumption
The fault injector or measurement tool was assumed correct before being checked.
F6
Rate uncertainty
A single batch percentage is treated as a stable rate without uncertainty.
Table 3: Observed measurement issues in the study.
Figure 1: Interactive trace inspection. The corpus view supports comparison across model runs, while the trace view exposes the evidence behind an individual detector output. In panel (b), the tool returns a corrupted value that the model later replaces without an explicit tool re-check.
Model
Task outcome
Verify
Override
Measurement caveat
claude-sonnet-5
30 / 0 / 0
5/30
4/30
Only perfect-accuracy row: no censoring, no errors, no misses. Overrides visible only at the trace level (F1).
claude-opus-4.8
28 / 2 / 0
2/30 (+3 claimed)
5/30
Override on 5 of 15 fault events; claimed-only verification appears three times; F1 evidence is not model-specific.
gpt-oss-120b
28 / 2 / 0
8/30
0/30
Highest accuracy of the open models, but the propagation denominator is small ( n=8 ): a single trace of noise moves the rate several points (F6).
gpt-oss-20b
24 / 2 / 4
4/30
1/30
12 of 17 fault events inconclusive—the propagation question is unanswerable for most faults on this run (F3).
Mistral-Large-3-675B
20 / 2 / 8
0/30 (+1 claimed)
3/30
Eight backend-error traces are recorded as not completed; recoveries occur with zero visible re-execution anywhere in the run (F1).
nemotron-3-ultra
20 / 0 / 10
11/30
16/30
All 10 misses are step-limit hits; all 20 completed runs are correct (F7). 56 of 155 fault events inconclusive (F3).
Table 4: Cross-model results on 30 questions under the baseline configuration. Task outcome is completed-correct/completed-wrong/not-completed. Verify and override are trace-level counts.
Model
Cell
Acc
Faults
Prop.
Verify
Override
nemotron-3-ultra
steps=25, tokens=1024
23/30
131/874
1/91
11/30
16/30
steps=25, tokens=4096
27/30
496/3524
0/460
9/30
21/30
steps=50, tokens=1024
21/30
174/1148
0/119
12/30
15/30
steps=50, tokens=4096
27/30
518/3627
0/459
6/30
24/30
steps=150, tokens=1024
24/30
237/1635
2/128
10/30
17/30
steps=150, tokens=4096
25/30
612/4401
0/467
9/30
21/30
Table 5: Constraint ablation (30 questions, fault seed=50). Two models under identical harness settings, varying only MAX_STEPS and MAX_TOKENS . Override counts traces flagged with candidate silent override. Absolute fault counts are not comparable across cells because longer-running cells make more tool calls.
Detector
Agreement
Rate
D-correct
67 / 67
100.0%
D-impact / recovery
19 / 19
100.0%
Fault relevance
18 / 18
100.0%
D-verify
67 / 81
82.7%
Table 6: Agreement between automated detector labels and human review. Per-detector n differs because not every measurement is applicable to every reviewed trace.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Disagreements / reviewed
nemotron-3-ultra
10 / 17
openai/gpt-oss-20b
3 / 14
claude-opus-4.8
1 / 10
openai/gpt-oss-120b
0 / 13
claude-sonnet-5
0 / 10
Mistral-Large-3-675B
0 / 17
Appendix
Table 7: D-verify disagreements among reviewed traces with a comparable verification label. These counts describe the reviewed cases and should not be interpreted as population disagreement rates.
Figure 2: Terminal summary for rapid triage. Per-trace outcomes, fault exposure, verification events, and flagged cases are summarized before detailed inspection.
Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation. We introduce TRACE, a protocol that turns a score change from a verdict into a testable diagnosis: it applies a targeted change to one part of an evaluation, compares paired runs, checks whether the agent's behavior changed, and rescores unchanged trajectories to test whether the scoring rule is responsible. In a controlled suite of 25 synthetic tasks, renaming tools lowers a scripted agent's score by 0.250 even though it performs exactly the same operations; restoring the original names at scoring time closes the entire gap, while the same mutation exposes a genuine behavioral failure in a second agent. On public τ2-bench tasks with four LLM agents, an initial 30-task study finds mixed reward changes whose one clear effect does not replicate. In a larger follow-up on 88 new tasks with repeated runs per condition, renaming tools or reformatting tool outputs leaves reward unchanged to within ±0.10 for seven of eight agent-change pairs, whereas tool names that deliberately mislead lower every agent's reward by 0.20-0.44, showing that the setup can detect real effects. Identical reruns flip 15-36% of task outcomes, so single-run comparisons cannot separate presentation effects from run-to-run variation. Two frontier LLM judges give consistent verdicts when a fixed trajectory is presented differently, yet disagree with each other on 57% of the same records, largely because one grades procedure rather than outcome. TRACE thus separates what a score change says about the agent from what it says about the measurement.
AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from τ2-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.
Reshabh K Sharma, Shraddha Barke, Benjamin Zorn
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA
Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at κ=0.82.
Chenyi Huang, Arun Josephraj Arokiaraj, Bin Wu +1
Department of Computer Science, University College London, London, United Kingdom · Centre for Artificial Intelligence, University College London, United Kingdom