AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. We developed Signals to identify such problems from agentic traces and suggest concrete remediations. We built the method around a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. The method distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. We evaluated the method on public agent executions and achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy.
Figures & tables
Figure 1: The Signals diagnostic workflow. We used the taxonomy to diagnose problems in execution records, assess their severity, and suggest remediation actions. Further executions can add evidence to existing Signals or reveal new issues.
Points
Consequence for a defect or credible exposure
Avoidable waste for an inefficiency
10
Small, localized adverse effect.
Small avoidable work bounded to one operation.
25
Material but reversible degradation.
Meaningful but bounded avoidable work.
40
Task failure or a materially wrong result.
Substantial or repeated avoidable work.
65
Major supported harm in a high-consequence domain.
Waste dominates cost or latency, or persists across stages or turns.
85
Severe, permanent, or irreversible harm.
Work continues until intervention or a hard limit and threatens viability.
Table 1: Severity factor values. Consequence and waste have separate definitions even when their point values coincide. Potential consequence applies only to an evidenced reachable path.
Scope band
Distinct affected sessions
Affected share (20+ analyzed sessions)
Points
Isolated
1
Less than 1%
0
Limited
2–3
1% to less than 10%
3
Repeated
4–10
10% to less than 30%
7
Broad
11–100
30% to less than 60%
12
Widespread
More than 100
At least 60%
15
Table 2: Affected-scope points. Use the higher band from the count and, when at least 20 sessions were analyzed, the percentage columns.
Taxonomy problem
Evidence supporting the diagnosis
Factual or logical error
POL123 and Policy POL123 identify the same policy. The added field label did not support the reported policy_reference conflict.
Table 3: The taxonomy diagnosis in the synthetic insurance-intake case. Generic policy identifiers replace the source values.
Diagnostic family
Types
Diagnostic focus
Instruction and specification
5
Following the accepted objective, explicit requirements, and permitted scope.
Knowledge, context, and retrieval
6
Obtaining, retaining, and citing the evidence needed for the task.
Reasoning, decision, and output
8
Producing supported, correct, complete results and truthful completion claims.
Tool, action, and integration
8
Choosing and calling tools, interpreting their results, and respecting intended effects.
Orchestration, coordination, and state
5
Routing work, transferring information and state, making progress, and stopping correctly.
Runtime and platform reliability
3
Meeting execution requirements despite provider, capacity, authentication, or configuration failures.
Table 4: The 45 problem types and their eight diagnostic families.
System
Precision
Recall
F1
One-pass direct generation
71.11%
76.46%
73.69%
Signals pipeline
92.90%
90.01%
91.43%
Table 5: Signal matching with the same model, taxonomy, evaluation cohort, and deterministic matcher.
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 170 trajectories across 11 diverse task settings, including structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AgentRx, an automated diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. AgentRx improves step localization by 75% on average over prior work, while providing failure category attribution.
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4 pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from τ2-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.
Reshabh K Sharma, Shraddha Barke, Benjamin Zorn
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA