AI agents can make incorrect decisions or waste resources even when their tools and infrastructure work normally. Their nondeterministic behavior makes these problems difficult to reproduce and diagnose. We developed Signals to identify such problems from agentic traces and suggest concrete remediations. We built the method around a taxonomy of 45 problem types across eight families, with definitions and evidence requirements for each type. The method distinguishes realized failures, risk exposures, and inefficiencies, groups related findings into engineering issues called Signals, and assesses their severity from the supporting evidence. Each Signal includes actions a developer can take and checks for evaluating the change. Further executions can add evidence to an existing Signal or reveal a new problem. We evaluated the method on public agent executions and achieved 91.43% F1 when matching Signals to reference issues, about 18 percentage points higher than a one-pass LLM control that used the same model and taxonomy.
Figures & tables
Figure 1: The Signals diagnostic workflow. We used the taxonomy to diagnose problems in execution records, assess their severity, and suggest remediation actions. Further executions can add evidence to existing Signals or reveal new issues.
Points
Consequence for a defect or credible exposure
Avoidable waste for an inefficiency
10
Small, localized adverse effect.
Small avoidable work bounded to one operation.
25
Material but reversible degradation.
Meaningful but bounded avoidable work.
40
Task failure or a materially wrong result.
Substantial or repeated avoidable work.
65
Major supported harm in a high-consequence domain.
Waste dominates cost or latency, or persists across stages or turns.
85
Severe, permanent, or irreversible harm.
Work continues until intervention or a hard limit and threatens viability.
Table 1: Severity factor values. Consequence and waste have separate definitions even when their point values coincide. Potential consequence applies only to an evidenced reachable path.
Scope band
Distinct affected sessions
Affected share (20+ analyzed sessions)
Points
Isolated
1
Less than 1%
0
Limited
2–3
1% to less than 10%
3
Repeated
4–10
10% to less than 30%
7
Broad
11–100
30% to less than 60%
12
Widespread
More than 100
At least 60%
15
Table 2: Affected-scope points. Use the higher band from the count and, when at least 20 sessions were analyzed, the percentage columns.
Taxonomy problem
Evidence supporting the diagnosis
Factual or logical error
POL123 and Policy POL123 identify the same policy. The added field label did not support the reported policy_reference conflict.
Table 3: The taxonomy diagnosis in the synthetic insurance-intake case. Generic policy identifiers replace the source values.
Diagnostic family
Types
Diagnostic focus
Instruction and specification
5
Following the accepted objective, explicit requirements, and permitted scope.
Knowledge, context, and retrieval
6
Obtaining, retaining, and citing the evidence needed for the task.
Reasoning, decision, and output
8
Producing supported, correct, complete results and truthful completion claims.
Tool, action, and integration
8
Choosing and calling tools, interpreting their results, and respecting intended effects.
Orchestration, coordination, and state
5
Routing work, transferring information and state, making progress, and stopping correctly.
Runtime and platform reliability
3
Meeting execution requirements despite provider, capacity, authentication, or configuration failures.
Table 4: The 45 problem types and their eight diagnostic families.
System
Precision
Recall
F1
One-pass direct generation
71.11%
76.46%
73.69%
Signals pipeline
92.90%
90.01%
91.43%
Table 5: Signal matching with the same model, taxonomy, evaluation cohort, and deterministic matcher.