AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did and why. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents. To address this gap, we present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM-generated explanations.
Figures & tables
Figure 1: Framework overview. The raw AI agent execution trace is processed through a sequential operational pipeline that constructs the structured report X(τ) . Trace normalization maps heterogeneous execution logs to a canonical step sequence, while requirement decomposition constructs the requirement registry R , to decompose user’s goal. At each step, an LLM extracts candidate behavioral commitments and evidence with explicit pointers to the original trace; deterministic validation filters unsupported or ill-typed updates and enforces the framework’s grounding constraints. Deterministic reconciliation then computes the per-step state transitions δt and requirement-level grounding reports Γt , yielding the structured tuples (Σt,δt,Γt) . Finally, the constrained narrator ψ renders X(τ) into per-step explanations E1,…,ET and a concise overall summary Eˉ .
β
ϵint
ϵext
Label
Interpretation
Concordant configurations (bahaviour matches available evidence)
1
1
1
Fully Grounded
Resolved with both internal and external evidence.
1
1
0
Self Grounded
Resolved from internal evidence only.
1
0
1
Externally Grounded
Resolved from external evidence only.
0
0
0
Genuine Gap
Unresolved and no supporting evidence available.
Discordant configurations (behaviour and evidence disagree)
Table 1: Diagnostic configurations derived from behaviour–evidence reconciliation. Here, β , ϵint , and ϵext denote behavioral committment, internal evidence, and external evidence, respectively, measured with respect to a specific requirement r at step t ; each equals 1 when the corresponding state is satisfied .
Structured report — failures (%)
Narrative — failures (%)
Benchmark
Agent
# traces
Our
LLM
Our–Our
LLM–Our
LLM-only
GAIA
Plan-and-act
116
0
1 (0.9%)
0
—
25 (21.6%)
GAIA
ReAct
116
0
9 (7.8%)
0
—
40 (34.5%)
GAIA
TRAIL
116
0
3 (2.6%)
0
—
28 (24.1%)
HotpotQA
Plan-and-act
100
0
1(1.0%)
0
—
5(5.0%)
HotpotQA
ReAct
100
0
0
0
—
1(1.0%)
Table 2: Evaluation corpus and pipeline reliability. For each of the five (benchmark × agent) subsets we report the number of traces processed and the number of failed generations per pipeline configuration (percentages relative to the row total). The Structured report block covers the intermediate representation ( X(τ) ), produced either by our deterministic pipeline ( Our ) or by a single-shot LLM prompt with the same schema ( LLM ). The Narrative block covers the final natural-language explanation ( Eˉ ), produced by: (i) Our–Our , our narrator applied to our structured report (the full pipeline described in Section 3.2 ); (ii) LLM-Our , our narrator applied to an directly LLM-generated version of the structured report; and (iii) LLM-only , an LLM prompted to summarise the raw trace directly in natural language.
Metric
Definition
Evaluation configuration
Scale
M1 – Faithfulness
The narrative accurately reflects the relevant behavior represented by its reference, without unsupported claims, omissions, or semantic distortions.
M1-a – Source report: Human + LLM judge
3-point
M1-b – Trace level: Human judge
3-point
M1-a_our – Our report: LLM judge
3-point
M2 – Exhaustiveness (Content)
The narrative contains all essential information needed to understand the agent’s behavior and the relevant steps leading to its outcome.
Human + LLM judge
4-point
M3 – Linguistic Quality (Delivery)
The narrative is clear, logically organized, concise, communicatively efficient, and easy to understand, without unnecessary or unsupported verbosity.
Human + LLM judge
4-point
M4 – Overall Quality (Holistic)
Overall, the narrative provides an appropriate and satisfactory explanation of the agent’s execution, considering both its content and presentation.
Human + LLM judge
4-point
Table 3: Narrative evaluation criteria and evaluation configurations.
Overall
≤10
11–30
≥31
METHOD
AGENT
n
mean
std
n
mean
std
n
mean
std
n
mean
std
OUR
ReAct
12
0.904
0.092
5
0.902
0.099
3
0.860
0.098
4
0.940
0.090
OUR
Plan-and-act
12
0.842
0.212
8
0.794
0.237
4
0.938
0.125
–
–
–
OUR
TRAIL
8
0.644
0.282
8
0.644
0.282
–
–
–
–
–
–
LLM
ReAct
11
0.075
0.131
5
0.116
0.161
3
0.083
0.144
3
0.000
0.000
LLM
Plan-and-act
12
0.057
0.149
8
0.062
0.177
4
0.045
0.090
–
–
–
Table 4: NFI: mean and standard deviation by method and agent, overall and by number-of-steps group.
Measure
Condition ( n )
H/J (%)
Agr. (%)
κw
Neg. rec.
M1-a
All (63)
95/98
94
-0.09
0% (0/3)
Our–Our (32)
100/97
97
−0.06†
n/a
LLM–Our (31)
90/100
90
−0.05
0% (0/3)
M1-a_our vs M1-b
All (86)
81/60
74
0.63
88% (14/16)
Our–Our (32)
100/97
97
0.09†
n/a
LLM–Our (31)
87/48
61
0.46
100% (4/4)
Table 5: Agreement between the human annotator and the LLM judge, pooled over conditions (bold) and split by condition. H/J = percentage of ratings on the agree side for the human and judge, respectively; Agr. = raw agreement on the agree/disagree collapse; κw = quadratic-weighted Cohen’s κ ; Neg. rec. = share of human-negative ratings also rated negative by the judge, with counts in parentheses. † marks rows with fewer than three human-negative ratings, for which κw is not informative; raw agreement and negative recall should be preferred. M1-a_our vs M1-b is a cross-level comparison: the judge scores the narrative against our structured report, whereas the human scores it against the original trace.
Human
LLM judge
H
Measure
n
Δ [95% CI]
n
W
T
L
Δ [95% CI]
p
pBH
H1
M2
26
0.85 [0.62, 1.07]
417
18
395
4
0.07 [0.02, 0.12]
0.0031
0.0031
H1
M3
26
1.04 [0.65, 1.36]
418
353
65
0
1.18 [1.12, 1.25]
0.0001
0.0001
H1
M4
26
1.35 [1.00, 1.64]
416
206
209
1
0.62 [0.55, 0.69]
0.0001
0.0001
H2
M2
31
1.03 [0.61, 1.45]
500
124
373
3
0.49 [0.41, 0.58]
0.0001
0.0001
H2
M3
31
0.39 [-0.03, 0.82]
500
30
438
32
-0.00 [-0.03, 0.03]
0.6418
0.8022
Table 6: Hypothesis tests on the LLM-as-judge corpus and comparison with human-evaluation effects. Inferential statistics are reported for the LLM-as-judge corpus, while human-evaluation effects are reported for descriptive comparison only.
H
Measure
ReAct
Plan-and-act
TRAIL
H1
M2
+0.08
+0.03
+0.15
H1
M3
+1.18
+1.11
+1.37
H1
M4
+0.63
+0.55
+0.77
H2
M2
+0.45
+0.59
+0.39
H2
M3
-0.01
-0.01
+0.03
H2
M4
+0.76
+0.93
+0.73
Table 7: H3 direction consistency for the LLM-as-judge corpus. Cell = mean paired difference within that agent architecture.
Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows
Arun Joshi
Independent Researcher Islington College (Alumini) Nepal
AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from τ2-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.
Reshabh K Sharma, Shraddha Barke, Benjamin Zorn
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA
In the field of Artificial Intelligence, an agent is a system which is able to autonomously make decisions in order to reach a desired goal. As these systems grow more prevalent in our day-to-day lives, there has been an increased need to add explainability features which can provide an account for an agent's behavior. We therefore propose a framework that outlines how to produce comprehensible explanations for policy-aware agents, or agents which have rule-enforcing policies incorporated in their decision-making framework. This framework is designed using insights from the social sciences on how to produce good explanations. It is implemented in the Answer Set Programming language while using Python to assist with information extraction and natural-language translation. Because these agents incur penalties when violating policies, we are able to leverage these penalties to detect undesirable events in scenarios that are counterfactual to the agents' original actions. This lends itself to creating contrastive explanations (e.g., "the agent performed this action because, had it not, undesirable event X would have occurred."), which formulate the core component for our explainability framework. The framework is evaluated using a survey wherein human participants provide feedback on our program-generated explanations.
Heather Merhout, Daniela Inclezan
Miami University · Miami University Oxford, Ohio, United States