Explaining AI Agents Through Execution Traces

Vittoria Vineis; Fabiano Veglianti; Lorenzo Antonelli; Claudia Di Carlo; Matteo Silvestri; Gabriele Tolomei

cs.AISep 5, 2026

Explaining AI Agents Through Execution Traces

Authors: Vittoria Vineis, Fabiano Veglianti, Lorenzo Antonelli, Claudia Di Carlo, Matteo Silvestri, Gabriele Tolomei

Organizations: Sapienza University of Rome, Rome, Italy

Abstract

AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did and why. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems, motivating a paradigm shift toward approaches specifically designed for AI Agents. To address this gap, we present a post-hoc XAI framework that transforms a lengthy agent's execution trace into a structured report and a faithful natural-language explanation explicitly grounded in its observable behavior. Because it relies solely on execution traces, the framework applies across different agent architectures, environments, and tasks. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming naive LLM-generated explanations.

Figures & tables

Panel of Figure 1 — Figure 1: Framework overview. The raw AI agent execution trace is processed through a sequential operational pipeline that constructs the structured report $\mathcal{X}(\tau)$ . Trace normalization maps heterogeneous execution logs to a canonical step sequence, while requirement decomposition constructs the requirement registry $\mathcal{R}$ , to decompose user’s goal. At each step, an LLM extracts candidate behavioral commitments and evidence with explicit pointers to the original trace; deterministic validation filters unsupported or ill-typed updates and enforces the framework’s grounding constraints. Deterministic reconciliation then computes the per-step state transitions $\delta_{t}$ and requirement-level grounding reports $\Gamma_{t}$ , yielding the structured tuples $(\Sigma_{t},\delta_{t},\Gamma_{t})$ . Finally, the constrained narrator $\psi$ renders $\mathcal{X}(\tau)$ into per-step explanations $E_{1},\ldots,E_{T}$ and a concise overall summary $\bar{E}$ .

$\beta$	$\epsilon^{int}$	$\epsilon^{ext}$	Label	Interpretation
Concordant configurations (bahaviour matches available evidence)
1	1	1	Fully Grounded	Resolved with both internal and external evidence.
1	1	0	Self Grounded	Resolved from internal evidence only.
1	0	1	Externally Grounded	Resolved from external evidence only.
0	0	0	Genuine Gap	Unresolved and no supporting evidence available.
Discordant configurations (behaviour and evidence disagree)

Table 1: Diagnostic configurations derived from behaviour–evidence reconciliation. Here,

\beta

\epsilon^{int}

, and

\epsilon^{ext}

denote behavioral committment, internal evidence, and external evidence, respectively, measured with respect to a specific requirement

r

at step

t

; each equals

1

when the corresponding state is satisfied .

Benchmark	Agent	# traces	Our	LLM	Our–Our	LLM–Our	LLM-only
			Structured report — failures (%)		Narrative — failures (%)
GAIA	Plan-and-act	116	0	1 (0.9%)	0	—	25 (21.6%)
GAIA	ReAct	116	0	9 (7.8%)	0	—	40 (34.5%)
GAIA	TRAIL	116	0	3 (2.6%)	0	—	28 (24.1%)
HotpotQA	Plan-and-act	100	0	$1\,(1.0\%)$	0	—	$5\,(5.0\%)$
HotpotQA	ReAct	100	0	0	0	—	$1\,(1.0\%)$

Table 2: Evaluation corpus and pipeline reliability. For each of the five (benchmark

\times

agent) subsets we report the number of traces processed and the number of failed generations per pipeline configuration (percentages relative to the row total). The Structured report block covers the intermediate representation (

\mathcal{X}(\tau)

), produced either by our deterministic pipeline ( Our ) or by a single-shot LLM prompt with the same schema ( LLM ). The Narrative block covers the final natural-language explanation (

\bar{E}

), produced by: (i) Our–Our , our narrator applied to our structured report (the full pipeline described in Section 3.2 ); (ii) LLM-Our , our narrator applied to an directly LLM-generated version of the structured report; and (iii) LLM-only , an LLM prompted to summarise the raw trace directly in natural language.

Metric	Definition	Evaluation configuration	Scale
M1 – Faithfulness	The narrative accurately reflects the relevant behavior represented by its reference, without unsupported claims, omissions, or semantic distortions.	M1-a – Source report: Human + LLM judge	3-point
		M1-b – Trace level: Human judge	3-point
		M1-a_our – Our report: LLM judge	3-point
M2 – Exhaustiveness (Content)	The narrative contains all essential information needed to understand the agent’s behavior and the relevant steps leading to its outcome.	Human + LLM judge	4-point
M3 – Linguistic Quality (Delivery)	The narrative is clear, logically organized, concise, communicatively efficient, and easy to understand, without unnecessary or unsupported verbosity.	Human + LLM judge	4-point
M4 – Overall Quality (Holistic)	Overall, the narrative provides an appropriate and satisfactory explanation of the agent’s execution, considering both its content and presentation.	Human + LLM judge	4-point

Table 3: Narrative evaluation criteria and evaluation configurations.

METHOD	AGENT	$n$	mean	std	$n$	mean	std	$n$	mean	std	$n$	mean	std
		Overall			$\leq 10$			11–30			$\geq 31$
OUR	ReAct	12	0.904	0.092	5	0.902	0.099	3	0.860	0.098	4	0.940	0.090
OUR	Plan-and-act	12	0.842	0.212	8	0.794	0.237	4	0.938	0.125	–	–	–
OUR	TRAIL	8	0.644	0.282	8	0.644	0.282	–	–	–	–	–	–
LLM	ReAct	11	0.075	0.131	5	0.116	0.161	3	0.083	0.144	3	0.000	0.000
LLM	Plan-and-act	12	0.057	0.149	8	0.062	0.177	4	0.045	0.090	–	–	–

Table 4: NFI: mean and standard deviation by method and agent, overall and by number-of-steps group.

Measure	Condition ( $n$ )	H/J (%)	Agr. (%)	$\kappa_{w}$	Neg. rec.
M1-a	All (63)	95/98	94	-0.09	0% (0/3)
	Our–Our (32)	100/97	97	$-0.06^{\dagger}$	n/a
	LLM–Our (31)	90/100	90	$-0.05$	0% (0/3)
M1-a_our vs M1-b	All (86)	81/60	74	0.63	88% (14/16)
	Our–Our (32)	100/97	97	$0.09^{\dagger}$	n/a
	LLM–Our (31)	87/48	61	0.46	100% (4/4)

Table 5: Agreement between the human annotator and the LLM judge, pooled over conditions (bold) and split by condition. H/J = percentage of ratings on the agree side for the human and judge, respectively; Agr. = raw agreement on the agree/disagree collapse;

\kappa_{w}

= quadratic-weighted Cohen’s

\kappa

; Neg. rec. = share of human-negative ratings also rated negative by the judge, with counts in parentheses. † marks rows with fewer than three human-negative ratings, for which

\kappa_{w}

is not informative; raw agreement and negative recall should be preferred. M1-a_our vs M1-b is a cross-level comparison: the judge scores the narrative against our structured report, whereas the human scores it against the original trace.

H	Measure	$n$	$\Delta$ [95% CI]	$n$	$W$	$T$	$L$	$\Delta$ [95% CI]	$p$	$p_{\mathrm{BH}}$
		Human		LLM judge
H1	M2	26	0.85 [0.62, 1.07]	417	18	395	4	0.07 [0.02, 0.12]	0.0031	0.0031
H1	M3	26	1.04 [0.65, 1.36]	418	353	65	0	1.18 [1.12, 1.25]	0.0001	0.0001
H1	M4	26	1.35 [1.00, 1.64]	416	206	209	1	0.62 [0.55, 0.69]	0.0001	0.0001
H2	M2	31	1.03 [0.61, 1.45]	500	124	373	3	0.49 [0.41, 0.58]	0.0001	0.0001
H2	M3	31	0.39 [-0.03, 0.82]	500	30	438	32	-0.00 [-0.03, 0.03]	0.6418	0.8022

Table 6: Hypothesis tests on the LLM-as-judge corpus and comparison with human-evaluation effects. Inferential statistics are reported for the LLM-as-judge corpus, while human-evaluation effects are reported for descriptive comparison only.

H	Measure	ReAct	Plan-and-act	TRAIL
H1	M2	+0.08	+0.03	+0.15
H1	M3	+1.18	+1.11	+1.37
H1	M4	+0.63	+0.55	+0.77
H2	M2	+0.45	+0.59	+0.39
H2	M3	-0.01	-0.01	+0.03
H2	M4	+0.76	+0.93	+0.73

Table 7: H3 direction consistency for the LLM-as-judge corpus. Cell = mean paired difference within that agent architecture.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Table 9

Explainable Artificial Intelligence

Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.

See all 320 papers

XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights
Mar 6, 2026Arun JoshiAgent Failure Analysis Explainable Artificial Intelligence
Independent Researcher Islington College (Alumini) Nepal
Willful Disobedience: Automatically Detecting Failures in Agentic Traces
Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent Evaluation AI Agent Auditing
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA
Explainability Framework for Policy-Aware Autonomous Agents
Jul 23, 2026Heather Merhout, Daniela InclezanExplainable Artificial Intelligence Counterfactual Explanations
Miami University · Miami University Oxford, Ohio, United States

Mar 6, 2026J/K move · Enter open · S save

XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights

Arun Joshi

Independent Researcher Islington College (Alumini) Nepal

Agent Failure Analysis Explainable Artificial Intelligence AI Coding Agents Explainability Evaluation

Open paper AskPDF arXiv

Large Language Model (LLM)-based coding agents show promise in automating software development tasks, yet they frequently fail in ways that are difficult for developers to understand and debug. While general-purpose LLMs like GPT can provide ad-hoc explanations of failures, raw execution traces remain challenging to interpret even for experienced developers. We present a systematic explainable AI (XAI) approach that transforms raw agent execution traces into structured, human-interpretable explanations. Our method consists of three key components: (1) a domain-specific failure taxonomy derived from analyzing real agent failures, (2) an automatic annotation system that classifies failures using defined annotation schema, (3) a hybrid explanation generator that produces visual execution flows, natural language explanations, and actionable recommendations. Through a user study with 20 participants (10 technical, 10 non-technical), we demonstrate that our approach enables users to identify failure root causes 2.8 times faster and propose correct fixes with 73% higher accuracy compared to raw execution traces. Importantly, our structured approach outperforms ad-hoc state of the art models explanations by providing consistent, domain-specific insights with integrated visualizations. Our work establishes a framework for systematic agent failure analysis, addressing the critical need for interpretable AI systems in software development workflows

Abstract

Explainable Artificial Intelligence

Explore similar work

XAI for Coding Agent Failures: Transforming Raw Execution Traces into Actionable Insights

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

Explainability Framework for Policy-Aware Autonomous Agents