An Agentic Approach to Generating XAI-Narratives

Yifan He; David Martens

cs.CLMar 20, 2026

An Agentic Approach to Generating XAI-Narratives

Authors: Yifan He, David Martens

Organizations: University of Antwerp, Belgium

Abstract

Explainable AI (XAI) research has experienced substantial growth in recent years. Existing XAI methods, however, have been criticized for being technical and expert-oriented, motivating the development of more interpretable and accessible explanations. In response, large language model (LLM)-generated XAI narratives have been proposed as a promising approach for translating post-hoc explanations into more accessible, natural-language explanations. In this work, we propose a multi-agent framework for XAI narrative generation and refinement. The framework comprises the Narrator, which generates and revises narratives based on feedback from multiple Critic Agents on faithfulness and coherence metrics, thereby enabling narrative improvement through iteration. We design five agentic systems (Basic Design, Critic Design, Critic-Rule Design, Coherent Design, and Coherent-Rule Design) and systematically evaluate their effectiveness across five LLMs on five tabular datasets. Results validate that the Basic Design, the Critic Design, and the Critic-Rule Design are effective in improving the faithfulness of narratives across all LLMs. Claude-4.5-Sonnet on Basic Design performs best, reducing the number of unfaithful narratives by 90% after three rounds of iteration. To address recurrent issues, we further introduce an ensemble strategy based on majority voting. This approach consistently enhances performance for four LLMs, except for DeepSeek-V3.2-Exp. These findings highlight the potential of agentic systems to produce faithful and coherent XAI narratives.

Figures & tables

Figure 1: The baseline narratives are fed into the Faithful Evaluator and the Coherence Agent in the beginning. Based on the Faithful Evaluator ’s output, the Faithful Critic offers revision suggestions to the Narrator . On the other side, the Coherence Agent identifies coherence-related issues and provides revision suggestions to the Narrator . The Narrator then generates an updated version of the narrative and continues the iteration until the stopping criterion is met.

Figure 2: Left: This is an example of the base prompt for the Narrator . The instance is from the Student dataset. Right: The Narrator generates a narrative based on the given SHAP input. However, in this example, it injects multiple faithfulness-related issues, including the wrong feature value on “goout” (being 5 times instead of 4 times in the given SHAP input), the wrong feature sign on “Walc” (being “positive” instead of “negative”), and the interchanged feature importance ranking between “goout” and “failures”. These issues are marked in red.

Figure 3: Left: The Faithful Evaluator ’s prompt that we use to extract information from a narrative. The narrative from Figure 2 is fed into the prompt. Middle: The Faithful Evaluator extracts the information into a dictionary, including the rank, sign, and value information for each feature. This is an intermediate output, which can be reused by the Faithful Critic . Right: The output from the Faithful Evaluator . The errors contained in the narrative are captured and reported by the Faithful Evaluator , shown in red.

Figure 4: Upper Left: The instance’s SHAP input table. Below Left: The extraction dictionary generated by the Faithful Evaluator . All mistakes are successfully extracted and shown in red. Right: The output from the Faithful Critic . It refers to the original SHAP table and outputs directional feedback, guiding the Narrator on how to revise.

Figure 5: Two examples of the feedback from the Coherence Agent . Left: Two narratives that contain coherence-related issues. Problems are highlighted in red. Right: The corresponding feedback from the Coherence Agent .

Panel of Figure 6 — Figure 6: Left: A narrative that is linguistically ambiguous on the importance of features. This is a difficult example for the Faithful Evaluator to correctly extract all rank information. Middle: Five LLM-based Faithful Evaluator extracts the information independently and a disagreement happens. For example, Llama 3.3 70B Instruct makes mistakes on the extraction dictionary (mark in red), whereas other LLMs do not. Right: With the ensemble strategy (majority voting), the extraction results from five LLMs are combined to make a final decision and mitigate extraction mistakes.

Concepts	Reason	Original Setting	New Setting
Iteration Round	How does the number of iteration rounds impact the result?	Round=2	Round=10
The Number of Features	How does the number of features included in the narrative impact the result?	Feature=4	Feature=8
No-Baseline Setting	What if we don’t use baseline narratives? How does this impact the result?	Baseline narratives from Deepseek-chat-0324	No baseline narratives
Different Baseline Narratives	How do the best and the worst baseline narratives impact the result?	Baseline narratives from Deepseek-chat-0324	1. Best Baseline narratives from DeepSeek-V3.2-Exp 2. Worst Baseline narratives from Llama 3.3 70B Instruct

Table 1: The key concepts of the additional experiments

Figure 7: This bar chart represents the three accuracies ( RA : rank accuracy, SA : sign accuracy, and VA : value accuracy) for different agentic system designs across the three rounds using Claude Sonnet 4.5.

Claude Sonnet 4.5 based Designs	Datasets	Faithfulness			No. Unf. Narr.
Claude Sonnet 4.5 based Designs	Datasets	RA	SA	VA	No. Unf. Narr.
Baseline	AVG.	(0.900 )	0.958	0.990	(31)
	diabetes	0.838	0.988	1.000	5
	stroke	(0.925)	0.938	0.975	(8)
	student	0.925	0.913	1.000	8
	fifa	0.875	0.963	1.000	7

Table 2: Faithfulness results for different agentic system designs based on Claude Sonnet 4.5. Arrows indicate the progression of the rank, sign, and value accuracy from Round 0 to Round 2. The last column shows the progression of the number of unfaithful narratives over rounds.

Category	Conditions
C1: Confusing Faithfulness Instruction	C1.1 Since the Faithful Evaluator feedback contains non-directional instruction, the Narrator does not clearly know how to revise accordingly. This occurs because the Faithful Evaluator only reveals the error type by how it is designed, without mentioning where the Narrator should change to.
	C1.2 (edge case) The Faithful Critic feedback contains information on reordering the mentioning of features while it already satisfies. This occurs when a narrative mentions all features following the order in the SHAP, while containing inappropriate descriptive phrases that violate the importance of that feature in the SHAP table (Figure 8 ).
C2: Extraction Mistake	The Faithful Evaluator reports non-existing faithfulness issues when a narrative is (arguably) faithful.
C3: Counterintuitive Feature Influence	When the given SHAP input contains counterintuitive feature effects, the Narrator is inclined to disregard and generate a narrative following common sense.
C4: Misleading Coherence Instruction	The Coherence Agent gives feedback with the intention to improve the narrative’s coherence, but it inadvertently encourages revisions that violate the SHAP input.
C5: Generation Misalignment	The Narrator fails to generate a faithful narrative with an identifiable cause from the four above. This is therefore attributed to one-time generation errors that LLMs may produce.

Table 3: Problem Categorization of Unfaithful Narratives

Panel of Figure 8 — Figure 8: Left: This is an narrative example from the Student dataset, index 14, from the Claude Sonnet 4.5-based Critic Design . This narrative mentions four features following the feature rank in the SHAP input. However, this narrative is unfaithful because the inappropriate use of descriptive terms (highlighted in orange boxes) implies that the feature importance of “failures” is higher than that of “goout”. This is an edge case in which the feature-mention order does not match the feature-importance order. Middle: The Faithful Evaluator extracts the rank of “failures” to be the second most important, and the feature “goout” to be the least important. It does so because it does not mechanically determine the rank based on the order in which features are mentioned; instead, it infers the ranking through a holistic linguistic understanding of the narrative. Right: The Faithful Critic instructs repositioning the two sentences. However, this is already satisfied in this narrative at the beginning.

Figure 9: This bar chart shows the distribution of problem categories across five system designs (Claude Sonnet 4.5).

Panel of Figure 10 — Figure 10: An Example of Extraction Mistake on Rank . Due to the subtle semantic differences of descriptive terms, the task of extracting the correct feature importance is difficult for the Faithful Evaluator .

Panel of Figure 11 — Figure 11: No Extraction Mistake on Rank . The Basic Design improves the clarity of the descriptive terms, making the extraction task easier.

Designs	LLMs	Faithfulness			Overall (round 2)
Designs	LLMs	RA	SA	VA	Overall (round 2)
Basic Design	Claude Sonnet 4.5	0.905 $\rightarrow$ 0.970 $\rightarrow$ 0.990	0.957 $\rightarrow$ 0.997 $\rightarrow$ 1.000	0.990 $\rightarrow$ 0.995 $\rightarrow$ 0.997	0.996
	Mistral Medium 3.1	0.898 $\rightarrow$ 0.870 $\rightarrow$ 0.887	0.950 $\rightarrow$ 0.965 $\rightarrow$ 0.970	0.990 $\rightarrow$ 0.995 $\rightarrow$ 0.995	0.951
	GPT-5	0.898 $\rightarrow$ 0.945 $\rightarrow$ 0.972	0.965 $\rightarrow$ 0.995 $\rightarrow$ 0.997	0.993 $\rightarrow$ 0.995 $\rightarrow$ 1.000	0.990
	Llama 3.3 70B Instruct	0.857 $\rightarrow$ 0.875 $\rightarrow$ 0.863	0.940 $\rightarrow$ 0.960 $\rightarrow$ 0.975	0.993 $\rightarrow$ 0.997 $\rightarrow$ 1.000	0.946
	DeepSeek-V3.2-Exp	0.905 $\rightarrow$ 0.943 $\rightarrow$ 0.940	0.960 $\rightarrow$ 0.988 $\rightarrow$ 0.980	0.992 $\rightarrow$ 0.988 $\rightarrow$ 0.993	0.971

Table 4: Faithfulness results for different agentic system designs based on different LLMs. Arrows indicate the progression of the rank, sign, and value accuracy from Round 0 to Round 2. The last column shows the overall accuracy in round 2 for each configuration.

Figure 12: This bar chart presents the three accuracies ( RA : rank accuracy, SA : sign accuracy, and VA : value accuracy) among five LLMs on the Critic-Rule Design .

DeepSeek-V3.2-Exp	Basic Design		Critic Design
Problem Categorization	number	instance	number	instance
C1: Confusing Faithfulness Instruction	10	diabetes: index 2,4,7,17 stroke: index 13 student: index 1, 12, 14 credit: index 6,10	1	student: index 14
C2: Extraction Mistake	3	stroke: index 2 student: index 5, 18	3	stroke: index 2 student: index 3, 5
C3: Counterintuitive Feature Influence	1	student: index 16	1	student: index 16
C4: Misleading Coherence Instruction
C5: Generation Misalignment

Table 5: The problem categorization on the round-2 unfaithful narratives reported by the Basic Design and the Critic Design based on DeepSeek-V3.2-Exp.

LLMs	Designs	Faithfulness			Overall (round 2)	No. Unf. Narr. (round 2)
LLMs	Designs	RA(o) $\|$ RA(e)	SA(o) $\|$ SA(e)	VA(o) $\|$ VA(e)	(o) $\|$ (e)	(o) $\|$ (e)
Claude Sonnet 4.5	Basic Design	0.990 $\|$ 1.000	1.000 $\|$ 1.000	0.997 $\|$ 0.997	0.996 $\|$ 0.999	3 $\|$ 1
	Critic Design	0.960 $\|$ 0.970	0.993 $\|$ 0.997	0.997 $\|$ 0.997	0.983 $\|$ 0.988	9 $\|$ 6
	Critic-Rule Design	0.960 $\|$ 0.970	0.997 $\|$ 0.997	0.997 $\|$ 0.997	0.985 $\|$ 0.988	8 $\|$ 6
	Coherent Design	0.860 $\|$ 0.872	0.980 $\|$ 0.990	0.997 $\|$ 1.000	0.946 $\|$ 0.954	28 $\|$ 23
	Coherent-Rule Design	0.878 $\|$ 0.892	0.985 $\|$ 0.993	0.997 $\|$ 1.000	0.953 $\|$ 0.962	25 $\|$ 20

Table 6: This table compares the round-2 accuracy obtained from the original experiments (o) and the ensemble experiments (e). Numbers are highlighted in red if the ensemble experiment’s result is worse than that of the original experiment. The final column reports the number of remaining unfaithful narratives from each.

Figure 13: A comparison between the problem categorization: original and ensemble experiments (Claude Sonnet 4.5).

LLMs	Faithfulness	Accuracies from Round 0 to Round 9	Overall (round 9 )
Claude Sonnet 4.5	RA	0.905 $\rightarrow$ 0.960 $\rightarrow$ 0.960 $\rightarrow$ 0.955 $\rightarrow$ 0.960 $\rightarrow$ 0.958 $\rightarrow$ 0.963 $\rightarrow$ 0.958 $\rightarrow$ 0.963 $\rightarrow$ 0.958	0.984
	SA	0.958 $\rightarrow$ 0.993 $\rightarrow$ 0.993 $\rightarrow$ 0.993 $\rightarrow$ 0.993 $\rightarrow$ 0.995 $\rightarrow$ 0.995 $\rightarrow$ 0.995 $\rightarrow$ 0.995 $\rightarrow$ 0.995
	VA	0.990 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 1.000 $\rightarrow$ 1.000 $\rightarrow$ 1.000 $\rightarrow$ 1.000 $\rightarrow$ 1.000
Mistral Medium 3.1	RA	0.898 $\rightarrow$ 0.943 $\rightarrow$ 0.983 $\rightarrow$ 0.985 $\rightarrow$ 0.985 $\rightarrow$ 0.985 $\rightarrow$ 0.985 $\rightarrow$ 0.990 $\rightarrow$ 0.990 $\rightarrow$ 0.990	0.995
	SA	0.953 $\rightarrow$ 0.985 $\rightarrow$ 0.990 $\rightarrow$ 0.990 $\rightarrow$ 0.995 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998
	VA	0.993 $\rightarrow$ 0.993 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998 $\rightarrow$ 0.998

Table 7: The round=10 experiments’ results across different LLMs ( Critic-Rule Design )

Panel of Figure 14 — Figure 14: The Evolution of Rank Accuracy (RA), Sign Accuracy (SA), and Value Accuracy (VA) across 10 rounds for different LLMs.

LLMs	No. Unf. Narr. (round 9)	C1: Confusing Faithfulness Instruction	C2: Extraction Mistake
Claude Sonnet 4.5	8	2	6
Mistral Medium 3.1	4	$/$	4
GPT-5	9	2	7
Llama 3.3 70B Instruct	5	$/$	5
DeepSeek-V3.2-Exp	1	$/$	1

Table 8: The round=10 experiments’ problem categorization ( Critic-Rule Design )

LLMs	Experiments	Faithfulness			Overall (round 2)
LLMs	Experiments	RA	SA	VA	Overall (round 2)
Claude Sonnet 4.5	Original Exp.	0.905 $\rightarrow$ 0.960 $\rightarrow$ 0.960	0.957 $\rightarrow$ 0.993 $\rightarrow$ 0.997	0.990 $\rightarrow$ 0.995 $\rightarrow$ 0.997	0.985
	Feature=8	0.750 $\rightarrow$ 0.924 $\rightarrow$ 0.939	0.959 $\rightarrow$ 0.991 $\rightarrow$ 0.993	0.993 $\rightarrow$ 0.994 $\rightarrow$ 0.998	0.976
	No Baseline	0.895 $\rightarrow$ 0.970 $\rightarrow$ 0.970	0.983 $\rightarrow$ 0.990 $\rightarrow$ 0.995	0.978 $\rightarrow$ 0.990 $\rightarrow$ 0.995	0.987
	Best Base.	0.940 $\rightarrow$ 0.965 $\rightarrow$ 0.975	0.958 $\rightarrow$ 0.983 $\rightarrow$ 0.990	0.993 $\rightarrow$ 0.998 $\rightarrow$ 1.000	0.988
	Worst Base.	0.775 $\rightarrow$ 0.965 $\rightarrow$ 0.970	0.968 $\rightarrow$ 0.988 $\rightarrow$ 0.993	0.985 $\rightarrow$ 0.993 $\rightarrow$ 0.998	0.987

Table 9: The additional experiments’ results across different LLMs ( Critic-Rule Design )

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

LLMs	Faithfulness	Mean	Min	Max	Std Dev.
Claude Sonnet 4.5	RA	0.905	0.905	0.905	0.000
	SA	0.958	0.957	0.960	0.001
	VA	0.990	0.990	0.990	0.000
Mistral Medium 3.1	RA	0.900	0.897	0.907	0.004
	SA	0.952	0.950	0.952	0.001
	VA	0.990	0.988	0.993	0.002

Appendix

Table 10: LLM extraction instability on round-0 rank, sign and value accuracy. The mean is computed on the corresponding accuracy across five agentic system designs.

LLMs	Agentic System Designs	Average (round 2)	Count
LLMs	Agentic System Designs	(o) $\|$ (e)	Count
Claude Sonnet 4.5	Basic Design	0.996 $\|$ 0.999	2
	Critic Design	0.983 $\|$ 0.988	0
	Critic-Rule Design	0.985 $\|$ 0.988	0
	Coherent Design	0.946 $\|$ 0.954	0
	Coherent-Rule Design	0.953 $\|$ 0.962	0

Appendix

Table 11: The number of occurrences of the best design for each LLM

Model	Model (check point)	Geograph	Open source	Cost in total (Euro) (o)	Cost in total (Euro) (e)
Claude Sonnet 4.5	claude-sonnet-4-5-20250929	US	No	10.7	31.8
Mistral Medium 3.1	mistral-medium-2508	Europe	No	1.7	4.5
GPT-5	gpt-5-chat-latest	US	No	4.7	10.6
Llama-3.3-70B	llama-3.3-70b-instruct	US	open-weight	0.04	1.2
Deepseek-V3.2-Exp	deepseek-v3.2-exp	China	open-weight	0.6	2.0

Appendix

Table 12: The comparison of the five LLMs. The cost reported in each cell represents the total expense incurred by running all five agentic designs on 100 instances, with up to three refinement iterations.

Explainable Artificial Intelligence

Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.

See all 320 papers

A Two-Stage LLM Framework for Accessible and Verified XAI Explanations
Apr 14, 2026Georgios Mermigkis, Dimitris Metaxakis, Marios Tyrovolas +4Explainable Artificial Intelligence Faithfulness of Language Model Explanations
Department of Computer Engineering and Informatics, University of Patras, Patras, Greece · Archimedes Unit, Athena Research Center, Athens, Greece · Department of Informatics and Telecommunications, University of Ioannina, Arta, Greece +1
PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations
Mar 6, 2026Vittoria Vineis, Matteo Silvestri, Lorenzo Antonelli +2Explainable Artificial Intelligence Faithfulness of Language Model Explanations
Sapienza University of Rome · Syllotips · TellmewAI s.r.l.
On the Importance and Evaluation of Narrativity in Natural Language AI Explanations
Apr 20, 2026Mateusz Cedro, David MartensNatural Language Explanation Generation Explainable Artificial Intelligence
University of Antwerp, Belgium

Apr 14, 2026J/K move · Enter open · S save

A Two-Stage LLM Framework for Accessible and Verified XAI Explanations

Georgios Mermigkis, Dimitris Metaxakis, Marios Tyrovolas, Argiris Sofotasios+3

Department of Computer Engineering and Informatics, University of Patras, Patras, Greece · Archimedes Unit, Athena Research Center, Athens, Greece · Department of Informatics and Telecommunications, University of Ioannina, Arta, Greece+1

Explainable Artificial Intelligence Faithfulness of Language Model Explanations Explanation Evaluation Explainability Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used to translate the technical outputs of eXplainable Artificial Intelligence (XAI) methods into accessible natural-language explanations. However, existing approaches often lack guarantees of accuracy, faithfulness, and completeness. At the same time, current efforts to evaluate such narratives remain largely subjective or confined to post-hoc scoring, offering no safeguards to prevent flawed explanations from reaching end-users. To address these limitations, this paper proposes a Two-Stage LLM Meta-Verification Framework that consists of (i) an Explainer LLM that converts raw XAI outputs into natural-language narratives, (ii) a Verifier LLM that assesses them in terms of faithfulness, coherence, completeness, and hallucination risk, and (iii) an iterative refeed mechanism that uses the Verifier's feedback to refine and improve them. Experiments across five XAI techniques and datasets, using three families of open-weight LLMs, show that verification is crucial for filtering unreliable explanations while improving linguistic accessibility compared with raw XAI outputs. In addition, the analysis of the Entropy Production Rate (EPR) during the refinement process indicates that the Verifier's feedback progressively guides the Explainer toward more stable and coherent reasoning. Overall, the proposed framework provides an efficient pathway toward more trustworthy and democratized XAI systems.

LLMs	Designs	Faithfulness			Overall (round 2)	No. Unf. Narr. (round 2)
LLMs	Designs	RA(o) $\|$ RA(e)	SA(o) $\|$ SA(e)	VA(o) $\|$ VA(e)	(o) $\|$ (e)	(o) $\|$ (e)
Claude Sonnet 4.5	Basic Design	0.990 $\|$ 1.000	1.000 $\|$ 1.000	0.997 $\|$ 0.997	0.996 $\|$ 0.999	3 $\|$ 1
	Critic Design	0.960 $\|$ 0.970	0.993 $\|$ 0.997	0.997 $\|$ 0.997	0.983 $\|$ 0.988	9 $\|$ 6
	Critic-Rule Design	0.960 $\|$ 0.970	0.997 $\|$ 0.997	0.997 $\|$ 0.997	0.985 $\|$ 0.988	8 $\|$ 6
	Coherent Design	0.860 $\|$ 0.872	0.980 $\|$ 0.990	0.997 $\|$ 1.000	0.946 $\|$ 0.954	28 $\|$ 23
	Coherent-Rule Design	0.878 $\|$ 0.892	0.985 $\|$ 0.993	0.997 $\|$ 1.000	0.953 $\|$ 0.962	25 $\|$ 20

Abstract

Explainable Artificial Intelligence

Explore similar work

A Two-Stage LLM Framework for Accessible and Verified XAI Explanations

PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations

On the Importance and Evaluation of Narrativity in Natural Language AI Explanations