MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

Yuyang Cheng; Raghav Kaushik Ravi; Srivarshinee Sridhar; Sriparna Saha; Akash Ghosh; Chirag Agarwal

cs.AIOct 1, 2026

MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

Authors: Yuyang Cheng, Raghav Kaushik Ravi, Srivarshinee Sridhar, Sriparna Saha, Akash Ghosh, Chirag Agarwal

Organizations: University of Virginia · Vellore Institute of Technology · Indian Institute of Technology, Patna

Abstract

Recent years have seen the employment of a plethora of machine learning (ML) models in high-stakes domains, but they remain largely opaque to the practitioners who act on their predictions. While post-hoc explanation methods offer a lens into this model behavior, wielding them effectively demands expertise most domain experts lack: navigating high-dimensional outputs, selecting the best explanations, and synthesizing evidence across disparate tools. To this end, we present MEA, a multi-agent framework that removes the explanation knowledge barrier entirely: a Proposer agent selects and configures explanation tools based on the question and modality, while an Actor agent is optimized end-to-end against faithfulness, transforming the outputs into natural language explanations grounded in model behavior across tabular, text, and vision modalities. Further, we introduce diverse question types spanning feature attribution, counterfactual reasoning, and spurious feature detection, each paired with a perturbation-based faithfulness metric. We find that frontier LLMs systematically produce unfaithful explanations. By optimizing against faithfulness rewards augmented with a modality-adaptive penalty, MEA consistently outperforms post hoc explainers, agentic, and closed-source baselines across six datasets, with reward-driven optimization yielding faithfulness gains of +28% (tabular), +21% (text), and +34% (vision) over the untrained backbone. More broadly, our findings suggest that AI agents themselves can serve as a scalable, adaptable interface to ML explainability, opening a path toward natural-language explainability that generalizes beyond the fixed, single-purpose tools that have long defined the field.

Figures & tables

Figure 1: Overview of Mea . Given a user question about model behavior, the Proposer agent reasons over the model prediction and input modality to select an explanation strategy, choosing from a multimodal (tabular, text, and vision) toolkit and agent reasoning tasks (grounding, reasoning, and comparison), the Actor agent then executes the strategy, processes raw explanation outputs, and produces a structured natural-language explanation. The explanation is evaluated via input perturbation against faithfulness metrics. In training, both agents are jointly optimized end-to-end via GRPO [ 14 ] , using a reward combining faithfulness and tool penalties.

Figure 2: Case study of Mea across three modalities. Each column shows a complete pipeline execution for Mea : the input, Proposer’s strategy and tool, Actor’s execution output, and final explanation with faithfulness evaluation.

Figure 3: Mea ’s performance. Faithfulness scores of Mea vs. baselines across three modalities: (A) tabular, (B) text, and (C) vision. Higher mean faithfulness scores indicate more faithful explanations. Results show that Mea outperform all baselines, including closed-source models and techniques like CoT, ToT, and ReACT.

Figure 4: Mea vs. Explainers. Mean faithfulness scores of identifying most salient input features across (A) tabular, (B) text, and (C) vision modalities. Mea outperforms all baselines on tabular and text inputs, and matches the best-performing method on vision inputs.

Dataset	Method	Flip Success	Avg. Faithfulness
Adult Census	Mea	100.0%	0.894
	DiCE	100.0%	0.631
	Growing Spheres	98.4%	0.499
Breast Cancer	Mea	95.5%	0.886
	DiCE	97.7%	0.580
	Growing Spheres	81.8%	0.412

Table 1: Mea vs. specialized counterfactual explainers on Q6 (flip prediction), aggregated across model architectures per dataset (

n=63

for Adult Census,

n=44

for Breast Cancer). Best per row group in bold .

Modality	Group	Base	Mea
Tabular	Feat. Attr.	0.447	0.783
	Counterfactual	0.422	0.499
	Spurious Feat.	0.264	0.349
	Overall	0.379	0.581
Text	Feat. Attr.	0.428	0.563
	Counterfactual	0.860	0.770

Table 2: OOD faithfulness on datasets/architectures unseen during training, averaged across question types per group. Base : Proposer–Actor without GRPO fine-tuning; Mea : full GRPO-tuned system. Best per row in bold .

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	Question
Feature Attribution - which parts of the input drove or suppressed the prediction?
Q1	Most Responsible	Which parts of the instance are most responsible for the model’s prediction?
Q2	Least Responsible	Which parts of the instance are least responsible for the model’s prediction?
Q3	Distinctive	Which specific parts of the instance distinguish its prediction from the next-best alternative?
Q4	Contrastive	Why are instances $A$ and $B$ given different predictions?
Counterfactual - how sensitive is the prediction to hypothetical modifications?

Appendix

Table 3: Overview of the ten explanation questions organised by category.

Question	Metric $m_{i}$	Intuition
Q1	$\max\bigl(0,P(\hat{y}\mid\mathbf{x})-P(\hat{y}\mid\mathbf{x}^{-r})\bigr)$	Masking the most responsible region should drop predicted-class confidence
Q2	$1-\bigl\lvert P(\hat{y}\mid\mathbf{x})-P(\hat{y}\mid\mathbf{x}^{-r})\bigr\rvert$	Masking the least responsible region should leave confidence unchanged
Q3	$\max\!\left(0,\min\!\left(1,\dfrac{\text{gap}_{\text{orig}}-\text{gap}_{\text{mod}}}{2*\text{gap}_{\text{orig}}}\right)\right)$	The top-1/top-2 margin should collapse after masking the distinctive region
Q4	$1-\mathrm{sim}(F_{A},F_{B})$	The agent should attribute different features to each instance
Q5	$\begin{cases}\max(0,\;\delta)&\hat{c}=1\ \text{(predicts change)}\\ \max(0,\;1-\lvert\delta\rvert)&\hat{c}=0\ \text{(predicts stable)}\end{cases}$	Probability change should agree in direction with the agent’s yes/no answer
Q6	$\begin{cases}\max(0,P(\hat{y}\mid\mathbf{x})-P(\hat{y}\mid\mathbf{x}_{+\Delta}))&\text{vision}\\ \max(0,\min(1,P(y_{t}\mid\mathbf{x}_{+\Delta})))&\text{text/tab}\end{cases}$	Applying the proposed change should raise the probability of the target class

Appendix

Table 4: Summary of evaluation metrics corresponding to different explanation faithfulness questions.

Figure 5: Dsitribution of data across modalities.

Figure 6: Training reward curves. (Left) Sequential training: Vertical dashed lines mark phase boundaries (Tab

\to

Text

\to

Vision); the vision reward barely rises, indicating collapsed rollout diversity. (Right) Hybrid training (all modalities interleaved): The recurring spikes are an artifact of the sampling seed, which concentrates relatively easy questions towards the end of each epoch.

Modality	Question Type	Base Agent	Mea
Tabular	Q8	0.132	0.184
	Q9	0.024	0.019
	Overall	0.081	0.106
Text	Q8	0.186	0.353
	Q9	0.066	0.075
	Overall	0.116	0.186

Appendix

Table 5: OOD faithfulness scores on held-out question types Q8 and Q9. Best per row in bold .

Modality	Question Group	Base Agent	Mea
Tabular	Feature Attribution	0.447	0.783
	Counterfactual	0.422	0.499
	Spurious Features	0.264	0.349
	Overall	0.379	0.581
Text	Feature Attribution	0.428	0.563
	Counterfactual	0.860	0.770

Appendix

Table 6: OOD faithfulness scores on different datasets and model architectures. Best performance is bolded .

Metric	Base Agent	Mea
Mean faithfulness	0.0316	0.0373
Median faithfulness	0.0150	0.0218
Max faithfulness	0.2398	0.2650
Class-flip rate (ablation changes prediction)	5/120	4/120

Appendix

Table 7: Base agent vs. Mea on Q11 (concept attribution), 120 questions.

Figure 7: Ablation on prompt robustness. Mean faithfulness scores of Mea evaluated on three paraphrased variants of the question taxonomy across (A) tabular, (B) text, and (C) vision modalities. Minimum variance indicates that Mea is robust to changes in question phrasing.

Q-Type	Variant 1	Variant 2	Variant 3
Q1	What was the most influential feature driving the model’s prediction?	Which feature played the biggest role in this prediction?	What did the model rely on most heavily to reach this conclusion?
Q2	Which input feature had the least effect on the model’s decision?	Which feature mattered least to the model’s prediction?	Which feature is least relevant to the model’s conclusion?
Q3	What features were most critical in setting this prediction apart from the runner-up class?	Which features tipped the balance toward this prediction over the runner-up?	What distinguishes this prediction from the second most likely alternative?
Q4	What differences in the inputs account for the model giving instances A and B different predictions?	Why does the model give different predictions to these two instances?	What input differences lead to contrasting predictions for these two instances?
Q5	Would the model’s output change if a specific feature were removed from this input?	If a particular feature were excluded, would the model’s prediction remain the same?	How sensitive is the model’s output to the presence of a specific feature?
Q6	What modifications to this instance would cause the model to output a different class?	How should this input be changed to flip the model’s prediction to a different class?	What changes to this instance would lead the model to predict a different outcome?

Appendix

Table 8: Three paraphrase variants of each question type ( Q-field only ; tabular modality shown). Dynamic values such as feature names, row IDs, and class labels (denoted { placeholder }) are extracted from the original question via regex and re-injected verbatim at generation time.

Figure 8: Ablation on critic agent. Mean faithfulness scores across three que1on categories for the base agent, base agent augmented with a critic agent, and Mea , evaluated on (A) tabular, (B) text, and (C) vision modalities. Mea consistently outperforms both baselines, demonstrating that RL-based training yields stronger improvements than inference-time critique.

Object	Field	Description
PROPOSER	analysis	Overall assessment of tool selection
	tool_analysis	Per-tool rating ( high/medium/low ) and recommendation ( keep/remove/replace )
	tools_to_keep	Tools to retain in the revised strategy
	tools_to_remove	Tools to drop
	tools_to_add	New tools to include
	strategy_suggestions	Free-text suggestions to improve tool selection

Appendix

Table 9: Output JSON schema of the critic agent.

Panel of Figure 9 — Figure 9: Generic actor preambles prepended to the actor prompt for each reasoning baseline across all question types.

Model	Strategy Mode	$n$	Mean Faith.	Pass Rate
Base Agent	Autonomous only	493	0.515	48.5%
	Tools only	475	0.589	54.9%
	Free choice (hybrid)	2622	0.516	49.4%
Mea	Autonomous only	498	0.668	66.7%
	Tools only	498	0.731	74.7%
	Free choice (hybrid)	2654	0.748	76.5%

Appendix

Table 10: Mean faithfulness by model and Proposer strategy mode.

Modality	Category	Baseline	Trained
Tabular	Feature Attribution	0.446	0.578
	Counterfactual	0.398	0.763
	Spurious Features	0.356	0.933
	Overall	0.419	0.671
Text	Feature Attribution	0.325	0.534
	Counterfactual	0.413	0.480

Appendix

Table 11: Faithfulness by question category, 4B backbone. Best per row in bold .

Category	Base	Mea	$\Delta$	$p$
Feature Attribution (Q1–4)	0.407	0.626	$+0.219$	$1.0\mathrm{e}{-6}$
Counterfactual (Q5–7)	0.653	0.941	$+0.288$	$1.2\mathrm{e}{-5}$
Spurious Features (Q10)	0.725	1.000	$+0.275$	$0.0022$

Appendix

Table 12: Faithfulness score by question category on the biased classifier, paired

t

-test by (q_type, row_no).

Baseline	$n$	Mean $\pm$ SEM	$\Delta$ ( Mea $-$ BL)	$t$	$p$	Sig.
SHAP	59	$0.018\pm 0.007$	$+0.481$	$16.26$	$<0.001$
LIME	96	$0.259\pm 0.027$	$+0.240$	$6.09$	$<0.001$
Integrated Gradients	96	$\mathbf{0.481\pm 0.029}$	$\mathbf{+0.018}$	$\mathbf{0.44}$	$\mathbf{0.661}$	n.s.
Guided Backprop	96	$\mathbf{0.497\pm 0.027}$	$\mathbf{+0.003}$	$\mathbf{0.07}$	$\mathbf{0.944}$	n.s.
SmoothGrad	96	$\mathbf{0.496\pm 0.030}$	$\mathbf{+0.003}$	$\mathbf{0.08}$	$\mathbf{0.935}$	n.s.
GradCAM	96	$0.131\pm 0.017$	$+0.368$	$10.95$	$<0.001$

Appendix

Table 13: Welch’s

t

-test: Mea vs. tool-only XAI baselines (vision, feature attribution Q1). Mea mean

=0.499\pm 0.029

(

n=96

Modality	Baseline	$n$	Mean $\pm$ SEM	$\Delta$ ( Mea $-$ BL)	$t$	$p$
Text	Base Agent	156	$0.949\pm 0.017$	$+0.051$	$3.09$	$0.002$
	GPT-5.4	157	$0.733\pm 0.035$	$+0.268$	$7.67$	$<0.001$
	Claude-Haiku-4.5	158	$0.665\pm 0.037$	$+0.335$	$9.11$	$<0.001$
	Gemini-3.1	160	$0.996\pm 0.001$	$+0.004$	$2.92$	$0.004$
Vision	Base Agent	148	$0.981\pm 0.004$	$+0.019$	$5.15$	$<0.001$
	GPT-5.4	156	$0.921\pm 0.006$	$+0.079$	$12.53$	$<0.001$

Appendix

Table 14: Welch’s

t

-test: Mea (

1.000\pm 0.000

n=160

) vs. baselines on spurious-feature questions (Q10).

Modality	Question Group	$n_{\text{pairs}}$	Paraphrase ( $\mu\pm$ SEM)	Mea ( $\mu\pm$ SEM)	$\Delta$	$t$	$p$	Sig.
Tabular	Feature Attribution	106	$0.683\pm 0.037$	$0.687\pm 0.037$	$-0.004$	$-0.63$	$0.532$	n.s.
	Counterfactual	32	$0.934\pm 0.023$	$0.939\pm 0.021$	$-0.005$	$-1.98$	$0.056$	n.s.
	Spurious Features	17	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.000$	–	–	n.s.
Text	Feature Attribution	126	$0.738\pm 0.030$	$0.746\pm 0.031$	$-0.008$	$-0.83$	$0.407$	n.s.
	Counterfactual	85	$0.947\pm 0.012$	$0.952\pm 0.012$	$-0.005$	$-0.73$	$0.469$	n.s.
	Spurious Features	37	$1.000\pm 0.000$	$1.000\pm 0.000$	$0.000$	–	–	n.s.

Appendix

Table 15: Paired

t

-test: Paraphrase vs. Mea (matched common samples).

\Delta=

Paraphrase

-

Mea .

Panel of Figure 10 — Figure 10: Complete prompt structure of the Critic Agent.

Panel of Figure 11 — Figure 11: Prompt structure for the Actor Agent. Task directives, constraints, and JSON schema keys are populated dynamically at runtime based on the specific question type (Q1-Q10) and data modality.

Panel of Figure 12 — Figure 12: Prompt structure for the Proposer Agent. Task directives are populated dynamically based on the active question type (Q1-Q10), ensuring the strategy generated aligns with the specific objective.

Explore similar work

May 27, 2026

Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

Explainable AI (XAI) helps users interpret model behavior and identify potential faults. Agentic XAI systems use Large Language Models (LLMs) to make explanations more accessible through natural-language interaction, but they can also produce plausible yet unfaithful explanations. This risk arises because unreliable XAI outputs for complex models can be amplified by LLMs and mislead users. We propose Faithful Agentic XAI (FAX), a framework that improves explanation faithfulness through explicit verification. FAX decomposes draft explanations into claims and cross-checks them against inherently faithful tools, filtering unsupported or contradictory claims before final generation. We also introduce CRAFTER-XAI-Bench, an open-world reinforcement learning benchmark with complex policies, diverse goals, and challenging scenarios for assessing model-specific faithfulness. On CRAFTER-XAI-Bench, FAX improves simulation faithfulness from 0.20 for the strongest baseline to 0.46 while maintaining high informativeness, relevance, and fluency. On three tabular benchmarks, FAX performs competitively with prior Agentic XAI baselines, but our analysis shows that these settings can conflate task accuracy with model-specific faithfulness. These findings show that explicit verification is essential for faithful Agentic XAI and that that faithfulness benchmarks must be designed to test explanations against the behavior of the target model itself.

Jaechang Kim, Sunung Mun, Seungjoon Lee +2

Graduate School of AI, POSTECH · Department of Computer Science and Engineering, POSTECH · Krafton

Explainable Artificial Intelligence AI Agent Benchmarks Explainability Evaluation

Ask PDF

May 2, 2026

Automated Interpretability and Feature Discovery in Language Models with Agents

We introduce an autonomous multiagent framework for mechanistic interpretability that automates both explaining and finding internal features in large language models. The system runs two coupled loops: (1) explanation refinement, where an agent proposes competing hypotheses and iteratively tests them with targeted prompt controls and a multi-metric evaluation; and (2) feature discovery, where an agent generates prompt sets, constructs a k-nearest-neighbor graph in activation space, and retrieves candidate features using statistical separability and semantic coherence criteria. On Gemma-2 family models and MLP neurons in weight-sparse transformers, our agent improves over one-shot auto-interpretations, discovers language-specific and safety-relevant features, and produces auditable explanation traces, showing that agent-driven empirical loops yield sharper and more falsifiable explanations than one-shot labels.

Arnau Marin-Llobet, Javier Ferrando

Harvard University

Mechanistic Interpretability

Ask PDF

Apr 15, 2026

Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance

Large language models (LLMs) achieve strong performance and have revolutionized NLP, but their lack of explainability keeps them treated as black boxes, limiting their use in domains that demand transparency and trust. A promising direction to address this issue is post-hoc text-based explanations, which aim to explain model decisions in natural language. Prior work has focused on generating convincing rationales that appear to be subjectively faithful, but it remains unclear whether these explanations are epistemically faithful, whether they reflect the internal evidence the model actually relied on for its decision. In this paper, we first assess the epistemic faithfulness of LLM-generated explanations via counterfactuals and show that they are often unfaithful. We then introduce a training-free method that enhances faithfulness by guiding explanation generation through attention-level interventions, informed by token-level heatmaps extracted via a faithful attribution method. This method significantly improves epistemic faithfulness across multiple models, benchmarks, and prompts.

Bar Alon, Itamar Zimerman, Lior Wolf

Blavatnik School of Computer Science, Tel Aviv University

Faithfulness of Language Model Explanations LLM Interpretability

Ask PDF