AGRI-Fidelity: Evaluating the Reliability of Listenable Explanations for Poultry Disease Detection

Sindhuja Madabushi; Arda Dogan; Jonathan Liu; Dian Chen; Dong S. Ha; Sook Shin; Sam H. Noh; Jin-Hee Cho

cs.LGMar 18, 2026

AGRI-Fidelity: Evaluating the Reliability of Listenable Explanations for Poultry Disease Detection

Authors: Sindhuja Madabushi, Arda Dogan, Jonathan Liu, Dian Chen, Dong S. Ha, Sook Shin, Sam H. Noh, Jin-Hee Cho

Organizations: Virginia Tech

Abstract

Existing XAI metrics measure faithfulness for a single model, ignoring model multiplicity where near-optimal classifiers rely on different or spurious acoustic cues. In noisy farm environments, stationary artifacts such as ventilation noise can produce explanations that are faithful yet unreliable, as masking-based metrics fail to penalize redundant shortcuts. We propose AGRI-Fidelity, a reliability-oriented evaluation framework for listenable explanations in poultry disease detection without spatial ground truth. The method combines cross-model consensus with cyclic temporal permutation to construct null distributions and compute a False Discovery Rate (FDR), suppressing stationary artifacts while preserving time-localized bioacoustic markers. Across real and controlled datasets, AGRI-Fidelity effectively provides reliability-aware discrimination for all data points versus masking-based metrics.

Figures & tables

Panel of Figure 1 — Figure 1: AGRI-Fidelity Framework for Reliability-Oriented Evaluation via Tiered Cross-Model Consensus and Permutation-Based FDR Validation.

Panel of Figure 2 — Figure 2: Cross-Model Attribution Consistency: Integrated gradients for a healthy sample in the denoised poultry dataset across CNN, MLP, LSTM, and ResNet consensus models.

Baseline	AI $\uparrow$	AD $\downarrow$	AG $\uparrow$	Sparse $\uparrow$	Comp. $\downarrow$	Faith. $\uparrow$	AGRI-Fidelity
							Fidelity $\uparrow$	Reliability $\uparrow$
LMAC-TD	34.0500 $\pm$ 3.1955	5.3698 $\pm$ 1.6220	21.3799 $\pm$ 0.8626	0.0140 $\pm$ 0.0033	12.2920 $\pm$ 0.0001	0.0739 $\pm$ 0.0243	0.9128 $\pm$ 0.0273	0.4285 $\pm$ 0.3958
L2I	30.1667 $\pm$ 3.9847	7.7010 $\pm$ 1.2752	20.3380 $\pm$ 2.9614	0.0661 $\pm$ 0.0086	12.2993 $\pm$ 0.0019	0.1471 $\pm$ 0.0300	0.5600 $\pm$ 0.0343	0.3503 $\pm$ 0.3832
CoughLIME	49.8724 $\pm$ 40.7621	44.5938 $\pm$ 43.4127	61.0600 $\pm$ 47.8459	0.3315 $\pm$ 0.2987	13.3498 $\pm$ 0.0031	0.1214 $\pm$ 0.1047	0.1501 $\pm$ 0.0601	0.6715 $\pm$ 0.3289
AudioLIME	27.6923 $\pm$ 3.7621	3.2820 $\pm$ 3.4127	2.6643 $\pm$ 2.8459	0.4011 $\pm$ 0.0187	0.2091 $\pm$ 0.0031	0.3619 $\pm$ 0.0368	0.9157 $\pm$ 0.0214	0.3987 $\pm$ 0.3849

Table 1: Quantitative Comparison of Baseline Explainers Across Standard Metrics and AGRI-Fidelity Components: Average Increase (AI

\uparrow

), Average Drop (AD

\downarrow

), Average Gain (AG

\uparrow

), Sparseness

\uparrow

, Complexity

\downarrow

, Faithfulness

\uparrow

, and AGRI-Fidelity (Fidelity

\uparrow

and Reliability

\uparrow

) for the Poultry dataset with a spurious feature.

Dataset	Model	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Spec $\uparrow$	FPR $\downarrow$
Chicken Denoised	CNN	0.8654	0.9306	0.7667	0.8231	0.9360	0.0640
	MLP	0.9096	0.9189	0.8781	0.8954	0.9322	0.0678
	LSTM	0.9615	0.9987	0.8889	0.9412	0.9998	0.0100
	ResNet	0.9385	0.9321	0.9436	0.9366	0.9313	0.0687
Chicken Spurious	CNN	0.9865	0.9805	0.9913	0.9855	0.9823	0.0177
	MLP	0.9365	0.9526	0.9036	0.9229	0.9618	0.0382

Table 2: Performance Comparison of Consensus Architectures Across Datasets: For each dataset, we report the performance of CNN, MLP, LSTM, and ResNet consensus models. Columns report Accuracy (Acc

\uparrow

), Precision (Prec

\uparrow

), Recall (Rec

\uparrow

), F1

\uparrow

, Specificity (Spec

\uparrow

), and False Positive Rate (FPR

\downarrow

Panel of Figure 3 — Figure 3: Fidelity–Reliability Quadrant Distributions Across Datasets. Each plot shows per-sample AGRI-Fidelity components using CoughLIME Explainer. Denoised datasets exhibit stronger concentration in the high-fidelity, high-reliability quadrant, whereas noisy datasets demonstrate reliability dispersion under spurious contamination.

Figure 4: Fidelity–Reliability Quadrant Analysis on the Controlled Spurious Dataset. Each point represents a sample plotted by its mean fidelity and reliability score. Dashed lines denote median splits, forming four interpretative quadrants.

Class	Total	Downward Trend	Near-1 Trend
Healthy	139	120	19
Unhealthy	121	40	81

Table 3: Distribution of FDR Trend Types Across Class Labels: For each class, we report the total number of samples and the count exhibiting Downward and Near-1 FDR trends. A Downward trend indicates rapid FDR decay with increasing consensus threshold, whereas a Near-1 trend indicates persistently high FDR across consensus levels.

Class	Trend	CNN	LSTM	MLP	ResNet
Healthy	Downward	0.1589	0.0206	0.0457	0.0159
Healthy	Near-One	0.121	0.0319	0.0137	0.0175
Unhealthy	Downward	0.5161	0.2194	0.1833	0.4263
Unhealthy	Near-One	0.7994	0.4519	0.7040	0.8542

Table 4: Average Fidelity Trends Across Classes and Models: For each class and trend type (Downward, Near-One), we report mean fidelity values for CNN, LSTM, MLP, and ResNet consensus models to enable cross-architecture comparison.

Explainability Evaluation

Explainability evaluation tests whether AI explanations are faithful, stable, understandable, useful, and reliable across models and settings.

See all 321 papers

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification
Aug 24, 2026Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao +2Explainability Evaluation Black-Box Model Auditing
Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar
FecalFed: Privacy-Preserving Poultry Disease Detection via Federated Learning
Apr 1, 2026Tien-Yu ChiClassification Federated Learning
Independent Researcher
Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness
May 27, 2026Jaechang Kim, Sunung Mun, Seungjoon Lee +2Explainable Artificial Intelligence AI Agent Benchmarks
Graduate School of AI, POSTECH · Department of Computer Science and Engineering, POSTECH · Krafton

Aug 24, 2026J/K move · Enter open · S save

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

Rosa Elysabeth Ralinirina, Jean Christian Ralaivao, Niaiko Michaël Ralaivao, Alain Josué Ratovondrahona+1

Doctoral School Modeling–Computer Science, University of Fianarantsoa, Madagascar

Explainability Evaluation Black-Box Model Auditing Feature Attribution SHAP Feature Attribution

Open paper AskPDF arXiv

SHAP and LIME are now standard tools for interpreting black-box predictions, yet their outputs can vary substantially when the input is perturbed by small amounts of noise--a problem we observed firsthand in our previous work on food security in Madagascar (Ralinirina et al., 2025). This variability raises the question of whether such explanations can be trusted at all. We address it by constructing an auditing protocol that measures two properties of any post-hoc explainer: robustness (how stable the explanation is under input perturbation) and fidelity (whether the features deemed important actually drive the model's prediction). These two quantities are combined into a single Trust Score. We run the protocol on a multi-sectoral dataset from Madagascar (83 features, 253 records, 4 malnutrition classes) using three classifiers and two explainers, plus their regularized counterparts. The results are sobering: models with AUC above 0.99 can produce numerically degenerate or flatly uninformative explanations, and fidelity scores lose discriminative power when the model is overfitted. These findings suggest that auditing XAI outputs is not optional but necessary, particularly when they inform decisions in sensitive domains.

Abstract

Explainability Evaluation

Explore similar work

A Formal Methodological Framework for Auditing Robustness and Fidelity in Explainable AI: From Application to Trust Certification

FecalFed: Privacy-Preserving Poultry Disease Detection via Federated Learning

Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness