Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

Thackshanaramana B

cs.LGMar 16, 2026

Hypothesis Class Determines Explanation: Why Accurate Models Disagree on Feature Attribution

Authors: Thackshanaramana B

Organizations: SRM Institute of Science and Technology, India

Abstract

The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.

Figures & tables

Claim	Empirical Evidence	Section
$\Delta\geq c>0$	$\Delta=0.261$ , $p<0.001$ , Cohen’s $d=0.92$	Table 3
Asymptotic persistence	Same-split: $\Delta=0.631$ , lottery rate 61.9%	Section 5.3
Split invariance	Within-class $\rho=1.000$ , cross-class $\rho=0.369$	Section 5.3
Dimensionality effect	$r=-0.251$ , $p<0.001$ ; $\rho_{\text{inter}}=0.287$ at $d>50$	Section 5

Table 1: Empirical validation of Theorem 1 . Each claim is directly testable; all four are confirmed.

Overall: Mean $\rho=0.572$ , Median $=0.615$ , SD $=0.293$
Metric	$\tau{=}0.3$	$\tau{=}0.4$	$\tau{=}0.5$	$\tau{=}0.6$	$\tau{=}0.7$	$\tau{=}0.8$
Lottery Rate	18.0%	25.0%	35.4%	45.7%	57.7%	71.9%
95% CI	$\pm$ 0.25	$\pm$ 0.28	$\pm$ 0.31	$\pm$ 0.32	$\pm$ 0.32	$\pm$ 0.29

Table 2: Lottery rates at multiple thresholds. Our primary threshold is

\tau=0.5

(feature rankings more dissimilar than similar).

	XGB	LGB	Cat	RF	LR
XGBoost	—	0.734	0.712	0.612	0.425
LightGBM		—	0.721	0.587	0.415
CatBoost			—	0.595	0.412
RandomForest				—	0.398
Logistic Reg.					—

Table 3: Mean pairwise explanation agreement (Spearman

\rho

). Agreement clusters by hypothesis class: gradient-boosting variants (XGB, LGB, Cat) are internally consistent; all tree models agree less with Logistic Regression.

Figure 1: Distribution of pairwise Spearman $\rho$ under the controlled same-split experiment. Within-class pairs (XGB vs. XGB, LR vs. LR) concentrate at $\rho=1.000$ ; cross-class pairs (XGB vs. LR) spread across the full range with median $\rho=0.364$ , well below the lottery threshold ( $\tau=0.5$ , dashed line). The structural gap $\Delta=0.636$ directly validates Theorem 1 (Claims 1–2): the Agreement Gap is bounded away from zero and persists when training variance is eliminated.

Comparison	Type	Mean $\rho$	Lottery Rate
Linear vs. Linear	Intra-class	0.827	6.7%
Tree vs. Tree	Intra-class	0.717	10.0%
Neural vs. Neural	Intra-class	0.552	34.4%
Linear vs. Tree	Cross-class	0.525	40.1%
Neural vs. Tree	Cross-class	0.509	48.8%
Linear vs. Neural	Cross-class	0.509	46.9%

Table 4: Mean Spearman

\rho

and lottery rate across all pairwise hypothesis class comparisons. Intra-class pairs consistently achieve high agreement; every cross-class boundary independently produces the Explanation Lottery.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	$d$	Acc.	Domain	Source
diabetes	768	8	0.781	Healthcare	OpenML-37
breast-cancer	699	9	0.967	Healthcare	OpenML-13
heart-disease	303	13	0.843	Healthcare	OpenML-43
credit-g	1,000	20	0.764	Finance	OpenML-31
credit-approval	690	15	0.870	Finance	OpenML-29
bank-marketing	45,211	16	0.899	Finance	OpenML-1461

Appendix

Table 5: Complete dataset specifications used in this study.

Feature Attribution

Feature attribution estimates which input features or internal components contribute to a model’s predictions.

See all 219 papers

Evaluating Explanation Methods by the Predictors They Induce
Sep 17, 2026Jacob Selbæk, Hugo L. HammerFeature Attribution SHAP Feature Attribution
Department of Computer Science, Oslo Metropolitan University, Oslo, Norway · Department of Holistic Systems, SimulaMet, Oslo, Norway · Department of Plastic and Reconstructive Surgery, Oslo University Hospital, Oslo, Norway
A Causal Argumentation Method for Explainability of Machine Learning Models
May 20, 2026Henry Salgado, Meagan R. Kendall, Martine CeberioExplainable Artificial Intelligence Causal Explanation
Department of Computer Science, The University of Texas at El Paso, El Paso, TX, USA · Department of Engineering Education and Leadership, The University of Texas at El2026 Paso, El Paso, TX, USA
All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models
Jul 10, 2026Pan LiExplainable Artificial Intelligence LLM Interpretability
Scheller College of Business, Georgia Tech.

Sep 17, 2026J/K move · Enter open · S save

Evaluating Explanation Methods by the Predictors They Induce

Jacob Selbæk, Hugo L. Hammer

Department of Computer Science, Oslo Metropolitan University, Oslo, Norway · Department of Holistic Systems, SimulaMet, Oslo, Norway · Department of Plastic and Reconstructive Surgery, Oslo University Hospital, Oslo, Norway

Feature Attribution SHAP Feature Attribution Explainability Evaluation Explanation Evaluation

Open paper AskPDF arXiv

Explanations of machine learning models are usually judged by criteria that are hard to compare. We propose a simpler test: if an explanation really describes how a model uses its features, it should be possible to rebuild the model's predictions from it. We turn each explanation into a predictor by reading each feature's effect and adding them up, and measure how well that predictor reproduces the model on unseen data. Nothing is fitted, so the score reflects the explanation itself. The test applies to any explanation that can be written as a function of the features; we demonstrate it on partial dependence plots (PDP), accumulated local effects (ALE), SHAP and LIME. We prove that summing partial dependence curves gives the best possible additive summary of a model when its features are independent, and that this fails when they are dependent. Across 13 real datasets and 9 synthetic designs and four model families, which method scores best depends entirely on feature dependence: where features are independent SHAP is slightly worse than PDP, exactly as the theory predicts; on dependent real data SHAP leads. Some widely used quality metrics even prefer a damaged explanation to an intact one.

Abstract

Feature Attribution

Explore similar work

Evaluating Explanation Methods by the Predictors They Induce

A Causal Argumentation Method for Explainability of Machine Learning Models

All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models