Integrating Feature Understandability into Feature Selection for Human-Centred Tabular Explanations
Authors: Nicola Rossberg, Bennett Kleinberg, Barry O'Sullivan, Luca Longo, Andrea Visentin
Organizations: School of Computer Science and Information Technology, University College Cork, Ireland · Centre for Research Training in Artificial Intelligence, University College Cork, Ireland · Department of Methodology and Statistics, Tilburg University · Insight Centre for Data Analytics, University College Cork, Ireland
Existing human-centred XAI research predominantly evaluates explanations after they have been generated. However, this fails to provide actionable insights into improving accessibility. This study incorporates human-based feature understandability into feature selection to improve the generation of human-centred explanations. Natural language explanations for machine learning models trained on tabular data often cite influential features in the input data to explain a given decision. As a consequence, users' ability to understand the produced explanation hinges on their understanding of the cited features. To quantify this natural-language feature understandability, Rossberg et al. proposed the Feature Understandability Scale, which assigns numerical `understandability scores' to features based on user feedback. To evaluate the degree to which these scores can improve machine learning explanation quality, this study collects understandability scores across two datasets, proposes an understandability-aware feature selection algorithm to select features that are both understandable and informative, and trains a random forest based on these feature subsets. The SHAP-based explanations generated by this model are evaluated using user feedback, and changes in explanation quality are assessed. The individual evaluation of the explanations showed that while the understandability-based explanations scored higher on average, the difference was not statistically significant. In contrast, the comparative evaluation found that participants preferred these explanations significantly more frequently than expected at the chance level. It is concluded that the integration of understandability can yield more understandable explanations at minimal cost to classification performance, although changes to quality require elucidation in larger cohorts.
Figures & tables
Figure 1: Graphical depiction of the theoretical basis on which the FUS implementation is expected to increase the understandability of explanations for tabular data.
Figure 2: Visualisation of the Repeated Monte Carlo Sampling, a) co-optimisation and b) informative setting of the feature selection, and machine learning model training methodology.
Figure 3: The Survey Flow of the evaluation procedure. Participants were randomly assigned to the comparative or individual evaluation, and the order of datasets was randomised for each condition. The individual setting used the explanation quality scale [ 67 ] to evaluate informative and co-optimised explanations separately, whereas the comparative setting prompted participants to select the more understandable explanation and provided prototypes of both co-optimised and informative explanations. Two or five instances were selected, respectively, and participants were asked to either use the scale to evaluate them (individual evaluation) or select the more understandable explanation (comparative evaluation).
Figure 4: Distribution of US across the two datasets
Telco Data
Mean
SD
Lower 95% CI
Higher 95% CI
Balanced Accuracy
0.733
0.009
0.731
0.736
F1
0.603
0.013
0.600
0.607
Precision
0.569
0.013
0.565
0.573
Recall
0.642
0.020
0.637
0.648
Medical Data
Table 1: Distributional Data of the Repeated Monte Carlo Sampling for both Datasets
Telco Dataset
Medical Dataset
Co-Optimised
Informative
Co-Optimised
Informative
Balanced Accuracy
75.25
75.35
75.54
75.40
F1
61.11
61.03
70.21
70.09
Precision
49.29
48.48
55.64
55.40
Recall
80.39
82.35
95.11
95.37
Mean US
4.16
3.96
3.79
3.54
Table 2: Results of the Co-Optimisation Procedure for both datasets
Telco Dataset
Medical Dataset
Co-Optimised
Informative
Co-Optimised
Informative
Internet Service, Tech Support , Total Charges, Monthly Charges, Contract
Tenure, Online Backup, Online Security, Tech Support , Contract
Table 3: Features selected for each dataset in the co-optimised and informative setting. The features are ranked by importance as determined by SHAP. Features selected by both settings are bolded.
Figure 5: Distribution of Scale ratings across the two datasets and settings.
Telco Dataset
Medical Dataset
Fixed Effects
Estimate
SE
95% CI
t (df)
p
Estimate
SE
95% CI
t (df)
p
(Intercept)
29.55
1.31
[26.95, 32.14]
22.64 (29.20)
< .001
31.16
0.99
[29.20, 33.12]
31.33 (61.86)
< .001
Exp. Type [Info]
− 1.03
0.75
[ − 2.51, 0.45]
− 1.38 (164.41)
.171
− 0.08
0.45
[ − 0.97, 0.80]
− 0.19 (167.18)
.851
Random Effects
SD (Coefficients)
SD (Int: Participant)
6.99
7.01
SD (Int: Example)
2.43
0.74
Table 4: Results of the mixed effects model analysis for the two datasets
Telco Dataset
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Sum
Mean
SD
Informative
13
10
9
8
11
17
15
11
9
12
115
11.5
2.84
Co-Optimised
18
20
21
23
17
12
17
19
19
19
185
18.5
2.92
Medical Dataset
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Q10
Sum
Mean
SD
Informative
5
13
7
4
15
12
17
9
9
9
100
10
4.22
Co-Optimised
26
18
23
27
11
19
15
20
20
21
200
20
4.78
Table 5: Results of the forced-choice analysis with the preferred option for each question in bold.
Telco Dataset
Medical Dataset
Fixed Effects
Estimate
SE
95% CI
z
p
Estimate
SE
95% CI
z
p
(Intercept)
0.60
0.20
[0.22, 1.02]
3.05
.002
0.92
0.28
[0.36, 1.55]
3.32
< .001
Derived Estimates
Back-transformed from the intercept
Odds ratio [95% CI]
1.82 [1.24, 2.67]
2.52 [1.46, 4.34]
P (co-optimised) [95% CI]
64.5% [55.3, 72.8]
71.6% [59.3, 81.3]
Random Effects
SD (Coefficients)
Table 6: Results of the mixed-effects logistic regression for the comparative evaluation (outcome: co-optimised explanation chosen = 1).
Number
Facet
Item
1
Actionability
I have learned something from the explanation
2
Causality rev.
The relationship between the input data and the predictions is vague
3
Comprehensibility rev.
The structure of the explanation is not clear
4
Efficiency rev.
It took me a long time to understand the explanation
5
Explicitness rev.
The explanation is incomprehensible
6
Informativeness
The explanation provides useful information
Table 7: Scale adapted from Vilone & Longo used for the individual explanation evaluation.
Strongly Disagree
Disagree
Neutral
Agree
Strongly Agree
I can understand the scale (units) of the feature.
x
I can easily understand if a given value of the feature is high or low.
x
I know what this feature measures.
x
I know what this feature represents.
x
I think that I can easily access a definition of the feature.
x
In my opinion, the feature should be used to predict the outcome.
x
Table 8: “Please rate the feature ‘Credit Score’ using the following questions. The scores of ‘Credit Score’ range from 18.4 to 118.65. The feature is used to predict whether or not a customer’s loan application will be approved. While rating the feature, consider if you would understand a loan decision, if it was justified using the feature. E.g. ’The customer did not get a loan because their Credit Score was 56.95’."
Despite the wide use of explainability techniques to attempt to understand the behavior of Artificial Intelligence (AI), the generated explanations may not always be reliable. An explanation can appear plausible to humans but fail to capture the internal reasoning of a model, particularly when dealing with complex tabular data. This paper studies the trustworthiness of local explainability techniques when applied to complex tabular classification tasks, considering evaluated metrics for three main properties: faithfulness to the model's predictions, robustness to input data variations, and complexity of the explanation itself. A benchmark was performed for Local Interpretable Model-Agnostic Explanations (LIME), Kernel SHapley Additive exPlanations (SHAP), and Feature Ablation techniques, across 32 datasets and different types of machine learning models. Model performance ranges were analyzed to identify two groups: consensus-correct, which are samples that all models predicted correctly, and consensus-wrong, samples that all models predicted incorrectly. The obtained results demonstrate that that the explanations are not always correlated with a model's predictive performance. Instead, dataset complexity and feature distributions seem to be the main factors affecting explanation quality and reliability.
Tomás Pereira, João Vitorino, Eva Maia +1
GECAD, ISEP, Polytechnic of Porto, Porto, Portugal
Explainable AI (XAI) often prioritizes model fidelity over user intelligibility, yielding explanations that satisfy technical transparency yet fail to support human sense-making. We propose a human-centered approach, MIND-X (Mental-model INtegrated Design for eXplainability), which aligns explanations with users' mental models, commonsense concepts, and judgment heuristics. Using fake review detection as a case study, we contribute: (1) a qualitative study (n=10) that elicits how people assess review credibility and produces an actionable, design-guiding taxonomy of perceptual cues; (2) a set of design principles that link those cues to explanation strategies and prompt LLMs to generate adaptive, language-based explanations; and (3) a quantitative evaluation (n=166) showing that these explanations offer distinct advantages over token-level transparency methods widely used in explainable text classification in terms of understandability, satisfaction, perceived trust, persuasiveness, and decision-making support. Rather than claiming general superiority, our evaluation illustrates how grounding explanations in users' everyday reasoning practices can produce more intelligible and context-sensitive accounts. Our findings advance human-centered explanation design beyond a mere "gloss of transparency," demonstrating that a mental-model-aligned design process can improve intelligibility and support accountability across user-centered explanation tasks.
Sidra Naveed, Md Shajalal, Gunnar Stevens +1
University of Siegen, Germany · Hochschule Bonn-Rhein-Sieg
The assumption that prediction-equivalent models produce equivalent explanations underlies many practices in explainable AI, including model selection, auditing, and regulatory evaluation. In this work, we show that this assumption does not hold. Through a large-scale empirical study across 24 datasets and multiple model classes, we find that models with identical predictive behavior can produce substantially different feature attributions. This disagreement is highly structured: models within the same hypothesis class exhibit strong agreement, while cross-class pairs (e.g., tree-based vs. linear) trained on identical data splits show substantially reduced agreement, consistently near or below the lottery threshold. We identify hypothesis class as the structural driver of this phenomenon, which we term the Explanation Lottery. We theoretically show that the resulting Agreement Gap persists under interaction structure in the data-generating process. This structural finding motivates a post-hoc diagnostic, the Explanation Reliability Score R(x), which predicts when explanations are stable across architectures without additional training. Our results demonstrate that model selection is not explanation-neutral: the hypothesis class chosen for deployment can determine which features are attributed responsibility for a decision.