Post-Hoc Feature Selection Layer for Neural Networks Interpretability

João Kenji Suwa; João Pedro Silveira e Silva; Bruno Iochins Grisci

cs.LGOct 22, 2026

Post-Hoc Feature Selection Layer for Neural Networks Interpretability

Authors: João Kenji Suwa, João Pedro Silveira e Silva, Bruno Iochins Grisci

Organizations: Institute of Informatics, UFRGS, Porto Alegre, Brazil

Abstract

The interpretability of complex neural networks remains a critical challenge, especially for models already deployed in high-stakes domains. To address this, we adapted the Feature Selection Layer (FSL) to optimize feature weights post-training, a variant we term post-hoc FSL. Our approach reframes the FSL as a lightweight, trainable module that integrates with already frozen pre-trained models on tabular datasets to highlight the features the original model considers most important. This post-hoc FSL learns feature relevance by fine-tuning its weights based on the original model's learned outputs. Crucially, this process is non-invasive, operating without altering the original model's architecture or its learned parameters. We conducted our experiments using both statistical and visual metrics, including accuracy, F1 score, recall, precision, weighted t-SNE and silhouette score, and also analyzed the stability of the post-hoc FSL on high-dimensional synthetic and real-world tabular datasets. We compared the post-hoc FSL feature weighting method using these metrics against the original embedded FSL and other post-hoc interpretability methods, such as Integrated Gradients, Noise Tunnel, DeepLIFT, Gradient SHAP, and Feature Ablation. Experimental results demonstrate that the post-hoc FSL feature weighting method successfully identified relevant features across the different datasets, maintaining the predictive power of the original neural network while enhancing its interpretability.

Figures & tables

Fig. 1 : Representation of post-hoc Feature Selection Layer. In this example, four input features are individually multiplied by their activated weights within the FSL before being passed to the pre-trained model. During fine-tuning, only the weights within the FSL are updated, while all parameters of the pre-trained model remain frozen.

Name	Feature type	Labels	Features	Informative	Folds	Samples	Test Split
XOR	Binary	2	50	2	11	500	20%
SynthA	Numerical	2	100	30	11	3000	10%
Spam	Numerical	2	3000	Unknown	11	5172	20%
Liver-22405	Numerical	2	22284	Unknown	15	48	10%
Breast-45827	Numerical	6	54676	Unknown	15	151	10%

TABLE I : Datasets used for the experiments

Model	Jaccard	Spearman	Pearson
Integrated Gradients	$0.855\pm 0.028$	$0.758\pm 0.038$	$0.969\pm 0.009$
Noise Tunnel	$0.954\pm 0.031$	$0.752\pm 0.038$	$0.974\pm 0.007$
Deep Lift	$0.976\pm 0.034$	$0.760\pm 0.037$	$0.971\pm 0.008$
Gradient SHAP	$0.939\pm 0.041$	$0.750\pm 0.041$	$0.975\pm 0.006$
Feature Ablation	$0.956\pm 0.050$	$0.759\pm 0.037$	$0.980\pm 0.005$
FSL	$\mathbf{1.000\pm 0.000}$	$\mathbf{0.805\pm 0.030}$	$\mathbf{0.980\pm 0.003}$

TABLE II : Results for stability metrics using the SynthA dataset. Bold indicates the highest average. Jaccard was calculated using the top-

30

weighted features.

Panel of Figure 4 — Fig. 2 : Weighted t-SNE of the post-hoc FSL and Kernel SHAP methods using the TabPFN and the SynthA dataset.

TABLE III : Prediction performance for real-world datasets. Bold indicates the highest average; best statistically significant results (

p<0.01

) from Kruskal-Wallis and Dunn’s tests are marked in orange.

Model	Spam	Liver	Breast
None	$0.090\pm 0.000$	$0.066\pm 0.000$	$0.192\pm 0.000$
Integrated Gradients	$0.202\pm 0.031$	$0.170\pm 0.031$	$0.451\pm 0.036$
Noise Tunnel	$0.189\pm 0.034$	$0.178\pm 0.044$	$0.447\pm 0.045$
Deep Lift	$0.202\pm 0.025$	$0.172\pm 0.049$	$0.449\pm 0.030$
Gradient SHAP	$0.209\pm 0.022$	$0.199\pm 0.042$	$0.450\pm 0.040$
Feature Ablation	$0.196\pm 0.029$	$0.181\pm 0.030$	$0.450\pm 0.033$

TABLE IV : Silhouette score for real-world dataset. Bold indicates the highest average; best statistically significant results (

p<0.01

) from Kruskal-Wallis and Dunn’s tests are marked in orange.

Panel of Figure 7 — Fig. 3 : Weighted t-SNE of all feature weighing metrics using the spam dataset.

Model	Jaccard	Spearman	Pearson
Integrated Gradients	$0.263\pm 0.078$	$\mathbf{0.649\pm 0.091}$	$0.739\pm 0.081$
Noise Tunnel	$0.226\pm 0.102$	$0.647\pm 0.092$	$0.728\pm 0.088$
Deep Lift	$0.263\pm 0.077$	$\mathbf{0.649\pm 0.091}$	$0.739\pm 0.081$
Gradient SHAP	$0.122\pm 0.047$	$0.599\pm 0.098$	$0.666\pm 0.084$
Feature Ablation	$0.261\pm 0.077$	$\mathbf{0.649\pm 0.092}$	$0.739\pm 0.081$
FSL	$0.174\pm 0.064$	$0.017\pm 0.016$	$0.454\pm 0.180$

TABLE V : Results for stability metrics using the Liver dataset. In bold is the highest average of each metric. Jaccard was calculated using the top-

100

weighted features.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 328 papers

A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks
Oct 16, 2026João Pedro Silveira e Silva, Márcio Dorn, Bruno Iochins GrisciFeature Selection Feature Attribution
Institute of Informatics, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · Center for Biotechnology, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · National Institute of Science and Technology - Forensic Science, Porto Alegre, RS, Brazil
SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation
Apr 28, 2026Ali Karkehabadi, Jamshid Hassanpour, Houman Homayoun +1Gradient-Based Attribution Neural Network Interpretability
University of California, Davis, USA · Georgia Institute of Technology, USA
From Weight Perturbation to Feature Attribution for Explaining Fully Connected Neural Networks
May 14, 2026Thodoris Lymperopoulos, Denia KanellopoulouFeature Attribution Perturbation-Based Feature Attribution
NCSR Demokritos Athens, Greece

Oct 16, 2026J/K move · Enter open · S save

A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks

João Pedro Silveira e Silva, Márcio Dorn, Bruno Iochins Grisci

Institute of Informatics, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · Center for Biotechnology, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · National Institute of Science and Technology - Forensic Science, Porto Alegre, RS, Brazil

Feature Selection Feature Attribution Neural Network Interpretability Multiclass Classification

Open paper AskPDF arXiv

In the age of data-driven decision-making, deep learning models have become powerful tools for uncovering patterns and making predictions that influence many aspects of society. However, their broader adoption in scientific and business applications is often limited by a lack of interpretability. Feature weighting methods, which assess the relevance of individual features, are among the most widely studied approaches for improving interpretability. While post hoc techniques have advanced significantly, especially in modeling nuanced relationships between features and labels in multiclass problems, embedded methods typically fall short in this regard. Embedded techniques integrate feature relevance learning directly into the model training process, but they are often unable to capture label-specific relationships. To address the limitation, we propose multiple extensions to the Feature Selection Layer, an embedded feature weighting mechanism for neural networks. Through extensive experimentation, we identify the most effective extension and benchmark it against established feature weighting methods. Our results show that the proposed method matches existing techniques in identifying globally relevant features and outperforms them in capturing label-specific relevance. By leveraging the internal dynamics of the trained neural network, it reveals how individual features contribute to the prediction of specific labels, requiring no post-processing beyond the training process.

Abstract

Neural Network Interpretability

Explore similar work

A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks

SaliencyDecor: Enhancing Neural Network Interpretability through Feature Decorrelation

From Weight Perturbation to Feature Attribution for Explaining Fully Connected Neural Networks