A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks

João Pedro Silveira e Silva; Márcio Dorn; Bruno Iochins Grisci

cs.LGOct 16, 2026

A Feature Selection Layer Approach for Multi-Class Classification on Neural Networks

Authors: João Pedro Silveira e Silva, Márcio Dorn, Bruno Iochins Grisci

Organizations: Institute of Informatics, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · Center for Biotechnology, Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil · National Institute of Science and Technology - Forensic Science, Porto Alegre, RS, Brazil

Abstract

In the age of data-driven decision-making, deep learning models have become powerful tools for uncovering patterns and making predictions that influence many aspects of society. However, their broader adoption in scientific and business applications is often limited by a lack of interpretability. Feature weighting methods, which assess the relevance of individual features, are among the most widely studied approaches for improving interpretability. While post hoc techniques have advanced significantly, especially in modeling nuanced relationships between features and labels in multiclass problems, embedded methods typically fall short in this regard. Embedded techniques integrate feature relevance learning directly into the model training process, but they are often unable to capture label-specific relationships. To address the limitation, we propose multiple extensions to the Feature Selection Layer, an embedded feature weighting mechanism for neural networks. Through extensive experimentation, we identify the most effective extension and benchmark it against established feature weighting methods. Our results show that the proposed method matches existing techniques in identifying globally relevant features and outperforms them in capturing label-specific relevance. By leveraging the internal dynamics of the trained neural network, it reveals how individual features contribute to the prediction of specific labels, requiring no post-processing beyond the training process.

Figures & tables

Figure 1: Representation of Listener extension. In this example, two listeners, one per label, accumulate weight updates for each of the three features (

aw_{1}\ldots aw_{3}

) based on the label in the current batch. The FSL remains unchanged. For

M

features and

L

labels,

y_{L}[{aw}_{1},\ldots,{aw}_{M}]

stores the accumulated weights per feature-label pair.

Figure 2: Representation of Residual extension. In this example, three weights were added for each feature on FSL, one per label. The colored lines represents the connection from the feature weights, already multiplied by the feature value, to their specific label output.

Figure 3: Representation of Output Aware extension. FSL is implemented with multiple weight columns, each corresponding to a specific label (yellow, blue, and red). During training, a one-hot encoded vector (

y_{1}

y_{2}

y_{3}

) indicates the expected label, which will have value

1

, while the others will be

0

. Each one-hot encoded vector multiplies the weight matrix, effectively zeroing out weights not related with the current input’s expected label. After being trained, the input features are multiplied by the sum of its feature weights.

Figure 4: Representation of Multiple Feature Selection Layer extension. Multiple selection layers are created, one for each label described by colors yellow, blue and red. The internal network is executed three times before backpropagation, one for each possible label. Only the feature weights related to the label being estimated are selected at each execution.

$\lambda$	XOR	SynthA	Isolet	Breast
$0.0000$	0.526 $\pm$ 0.160	0.704 $\pm$ 0.019	0.943 $\pm$ 0.007	0.961 $\pm$ 0.022
$0.0001$	0.588 $\pm$ 0.194	0.732 $\pm$ 0.053	0.947 $\pm$ 0.004	0.981 $\pm$ 0.026
$0.0010$	0.741 $\pm$ 0.268	0.894 $\pm$ 0.010	0.946 $\pm$ 0.004	0.961 $\pm$ 0.022
$0.0100$	1.000 $\pm$ 0.000	0.918 $\pm$ 0.008	0.940 $\pm$ 0.005	0.942 $\pm$ 0.021
$0.1000$	1.000 $\pm$ 0.000	0.926 $\pm$ 0.007	0.907 $\pm$ 0.007	0.902 $\pm$ 0.070
$1.0000$	1.000 $\pm$ 0.000	0.746 $\pm$ 0.032	0.800 $\pm$ 0.017	0.971 $\pm$ 0.026

Table 1: Test F1-score, mean

\pm

standard deviation over 5 executions, for Multiple FSL across varying regularization strengths

\lambda

. In bold is the best average per dataset. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 5 — Figure 5: Effect of regularization strength. Test F1 (left axis, solid) and high-dimensional silhouette score (right axis, dashed) as a function of regularization strength $\lambda$ , per dataset (mean $\pm$ std over 5 executions). Where the two curves diverge, the $\lambda$ that maximizes classification accuracy is not the $\lambda$ that maximizes feature-space separability.

Dataset	Activation	Test F1	Test Loss	2D Silhouette
XOR	ReLU	1.0000 $\pm$ 0.0000	0.0004 $\pm$ 0.0007	0.1931 $\pm$ 0.0195
	Sigmoid	0.4235 $\pm$ 0.0746	0.7011 $\pm$ 0.0083	0.0028 $\pm$ 0.0024
	Tanh	0.9980 $\pm$ 0.0045	0.0029 $\pm$ 0.0061	0.1974 $\pm$ 0.0177
SynthA	ReLU	0.9168 $\pm$ 0.0111	0.4945 $\pm$ 0.0733	0.1656 $\pm$ 0.0240
	Sigmoid	0.7090 $\pm$ 0.0119	0.7701 $\pm$ 0.0168	0.1472 $\pm$ 0.0178
	Tanh	0.9187 $\pm$ 0.0097	0.5199 $\pm$ 0.0428	0.1971 $\pm$ 0.0693

Table 2: Effect of the activation function on Multiple FSL across datasets calculated over 5 executions. In bold is the best average per dataset and metric, among activation functions. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Features	Init. Weight	Test F1	Test Loss	2D Silhouette
50	$1$	1.000 $\pm$ 0.000	0.000 $\pm$ 0.000	0.648 $\pm$ 0.021
50	$1/m$	0.667 $\pm$ 0.456	0.440 $\pm$ 0.602	0.682 $\pm$ 0.027
500	$1$	1.000 $\pm$ 0.000	0.000 $\pm$ 0.000	0.725 $\pm$ 0.008
500	$1/m$	0.833 $\pm$ 0.373	0.220 $\pm$ 0.493	0.726 $\pm$ 0.007
5000	$1$	0.998 $\pm$ 0.001	0.003 $\pm$ 0.002	0.600 $\pm$ 0.038
5000	$1/m$	0.997 $\pm$ 0.002	0.016 $\pm$ 0.013	0.601 $\pm$ 0.072

Table 3: Comparison of Multiple FSL with weight init of

1

and

1/m

(

m

is the number of input features) across four synthetic datasets and the real world Breast dataset. In bold is the best average per number of features. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Name	Feature Type	Labels	Features	Relevant	Per label	Samples
XOR	Binary	2	50	2	2	500
SynthA	Numerical	3	100	30	30	3000
SynthB	Numerical	3	300	30	10	3000
SynthC	Numerical	3	300	30	15	3000

Table 4: Synthetic datasets with defined number of relevant features (Relevant) and relevant features per label (Per label).

Dataset	Features	Labels	Identifier	Samples
Liver 22405	22284	normal	0	24
Liver 22405	22284	heptocarcinoma	1	24
Colorectal 41657	33468	normal	0	12
		high_grade_dysplasia	1	28
		low_grade_dysplasia	2	21
		adenocarcinoma	3	25

Table 5: Real-world datasets selected from CuMiDa and the UCI Machine Learning Repository. Liver, Isolet (298-300 samples per class), and Letter Recognition (734-813 samples per class) are class-balanced, whereas Colorectal and Breast exhibit greater imbalance across classes.

Algorithm	Stability metric	Features	Score
FSL	Pearson	50	.990
	Spearman	50	.098
	Jaccard	2	1
Listener	Pearson	50	.987
	Spearman	50	.150
	Jaccard	2	1

Table 6: Stability comparison between algorithms feature weights on XOR dataset. Jaccard similarity measures set overlap, ranging from 0 (no overlap) to 1 (identical sets), being used to assess the similarity between the top-n highest-weighted features. Spearman accesses ranking similarity, ranging from -1 (perfect inverse correlation) to 1 (perfect correlation). Pearson measures linear correlation between the feature weights, ranging from -1 (perfect negative correlation) to 1 (perfect positive correlation) where 0 means no correlation. In bold are the best values for each metric.

Features	Labels	FSL	Listener	Output Aware	Residual	Multiple FSL
50	2	15.90 $\pm$ 1.50	21.32 $\pm$ 1.09	19.95 $\pm$ 1.43	16.54 $\pm$ 1.29	16.44 $\pm$ 1.47
	5	14.60 $\pm$ 0.42	19.37 $\pm$ 0.80	17.60 $\pm$ 0.51	15.88 $\pm$ 0.38	15.64 $\pm$ 0.44
	10	17.03 $\pm$ 1.57	27.15 $\pm$ 1.73	24.62 $\pm$ 1.95	18.25 $\pm$ 1.79	17.95 $\pm$ 1.76
	20	15.30 $\pm$ 1.20	26.43 $\pm$ 2.71	24.37 $\pm$ 2.84	17.08 $\pm$ 1.60	16.62 $\pm$ 1.44
500	2	17.83 $\pm$ 1.44	39.24 $\pm$ 2.33	35.86 $\pm$ 2.13	18.73 $\pm$ 1.29	18.38 $\pm$ 1.15
	5	15.17 $\pm$ 2.08	33.12 $\pm$ 3.42	29.87 $\pm$ 3.78	14.94 $\pm$ 2.04	14.66 $\pm$ 2.42

Table 7: Execution time (s) of each proposed extension across varying numbers of features and labels. Values represent the mean

\pm

standard deviation over 5 executions. In bold is the best average per dataset. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Features	Labels	FSL	Listener	Output Aware	Residual	Multiple FSL
50	2	18.1	18.1	18.1	18.1	18.4
	5	18.1	18.2	18.2	18.1	19.0
	10	18.1	18.2	18.2	18.2	20.2
	20	18.1	18.2	18.2	18.2	22.4
500	2	24.8	25.7	25.7	24.8	26.1
	5	24.8	25.9	25.9	24.9	29.9

Table 8: Peak GPU memory usage (MB) of each proposed extension across varying numbers of features and labels. Standard deviation is omitted because the variation in peak GPU allocation is minimal, given the same architecture and dataset, the memory footprint remains nearly identical across all executions. All peak GPU memory allocations were measured using an NVIDIA GeForce RTX 5070 Ti (16 GB VRAM). In bold is the best average per dataset and metric.

Algorithm	Label	SynthA	SynthB	SynthC
FSL	0	.887 $\pm$ .015	.934 $\pm$ .013	.878 $\pm$ .012
	1	.885 $\pm$ .013	.942 $\pm$ .011	.863 $\pm$ .016
	2	.888 $\pm$ .014	.924 $\pm$ .013	.865 $\pm$ .012
Listener	0	.888 $\pm$ .007	.945 $\pm$ .011	.890 $\pm$ .016
	1	.883 $\pm$ .009	.946 $\pm$ .021	.867 $\pm$ .018
	2	.889 $\pm$ .009	.934 $\pm$ .013	.880 $\pm$ .016

Table 9: Per label F1 scores of FSL and proposed extensions on the SynthA, SynthB, and SynthC datasets. The scores were calculated across 11 independent executions using test dataset. For each reported metric we present the average and standard deviation. In bold is the highest average of each dataset and label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 6 — Figure 6: Analysis of PIFS and PSFI on XOR, SynthA, SynthB and SynthC datasets between original FSL and proposed extensions for increasing selection subsets. All values scores are the average over rankings generated by 11 independent models execution. Vertical axis represents the scores constrained from 0 to 1 as both measures are percentages. Horizontal axis represents the subsets size selected from the top-n features with the highest weights.

Panel of Figure 7 — Figure 7: Feature erasure (occlusion) based on SynthC using feature weights related to label $0$ (blue line). Each line represents the F1 score trend related to a single label as we erase the top- $n$ features based on their weights in descending order. All values were calculated using the test dataset.

Panel of Figure 8 — Figure 8: Feature ranking position related to label $0$ across proposed extensions executions on SynthC. All rankings were created from the feature weights related to label 0 in descending order. Each algorithm was executed 11 times. The vertical axis represents each individual execution, while the horizontal axis shows the ranking position of each feature, from 0 (top) to 299 (bottom), within that run. Different colors represent different features. Given the high density of lines, this visualization is not intended for tracking individual features. Instead, the objective is to highlight a macro-level pattern: the clear distinction between relevant features, which form stable, straight bands consistently ranked at the top, and noisy features, which exhibit highly chaotic and fluctuating trajectories at the bottom of the graph. Additionally, the results demonstrate how the positions of both informative and noisy features vary across executions within each group.

Panel of Figure 9 — Figure 9: Visualization of weighted t-SNE using label 0 (red dots) feature weights from Breast dataset. Each label is represented by a distinct color, being red assigned to label $0$ . 9(a) presents the original t-SNE without feature weights. 9(b) , 9(c) , 9(d) and 9(e) represents the weighted t-SNE for all proposed extensions using the assigned weights for each feature related to label $0$ .

Figure 10: Comparing SVC prediction performance on top-1000 features ranked by weight from FSL and proposed extensions for Breast dataset. Vertical axis shows the F1 score on test dataset. Horizontal axis shows the number of features being used, from 1 to 1000. Feature subsets were defined by the general feature weight given for each specific method. For each method, two comparisons were conducted: one using only feature subsets, and another using weighted features.

Panel of Figure 11 — Figure 11: Impact of feature erasure based on label 0 feature weights for Breast dataset. The features were erased from the dataset following a ranking based on the feature weights given for label 0. The horizontal axis shows the number of erased features, while the vertical axis displays the F1 score calculated from the test dataset.

Algorithm	Pearson	Spearman	Jaccard
Decision Tree	0.833 $\pm$ .044	0.607 $\pm$ .097	0.633 $\pm$ .079
Random Forest	0.987 $\pm$ .002	0.688 $\pm$ .042	0.935 $\pm$ .000
Kruskal Wallis	0.999 $\pm$ .000	0.999 $\pm$ .000	1.000 $\pm$ .000
ReliefF	0.997 $\pm$ .001	0.977 $\pm$ .007	0.748 $\pm$ .061
Lasso	0.992 $\pm$ .002	0.964 $\pm$ .012	0.925 $\pm$ .047
LIME	0.755 $\pm$ .043	0.475 $\pm$ .101	0.489 $\pm$ .067

Table 10: Stability comparison between algorithms feature weights on SynthA dataset. Pearson and Spearman was measured using all features (100), while Jaccard was measured using the top-30 features given by the feature weighting algorithm, which is the number of informative features. In bold is the highest average of each dataset and label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 12 — Figure 12: Evaluation of PIFS and PSFI on the SynthA dataset for Multiple FSL and compared feature weighting algorithms. All scores are averages over 11 independent runs per algorithm. Vertical axis represents scores constrained from $0$ to $1$ , and the horizontal axis shows subset sizes based on the top- $n$ features by descending feature weight. PSFI is represented by orange lines, PIFS by blue lines. With 30 features, a perfect result has PSFI stable at 1.0 up to the top 30 features and PIFS reaching 1.0 at the 30th feature.

Figure 13: Comparing SVC prediction performance trained with top-

n

features ranked by weight from FSL and proposed extensions for SynthA dataset. Vertical axis shows the F1 score on test dataset. Horizontal axis shows the number of features being used. Feature subsets were defined by the general feature weight given for each specific algorithm. For each algorithm, two comparisons were conducted: one using only feature subsets, and another using weighted features.

Panel of Figure 14 — Figure 14: Comparing SVC prediction performance trained with top- $n$ features ranked by weight from FSL and proposed extensions for SynthA dataset. Multiple feature subsets of increasing size are generated based on the feature weights assigned by each method. These subsets are then used to train an SVC model, and its prediction score is reported. Each algorithm’s scores are presented separately for a better individual analysis.

Panel of Figure 15 — Figure 15: Heatmap visualization of top-30 features for Lasso, LIME, DeepSHAP, and Multiple FSL, based on their label-specific weights for label 2 of the SynthB dataset. Normalized feature weights for each algorithm, ranked by their label 2 (third column) feature weights. Feature names prefix indicate if they are informative or noisy, followed by an identification letter sequence, which ensures that all names are unique, and the label which they are related. For example, informative_for_labels_A_1 is the informative feature A that is related to label 1, while noisyAI is a noisy feature identified by the letters “AI”.

Panel of Figure 16 — Figure 16: Liver dataset weighted t-SNE visualization using general feature weights. 16(a) presents the t-SNE visualization. The other figures presents the weighted t-SNE using the general feature weights extracted from each algorithm. The two classes are displayed in red circles (0) and blue diamonds (1).

Weight source	HD silhouette	2D silhouette
None	.038	.049
Decision Tree	.409 $\pm$ .073	.498 $\pm$ .056
Random Forest	.220 $\pm$ .019	.517 $\pm$ .047
Kruskal Wallis	.148 $\pm$ .013	.305 $\pm$ .061
ReliefF	.054 $\pm$ .003	.066 $\pm$ .016
Lasso	.382 $\pm$ .038	.661 $\pm$ .033

Table 11: Average HD and 2D silhouette coefficients for Liver dataset. All feature weights were provided from the general feature weights, not being label-specific. Averages were calculated with the weighted t-SNE results given the feature weights generated by each algorithm through 11 different executions. In bold is the highest average for each column. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Model	XOR	SynthB	Breast	Isolet	Letter Recognition
Base Neural Network	4.59 $\pm$ 0.60	39.40 $\pm$ 4.22	14.83 $\pm$ 0.31	13.88 $\pm$ 0.53	98.19 $\pm$ 1.65
Decision Tree	0.00 $\pm$ 0.00	0.76 $\pm$ 0.21	1.24 $\pm$ 0.11	2.13 $\pm$ 0.05	0.04 $\pm$ 0.00
Random Forest	0.06 $\pm$ 0.00	1.94 $\pm$ 0.07	0.66 $\pm$ 0.01	5.73 $\pm$ 0.05	0.85 $\pm$ 0.01
Kruskal Wallis	0.02 $\pm$ 0.00	0.16 $\pm$ 0.03	18.09 $\pm$ 0.56	0.83 $\pm$ 0.00	0.02 $\pm$ 0.00
ReliefF	0.02 $\pm$ 0.01	0.23 $\pm$ 0.04	0.47 $\pm$ 0.01	1.84 $\pm$ 0.02	4.06 $\pm$ 0.06
Lasso	0.00 $\pm$ 0.00	0.06 $\pm$ 0.01	7.00 $\pm$ 0.74	2.57 $\pm$ 0.51	0.15 $\pm$ 0.27

Table 12: Execution times (s) for each feature weighting method across the XOR, SynthB, Breast, Isolet, and Letter Recognition datasets. Values represent the mean

\pm

standard deviation over 11 executions. To ensure a fair comparison with embedded methods, the times reported for post-hoc methods represent the sum of the base model training time and the method’s execution time. Note that SAGE’s computational cost is not reported for the Breast dataset because it resulted in an out of memory error. The method’s default Kernel estimator requires tracking an

\mathcal{O}(M^{2})

covariance matrix, which demands tens of gigabytes of RAM for the 54,675 features of the Breast dataset, exceeding standard hardware limits. In bold is the best average per dataset. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Model	XOR	SynthB	Breast	Isolet	Letter Recognition
Base Neural Network	19.8	23.3	83.1	35.8	23.7
DeepSHAP	21.7	39.7	97.0	35.8	34.3
LassoNet	20.2	29.3	102.6	47.9	62.1
TabNet	17.7	21.1	12063.8	26.2	28.7
FSL	19.8	24.0	83.7	38.2	23.8
Multiple FSL	21.0	27.1	127.7	114.3	144.8

Table 13: Peak GPU memory usage (MB) for each feature weighting method across the XOR, SynthB, Breast, Isolet, and Letter Recognition datasets. Standard deviation is omitted because the variation in peak GPU allocation is minimal, given the same architecture and dataset, the memory footprint remains nearly identical across all executions. All peak GPU memory allocations were measured using an NVIDIA GeForce RTX 5070 Ti (16 GB VRAM). In bold is the best average per dataset and metric.

Label	Baseline	DeepSHAP	TabNet	Multiple FSL
A	-0.182 $\pm$ 0.000	-0.173 $\pm$ 0.027	-0.128 $\pm$ 0.039	-0.181 $\pm$ 0.031
B	-0.103 $\pm$ 0.000	-0.039 $\pm$ 0.069	-0.002 $\pm$ 0.025	-0.041 $\pm$ 0.063
C	0.144 $\pm$ 0.000	0.135 $\pm$ 0.052	0.146 $\pm$ 0.100	0.179 $\pm$ 0.034
D	-0.052 $\pm$ 0.000	-0.041 $\pm$ 0.038	-0.017 $\pm$ 0.033	0.003 $\pm$ 0.093
E	-0.048 $\pm$ 0.000	-0.034 $\pm$ 0.043	0.038 $\pm$ 0.036	-0.131 $\pm$ 0.050
F	0.020 $\pm$ 0.000	0.097 $\pm$ 0.038	0.116 $\pm$ 0.054	0.104 $\pm$ 0.050

Table 14: Isolet dataset 2D silhouette coefficients per label. Averages were calculated over 11 executions. A one-vs-all (OVA) strategy was adopted, where all labels other than the target label were collapsed into a single category. In bold is the highest average. Best results with statistical significance (p < 0.01) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 17 — Figure 17: Impact of the most relevant label-specific features on predictive performance in the Letter Recognition dataset. The graph illustrates the effect of initially erasing all features and then sequentially re-enabling them, one by one, according to the label-specific feature weight ranking for label 0. The red line denotes the F1-score of the target label, while the gray lines represent the F1-scores of all other labels.

Dataset	Baseline	LassoNet	TabNet	FSL	Multiple FSL
XOR	0.078 $\pm$ 0.051	0.003 $\pm$ 0.001	0.067 $\pm$ 0.028	0.003 $\pm$ 0.007	0.003 $\pm$ 0.006
SynthA	0.086 $\pm$ 0.027	0.044 $\pm$ 0.017	0.101 $\pm$ 0.025	0.073 $\pm$ 0.010	0.062 $\pm$ 0.010
SynthB	0.090 $\pm$ 0.038	0.087 $\pm$ 0.067	0.067 $\pm$ 0.018	0.037 $\pm$ 0.011	0.029 $\pm$ 0.005
SynthC	0.076 $\pm$ 0.032	0.041 $\pm$ 0.013	0.054 $\pm$ 0.020	0.086 $\pm$ 0.007	0.066 $\pm$ 0.016
Liver	0.094 $\pm$ 0.142	0.050 $\pm$ 0.074	0.111 $\pm$ 0.074	0.033 $\pm$ 0.080	0.023 $\pm$ 0.053
Colorectal	0.172 $\pm$ 0.080	0.190 $\pm$ 0.046	0.109 $\pm$ 0.091	0.129 $\pm$ 0.064	0.136 $\pm$ 0.079

Table 15: Expected Calibration Error (ECE) comparison across datasets. Lower values indicate better calibration. The proposed Multiple FSL consistently achieves highly calibrated probabilities without the need for post-hoc adjustments. For a complete comparison with additional baselines, please refer to Table C.9 . In bold is the lowest average. Best results with statistical significance (p < 0.01) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 18 — Figure 18: Colorectal dataset weighted t-SNE visualization using feature weights related to label 1. 18(a) presents a t-SNE visualization without feature weights. Label 1 is represented in blue diamonds.

Panel of Figure 19 — Figure 19: Impact of feature erasure based on label 4 feature weights for Breast dataset. The features were erased from the dataset following a ranking based on the feature weights given for label 4 by each algorithm. The horizontal axis shows the number of erased features, while the vertical axis displays the F1 score calculated from the test dataset.

Weight source	HD silhouette	2D silhouette
None	.024	.090
Lasso	.024 $\pm$ .000	.060 $\pm$ .032
LIME	.024 $\pm$ .000	.036 $\pm$ .030
DeepSHAP	.048 $\pm$ .008	.141 $\pm$ .042
Multiple FSL	.143 $\pm$ .025	.446 $\pm$ .027

Table 16: Colorectal dataset HD and 2D silhouette coefficients with feature weights related to label 1. Averages were calculated over 11 executions. “None” represents the t-SNE without using weighted features, being a fixed value. Averages were calculated with the weighted t-SNE results given the feature weights generated by each algorithm through 11 different executions. A one-vs-all (OVA) strategy was adopted, where all labels other than label 1 were converted into a single label. In bold is the highest average. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Algorithm	General	Label 0	Label 1	Label 2
LassoNet	0.948 $\pm$ .031	0.936 $\pm$ .067	1.0 $\pm$ .000	0.882 $\pm$ .098
TabNet	0.530 $\pm$ .071	0.518 $\pm$ .147	0.591 $\pm$ .130	0.509 $\pm$ .138
SAGE	0.951 $\pm$ .035	-	-	-
Multiple FSL	0.921 $\pm$ .040	0.982 $\pm$ .040	1.0 $\pm$ .000	1.0 $\pm$ .000

Table 17: PIFS comparison between Multiple FSL, LassoNet, TabNet and SAGE with SynthB dataset. The measure considers a subset of 30 features, which is the amount of informative features on SynthB dataset. SAGE is the only method unable to define label-specific weights. In bold is the highest average of each dataset and label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 20 — Figure 20: Breast dataset weighted t-SNE visualization using feature weights from Multiple FSL. 20(a) presents the t-SNE visualization. 20(b) presents a weighted t-SNE visualization with general weights. 20(c) to 20(h) presents the weighted t-SNE visualization with label-specific weights.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 38 — Figure C.1 : t-SNE and “perfect” weighted t-SNE of each synthetic datasets. The “perfect” weighted t-SNE is achieved by assigning a weight of 1 for all informative features, while irrelevant features were assigned a weight of 0.

Panel of Figure 39 — Figure C.2 : t-SNE of CuMiDa datasets. Analysis of each CuMiDa datasets used on this work. A weighted t-SNE approach was not implemented due to the lack of prior knowledge about the relevance of each feature.

Figure C.3 : Evaluation of PIFS and PSFI on the XOR dataset between original FSL and proposed extensions. All scores are the average over rankings generated by 11 independent execution. Vertical axis represents the score constrained from

0

1

as both measures are percentages. Horizontal axis represents the subset size being evaluated. All subsets are defined by the top-

n

features from a ranking based on the feature weights generated by each algorithm.

Panel of Figure 41 — Figure C.4 : Heatmap visualization with the 15 highest weighted features for XOR dataset. Normalized weights for each feature ranked from the highest general weight to the lowest. “General” represents the average label-specific weights for the proposed extensions Residual, Output Aware and Multiple FSL.

Panel of Figure 42 — Figure C.5 : Feature ranking positions across FSL and proposed extensions executions on XOR dataset. All rankings were created by sorting the features in descending order of their weights. Each algorithm was executed 11 times (horizontal axis). The horizontal axis represents each individual execution, while the vertical axis shows the rank position of each feature, from 0 (top) to 49 (bottom), within that execution. Each color represent a different feature, with blue and orange lines at the top indicating the two relevant features.

Panel of Figure 43 — Figure C.6 : Visualization of XOR dataset with weighted t-SNE using the feature weights attributed by original FSL and the proposed extensions. 6(a) presents a t-SNE visualization of the data without feature weights. In contrast, 6(b) , 6(c) , 6(d) , 6(e) and 6(f) employ weighted t-SNE, utilizing the generated weights from original FSL and each proposed extension to enhance the separation between clusters. False labels (0) are represented in red, while true labels (1) are represented in blue.

Panel of Figure 44 — Figure C.7 : Analysis of PIFS and PSFI on SynthA, SynthB and SynthC datasets between original FSL and proposed extensions for increasing selection subsets. All values scores are the average over rankings generated by 11 independent models execution. Vertical axis represents the scores constrained from 0 to 1 as both measures are percentages. Horizontal axis represents the subsets size selected from the top-n features with the highest weights. 7(a) represents the evaluation over SynthA dataset, 7(b) over SynthB and 7(c) over SynthC.

Weight source	HD silhouette	2D silhouette
None	.0030	-.013
FSL	.0105 $\pm$ .0005	.1755 $\pm$ .0560
Listener	.0109 $\pm$ .0003	.1291 $\pm$ .0433
Residual	.0109 $\pm$ .0004	.1755 $\pm$ .0560
Output Aware	.0128 $\pm$ .0005	.1696 $\pm$ .0693
Multiple FSL	.0123 $\pm$ .0004	.1773 $\pm$ .0922

Appendix

Table C.1 : SynthA dataset HD and 2D silhouette coefficients based on the feature weights attributed by original FSL and the proposed extensions. All feature weights were provided from the general weights, not being label-specific. High-dimensional (HD) silhouette coefficients are calculated before the dimension reduction performed by weighted t-SNE, while 2D silhouette coefficients are calculated after. “None” represents the silhouette coefficients without adding feature weights. In bold is the highest average. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 46 — Figure C.8 : SynthA weighted t-SNE visualization using the general feature weights attributed by original FSL and the proposed extensions. 8(a) presents a t-SNE visualization of the data without feature weights. In contrast, 8(b) , 8(c) , 8(d) , 8(e) and 8(f) employ weighted t-SNE, utilizing the generated weights from original FSL and all proposed extensions to enhance the separation between clusters. Label 0 is represented in red, label 1 in blue and label 3 in yellow.

Panel of Figure 47 — Figure C.9 : Heatmap visualization of top-30 general features for each extension from SynthB dataset. Normalized feature weights for each algorithm, ranked by their overall general weight. Feature names prefix indicate if they are informative or noisy, followed by an identification letter sequence, which ensures that all names are unique, and the label which they are related. For example, informative_for_labels_A_1 is the informative feature identified by the letter ’A’ that is related to label 1, while noisyAI is a noisy feature identified by the letter sequence “AI”.

Algorithm	Label 0	Label 1	Label 2
Listener	.834	.727	.886
Residual	.398	.400	.862
Output Aware	.463	.513	.599
Multiple FSL	.341	.626	.811

Appendix

Table C.2 : Analysis of feature weights for informative_per_label_L_0_1 from SynthC dataset. Each column represents the feature weight given by each algorithm to the feature informative_per_label_L_0_1 . Although the feature is directly related to label 0 and label 1, all extensions gave a higher weight, highlighted in bold, related to label 2.

Algorithm	Label	Precision	Recall	F1 score
FSL	0	.990 $\pm$ .032	.977 $\pm$ .048	.983 $\pm$ .029
	1	.951 $\pm$ .084	.985 $\pm$ .048	.965 $\pm$ .049
	2	1.0 $\pm$ .0	1.0 $\pm$ .0	1.0 $\pm$ .0
	3	1.0 $\pm$ .0	1.0 $\pm$ .0	1.0 $\pm$ .0
	4	.974 $\pm$ .055	1.0 $\pm$ .0	.986 $\pm$ .030
	5	1.0 $\pm$ .0	.939 $\pm$ .080	.967 $\pm$ .044

Appendix

Table C.3 : Individual predictive performance of FSL and proposed extensions on Breast dataset per label. The average scores were calculated across 11 independent executions using test dataset. For each reported metric we present the average and standard. In bold is the highest average of each metric and label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 50 — Figure C.10 : SynthA weighted t-SNE visualization using the feature weights attributed by Multiple FSL and compared algorithms. 10(a) presents a t-SNE visualization of the data without feature weights. In contrast other figures employ weighted t-SNE, utilizing the generated weights from each feature weighting algorithm.

Panel of Figure 51 — Figure C.11 : SynthB weighted t-SNE visualization using the feature weights attributed to label 1. 11(a) presents a t-SNE visualization of the data without feature weights. In contrast other figures employ weighted t-SNE, utilizing the generated weights from each algorithm specifically related to label 1 (blue diamonds).

Algorithm	Pearson	Spearman	Jaccard
Decision Tree	0.728 $\pm$ .087	0.330 $\pm$ .040	0.536 $\pm$ .044
Random Forest	0.989 $\pm$ .003	0.342 $\pm$ .039	0.947 $\pm$ .026
Kruskal Wallis	0.999 $\pm$ .000	1.000 $\pm$ .000	1.000 $\pm$ .000
ReliefF	0.987 $\pm$ .001	0.924 $\pm$ .008	0.976 $\pm$ .033
Lasso	0.989 $\pm$ .001	0.840 $\pm$ .035	0.753 $\pm$ .061
LIME	0.874 $\pm$ .026	0.162 $\pm$ .052	0.388 $\pm$ .050

Appendix

Table C.4 : Stability comparison between algorithms feature weights on SynthB dataset. Jaccard similarity measures set overlap, ranging from 0 (no overlap) to 1 (identical sets), being used to assess the similarity between the top-n highest-weighted features. Spearman accesses ranking similarity, ranging from -1 (perfect inverse correlation) to 1 (perfect correlation). Pearson measures linear correlation between the feature weights, ranging from -1 (perfect negative correlation) to 1 (perfect positive correlation) where 0 means no correlation. Pearson and Spearman was measured using all features (100), while Jaccard was measured using the top-30 features given by the feature weighting algorithm, which is the number of informative features. In bold is the highest average of each dataset and label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Weight source	HD silhouette	2D silhouette
None	.024	.090
Lasso	.187 $\pm$ .011	.487 $\pm$ .047
LIME	.072 $\pm$ .001	.207 $\pm$ .026
DeepSHAP	.096 $\pm$ .005	.298 $\pm$ .030
Multiple FSL	.159 $\pm$ .015	.561 $\pm$ .021

Appendix

Table C.5 : Colorectal dataset HD and 2D silhouette coefficients with general feature weights. “None” represents the t-SNE without using weighted features, being a fixed value. Averages were calculated with the weighted t-SNE results given the feature weights generated by each algorithm through 11 different executions. For the comparison was used the general feature weights without an specific related label. In bold is the highest average. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Panel of Figure 54 — Figure C.12 : Colorectal dataset weighted t-SNE visualization using general feature weights. 12(a) presents a t-SNE visualization without feature weights.

Panel of Figure 55 — Figure C.13 : Weighted tSNE applied to SynthB using label 1 weights from Multiple FSL, LassoNet and TabNet. Label 1 samples are presented as blue squares.

Label	Baseline	Multiple FSL	TabNet	LassoNet	Lasso
A	0.989 $\pm$ 0.005	0.993 $\pm$ 0.003	0.941 $\pm$ 0.019	0.988 $\pm$ 0.005	0.603 $\pm$ 0.004
B	0.945 $\pm$ 0.006	0.940 $\pm$ 0.012	0.859 $\pm$ 0.025	0.928 $\pm$ 0.009	0.546 $\pm$ 0.011
C	0.970 $\pm$ 0.010	0.976 $\pm$ 0.005	0.908 $\pm$ 0.019	0.946 $\pm$ 0.010	0.492 $\pm$ 0.003
D	0.959 $\pm$ 0.011	0.957 $\pm$ 0.010	0.855 $\pm$ 0.027	0.935 $\pm$ 0.005	0.555 $\pm$ 0.007
E	0.953 $\pm$ 0.015	0.950 $\pm$ 0.009	0.885 $\pm$ 0.022	0.951 $\pm$ 0.006	0.085 $\pm$ 0.023
F	0.956 $\pm$ 0.008	0.952 $\pm$ 0.010	0.858 $\pm$ 0.024	0.920 $\pm$ 0.019	0.475 $\pm$ 0.015

Appendix

Table C.6 : Label-specific F1-score, mean

\pm

standard deviation over 11 executions, for Letter Recognition test dataset. In bold is the best average per label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Label	Baseline	Multiple FSL	TabNet	LassoNet	Lasso
A	0.953 $\pm$ 0.017	0.964 $\pm$ 0.017	0.974 $\pm$ 0.017	0.972 $\pm$ 0.012	0.752 $\pm$ 0.014
B	0.764 $\pm$ 0.046	0.869 $\pm$ 0.027	0.899 $\pm$ 0.030	0.865 $\pm$ 0.022	0.791 $\pm$ 0.017
C	0.982 $\pm$ 0.010	0.996 $\pm$ 0.006	0.982 $\pm$ 0.014	0.975 $\pm$ 0.008	0.882 $\pm$ 0.009
D	0.836 $\pm$ 0.056	0.905 $\pm$ 0.025	0.908 $\pm$ 0.023	0.899 $\pm$ 0.018	0.767 $\pm$ 0.012
E	0.871 $\pm$ 0.048	0.893 $\pm$ 0.033	0.918 $\pm$ 0.022	0.913 $\pm$ 0.020	0.862 $\pm$ 0.014
F	0.900 $\pm$ 0.037	0.916 $\pm$ 0.021	0.957 $\pm$ 0.035	0.969 $\pm$ 0.016	0.720 $\pm$ 0.020

Appendix

Table C.7 : Label-specific F1-score, mean

\pm

standard deviation over 11 executions, for Isolet test dataset. In bold is the best average per label. Best results with statistical significance (

p<0.01

) according to the Kruskal-Wallis and Dunn’s test are highlighted in orange.

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 196 papers

Post-Hoc Feature Selection Layer for Neural Networks Interpretability
Oct 22, 2026João Kenji Suwa, João Pedro Silveira e Silva, Bruno Iochins GrisciNeural Network Interpretability Feature Attribution
Institute of Informatics, UFRGS, Porto Alegre, Brazil
From Weight Perturbation to Feature Attribution for Explaining Fully Connected Neural Networks
May 14, 2026Thodoris Lymperopoulos, Denia KanellopoulouFeature Attribution Perturbation-Based Feature Attribution
NCSR Demokritos Athens, Greece
Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance
May 29, 2026Doğukan Bağcı, Bernt Schiele, Simone Schaub-Meyer +2Disentangled Representation Learning Representation Disentanglement
Max Planck Institute for Informatics, SIC · Department of Computer Science, TU Darmstadt

Oct 22, 2026J/K move · Enter open · S save

Post-Hoc Feature Selection Layer for Neural Networks Interpretability

João Kenji Suwa, João Pedro Silveira e Silva, Bruno Iochins Grisci

Institute of Informatics, UFRGS, Porto Alegre, Brazil

Neural Network Interpretability Feature Attribution Feature Selection

Open paper AskPDF arXiv

The interpretability of complex neural networks remains a critical challenge, especially for models already deployed in high-stakes domains. To address this, we adapted the Feature Selection Layer (FSL) to optimize feature weights post-training, a variant we term post-hoc FSL. Our approach reframes the FSL as a lightweight, trainable module that integrates with already frozen pre-trained models on tabular datasets to highlight the features the original model considers most important. This post-hoc FSL learns feature relevance by fine-tuning its weights based on the original model's learned outputs. Crucially, this process is non-invasive, operating without altering the original model's architecture or its learned parameters. We conducted our experiments using both statistical and visual metrics, including accuracy, F1 score, recall, precision, weighted t-SNE and silhouette score, and also analyzed the stability of the post-hoc FSL on high-dimensional synthetic and real-world tabular datasets. We compared the post-hoc FSL feature weighting method using these metrics against the original embedded FSL and other post-hoc interpretability methods, such as Integrated Gradients, Noise Tunnel, DeepLIFT, Gradient SHAP, and Feature Ablation. Experimental results demonstrate that the post-hoc FSL feature weighting method successfully identified relevant features across the different datasets, maintaining the predictive power of the original neural network while enhancing its interpretability.

Abstract

Feature Selection

Explore similar work

Post-Hoc Feature Selection Layer for Neural Networks Interpretability

From Weight Perturbation to Feature Attribution for Explaining Fully Connected Neural Networks

Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance