Assessing feature scorer results on high-dimensional datasets with t-SNE
Authors: Bruno Iochins Grisci, Mario Inostroza-Ponta, Márcio Dorn
Organizations: Institute of Informatics, Federal University of Rio Grande do Sul, Av. Bento Gonçalves 9500, Porto Alegre, 91501-970, RS, Brazil · Computer Engineering Department, University of Santiago de Chile, Av. Libertador Bernardo O’Higgins, Santiago, 9170022, RM, Chile · Center for Biotechnology, Federal University of Rio Grande do Sul, Av. Bento Gonçalves, Porto Alegre, 91501-970, RS, Brazil · National Institute of Science and Technology - Forensic Science, Porto Alegre, RS, Brazil
While vast literature on high-dimensional data visualization is available, there are not many works regarding the visualization of feature scorers and their results. Feature scorers are algorithms that assign numerical importance to each feature of multi-dimensional datasets. These importance scores can be used in several applications, such as feature selection, knowledge discovery, and machine learning interpretability. There are several feature scorers to choose from, and often no single metric or ground truth is available to guarantee the quality of their results. In this scenario, visualization can become valuable to support the decision of which method to choose and how good its results are. For this goal, this work presents "weighted t-SNE." It modifies the relationship between data points in the embedded 2D space to reflect the importance of each dimension of the original datasets as assessed by a feature scorer. This research discusses how to implement weighted t-SNE, proposes the silhouette coefficient as a numerical evaluation of the results, and shows several examples of its use in practice. Synthetic and real-world tabular datasets are used in the experiments together with nine feature scorers, ranging from Mutual Information to neural networks. Each feature scorer produces unique visualizations, and weighted t-SNE can be used to compare and choose the one that better suits a given dataset and task. Weighted t-SNE can also visually show the importance of features learned by machine learning models and help us see how they are organizing the data, increasing their interpretability.
Figures & tables
Figure 1: Comparison between the visualization using weighted t-SNE of a synthetic regression dataset before and after considering the feature importance as learned by a neural network. Each figure is the visualization of the 2D embedding of a regression dataset (negative target values in blue, positive values in red) composed mostly of irrelevant features. On the left is the regular visualization, in which the samples are mixed. On the right, it is possible to visualize how the feature importance learned by a neural network disentangles the samples. This experiment is further detailed in Section 4 . Interactive versions of all the plots in this publication are available at https://sbcblab.github.io/wtsne/ .
Dataset
Samples
Features
Relevant
Classes
Origin
Reference
XOR
500
50
2
2
Synthetic
[ 69 , 10 ]
Synth
100
5000
50
2
Synthetic
[ 70 , 71 ]
Liver
48
22 284
Unknown
2
Microarray
[ 2 ]
Prostate
115
12 647
Unknown
2
Microarray
[ 2 ]
Regression
1000
100
4
–
Synthetic
[ 72 , 71 ]
Mouse cortex
23 822
45 769
Unknown
23
RNA-seq
[ 73 , 47 ]
Table 1: Description of the datasets used in the weighted t-SNE experiments. Relevant is the number of features which are relevant to the task. All datasets are numerical and tabular.
Scorer
Type
Correlation
Reference
Kruskal–Wallis
Filter
Univariate
[ 30 ]
Mutual Information
Filter
Univariate
[ 32 ]
mRMR
Filter
Multivariate
[ 31 ]
ReliefF
Filter
Multivariate
[ 33 ]
Lasso
Embedded
Multivariate
[ 35 ]
Decision Tree
Embedded
Multivariate
[ 77 ]
Table 2: Properties of the feature scorers used in the experiments. The type refers to how the scorer works and the correlation to how interactions between features are treated.
Figure 2: Weighted t-SNE visualization for the XOR dataset. Each figure is the visualization of the 2D embedding with weighted t-SNE for each of the nine feature scorers, plus the original data in “no scoring”. Each red or blue point represents one sample from one of the two classes. The color legend is shown in Fig. 2(a) and omitted in the other figures.
Figure 3: Weighted t-SNE visualization for the synthetic 2-classes dataset. Each red or blue point represents one sample from one of the two classes. The color legend is shown in Fig. 3(a) and omitted in the other figures.
Figure 4: Weighted t-SNE visualization for the liver cancer dataset. Each red or blue point represents one sample from one of the two classes. The color legend is shown in Fig. 4(a) and omitted in the other figures.
Figure 5: Weighted t-SNE visualization for the prostate cancer dataset. Each red or blue point represents one sample from one of the two classes. The color legend is shown in Fig. 5(a) and omitted in the other figures.
Figure 6: Comparison between the Kruskal–Wallis Filter and a “perfect” selection. Each figure is the visualization of the 2D embedding with weighted t-SNE for the results of Kruskal–Wallis Filter and a “perfect” selection. This visualization refers to the synthetic dataset with 5000 features, in which only 50 are relevant. The “perfect” solution weights all relevant features with a score of one and all noisy features with a score of zero. Details as in Fig. 3 .
XOR
Synth
Liver
Prostate
HD
2D
KL
kNN
HD
2D
KL
kNN
HD
2D
KL
kNN
HD
2D
KL
kNN
No scoring
0.000
0.003
2.12
0.58
0.001
− 0.009
0.16
0.63
0.038
0.038
0.14
0.68
0.008
0.046
0.51
0.58
Kruskal–Wallis
0.004
0.010
0.48
0.51
0.041
0.609
0.56
1.00
0.183
0.449
0.15
0.94
0.072
0.196
0.35
0.71
Mutual Information
0.000
− 0.001
1.19
0.54
0.006
0.180
0.69
0.66
0.133
0.234
0.15
0.94
0.025
0.121
0.49
0.68
mRMR
0.000
− 0.002
0.74
0.54
0.008
0.030
0.77
0.56
0.228
0.560
0.18
0.94
0.031
0.156
0.54
0.66
ReliefF
0.104
0.204
0.80
1.00
0.005
0.433
0.33
0.97
0.128
0.333
0.18
0.94
0.018
0.126
0.48
0.68
Table 3: Silhouette coefficient, KL divergence, and F1-score of a kNN trained on the data for four distinct datasets and nine feature scorers. The HD columns show the silhouette coefficient for the original high-dimensional space, the 2D columns show the silhouette coefficient for the 2D embedding representation learned by t-SNE, and the KL columns show the Kullback–Leibler divergence from the t-SNE optimization. For each scorer, the silhouette coefficient is computed considering the scalarization of the dimensions by their importance, except for “no scoring”. The kNN column shows the F1-score on a test set (33% of samples) of a kNN trained on the datasets using the importance scores.
XOR
Synth
Liver
Prostate
Regression
Mouse cortex
No scoring
0.814
0.653
0.939
0.809
0.714
0.548
Kruskal–Wallis
0.997
0.863
0.949
0.896
–
–
Mutual Information
0.978
0.728
0.949
0.843
–
–
mRMR
0.990
0.792
0.937
0.821
–
–
ReliefF
0.978
0.701
0.954
0.826
–
–
Lasso
0.934
0.648
0.943
0.815
–
0.995
Table 4: Trustworthiness T(7) of the 2D t-SNE and weighted t-SNE projections for all datasets and feature scorers.
Figure 7: Weighted t-SNE visualization for the mouse cortex dataset. Each figure is the visualization of the 2D embedding with weighted t-SNE for three machine learning models, plus the original data in “no scoring”. The captions describe the F1-score of each model and the silhouette coefficient (SC) of each 2D embedding. Each color corresponds to a different class. Following the scheme from Kobak and Berens [47] , warm colors correspond to inhibitory neurons, cold colors correspond to excitatory neurons, and brown or gray colors correspond to non-neural cells.
Figure 8: Comparison between the weighted t-SNE and the activation of the neurons in the last layer of a neural network. On the left is the same weighted t-SNE presented in Fig. 2(j) , and on the right is the neurons’ activation of the last layer (before the softmax function) of the same neural network. Because this layer only has two neurons, the activation of the first neuron is on the x -axis and the activation of the second neuron is on the y -axis. Visualization details of the XOR dataset as in Fig. 2 .
Figure 9: Visualization of the training of a neural network with the XOR dataset. Each figure shows the weighted t-SNE for the feature scores of a specific training epoch of a neural network. Each caption describes the training epoch, the training loss, and the silhouette coefficient (SC) of the 2D embedding. Visualization details of the XOR dataset as in Fig. 2 . An animation of this plot is available at https://sbcblab.github.io/wtsne/ .
Figure 10: Visualization of the training of a neural network with the mouse cortex dataset. Each figure shows the weighted t-SNE for the feature scores of a specific training epoch of a neural network. Each caption describes the training epoch, the training loss, and the silhouette coefficient (SC) of the 2D embedding. Visualization details of the mouse cortex dataset as in Fig. 7 . An animation of this plot is available at https://sbcblab.github.io/wtsne/ .
Dataset
Correlation
XOR
− 0.85
Synth
− 0.37
Liver
− 0.81
Prostate
− 0.67
Regression
− 0.74
Mouse cortex
− 0.98
Table 5: Pearson correlation between training loss of neural network and the corresponding silhouette coefficient of the weighted t-SNE 2D embedding.
Figure 11: Comparison using weighted t-SNE between two random forests with the same hyperparameters. Both of them achieved the same training F1-score of 0.99 and test F1-score of 0.93. Even so, they scored features differently, resulting in distinct visualizations. The two random forests were trained to label the red samples from the blue and cyan samples in a binary classification.
Random Forest A
Random Forest B
Feature
Importance
Feature
Importance
rel14
0.150922
rel13
0.126705
rel11
0.144392
rel15
0.112081
rel15
0.116410
rel11
0.111083
rel12
0.100376
rel14
0.075461
rel7
0.046047
rel9
0.068682
Table 6: The feature importance of the ten top ranking features of the two random forests from Fig. 11 .
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Scorer
Type
Correlation
Reference
Kruskal-Wallis
Filter
Univariate
[ 30 ]
Mutual Information
Filter
Univariate
[ 32 ]
mRMR
Filter
Multivariate
[ 31 ]
ReliefF
Filter
Multivariate
[ 33 ]
Lasso
Embedded
Multivariate
[ 35 ]
Decision Tree
Embedded
Multivariate
[ 77 ]
Appendix
Table A.1 : Properties of the feature scorers used in the experiments. The type refers to how the scorer works and the correlation to how interactions between features are treated.