Across a wide range of applications, kernel measures are used to tune hyperparameters, select among candidate kernels, and assess the suitability of kernels for specific learning tasks. For classification, kernel measures are used to predict which kernels may produce accurate kernel classifiers. Many existing kernel measures, such as alignment to the ideal kernel, are designed to reward kernels which captures all samples from the same class as similar. While effective in many settings, this requirement is unnecessarily restrictive for classification, where classes may exhibit substantial internal structure. In this work, the "Confidence Score", a new kernel measure, is proposed. Unlike alignment-to-the-ideal kernel measures, the Confidence Score measures the alignment between an encoding of the ground truth and that ground truth projected by the kernel, allowing kernels that preserve intra-class structure to achieve high scores. Through experiments, it is demonstrated that the Confidence Score exhibits a higher correlation with classification accuracy across a variety of kernels on both real-world and synthetic datasets, compared to select other kernel measures.
Figures & tables
CKA ( J→∞ )
RFSC ( J→∞ )
CS ( J→∞ )
Acc.
Kideal
1 (1)
λC−1 ( λC−1 )
Eq 11 (1)
1
Kmin
(2J−4)2 (0)
JC−1 (0)
1 (1)
1
Kident
J−11 (0)
JC−1 (0)
0 (0)
0
Table 1: The CKA from Equation 2 , RFSC from Equation 7 , CS from Equation 9 , and leave-one-out classification accuracy using three example kernels. The limit as J→∞ is presented in parentheses.
Dataset Name
Size
Dimensions
Num. Classes
Banknote
1372
4
2
Bioddeg
1055
41
2
Diabetes
768
8
2
Glass
214
9
7
Heart
270
13
2
Iris
150
4
3
Table 2: A summary of the datasets used in this research.
Figure 1: The CS, CKA, and leave-one-out classification accuracy over a range of bandwidth values on eight datasets. The maxima of the CS and CKA are denoted by purple diamonds. Since there is no unique maximum classification accuracy for most datasets, the maximum accuracy is not marked.
Dataset
Accuracy at Max CKA
Accuracy at Max CS
Max Accuracy
Clusters
0.922
0.967
0.978
Moons
0.919
0.981
0.986
Iris
0.951
0.973
0.982
Glass
0.771
0.893
0.899
Wine
0.978
0.989
0.989
Lympho
0.774
0.922
0.926
Table 3: Classification accuracy (from regularized regression classifier) determined by bandwidth from maximum kernel measures compared to maximum accuracy across all bandwidths, from data presented in Fig. 3 .
Figure 2: The CS, CKA, and leave-one-out classification accuracy of a Gaussian kernel classifier across a range of bandwidth values on 10 partitions of the MNIST dataset.
Dataset
p-value CS vs CKA
p-value CS vs RFSC
Banknote
0.00182
–
Biodeg
0.00001
0.00060
Diabetes
0.00000
0.00917
Glass
0.00004
0.00000
Heart
0.00005
0.00033
Iris
0.00001
0.00000
Table 4: Maximum p-value obtained from the Cocor tests comparing the correlation of classification accuracy with CS against the corresponding correlations obtained with CKA and RFSC.
Figure 3: Synthetic datasets.
Figure 4: Heat maps of kernels on the ‘Clusters’ dataset (top) and ‘Noisy Crescent Moons’ dataset (bottom). The ideal kernels are on the left, the LVBL kernels in centre, and the Learned to Align with the Ideal (LAI) kernels [ 14 ] are on the right. Blue indicates minimum similarity and red indicates maximum similarity.
CS
CKA
LKA
HKA
Acc.
Cluster LVBL
0.937
0.190
0.426
0.308
0.978
Clusters LAI
0.898
0.283
0.543
0.413
0.933
Moons LVBL
0.867
0.158
0.378
0.268
0.961
Moons LAI
0.821
0.259
0.487
0.373
0.931
Table 5: Ideal kernel alignment-based and confidence based kernel measures on kernels, classification accuracy, on synthetic datasets from Fig. 3 .