Across a wide range of applications, kernel measures are used to tune hyperparameters, select among candidate kernels, and assess the suitability of kernels for specific learning tasks. For classification, kernel measures are used to predict which kernels may produce accurate kernel classifiers. Many existing kernel measures, such as alignment to the ideal kernel, are designed to reward kernels which captures all samples from the same class as similar. While effective in many settings, this requirement is unnecessarily restrictive for classification, where classes may exhibit substantial internal structure. In this work, the "Confidence Score", a new kernel measure, is proposed. Unlike alignment-to-the-ideal kernel measures, the Confidence Score measures the alignment between an encoding of the ground truth and that ground truth projected by the kernel, allowing kernels that preserve intra-class structure to achieve high scores. Through experiments, it is demonstrated that the Confidence Score exhibits a higher correlation with classification accuracy across a variety of kernels on both real-world and synthetic datasets, compared to select other kernel measures.
Figures & tables
CKA ( J→∞ )
RFSC ( J→∞ )
CS ( J→∞ )
Acc.
Kideal
1 (1)
λC−1 ( λC−1 )
Eq 11 (1)
1
Kmin
(2J−4)2 (0)
JC−1 (0)
1 (1)
1
Kident
J−11 (0)
JC−1 (0)
0 (0)
0
Table 1: The CKA from Equation 2 , RFSC from Equation 7 , CS from Equation 9 , and leave-one-out classification accuracy using three example kernels. The limit as J→∞ is presented in parentheses.
Dataset Name
Size
Dimensions
Num. Classes
Banknote
1372
4
2
Bioddeg
1055
41
2
Diabetes
768
8
2
Glass
214
9
7
Heart
270
13
2
Iris
150
4
3
Table 2: A summary of the datasets used in this research.
Figure 1: The CS, CKA, and leave-one-out classification accuracy over a range of bandwidth values on eight datasets. The maxima of the CS and CKA are denoted by purple diamonds. Since there is no unique maximum classification accuracy for most datasets, the maximum accuracy is not marked.
Dataset
Accuracy at Max CKA
Accuracy at Max CS
Max Accuracy
Clusters
0.922
0.967
0.978
Moons
0.919
0.981
0.986
Iris
0.951
0.973
0.982
Glass
0.771
0.893
0.899
Wine
0.978
0.989
0.989
Lympho
0.774
0.922
0.926
Table 3: Classification accuracy (from regularized regression classifier) determined by bandwidth from maximum kernel measures compared to maximum accuracy across all bandwidths, from data presented in Fig. 3 .
Figure 2: The CS, CKA, and leave-one-out classification accuracy of a Gaussian kernel classifier across a range of bandwidth values on 10 partitions of the MNIST dataset.
Dataset
p-value CS vs CKA
p-value CS vs RFSC
Banknote
0.00182
–
Biodeg
0.00001
0.00060
Diabetes
0.00000
0.00917
Glass
0.00004
0.00000
Heart
0.00005
0.00033
Iris
0.00001
0.00000
Table 4: Maximum p-value obtained from the Cocor tests comparing the correlation of classification accuracy with CS against the corresponding correlations obtained with CKA and RFSC.
Figure 3: Synthetic datasets.
Figure 4: Heat maps of kernels on the ‘Clusters’ dataset (top) and ‘Noisy Crescent Moons’ dataset (bottom). The ideal kernels are on the left, the LVBL kernels in centre, and the Learned to Align with the Ideal (LAI) kernels [ 14 ] are on the right. Blue indicates minimum similarity and red indicates maximum similarity.
CS
CKA
LKA
HKA
Acc.
Cluster LVBL
0.937
0.190
0.426
0.308
0.978
Clusters LAI
0.898
0.283
0.543
0.413
0.933
Moons LVBL
0.867
0.158
0.378
0.268
0.961
Moons LAI
0.821
0.259
0.487
0.373
0.931
Table 5: Ideal kernel alignment-based and confidence based kernel measures on kernels, classification accuracy, on synthetic datasets from Fig. 3 .
The performance and ease of use of deep learning-based binary classifiers have improved significantly in recent years. This has opened up the potential for automating critical inspection tasks, which have traditionally only been trusted to be done manually. However, the application of binary classifiers in critical operations depends on the estimation of reliable confidence bounds such that system performance can be ensured up to a given statistical significance. We present Wilson Score Kernel Density Classification, which is a novel kernel-based method for estimating confidence bounds in binary classification. The core of our method is the Wilson Score Kernel Density Estimator, which is a function estimator for estimating confidence bounds in Binomial experiments with conditionally varying success probabilities. Our method is evaluated in the context of selective classification on four different datasets, illustrating its use as a classification head of any feature extractor, including vision foundation models. Our proposed method shows similar performance to Gaussian Process Classification, but at a lower computational complexity.
Thorbjørn Mosekjær Iversen, Zebin Duan, Frederik Hagelskjær
SDU Robotics, The Maersk Mc-Kinney Moller Institute, University of Southern
Kernel methods separate data representation from decision-making, but typically require the kernel to be chosen in advance. We show that this kernel can instead be learned by alignment, and develop the resulting framework through the recently introduced Collaborative Learning and Inference (CLaI). We show that Collaborative Learning can be viewed as a kernel alignment process, in which an embedding is trained so that its induced similarity matches a label-derived target kernel. We also prove that Collaborative Inference is equivalent to kernel Bayes classification with Parzen-window density estimation. Motivated by these perspectives, we generalise CLaI by replacing cosine similarity with a learned Mahalanobis distance and extend it to multiclass classification. On CIFAR-10, PathMNIST, and SleepEDF, the Mahalanobis formulation improves accuracy, converges faster, and yields lower calibration error than the cosine-based variant. Auxiliary experiments further support these connections, showing that CLaI produces latent signals of the same form as a Gaussian process, while achieving competitive calibration on sepsis prediction. Together, these results establish a principled learned-kernel framework that unifies representation learning, kernel alignment, and Bayesian classification, and extends naturally to the multiclass setting.
Multivariate conformal prediction requires nonconformity scores that compress residual vectors into scalars while preserving certain implicit geometric structure of the residual distribution. We introduce a Multivariate Kernel Score (MKS) that produces prediction regions that explicitly adapt to this geometry. We show that the proposed score resembles the Gaussian process posterior variance, unifying Bayesian uncertainty quantification with the coverage guarantees of frequentist-type. Moreover, the MKS can be decomposed into an anisotropic Maximum Mean Discrepancy (MMD) that interpolates between kernel density estimation and covariance-weighted distance. We prove finite-sample coverage guarantees and establish convergence rates that depend on the effective rank of the kernel-based covariance operator rather than the ambient dimension, enabling dimension-free adaptation. On regression tasks, the MKS reduces the volume of prediction regions significantly, compared to ellipsoidal baselines while maintaining nominal coverage, with larger gains at higher dimensions and tighter coverage levels.