Estimation of Confidence Bounds in Binary Classification using Wilson Score Kernel Density Estimation

Thorbjørn Mosekjær Iversen; Zebin Duan; Frederik Hagelskjær

cs.LGFeb 24, 2026

Estimation of Confidence Bounds in Binary Classification using Wilson Score Kernel Density Estimation

Authors: Thorbjørn Mosekjær Iversen, Zebin Duan, Frederik Hagelskjær

Organizations: SDU Robotics, The Maersk Mc-Kinney Moller Institute, University of Southern

Abstract

The performance and ease of use of deep learning-based binary classifiers have improved significantly in recent years. This has opened up the potential for automating critical inspection tasks, which have traditionally only been trusted to be done manually. However, the application of binary classifiers in critical operations depends on the estimation of reliable confidence bounds such that system performance can be ensured up to a given statistical significance. We present Wilson Score Kernel Density Classification, which is a novel kernel-based method for estimating confidence bounds in binary classification. The core of our method is the Wilson Score Kernel Density Estimator, which is a function estimator for estimating confidence bounds in Binomial experiments with conditionally varying success probabilities. Our method is evaluated in the context of selective classification on four different datasets, illustrating its use as a classification head of any feature extractor, including vision foundation models. Our proposed method shows similar performance to Gaussian Process Classification, but at a lower computational complexity.

Figures & tables

Fig. 1 : Overview of the Wilson Score Kernel Density Classifier when used for image classification. The image is fed to a feature extractor, e.g. a vision foundation model, and mapped to a lower-dimensional space. Confidence bounds for the class probability are computed from a set of labeled training samples using the Wilson Score Kernel Density Estimator.

Panel of Figure 2 — Fig. 2 : Intuitive illustration of WS-KDE. (a) The Wilson Score method could in principle be used for estimating confidence bounds by binning the feature space and treating the samples in each bin as a part of separate Binomial tests. (b) WS-KDE provides a more elegant solution by combining the Wilson Score method with kernel smoothing.

Panel of Figure 3 — Fig. 3 : As part of an automated assembly, the robot inserts a part into a fixture. A vision-based classifier is used to infer in the insertion was successful, before the assembly process can continue.

Panel of Figure 4 — Fig. 4 : Precision/Recall reject plots for four of the experiments. In plots b and c ResNet18 is used as the feature extractor, while plot d uses Dinov3. In b-d dimensionality reduction is done with UMAP. The lines indicate the mean, and the shaded area indicates the 5% and 95% quantiles over 50 repeated experiments. The vertical lines indicate at which coverage the corresponding lower confidence bound is at $\tau=95\%$ . The precision to the right of this line must be above $\tau$ for the classifier to be reliable.

			AUPRC $\uparrow$	AURRC $\uparrow$	$t_{\text{optim}}\ (s)\downarrow$	$t_{\text{infer}}\ (s)\downarrow$
Dataset	Dim. reduction	Method	mean [std]	mean [std]	mean [std]	mean [std]
Bank [ 16 ]	None	GPC	0.996 [0.004]	1.0 [0.0]	12.1 [0.7]	0.04 [0.01]
		WS-KDC	0.994 [0.006]	0.9999 [0.0007]	0.23 [0.03]	0.05 [0.02]
Cats & Dogs [ 17 ] (1k)	ResNet18 + pca	GPC	0.97 [0.01]	0.98 [0.01]	4.6 [0.4]	0.03 [0.01]
		WS-KDC	0.97 [0.01]	0.98 [0.01]	0.22 [0.01]	0.04 [0.03]
	ResNet18 + umap	GPC	0.97 [0.01]	0.99 [0.01]	4.5 [0.4]	0.03 [0.01]

TABLE I : Summary of experimental results for WS-KDC and GPC. The experiments cover four datasets and various feature extractions and dimensionality reduction methods. The data shows that the selective classification performance expressed as the area under the precision/recall reject curve (AUPRC/AURRC) was similar for the two methods. The only considerable difference between the two methods is the optimization time, where WS-KDE is significantly faster.

Fig. 5 : Plot of the mapping from ResNet18 classification confidences to the confidence bounds estimated by the WS-KDC and GPC during a single run of the Cats & Dogs ResNet18 + fc (1k) experiment.

Fig. 6 : Double logarithmic Plot of the mean of the optimization and inference time of WS-KDC and GPC when applied to various-sized subsets of the chestMNIST ResNet18 + umap experiment. Error bars indicate the standard deviation. The plot stops at 4000 for GPC as the mean optimization time was 525 seconds. In comparison, WS-KDC was optimized in 1.5 seconds.

Binary Classification

Binary classification assigns examples to one of two classes while balancing accuracy, calibration, class imbalance, and the costs of different errors.

See all 66 papers

Computationally lightweight classifiers with frequentist bounds on predictions
Mar 23, 2026Shreeram Murali, Cristian R. Rojas, Dominik BaumannClassification Uncertainty Quantification
Cyber-physical Systems Group, Aalto University · Decision and Control Systems, KTH Royal Institute of Technology
Improving Selective Classification with Pairwise Queries for Binary Classification
May 28, 2026Harsh Vardhan, Sunav Choudhary, Natwar Modani +1Pairwise Comparison Selective Prediction
CSE, UCSD, 9500 Gilman Drive, La Jolla, 92092, CA, USA. · Adobe Research, 345 Park Ave, San Jose, 95110, CA, USA. · Adobe Research, Marathahalli, Bengaluru, 560087, Karnataka, India. +1
ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning
Aug 10, 2026Ange-Clément Akazan, Ineza Remy Mugenga, Abebe Geletu +2Chance-Constrained Optimization Classification
School of Agriculture and Science, University of KwaZulu-Natal, Pietermaritzburg, KwaZulu-Natal, South Africa · 2AIMS Research and Innovation Centre, African Institute for Mathematical Sciences, Kigali, Rwanda · 3African Institute for Mathematical Sciences, Kigali, Rwanda

Mar 23, 2026J/K move · Enter open · S save

Computationally lightweight classifiers with frequentist bounds on predictions

Shreeram Murali, Cristian R. Rojas, Dominik Baumann

Cyber-physical Systems Group, Aalto University · Decision and Control Systems, KTH Royal Institute of Technology

Classification Uncertainty Quantification

Open paper AskPDF arXiv

While both classical and neural network classifiers can achieve high accuracy, they fall short on offering uncertainty bounds on their predictions, making them unfit for safety-critical applications. Existing kernel-based classifiers that provide such bounds scale with

\mathcal O (n^{\sim3})

in time, making them computationally intractable for large datasets. To address this, we propose a novel, computationally efficient classification algorithm based on the Nadaraya-Watson estimator, for whose estimates we derive frequentist uncertainty intervals. We evaluate our classifier on synthetically generated data and on electrocardiographic heartbeat signals from the MIT-BIH Arrhythmia database. We show that the method achieves competitive accuracy

>

\SI{96}{\percent} at

\mathcal O(n)

and

\mathcal O(\log n)

operations, while providing actionable uncertainty bounds. These bounds can, e.g., aid in flagging low-confidence predictions, making them suitable for real-time settings with resource constraints, such as diagnostic monitoring or implantable devices.

Abstract

Binary Classification

Explore similar work

Computationally lightweight classifiers with frequentist bounds on predictions

Improving Selective Classification with Pairwise Queries for Binary Classification

ReliableNet: A Chance-Constrained Approach to Trustworthy Classification in Deep Learning