A probability calibrator for machine-learning classifiers
Authors: K. Dyrland, A. S. Lundervold, P. G. L. Porta Mana
Organizations: Dept of Computer science, Electrical Engineering and Mathematical Sciences, Western Norway University of Applied Sciences, Bergen, Norway · †& Mohn Medical Imaging and Visualization Centre, Dept of Radiology, Haukeland University Hospital, Bergen, Norway
In fields such as medicine and drug discovery, the ultimate goal of a classification is not to guess a class, but to choose the optimal course of action among a set of possible ones, usually not in one-one correspondence with the set of classes. This decision-theoretic problem requires proper probabilities for the classes. Proper probabilities conditional on the features are computationally almost impossible to find in many important cases. The main idea of the present work is to calculate probabilities conditional not on the features, but on the trained classifier's output. This calculation is cheap, needs to be made only once, and provides an output-to-probability "calibrator" or "transducer" that can be applied to all future outputs of the classifier. In conjunction with problem-dependent utilities, the probabilities of the transducer allow us to find the optimal choice among the classes or among a set of more general decisions, by means of expected-utility maximization. This idea is demonstrated in a simplified drug-discovery problem with a highly imbalanced dataset. The transducer and utility maximization together always lead to improved results, sometimes close to theoretical maximum, for all sets of problem-dependent utilities. The one-time-only calculation of the transducer also provides, automatically: (i) a quantification of the uncertainty about the transducer itself; (ii) the expected utility of the augmented algorithm (including its uncertainty), which can be used for algorithm selection; (iii) the possibility of using the algorithm in a "generative mode", useful if the training dataset is biased.
Figures & tables
Figure 1: Probabilities of class 1 ( “active” , blue solid curve ) and class 0 ( “inactive” , red dashed curve ) conditional on the random-forest output. Their extremal values are 0.0014 and 0.929. The random-forest output clearly cannot be interpreted as a probability. The shaded region around each curve represents its 12.5%–87.5% range of possible variability if more data were used to calculate the probabilities.
Figure 2: Probability of class 1 conditional on the convolutional-neural-network outputs. Its extremal values are 0.0014 and 0.923. Plot on the side: softmax function of the same outputs, for comparison.
Figure 3: Cross-section of the probability surface of fig. 2 across the bisector of the II and IV quadrants of output space. The shaded region represents the 12.5%–87.5% range of possible variability upon increase of the calibration dataset. The cross-section of the softmax function ( grey dashed curve ) is also shown for comparison.
[[r]1001]
[[r]10−1010]
[[r]1−10010]
[[r]10−1001]
standard method
[[r]32253779.5246.5]
Random Forest
augmentation
[[r]32075538288]
[[r]30502127319]
[[r]32075540286]
[[r]326203260]
standard method
[[r]31659749277]
Neural Network
augmentation
[[r]31897365261]
[[r]288238012314]
[[r]31897366260]
[[r]326203260]
Table 1: Confusion matrices from demonstration dataset
Figure 4: Space of utility matrices (modulo equivalence) for binary classification.
Figure 5: Rescaled utility yields obtained using the two classifiers in the standard way, vs those obtained with augmentation, for a uniform distribution of possible utility matrices over the utility-matrix space of fig. 4 . The augmentation always leads to an improved utility yield, especially in cases where the standard method has a low or high performance. Owing to noise coming from numerical rounding, in some cases the yield from augmentation may appear lower than from the standard method (points below the dashed grey line).
[[r]1001]
[[r]10−1010]
[[r]1−10010]
[[r]10−1001]
Random Forest
0.973
1.68
9.59
9.08
Neural Net
0.962
1.62
9.56
9.08
Table 3: Expected utilities for the two algorithms of § 4
Figure 6: Probability distributions of the long-run utility yields of random forest and convolutional neural network in the four cases of § 4
Figure 7: Probability densities of the random-forest output conditional on class 1 ( “active” , blue solid curve ) and on class 0 ( “inactive” , red dashed curve , truncated). The shaded region around each curve represents its 12.5%–87.5% range of possible variability upon increase of the calibration dataset.
Figure 8: Rescaled utility yields obtained using the two classifiers in the standard way, vs those obtained with augmentation in generative mode, on an altered dataset with very different class balance from the training and calibration sets. The distribution of possible utility matrices is uniform over their space, as before. The utility yields of the standard method have worsened with respect to those of fig. 5 , as can be seen from the histogram tails. The augmentation in generative mode, however has not suffered from this dataset mismatch.
Figure 9: Samples ( thin light-blue curves ) of probable transducer curves (for class 1) that could be obtained if we had more calibration data in the demonstration of § 4 . They are samples obtained from the distribution w(F\nonscript∣\nonscriptD) of eq. ( 20 ). According to the probability calculus, the curve to be used for a new unit is their average ( thicker dark-blue curve ), which is the same as plotted in fig. 1 .
Figure 10: Decision tree for the choice of algorithm. From left to right: Decision node about the algorithm (green), uncertainty node about the algorithm’s output (yellow), decision node (e.g. about class) for the inspected item (blue), uncertainty node about the class (red). Only some example nodes and branches are labelled.
Probability calibration measures and improves whether predicted probabilities match observed outcome frequencies, supporting reliable decisions under uncertainty.
The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.
In multiclass probabilistic prediction, Utility Calibration (UC), which focuses auditing on specified utilities, has recently received attention as a way to guarantee downstream decisions while controlling computational and sample requirements. At the same time, some multiclass problems have meaningful label hierarchies that play important roles in medicine and image classification, yet how UC evaluates utility within a hierarchy remains insufficiently understood. We show that the difference between realized utility and predicted mean utility admits an exact decomposition into a sum of contributions from the internal nodes of the label tree. This decomposition shows that positive and negative contributions from different nodes can cancel, and that even when UC is small, the utility errors remaining in parts of the hierarchy need not be small. To address this problem, we propose Hierarchical Utility Calibration (HUC), which evaluates each node contribution before summation while retaining the same target utility, subgroup, and predicted-utility interval. We further provide finite-sample evaluation over all predicted-utility intervals and propose HUC-Boost, which updates only violated internal nodes, with theoretical guarantees for both.
Futoshi Futami, Jerry Huang, Ichiro Takeuchi
The University of Osaka · The University of Tokyo · RIKEN Center for Advanced Intelligence Project (AIP) +3
Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.
Yuyang Dai, Yuting Tang, Futoshi Futami +1
Florida State University · The University of Tokyo · RIKEN AIP +1