Uncertainty Quantification in Pairwise Difference Learning for Classification
Authors: Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier
Organizations: IDIADA Fahrzeugtechnik GmbH, Munich, Germany. · Dr. Ing. h.c. F. Porsche AG, Stuttgart, Germany. · Institute of Informatics, University of Munich (LMU), Germany. · Munich Center for Machine Learning (MCML), Munich, Germany. · German Centre for Artificial Intelligence (DFKL, DSA), Kaiserslautern, Germany.
Instead of learning a mapping from instances to outcomes in the standard way, the key idea of pairwise difference learning (PDL) is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. We propose a version of PDL for the task of classification and propose a suitable meta-learning technique for this purpose. Moreover, we address the problem of uncertainty quantification, i.e., of representing and quantifying the uncertainty of a PDL classifier's predictions in a faithful manner. Building on recent work on the decomposition of total uncertainty into aleatoric and epistemic components, we propose different ways of extending corresponding measures to the setting of PDL, depending on how and to what extent uncertainty is represented by the underlying base learner. In large-scale empirical studies, we analyze both the predictive accuracy and the uncertainty awareness of our PDL classifier.
Figures & tables
Base learner γ
Averaging
Integrating
Deterministic
First-order predictor
(Second-order predictor)
pdc:X→ΔK
Probabilistic
First-order predictor
Second-order predictor
pdc:X→ΔK
PDC:X→P(ΔK)
Bayesian
Second-order predictor
(Third-order predictor)
PDC:X→P(ΔK)
Table 1: Different types of predictors obtained from different types of base learners in PDL through averaging or integrating their predictions.
Figure 1: Comparison of learned patterns: (a) Deterministic predictor (1-NN) and its dual pdc1-NN ; (b) Probabilistic predictor (DT) and its dual PDCDT .
Figure 2: Second-order probability distributions predicted by PDCRF using, from left to right, the UL, ULP, and UP approaches. Each representation impacts the uncertainty measurements differently.
Figure 3: Distribution of key characteristics (minimum, mean, maximum) for the 99 OpenML classification datasets.
Significant wins
Wins
Average Test Macro F1
Classifier
base
PDC
base
PDC
base
± sem
PDC
± sem
DecisionTree
2
50
22
76
0.7694
0.0037
0.7982
0.0034
ExtraTree
1
61
9
90
0.7434
0.0037
0.7987
0.0035
Bagging
3
26
27
70
0.7906
0.0035
0.8062
0.0034
RandomForest
5
27
22
73
0.7933
0.0035
0.8073
0.0034
ExtraTrees
6
24
21
77
0.7951
0.0036
0.8113
0.0035
Table 2: Comparing baseline classifiers to PDC using 99 datasets.
Figure 4: Distribution of Rank Correlation with probabilistic learners.
Figure 5: Distribution of Rank Correlation with Bayesian learners.
Figure 6: Estimation of TU: Baseline DT and Bagging(MLP) models fail to capture total uncertainty accurately, whereas their PDC counterparts yield robust TU estimates.
Figure 7: Splitting of AU and EU. RandomForest incorrectly allocates AU to an empty region, whereas both PDC(DT) and PDC(RandomForest) consistently assign the highest AU to the noisy center and the highest EU to the sparse region.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Estimator
Unique contribution
Wins
ExtraTree
0
0
RandomForest
0.00002
1
Bagging
0.00004
2
DecisionTree
0.00020
10
ExtraTrees
0.00041
9
PDC
0.01312
75
Appendix
Table 3: Unique contribution of each estimator to the average Macro F1 score using the best optimized model on each dataset.
# CV
Baseline Macro F1
PDC Macro F1
Test
Test
Estimator
runs
Train
Test
Train
Test
ΔF1
p-value
DecisionTree
14
0.950
0.884
0.955
0.895
0.011
10−05
ExtraTree
11
0.915
0.844
0.924
0.861
0.017
10−04
ExtraTrees
26
0.985
0.828
0.991
0.853
0.025
10−16
RandomForest
18
0.992
0.855
0.997
0.881
0.026
10−11
Bagging
20
0.998
0.835
0.999
0.859
0.024
10−15
Appendix
Table 4: Comparing test Macro F1 on the subset of runs where train scores are not significantly different.
Figure 8: Accuracy rejection curve for the Dataset 772.
Figure 9: Accuracy rejection curve for the Dataset 991.
Figure 10: Accuracy rejection curve for the Dataset 1462.
Figure 11: Accuracy rejection curve for the Dataset 40704.
Figure 12: Accuracy rejection curve for the Dataset 44490.
Figure 13: ECE rejection curve for the Dataset 772.
Figure 14: ECE rejection curve for the Dataset 991.
Figure 15: ECE rejection curve for the Dataset 1462.
Figure 16: ECE rejection curve for the Dataset 40704.
Figure 17: ECE rejection curve for the Dataset 44490.
Pairwise difference learning (PDL) has recently been introduced as a new meta-learning technique for regression. Instead of learning a mapping from instances to outcomes in the standard way, the key idea is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. This paper extends PDL toward the task of classification and proposes a meta-learning technique for inducing a PDL classifier by solving a suitably defined (binary) classification problem on a paired version of the original training data. We analyze the performance of the PDL classifier in a large-scale empirical study and find that it outperforms state-of-the-art methods in terms of prediction performance. Last but not least, we provide an easy-to-use and publicly available implementation of PDL in a Python package.
Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier
IDIADA Fahrzeugtechnik GmbH, Munich, Germany · Dr. Ing. h.c. F. Porsche AG Stuttgart, Germany · Ludwig-Maximilians-Universität Munich, Germany
Single-pass uncertainty quantification (UQ) methods for classification represent uncertainty by predicting a tractable distribution over the class probability vector. While existing approaches primarily focus on enhancing the expressiveness of this distribution, they often provide limited insight into how predictive uncertainty is structured and aggregated, resulting in weak interpretability. We introduce the courtroom analogy, which conceptualizes uncertainty-aware classification as a structured debate among class-specific advocates. Each advocate forms a probabilistic opinion, and a final verdict is reached by aggregating these opinions using input-dependent plausibility weights. In this framework, each advocate's opinion is modeled as a Dirichlet distribution whose concentration parameter is decomposed into shared evidence and class-specific advocacy. This yields a structured mixture of Dirichlet distributions with semantically interpretable parameters. To instantiate this formulation, we propose Mixture of Dirichlet EXperts (MoDEX), a single-pass neural architecture that predicts the courtroom parameters, enabling efficient and expressive UQ while explicitly modeling uncertainty aggregation. We demonstrate that MoDEX enjoys strong theoretical properties and achieves state-of-the-art UQ performance across diverse benchmarks, yielding interpretable uncertainty estimates with meaningful semantics.
Taeseong Yoon, Heeyoung Kim
Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea.
Credal sets, i.e., closed convex sets of probability measures, provide a natural framework to represent aleatoric and epistemic uncertainty in machine learning. Yet how to quantify these two types of uncertainty for a given credal set, particularly in multiclass classification, remains underexplored. In this paper, we propose a distance-based approach to quantify total, aleatoric, and epistemic uncertainty for credal sets. Concretely, we introduce a family of such measures within the framework of Integral Probability Metrics (IPMs). The resulting quantities admit clear semantic interpretations, satisfy natural theoretical desiderata, and remain computationally tractable for common choices of IPMs. We instantiate the framework with the total variation distance and obtain simple, efficient uncertainty measures for multiclass classification. In the binary case, this choice recovers established uncertainty measures, for which a principled multiclass generalization has so far been missing. Empirical results confirm practical usefulness, with favorable performance at low computational cost.
Xabier Gonzalez-Garcia, Siu Lun Chau, Julian Rodemann +6
Dept. of Statistics, CS & Mathematics, Public Univ. of Navarre · EPIC Lab, Nanyang Technological Univ. Singapore · Rational Intelligence Lab, CISPA Helmholtz Center for Information Security +7