Uncertainty Quantification in Pairwise Difference Learning for Classification

Mohamed Karim Belaid; Maximilian Rabus; Eyke Hüllermeier

cs.LGOct 15, 2026

Uncertainty Quantification in Pairwise Difference Learning for Classification

Authors: Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier

Organizations: IDIADA Fahrzeugtechnik GmbH, Munich, Germany. · Dr. Ing. h.c. F. Porsche AG, Stuttgart, Germany. · Institute of Informatics, University of Munich (LMU), Germany. · Munich Center for Machine Learning (MCML), Munich, Germany. ·

Abstract

Instead of learning a mapping from instances to outcomes in the standard way, the key idea of pairwise difference learning (PDL) is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. We propose a version of PDL for the task of classification and propose a suitable meta-learning technique for this purpose. Moreover, we address the problem of uncertainty quantification, i.e., of representing and quantifying the uncertainty of a PDL classifier's predictions in a faithful manner. Building on recent work on the decomposition of total uncertainty into aleatoric and epistemic components, we propose different ways of extending corresponding measures to the setting of PDL, depending on how and to what extent uncertainty is represented by the underlying base learner. In large-scale empirical studies, we analyze both the predictive accuracy and the uncertainty awareness of our PDL classifier.

Figures & tables

Base learner $\gamma$	Averaging	Integrating
Deterministic	First-order predictor	(Second-order predictor)
	$\text{pdc}:\mathcal{X}\to\Delta_{K}$
Probabilistic	First-order predictor	Second-order predictor
	$\text{pdc}:\mathcal{X}\to\Delta_{K}$	$\text{PDC}:\mathcal{X}\to\mathbb{P}(\Delta_{K})$
Bayesian	Second-order predictor	(Third-order predictor)
	$\text{PDC}:\mathcal{X}\to\mathbb{P}(\Delta_{K})$

Table 1: Different types of predictors obtained from different types of base learners in PDL through averaging or integrating their predictions.

Panel of Figure 1 — Figure 1: Comparison of learned patterns: (a) Deterministic predictor (1-NN) and its dual $\text{pdc}_{\text{1-NN}}$ ; (b) Probabilistic predictor (DT) and its dual $\text{PDC}_{\text{DT}}$ .

Figure 2: Second-order probability distributions predicted by $\text{PDC}_{RF}$ using, from left to right, the UL, ULP, and UP approaches. Each representation impacts the uncertainty measurements differently.

Figure 3: Distribution of key characteristics (minimum, mean, maximum) for the 99 OpenML classification datasets.

	Significant wins		Wins		Average Test Macro F1
Classifier	base	PDC	base	PDC	base	$\pm$ sem	PDC	$\pm$ sem
DecisionTree	2	50	22	76	0.7694	0.0037	0.7982	0.0034
ExtraTree	1	61	9	90	0.7434	0.0037	0.7987	0.0035
Bagging	3	26	27	70	0.7906	0.0035	0.8062	0.0034
RandomForest	5	27	22	73	0.7933	0.0035	0.8073	0.0034
ExtraTrees	6	24	21	77	0.7951	0.0036	0.8113	0.0035

Table 2: Comparing baseline classifiers to PDC using 99 datasets.

Panel of Figure 4 — Figure 4: Distribution of Rank Correlation with probabilistic learners.

Panel of Figure 5 — Figure 5: Distribution of Rank Correlation with Bayesian learners.

Panel of Figure 6 — Figure 6: Estimation of TU: Baseline DT and Bagging(MLP) models fail to capture total uncertainty accurately, whereas their PDC counterparts yield robust TU estimates.

Panel of Figure 7 — Figure 7: Splitting of AU and EU. RandomForest incorrectly allocates AU to an empty region, whereas both PDC(DT) and PDC(RandomForest) consistently assign the highest AU to the noisy center and the highest EU to the sparse region.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Estimator	Unique contribution	Wins
ExtraTree	0	0
RandomForest	0.00002	1
Bagging	0.00004	2
DecisionTree	0.00020	10
ExtraTrees	0.00041	9
PDC	0.01312	75

Appendix

Table 3: Unique contribution of each estimator to the average Macro F1 score using the best optimized model on each dataset.

	# CV	Baseline Macro F1		PDC Macro F1		Test	Test
Estimator	runs	Train	Test	Train	Test	$\Delta F1$	p-value
DecisionTree	14	0.950	0.884	0.955	0.895	0.011	$10^{-05}$
ExtraTree	11	0.915	0.844	0.924	0.861	0.017	$10^{-04}$
ExtraTrees	26	0.985	0.828	0.991	0.853	0.025	$10^{-16}$
RandomForest	18	0.992	0.855	0.997	0.881	0.026	$10^{-11}$
Bagging	20	0.998	0.835	0.999	0.859	0.024	$10^{-15}$

Appendix

Table 4: Comparing test Macro F1 on the subset of runs where train scores are not significantly different.

Panel of Figure 8 — Figure 8: Accuracy rejection curve for the Dataset 772.

Panel of Figure 9 — Figure 9: Accuracy rejection curve for the Dataset 991.

Panel of Figure 10 — Figure 10: Accuracy rejection curve for the Dataset 1462.

Panel of Figure 11 — Figure 11: Accuracy rejection curve for the Dataset 40704.

Panel of Figure 12 — Figure 12: Accuracy rejection curve for the Dataset 44490.

Panel of Figure 13 — Figure 13: ECE rejection curve for the Dataset 772.

Panel of Figure 14 — Figure 14: ECE rejection curve for the Dataset 991.

Panel of Figure 15 — Figure 15: ECE rejection curve for the Dataset 1462.

Panel of Figure 16 — Figure 16: ECE rejection curve for the Dataset 40704.

Panel of Figure 17 — Figure 17: ECE rejection curve for the Dataset 44490.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 636 papers

Pairwise Difference Learning for Classification
Jun 28, 2024Mohamed Karim Belaid, Maximilian Rabus, Eyke HüllermeierClassification Meta-Learning
IDIADA Fahrzeugtechnik GmbH, Munich, Germany · Dr. Ing. h.c. F. Porsche AG Stuttgart, Germany · Ludwig-Maximilians-Universität Munich, Germany
Courtroom Analogy: New Perspective on Uncertainty-Aware Classification
May 25, 2026Taeseong Yoon, Heeyoung KimDirichlet Process Mixture Models Uncertainty Quantification
Department of Industrial and Systems Engineering, Korea Advanced Institute of Science and Technology, Daejeon, South Korea.
Quantification of Credal Uncertainty: A Distance-Based Approach
Mar 28, 2026Xabier Gonzalez-Garcia, Siu Lun Chau, Julian Rodemann +6Uncertainty Quantification Aleatoric Uncertainty
Dept. of Statistics, CS & Mathematics, Public Univ. of Navarre · EPIC Lab, Nanyang Technological Univ. Singapore · Rational Intelligence Lab, CISPA Helmholtz Center for Information Security +7

Jun 28, 2024J/K move · Enter open · S save

Pairwise Difference Learning for Classification

Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier

IDIADA Fahrzeugtechnik GmbH, Munich, Germany · Dr. Ing. h.c. F. Porsche AG Stuttgart, Germany · Ludwig-Maximilians-Universität Munich, Germany

Classification Meta-Learning Supervised Learning Binary Classification

Open paper AskPDF arXiv

Pairwise difference learning (PDL) has recently been introduced as a new meta-learning technique for regression. Instead of learning a mapping from instances to outcomes in the standard way, the key idea is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. This paper extends PDL toward the task of classification and proposes a meta-learning technique for inducing a PDL classifier by solving a suitably defined (binary) classification problem on a paired version of the original training data. We analyze the performance of the PDL classifier in a large-scale empirical study and find that it outperforms state-of-the-art methods in terms of prediction performance. Last but not least, we provide an easy-to-use and publicly available implementation of PDL in a Python package.

Abstract

Uncertainty Quantification

Explore similar work

Pairwise Difference Learning for Classification

Courtroom Analogy: New Perspective on Uncertainty-Aware Classification

Quantification of Credal Uncertainty: A Distance-Based Approach