Pairwise Difference Learning for Classification

Mohamed Karim Belaid; Maximilian Rabus; Eyke Hüllermeier

cs.LGJun 28, 2024

Pairwise Difference Learning for Classification

Authors: Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier

Organizations: IDIADA Fahrzeugtechnik GmbH, Munich, Germany · Dr. Ing. h.c. F. Porsche AG Stuttgart, Germany · Ludwig-Maximilians-Universität Munich, Germany

Abstract

Pairwise difference learning (PDL) has recently been introduced as a new meta-learning technique for regression. Instead of learning a mapping from instances to outcomes in the standard way, the key idea is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. This paper extends PDL toward the task of classification and proposes a meta-learning technique for inducing a PDL classifier by solving a suitably defined (binary) classification problem on a paired version of the original training data. We analyze the performance of the PDL classifier in a large-scale empirical study and find that it outperforms state-of-the-art methods in terms of prediction performance. Last but not least, we provide an easy-to-use and publicly available implementation of PDL in a Python package.

Figures & tables

Figure 1: Illustration of the PDL classifier.

Figure 2: Comparing learned patterns using PDL classifiers and baseline models.

Figure 3: Distribution of key characteristics of the 99 OpenML classification datasets (minimum, mean, maximum).

Figure 4: Comparing average Macro F1 score of optimized baseline classifiers and PDL classifiers.

	Significant wins		Wins		Average Test Macro F1
Classifier	base	PDC	base	PDC	base	$\pm$ sem	PDC	$\pm$ sem
Bagging	3	26	27	70	0.7906	0.0035	0.8062	0.0034
DecisionTree	2	50	22	76	0.7694	0.0037	0.7982	0.0034
ExtraTree	1	61	9	90	0.7434	0.0037	0.7987	0.0035
ExtraTrees	6	24	21	77	0.7951	0.0036	0.8113	0.0035
GradientBoosting	9	23	25	72	0.7839	0.0037	0.7903	0.0039

Table 1: Comparing baseline classifiers to PDC using 99 datasets.

Estimator	Unique contribution	Wins
ExtraTree	0	0
HistGradientBoosting	0	0
RandomForest	0.00002	1
Bagging	0.00004	2
GradientBoosting	0.00006	2
DecisionTree	0.00020	10

Table 2: Unique contribution of each estimator to the average Macro F1 score using the best optimized model on each dataset.

	# CV	Baseline Macro F1		PDC Macro F1		Test	Test
Estimator	runs	Train	Test	Train	Test	$\Delta F1$	p-value
Bagging	20	0.998	0.835	0.999	0.859	0.024	$10^{-15}$
DecisionTree	14	0.950	0.884	0.955	0.895	0.011	$10^{-05}$
ExtraTree	11	0.915	0.844	0.924	0.861	0.017	$10^{-04}$
ExtraTrees	26	0.985	0.828	0.991	0.853	0.025	$10^{-16}$
GradientBoosting	58	0.930	0.822	0.926	0.840	0.018	$10^{-17}$

Table 3: Comparing test Macro F1 on the subset of runs where train scores are not significantly different.

Figure 5: Effect of the anchor set size on PDC’s loss relative to the baseline.

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 401 papers

Uncertainty Quantification in Pairwise Difference Learning for Classification
Oct 15, 2026Mohamed Karim Belaid, Maximilian Rabus, Eyke HüllermeierUncertainty Quantification Uncertainty Decomposition
IDIADA Fahrzeugtechnik GmbH, Munich, Germany. · Dr. Ing. h.c. F. Porsche AG, Stuttgart, Germany. · Institute of Informatics, University of Munich (LMU), Germany. +2
Efficient Pairwise Difference Learning: A Comparative Study of Encoding Strategies and Training Pair Selection
Oct 15, 2026Bernard Georges, Mohamed Karim Belaid, Manuel RodriguesData-Efficient Learning Training Data Selection
University of Minho, ALGORITMI CENTER LASI, Braga, Portugal · IDIADA Fahrzeugtechnik GmbH, Munich, Germany
Binary Classification from Coupled Pairwise Labels
Mar 20, 2026Tomoya Tate, Kosuke Sugiyama, Masato UchidaPairwise Comparison Binary Classification
Waseda University, Tokyo, Japan

Oct 15, 2026J/K move · Enter open · S save

Uncertainty Quantification in Pairwise Difference Learning for Classification

Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier

IDIADA Fahrzeugtechnik GmbH, Munich, Germany. · Dr. Ing. h.c. F. Porsche AG, Stuttgart, Germany. · Institute of Informatics, University of Munich (LMU), Germany.+2

Uncertainty Quantification Uncertainty Decomposition

Open paper AskPDF arXiv

Instead of learning a mapping from instances to outcomes in the standard way, the key idea of pairwise difference learning (PDL) is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. We propose a version of PDL for the task of classification and propose a suitable meta-learning technique for this purpose. Moreover, we address the problem of uncertainty quantification, i.e., of representing and quantifying the uncertainty of a PDL classifier's predictions in a faithful manner. Building on recent work on the decomposition of total uncertainty into aleatoric and epistemic components, we propose different ways of extending corresponding measures to the setting of PDL, depending on how and to what extent uncertainty is represented by the underlying base learner. In large-scale empirical studies, we analyze both the predictive accuracy and the uncertainty awareness of our PDL classifier.

Abstract

Classification

Explore similar work

Uncertainty Quantification in Pairwise Difference Learning for Classification

Efficient Pairwise Difference Learning: A Comparative Study of Encoding Strategies and Training Pair Selection

Binary Classification from Coupled Pairwise Labels