Pairwise Difference Learning for Classification (PDC) enhances supervised learning in low-data regimes by training a binary similarity classifier on pairs of instances rather than individual samples, effectively inflating the training set from n to O(n2) examples. However, the PDC framework leaves critical implementation choices underspecified, particularly regarding input representation and pair selection strategies. This work systematically investigates these design decisions through extensive empirical evaluation. We compare four feature encoders (paired features, triplet with relative difference, absolute difference, and a novel triplet with absolute difference) alongside four pair selection strategies ranging from exhaustive to kernel-thinning-based schemes. Experiments across nearly 200 benchmark classification datasets using Decision Tree, Multilayer Perceptron, Gaussian Naive Bayes, and Support Vector Classifier as base models reveal that the proposed triplet with absolute difference encoder (φ3dabs) achieves the highest average F1 scores. More notably, restricting training to upper-triangle pairs reduces computational time by approximately 40% with negligible accuracy loss. Kernel thinning-based selection provided no accuracy benefits while substantially increasing computational overhead. Overall, PDC configurations outperformed base models in 94% of tested cases, though no single configuration dominated universally, suggesting dataset-specific optimization remains an open challenge.
Figures & tables
Encoder
Output vector ϕ(xi,xj)
ϕ2d(xi,xj)
(22,8,38,72)
ϕ3drel(xi,xj)
(22,8,38,72,−16,−64)
ϕabs(xi,xj)
(16,64)
ϕ3dabs(xi,xj)
(22,8,38,72,16,64)
Table 1: Example of the four pair encoders on two Titanic passengers with features Age (years) and Fare (US$).
Table 3: F1 score Comparison of different input combinations
Figure 1: Histogram of the F1 score difference of between a method and ϕ3drel 1 1 1 The notation for this figure is based on the original work with: ϕ1=ϕabs , ϕ2=ϕ3dabs and δ0=δ■
Baseline
Before Pairing (m=1)
After Pairing (m=1)
After Pairing (m=2)
Results (80 DS)
0.838
0.811
0.838
0.825
Time [min]
7
7
209
147
Table 4: Comparison of F1 and run time across Different Kernel Thinning Settings for 80 Datasets per experiment
Pairwise difference learning (PDL) has recently been introduced as a new meta-learning technique for regression. Instead of learning a mapping from instances to outcomes in the standard way, the key idea is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. This paper extends PDL toward the task of classification and proposes a meta-learning technique for inducing a PDL classifier by solving a suitably defined (binary) classification problem on a paired version of the original training data. We analyze the performance of the PDL classifier in a large-scale empirical study and find that it outperforms state-of-the-art methods in terms of prediction performance. Last but not least, we provide an easy-to-use and publicly available implementation of PDL in a Python package.
Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier
IDIADA Fahrzeugtechnik GmbH, Munich, Germany · Dr. Ing. h.c. F. Porsche AG Stuttgart, Germany · Ludwig-Maximilians-Universität Munich, Germany
Deep neural networks often fail at small-sample regression, collapsing to low-variance predictions that miss individual differences. Here, we introduce a pairwise comparison framework that reframes regression as aggregated relative rankings. This method improves data efficiency across model architectures and dataset sizes. On the UTKFace dataset (23k images), pairwise learning generates over one billion training comparisons without additional supervision. The paired ranking allows a lightweight ResNet10 trained from scratch on home hardware to achieve a 3.64-year MAE-surpassing previous models, including those with large-scale pretraining. For classification into categories, pairwise supervision consistently outperforms an otherwise identical 10-way crossentropy model on STL10 with 500-2000 labeled examples, improving accuracy by up to 5.2%. Overall, pairwise supervision enhances data efficiency and enables strong regression and classification performance when labeled data are limited, particularly in clinical settings.
Robert Bain, Marjan Rashidi, Elizabeth Chrastil
Department of Neurobiology & Behavior, University of California, Irvine, Irvine, CA 92697, USA · Department of Cognitive Sciences, University of California, Irvine, Irvine, CA 92697, USA
Instead of learning a mapping from instances to outcomes in the standard way, the key idea of pairwise difference learning (PDL) is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. We propose a version of PDL for the task of classification and propose a suitable meta-learning technique for this purpose. Moreover, we address the problem of uncertainty quantification, i.e., of representing and quantifying the uncertainty of a PDL classifier's predictions in a faithful manner. Building on recent work on the decomposition of total uncertainty into aleatoric and epistemic components, we propose different ways of extending corresponding measures to the setting of PDL, depending on how and to what extent uncertainty is represented by the underlying base learner. In large-scale empirical studies, we analyze both the predictive accuracy and the uncertainty awareness of our PDL classifier.
Mohamed Karim Belaid, Maximilian Rabus, Eyke Hüllermeier
IDIADA Fahrzeugtechnik GmbH, Munich, Germany. · Dr. Ing. h.c. F. Porsche AG, Stuttgart, Germany. · Institute of Informatics, University of Munich (LMU), Germany. +2