Pairwise Difference Learning for Classification

Instead of learning a mapping from instances to outcomes in the standard way, the key idea of pairwise difference learning (PDL) is to learn a function that takes two instances as input and predicts the difference between the respective outcomes. Given a function of this kind, predictions for a query instance are derived from every training example and then averaged. We propose a version of PDL for the task of classification and propose a suitable meta-learning technique for this purpose. Moreover, we address the problem of uncertainty quantification, i.e., of representing and quantifying the uncertainty of a PDL classifier's predictions in a faithful manner. Building on recent work on the decomposition of total uncertainty into aleatoric and epistemic components, we propose different ways of extending corresponding measures to the setting of PDL, depending on how and to what extent uncertainty is represented by the underlying base learner. In large-scale empirical studies, we analyze both the predictive accuracy and the uncertainty awareness of our PDL classifier.

Pairwise Difference Learning for Classification (PDC) enhances supervised learning in low-data regimes by training a binary similarity classifier on pairs of instances rather than individual samples, effectively inflating the training set from $n$ to $O(n^2)$ examples. However, the PDC framework leaves critical implementation choices underspecified, particularly regarding input representation and pair selection strategies. This work systematically investigates these design decisions through extensive empirical evaluation. We compare four feature encoders (paired features, triplet with relative difference, absolute difference, and a novel triplet with absolute difference) alongside four pair selection strategies ranging from exhaustive to kernel-thinning-based schemes. Experiments across nearly 200 benchmark classification datasets using Decision Tree, Multilayer Perceptron, Gaussian Naive Bayes, and Support Vector Classifier as base models reveal that the proposed triplet with absolute difference encoder ($φ_{\mathrm{3dabs}}$) achieves the highest average F1 scores. More notably, restricting training to upper-triangle pairs reduces computational time by approximately 40% with negligible accuracy loss. Kernel thinning-based selection provided no accuracy benefits while substantially increasing computational overhead. Overall, PDC configurations outperformed base models in 94% of tested cases, though no single configuration dominated universally, suggesting dataset-specific optimization remains an open challenge.

Even when it is difficult to assign absolute class labels to individual instances, relational information may still be available, such as whether two instances belong to the same class or which instance is more likely to belong to the positive class. In this study, we refer to these two types of information as Similarity/Dissimilarity (SD) labels and Pairwise Comparison (Pcomp) labels, respectively, and consider binary classification that uses both types of relational information from the same instance pairs. SD learning uses the distinction between similar and dissimilar pairs but does not use the ordering within each pair, whereas Pcomp learning uses the ordering within each pair but does not distinguish between similar and dissimilar pairs. We therefore propose SD-Pcomp learning, whose objective function simultaneously preserves the structures of both SD learning and Pcomp learning. The proposed objective function admits two decompositions: one consists of an SD estimator plus a term that represents ordering information from Pcomp labels, and the other consists of a Pcomp estimator plus a term that represents pair-type information from SD labels. These decompositions clarify how the complementary information provided by SD and Pcomp labels is integrated into the proposed objective function. Experiments on eight datasets compare the proposed method with SD learning, Pcomp learning, and a method that takes a convex combination of their objective functions. We evaluate the effect of using both types of relational information on classification performance in terms of classification accuracy and AUC.