The Optimal Ranking Tradeoffs Between Two Ranking Scores in Two-Class Classification

Sébastien Piérard; Adrien Deliège; Marc Van Droogenbroeck

cs.PFOct 16, 2026

The Optimal Ranking Tradeoffs Between Two Ranking Scores in Two-Class Classification

Authors: Sébastien Piérard, Adrien Deliège, Marc Van Droogenbroeck

Organizations: Montefiore Institute, University of Liège, Liège, Belgium

Abstract

The accuracy, true negative rate (negative class recall), true positive rate (positive class recall), negative predictive value (negative class precision), positive predictive value (positive class precision), class-specific intersections over unions, class-specific F-scores, Cohen's kappa, and balanced accuracy are scores that one intuitively wants to maximize when developing machine learning methods. However, they lead to different rankings of such methods. So, how can we establish a ranking that is an optimal tradeoff between any pair of these scores? It turns out that all these scores are special cases of ranking scores: a continuum of scores parameterized by the relative importance values assigned to the true negative, false positive, false negative, and true positive outcomes. Since all these scores are known to induce meaningful rankings, we seek the optimal tradeoffs among them. We consider the tradeoff between two scores to be optimal when it minimizes the Fréchet variance regarding the Kendall distance (also known as the bubble-sort distance) between rankings. For any given set of classifiers to rank, the optimal tradeoff lies at the center of the shortest (geodesic) path on the manifold of all rankings induced by the ranking scores. However, in the parameter space of the ranking scores, it is rarely achieved by averaging the importance values corresponding to the initial scores. In this paper, we present an exact analytical solution for determining a ranking score that is an optimal tradeoff for any given pair of initial scores, in the common case where the class priors are the same for all compared performances. We also experimentally investigate what happens when performance follows some distributions with fixed priors.

Figures & tables

score	classic formula	$I_{tn}$	$I_{fp}$	$I_{fn}$	$I_{tp}$	$a^{(\pi)}(I)$	$b^{(\pi)}(I)$
$TNR=Re^{-}$	$P(Y=\hat{Y}\|Y=c_{-})$	$1$	$1$	$0$	$0$	$0$	$0$
$TPR=Re^{+}$	$P(Y=\hat{Y}\|Y=c_{+})$	$0$	$0$	$1$	$1$	$1$	$1$
$NPV=Pr^{-}$	$P(Y=\hat{Y}\|\hat{Y}=c_{-})$	$1$	$0$	$1$	$0$	$0$	$1$
$PPV=Pr^{+}$	$P(Y=\hat{Y}\|\hat{Y}=c_{+})$	$0$	$1$	$0$	$1$	$1$	$0$
$SNPV$	$\frac{TNR}{TNR+FNR}=\frac{NPV\pi_{+}}{NPV(\pi_{+}-\pi_{-})+\pi_{-}}$	$\pi_{+}$	$0$	$\pi_{-}$	$0$	$0$	$1$
$SPPV$	$\frac{TPR}{FPR+TPR}=\frac{PPV\pi_{-}}{PPV(\pi_{-}-\pi_{+})+\pi_{+}}$	$0$	$\pi_{+}$	$0$	$\pi_{-}$	$1$	$0$

Table 1: Some ranking scores, when the class priors

(\pi_{-},\pi_{+})

are fixed, with corresponding importance values, and the coordinates of the induced performance ordering in the

(a^{(\pi)},b^{(\pi)})

space.

Figure 1: The tetrahedron of ranking scores, or the simplex of importance values in two-class classification. Be careful not to confuse this with the tetrahedron of performances or confusion simplex ( Chicco et al., 2021 ; Chicco & Jurman, 2023 ; Lovell et al., 2023 ; Brzezinski et al., 2017 ; Brzezinski et al., 2020 ; Brzezinski et al., 2018 ) . In this one, each point corresponds to a score, not a performance.

Panel of Figure 2 — Figure 2: Optimality $\mathcal{O}$ obtained with each of the three Pythagorean means ( $\mu_{A}$ : arithmetic, $\mu_{G}$ : geometric, $\mu_{H}$ : harmonic) and with our method, for seven pairs of scores and three sets of $1000$ performances corresponding to $\pi_{+}=0.01$ . Our method always yields an optimal result ( $\mathcal{O}=1$ , in blue), but arbitrarily chosen means are only approximately optimal in a few specific cases. In fact, the optimality value is often barely above 0.5 (the worst, in red). The meaningful rankings ( i.e. , the tradeoffs satisfying the axioms of performance-based ranking ( Piérard et al., 2025c ) ) are in bold.

Panel of Figure 3 — Figure 3: Case studies with large performances sets randomly drawn from reference distributions with fixed priors ( $\pi_{+}=0.2$ ). The colored markers indicate, in the $(a^{(\pi)},b^{(\pi)})$ space, the performance orderings that are optimal tradeoffs for the pairs $PPV\cdot TPR$ ( $\blacktriangleright$ ), $NPV\cdot TNR$ ( $\blacktriangleleft$ ), $NPV\cdot PPV$ ( $\blacksquare$ ), $TNR\cdot TPR$ ( $\filleddiamond$ ), $IoU^{-}\cdot IoU^{+}$ ( ✖ ), $PPV\cdot TNR$ ( $\blacktriangledown$ ), and $NPV\cdot TPR$ ( $\blacktriangle$ ).

Panel of Figure 4 — Figure 4: Case studies with 4 real-world performance sets. For each of them, the performances are shown in the ROC space, and the optimal tradeoffs for 7 pairs of scores are shown in $(a^{(\pi)},b^{(\pi)})$ .

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 410 papers

Jun 24, 2026

Scoring Is Not Enough: Addressing Gaps in Utility-fairness Trade-offs for Ranking

Scoring functions are used to represent the relevance of individual documents. In modern information retrieval or recommendation systems, they are often learned from data and play a pivotal role in ranking sets of documents or items in a way that maximizes utility to a query or user. With the recent interest in algorithmic fairness, the success of scoring has naturally led to methods that learn scores that simultaneously trade off fairness and utility. In this work, we show that in stark contrast with utility-centric objectives, scoring is sub-optimal in achieving all utility-fairness trade-offs. We establish this with a series of counter-examples with a generic fairness formulation. We show that the issue persists whether we have a deterministic scoring function or a randomized one, or whether we measure fairness at the scope of a single query or across multiple queries. On the positive side, we empirically demonstrate that semi-greedy post-processing has the potential to achieve much better trade-offs, often approaching the ideal of exhaustive post-processing in a tractable way.

Shubham Singh, Ian A. Kash, Mesrob I. Ohannessian

Department of Computer Science University of Illinois Chicago Chicago, IL 60607 · Department of Electrical and Computer Engineering University of Illinois Chicago Chicago, IL 60607

Learning to Rank Algorithmic Fairness

Ask PDF

May 22, 2025

Accuracy vs. Accuracy: Computational Tradeoffs Between Classification Rates and Utility

We revisit the foundations of fairness and its interplay with utility and efficiency in settings where the training data contain richer labels, such as individual types, rankings, or risk estimates, rather than just binary outcomes. In this context, we propose algorithms that achieve stronger notions of evidence-based fairness than are possible in standard supervised learning. Our methods support classification and ranking techniques that preserve accurate subpopulation classification rates, as suggested by the underlying data distributions, across a broad class of classification rules and downstream applications. Furthermore, our predictors enable loss minimization, whether aimed at maximizing utility or in the service of fair treatment. Complementing our algorithmic contributions, we present impossibility results demonstrating that simultaneously achieving accurate classification rates and optimal loss minimization is, in some cases, computationally infeasible. Unlike prior impossibility results, our notions are not inherently in conflict and are simultaneously satisfied by the Bayes-optimal predictor. Furthermore, we show that each notion can be satisfied individually via efficient learning. Our separation thus stems from the computational hardness of learning a sufficiently good approximation of the Bayes-optimal predictor. These computational impossibilities present a choice between two natural and attainable notions of accuracy that could both be motivated by fairness.

Noga Amit, Omer Reingold, Guy N. Rothblum

UC Berkeley · Stanford · Apple

Algorithmic Fairness Statistical Learning Theory

Ask PDF

Sep 20, 2026

Density-Ratio Rescoring for Imbalanced Classification Using Raking Duals and Classifier Scores

Density-Ratio Rescoring (DRR) augments a classifier trained at the original class prior with a survey-raking dual score. Raking reweights the majority sample to match minority feature moments within a tolerance. DRR marginally standardizes the dual and base scores and combines them with a fixed weight of one half, using the fitted dual directly for prediction without resampling or refitting the base classifier. Under exact population matching and a correctly specified log-linear tilt model, the dual equals the log density ratio up to an additive constant. A class-separation analysis characterizes the signal strength and correlation conditions under which fusion improves separation under common within-class covariance. On 24 tabular benchmarks, evaluated over 30 trials and five base learners, DRR at the D=128 random-feature setting improves average precision over the standardized base on every dataset, with a mean gain of 0.034. It exceeds the shared-dual raking-and-relabeling resampler on 22 of 24 datasets, with a mean gain of

0.092

, and on all eight one-versus-rest tasks of a shared gene-expression cohort. These results demonstrate the effectiveness of using raking duals as reusable scores for improving rare-class ranking while retaining classifiers trained at the original prior.

Dongha Kim, Seunghwan Park

School of Mathematics, Statistics and Data Science, Sungshin Women’s University, 2, Bomun-ro 34da-gil, Seongbuk-gu, Seoul, 02844, Republic of Korea · Department of Information Statistics, Kangwon National University, 1, Kangwondaehak-gil, Chuncheon-si, Gangwon-do, 24341, Republic of Korea

Class-Imbalanced Learning Density Ratio Estimation Binary Classification Classification

Ask PDF

Abstract

Classification

Explore similar work

Scoring Is Not Enough: Addressing Gaps in Utility-fairness Trade-offs for Ranking

Accuracy vs. Accuracy: Computational Tradeoffs Between Classification Rates and Utility

Density-Ratio Rescoring for Imbalanced Classification Using Raking Duals and Classifier Scores