Discrimination, Calibration, and Ranking: A Unified Matrix Framework

Ronaldo Cristiano Prati; Farangis Johari

cs.LGNov 1, 2026

Discrimination, Calibration, and Ranking: A Unified Matrix Framework

Authors: Ronaldo Cristiano Prati, Farangis Johari

Organizations: Centro de Matemática, Computação e Cognição Universidade Federal do ABC (UFABC) Santo André, SP, Brazil · Universidade Federal do ABC (UFABC) Santo André, SP, Brazil

Abstract

Discrimination, calibration, and ranking are the three pillars of classifier evaluation, yet no single algebraic language spans all three. A recent indicator-matrix account unifies discrimination measures (precision, recall, F1, Matthews correlation) across binary, multiclass, multilabel and related settings, using Boolean cell operations on indicator label matrices, generalised to soft labels by t-norms. Calibration measures (Brier score, log-loss) and ranking measures (nDCG, AUC) fall outside it. We close the gap with two further classes of cell operators: deviation operators phi(y - yhat), which give a Calibration Framework, and rank- and slice-dependent operators (positional gains for DCG, column concordance cells for AUC), which give a Ranking Framework. Both inherit the micro/macro/exemplar aggregation architecture. For calibration, the three Brier aggregations collapse to one scalar; the Murphy decomposition holds in matrix form (unbinned, binned and isotonic); and, within the Brier-type class-indexed family, a measure is skew-invariant if and only if it depends only on the class-conditional errors alpha_j and beta_j, the calibration analogues of recall and specificity. For ranking, raw DCG collapses across aggregations while nDCG does not; per-class nDCG and per-class AUC are related by a row-column duality; micro-AUC differs from pooled ROC-AUC; raw positional scores are never strictly proper, and nDCG and AUC need not be proper. The skew-invariant per-class building blocks share one class-conditional expectation template. Experiments on two multilabel benchmarks illustrate the distinctions. Many of the results are machine-checked in Lean 4; the proofs, the experiments and the aggmetrics.scores package are public: https://github.com/rcprati/discrimination-calibration-ranking

Figures & tables

Figure 1: Taxonomy of the three cell operator classes. Each panel shows the operator form, its dependency structure (illustrated by the highlighted cell in the

3\times 3

matrix icon), canonical evaluation measures, proper scoring rule status, skew-invariant basis, and micro = macro collapse behaviour. Framework 3 (Positional) splits into a row-dependent subfamily (nDCG, rank within a row) and a column-dependent subfamily (AUC, rank within a column), unified by the row–column duality theorem (Section 6 ).

T

denotes a triangular norm;

\phi

is a convex symmetric function with

\phi(0)=0

;

w(\cdot)

denotes a non-increasing discount weight for row nDCG, whereas the column AUC mean-rank construction admits the increasing rank cost

v(r)=r

. The cross under “strictly proper” in the Positional panel covers both subfamilies: raw positional scores are never strictly proper. For non-increasing discounts and pairwise distinct true marginals, truthful reporting maximises their expected raw score among competing reports with distinct entries (Proposition 68 ). Under the min-rank convention, allowing tied reports can invalidate weak properness; constant discounts are an exception, since every report then has the same expected raw score. Normalised nDCG and AUC need not be proper (Remark 69 , Proposition 84 ).

Term	Column axis ( $\Upsigma_{\mathrm{m}}$ )	Row axis ( $\Upsigma_{\mathrm{n}}$ )	Interpretation
UNC	$\bar{y}_{j}(1-\bar{y}_{j})$	$\bar{Y}_{i}(1-\bar{Y}_{i})$	Inherent difficulty: class prevalence (col) vs. example label density (row)
RES	Spread of the label frequency across the groups of equal predicted value in a column	Same, across the groups of equal predicted value in a row	Discrimination: separates pos. from neg. labels?
REL	Mean-sq. calibration error per class	Mean-sq. calibration error per example	Calibration: confidence matches frequency?

Table 1: The two Murphy decompositions induced by the column and row aggregation operators. Both partition the same per-cell Brier score, but condition on different axes. Column-axis UNC is the class-prevalence Bernoulli variance; row-axis UNC is the example label-density Bernoulli variance. The two are generally unequal (see Proposition 37 ).

Aggregation	Raw DCG	Normalisation	Normalised nDCG
$\Upsigma_{\mathrm{1}}$ (micro)	$\mathrm{DCG}^{\mathrm{micro}}$	$\overline{\mathrm{IDCG}}$	$\mathrm{nDCG}^{\mathrm{micro}}$
$\Upsigma_{\mathrm{m}}$ (class)	$\mathrm{DCG}_{j}^{\mathrm{class}}$	$\pi_{j}\,w(1)$	$\mathrm{nDCG}_{j}^{\mathrm{class}}$ (novel)
$\Upsigma_{\mathrm{n}}$ (exemplar)	$\mathrm{DCG}_{i}^{\mathrm{ex}}$	$\mathrm{IDCG}_{i}/m$	$\mathrm{nDCG}_{i}$ (standard)

Table 2: DCG family via the three aggregation operators. “Novel” indicates measures not previously expressed within the matrix framework. The standard nDCG of the IR literature ( Järvelin and Kekäläinen, 2002 ) corresponds to the exemplar axis.

Property	Discrimination	Calibration	Ranking
Operator dependency	Pointwise	Pointwise	Row (nDCG) / Col (AUC)
Collapses (raw cell sums / normalised ratios)	Yes / No	Yes / Yes	Yes / No
Strictly proper scoring rule	$\times$	Depends on loss (Brier: strict; MAE: not proper)	nDCG: not generally proper; AUC: non-strict and conditional
Skew-invariant basis	Recall, specificity	$(\alpha_{j},\beta_{j})$	$\mathrm{nDCG}_{j}^{\mathrm{class}}$ , $\mathrm{AUC}_{j}$
Common structure of basis	Class-conditional rates	Positive/negative conditional error means	Positive-class conditional rank quality
Hard-prediction limit	Precision, recall	Hamming loss (complement: label accuracy)	Precision@ $k$ (indicator weight)

Table 3: Structural properties of the three frameworks. For Ranking, nDCG and AUC are row-dependent and column-dependent subfamilies with raw aggregation collapse but generally different normalised summaries. Properness depends on the functional and its assumptions: normalised nDCG need not be proper, while AUC is never strictly proper and may fail truthful maximality under dependent labels. The skew-invariant bases use the appropriate conditioning strata, not only the positive examples.

Property	Discrimination	Calibration	Ranking
Pointwise in $(Y_{i,j},\hat{Y}_{i,j})$	✓	✓	$\times$
$D(y,y)=0$ for all $y$	$\times$	✓	n/a ∗
Strictly proper scoring rule	$\times$	✓ ‡	$\times$ †
Marginal consistency (Thm. 8 , Prati 2027 )	$T_{P}$ only	n/a	n/a
Micro = macro = exemplar (scalar)	$\times$	✓	raw ✓; norm. $\times$
Skew-invariant measures exist	✓	✓(via $(\alpha_{j},\beta_{j})$ )	✓(per-class) §

Table 4: Properties of the three cell operator classes. Operator types: Discrimination = coincidence (Boolean conjunction, generalised by

t

-norms); Calibration = deviation (

\phi(y-\hat{y})

); Ranking = positional (

Y\cdot w(\mathrm{rank})

). ∗ A positional operator is a gain, not a loss:

D=Y_{i,j}\,w(\mathrm{rank})

vanishes only when

Y_{i,j}=0

(irrelevant label) or when the weight does, and is largest,

D=w(1)\,Y_{i,j}

, when the label is ranked first. † Raw positional operators are proper among reports with distinct entries (the truthful report maximises the expected score) but never strictly proper: any order-preserving distortion of the probabilities attains the same score (Proposition 68 ). With ties ranked by the min-rank rule, raw positional operators need not be proper (non-constant discounts admit counterexamples, while a constant weight makes every report a maximiser); normalised nDCG is not proper in general (Remark 69 ). ‡ For the symmetric deviation operators

\phi(y-\hat{y})

with

\phi

even, convex, finite-valued and continuous on

[-1,1]

and differentiable on

(0,1)

, properness holds exactly when

\phi^{\prime}(t)/t

is symmetric about

\tfrac{1}{2}

; strictly positive weights additionally guarantee strict properness (Corollary 43 (c)). Examples include squared error and the nonquadratic loss of Appendix C ; log-loss satisfies the weight condition only in the extended-valued sense of Corollary 43 (b). Convexity alone is insufficient:

\phi(t)=|t|

(MAE) yields a non-proper operator whose minimiser is the conditional median rather than the conditional mean (Remark 45 ). § Per-class nDCG

{}_{j}^{\mathrm{class}}

and per-class AUC j are skew-invariant (Propositions 71 and 82 ); exemplar and micro summaries are not in general.

Figure 2: Experiment 1 (Emotions). (a) Scores across frameworks for five classifiers. (b) Rank heatmap (green = 1st, red = last). No two metrics produce the same classifier ordering.

Figure 3: Experiment 2 (Emotions). (a) Brier score aggregations (micro, macro, exemplar) are numerically identical, confirming the collapse theorem. (b) nDCG aggregations: macro-class nDCG is well separated from the other two (

\approx 0.72

against

\approx 0.88

–

0.90

) and micro and exemplar differ slightly, confirming non-collapse for normalised ranking measures.

Figure 4: Experiment 3 (Emotions). Varying the prevalence of the first label from its natural

0.28

up to

0.81

(by removing negatives). (a) Per-class

\mathrm{nDCG}_{j}^{\mathrm{class}}

is exactly constant in this subsampling design, whereas empirical per-class

\mathrm{AUC}_{j}

remains approximately stable, with finite-sample variation (left axis), while the raw mean rank

M_{0}

(right axis) drifts strongly;

\mathrm{AUC}_{0}

is its affine normalisation (Proposition 76 ). (b) Solid: micro-AUC and macro-BS under row subsampling, which couples the classes. Dashed: per-class-summary convention (micro-AUC by scaling the positives of class 0, macro-BS by the mixture formula, the other classes held fixed). Shaded bands:

\pm 1

s.d. over 10 subsamples.

Figure 5: Experiment 4 (Emotions, classifier LR). Predicted probabilities distorted as

\hat{P}\to\hat{P}^{\gamma}

, an order-preserving transformation for every

\gamma>0

. (a) macro-AUC and macro-nDCG class are exactly flat (spread

0.00

) across the entire sweep. (b) Macro-Brier score is minimised near

\gamma=1

(the untouched probabilities), consistent with a strictly proper scoring rule applied to nearly calibrated scores. (c) Macro-MAE is minimised at

\gamma=2.16

, well away from

\gamma=1

, consistent with Remark 45 : MAE is not a proper scoring rule, and here it favours the downward-shifted scores of the power map (

\hat{P}^{\gamma}<\hat{P}

for

\gamma>1

Figure 6: Experiment 5 (Yeast, 14 labels, 5 classifiers). Per-class binned Murphy decomposition $\widehat{\mathrm{BS}}_{j}=\mathrm{UNC}_{j}-\widehat{\mathrm{RES}}_{j}+\widehat{\mathrm{REL}}_{j}$ . Top row: $\widehat{\mathrm{REL}}_{j}$ (miscalibration, Reds) and $\widehat{\mathrm{RES}}_{j}$ (resolution, Greens) heatmaps. Bottom: $\mathrm{UNC}_{j}$ bar chart, identical across classifiers (data property only).

	Reliability (lower is better)		Resolution (higher is better)
Classifier	binned	isotonic	binned	isotonic
LR	0.0024	0.0053	0.0217	0.0255
RF	0.0039	0.0067	0.0277	0.0317
GBT	0.0040	0.0063	0.0265	0.0293
MLP (uncal.)	0.0056	0.0081	0.0266	0.0295
MLP (cal.)	0.0025	0.0056	0.0257	0.0297

Table 5: Experiment 5 (Yeast), binned (10 equal-width bins) and isotonic Murphy terms, averaged over the 14 labels. The isotonic terms satisfy

\mathrm{BS}_{j}=\mathrm{UNC}_{j}-\mathrm{RES}_{j}^{\mathrm{iso}}+\mathrm{REL}_{j}^{\mathrm{iso}}

for the original scores.

Figure 7: Experiment 6. (a) Mean

\alpha_{i}

(solid) and

\beta_{i}

(dashed) by label cardinality

k_{i}

on Yeast; only bins with

\geq 15

examples are shown. (b) Empirical CDFs of

\mathrm{BS}_{i}^{\mathrm{example}}

on Emotions; dotted verticals mark each classifier’s mean. The means are close (

0.131

–

0.142

) but the CDFs differ.

Figure 8: Experiment 7 (Yeast,

m=14

). (a) Scatter of label-mean

\mathrm{AUC}_{j}

vs.

\mathrm{nDCG}_{j}^{\mathrm{class}}

, annotated with label names; light dots show per-classifier spread. Most labels fall below the diagonal

y=x

; Class12 and Class13, the two most prevalent labels, are exceptions (

\mathrm{nDCG}_{j}^{\mathrm{class}}>\mathrm{AUC}_{j}

). (b) Dumbbell plot for RF: circle =

\mathrm{AUC}_{j}

(column-rank), diamond =

\mathrm{nDCG}_{j}^{\mathrm{class}}

(row-rank), sorted by mean

\mathrm{AUC}_{j}

; line length = row-rank deficit.

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 425 papers

May 5, 2026

The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality

The Brier score conflates two distinct properties of probabilistic predictions: reliability (calibration error) and resolution (discriminatory power). We introduce the Manokhin Probability Matrix, a BCG-style two-dimensional diagnostic framework that separates them. Classifiers are placed on a 2x2 grid by Spiegelhalter Z-statistic and AUC-ROC expected rank, then assigned to one of four archetypes: Eagle (good on both axes), Bull (strong discrimination, poor calibration), Sloth (well-calibrated, weak discriminator), and Mole (poor on both). Each archetype carries a distinct prescription. We populate the matrix from a large-scale empirical study spanning 21 classifiers, 5 post-hoc calibrators, and 30 real-world binary classification tasks from the TabArena-v0.1 suite. The assignment is unambiguous. CatBoost, TabICL, EBM, TabPFN, GBC, and Random Forest are Eagles. XGBoost, LightGBM, and HGB are Bulls; Venn-Abers calibration cuts log-loss by 6.5 to 12.6% on Bulls but degrades Eagles by 2.1%. SVM, LR, LDA, and the empirical base-rate predictor are Sloths. MLP, KNN, Naive Bayes, and ExtraTrees are Moles. A theoretical asymmetry follows: no order-preserving post-hoc calibrator can add discriminatory power (Proposition 1), so calibration is the fixable part and discrimination is the hard part. The practical rule is direct: do not optimise aggregate Brier score without first decomposing it; optimise discrimination first, then fix calibration post-hoc. Code and raw experimental data are available at https://github.com/valeman/classifier_calibration.

Valery Manokhin

Independent Researcher

Post-Hoc Calibration Probability Calibration Model Calibration Binary Classification Classification

Ask PDF

Jul 4, 2026

A Unified Algebraic Framework for Classification Performance Evaluation

We propose a unified algebraic framework for classification performance evaluation that encompasses binary, multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings within a single formalism. The foundation is a representation of actual and predicted labels as binary indicator matrices, combined with three aggregation operators -- global, column-wise, and row-wise -- that correspond exactly to micro, macro/weighted, and exemplar averaging. Any binary performance measure expressed in terms of true/positive/negative counts extends automatically to all settings by substituting these operators, generating multiclass and multilabel versions without measure-specific derivations. The framework further accommodates soft classifier outputs via argmax or thresholding, soft ground truth via triangular norms, ordinal classification via membership functions or cumulative encodings, and cost-sensitive evaluation via a cost matrix that subsumes MAE and MSE as special cases. We establish several theoretical results: micro-averaging equals denominator-weighted macro-averaging; the product

t

-norm is the unique one preserving the confusion-matrix partition; skew-invariant measures are characterised as functions of recall and specificity; and micro-precision, micro-recall, and micro-

F_1

are all equal to accuracy in multiclass settings. Empirical illustrations on synthetic and real data confirm the theoretical findings.

Ronaldo C. Prati

Universidade Federal do ABC (UFABC), Av. dos Estados, 5001, Santo André, SP, 09210-580, Brazil

Classification

Ask PDF

May 28, 2026

Calibrated Preference Learning: The Case of Label Ranking

Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.

Santo M. A. R. Thies, Viktor Bengs, Timo Kaufmann +2

Data Science and its Applications, Deutsches Forschungsinstitut für Künstliche Intelligenz (DFKI), Kaiserslautern, Germany · Munich Center for Machine Learning, Munich (MCML), Germany · Ludwig Maximilian Universität München, Munich, Germany

Probability Calibration Learning to Rank Preference Learning

Ask PDF

Abstract

Classification

Explore similar work

The Manokhin Probability Matrix: A Diagnostic Framework for Classifier Probability Quality

A Unified Algebraic Framework for Classification Performance Evaluation

Calibrated Preference Learning: The Case of Label Ranking