Classifier Pooling for Modern Ordinal Classification

Noam H. Rotenberg; Andreia V. Faria; Brian Caffo

cs.LGMar 18, 2026

Classifier Pooling for Modern Ordinal Classification

Authors: Noam H. Rotenberg, Andreia V. Faria, Brian Caffo

Organizations: Department of Biomedical Engineering, Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA. · Department of Radiology, School of Medicine, Johns Hopkins University, Baltimore, MD, USA. · 3*Department of Biostatistics, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, USA.

Abstract

Ordinal data is widely prevalent in clinical and other domains, yet there is a lack of both modern, machine-learning based methods and publicly available software to address it. In this paper, we present a model-agnostic method of ordinal classification, which can apply any non-ordinal classification method in an ordinal fashion. We also provide an open-source implementation of these algorithms, in the form of a Python package. We apply these models on multiple real-world datasets to show their performance across domains. We show that they often outperform non-ordinal classification methods, especially when the number of datapoints is relatively small or when there are many classes of outcomes. This work, including the developed software, facilitates the use of modern, more powerful machine learning algorithms to handle ordinal data.

Figures & tables

Figure 1: 1a : Fitting paradigm for thresholded ordinal classification ( Algorithm 1 ), paired with the difference prediction paradigm (fig. 1b ; Algorithm 2 ). 1c : Prediction paradigm of tree-based ordinal classification ( Algorithm 3 ). To find

P(Y=i)

, multiply all of the conditional probabilities above the node

Y=i

. Equivalence of this product to

P(Y=i)

is shown in the Appendix.

		Dataset 1: Fetal Health			Dataset 2: Car Quality			Dataset 3: Wine Quality			Dataset 4: RetinaMNIST radiomics			Dataset 5: MNIST radiomics (negative control)
Method	Method Subtype	AW	PC	AUC OVR	AW	PC	AUC OVR	AW	PC	AUC OVR	AW	PC	AUC OVR	AW	PC	AUC OVR
Logistic Regression	OVR	0.744	0.914	0.925	0.657	0.899	0.963	0.968	0.999	0.998	0.215	0.367	0.637	0.224	0.280	0.690
	Ordered logit model	0.698	0.911	0.947	0.607	0.909	0.945	0.908	0.999	0.929
	Difference	0.757	0.906	0.936	0.686	0.907	0.966	0.984	0.999	0.998	0.306	0.594	0.698	0.221	0.294	0.690
	Tree-based	0.763	0.912	0.935	0.662	0.913	0.964	0.984	0.999	0.997	0.320	0.604	0.708	0.191	0.325	0.648
Ordered probit model		0.702	0.914	0.947	0.586	0.910	0.945	0.908	0.999	0.930				0.124	0.126	0.528

Table 2: Results of Experiment 1 ; evaluation of classification methods, including difference and tree-based ordinal classification, across various datasets with ordinal outcome variables. Models that failed to train (defined as when the AW was less than chance) were marked by *. AW = accuracy, weighted by inverse class size; PC = polychoric correlation; OVR = one vs. rest; AUC OVR = multi-class area under the receiver operating characteristic curve, calculated with an OVR paradigm.

Method	Method Subtype	$\Delta$ AW	$\Delta$ PC	$\Delta$ AUC OVR
Logistic Regression	OVR	-0.358	-0.026	0.025
	Ordered logit model	-0.272	0.025	0.033
	Difference	-0.322	-0.010	0.027
	Tree-based	-0.302	0.015	0.028
Ordered probit model		-0.311	0.012	0.031
Gaussian Naïve Bayes	Multiclass	-0.082	0.074	0.029

Table 4: Results of Experiment 2 ; change in performance from 3 outcome classes on the full dataset to 7 outcome classes on the 50% dataset (i.e., performance on 7-class groupings on 50% of the train dataset minus performance on 3-class groupings on full train dataset).

\Delta

AW = change in accuracy, weighted by inverse class size;

\Delta

PC = change in polychoric correlation; OVR = one vs. rest;

\Delta

AUC OVR = change in multi-class area under the receiver operating characteristic curve, calculated with an OVR paradigm.

Best split index calculation method		Even Split			Best Classifier (via 4-fold validation)**			Last index			Middle index
	Best split index	4			0***			6			3
Method	Method Subtype	AW	PC	AUC OVR	AW	PC	AUC OVR	AW	PC	AUC OVR	AW	PC	AUC OVR
Logistic Regression	Difference	0.310	0.767	0.824	0.310	0.767	0.824	0.310	0.767	0.824	0.310	0.767	0.824
	Tree-based	0.268	0.764	0.813	0.307	0.746	0.815	0.269	0.782	0.816	0.285	0.747	0.814
Gaussian Naïve Bayes	Difference	0.346	0.689	0.731	0.346	0.689	0.731	0.346	0.689	0.731	0.346	0.689	0.731
	Tree-based	0.342	0.684	0.740	0.342	0.685	0.736	0.342	0.683	0.734	0.342	0.685	0.741

Table 5: Results of Experiment 3 ; comparison of methods for choosing the best split index hyperparamter on Dataset 6 (abalone rings). The underlined values are the maximum for each metric, for each classifier, for each method, if the maximum value is unique. AW = accuracy, weighted by inverse class size; PC = polychoric correlation; OVR = one vs. rest; AUC OVR = multi-class area under the receiver operating characteristic curve, calculated in an OVR paradigm. **Note that the results for this hyperparameter are not the same as in Experiment 2 because the variables were regrouped, as described in Methods: Evaluation . ***Note that the index is dependent on classifier performance, but in this case, the index was 0 for all classifiers.

Ordinal Regression

Ordinal regression predicts ordered categories while accounting for the fact that some mistakes are more severe than others.

See all 38 papers

ADABORD: a novel AdaBoost approach for ordinal classification
Jul 23, 2026Rafael Ayllón-Gavilán, Francisco José Martínez-Estudillo, David Guijo-Rubio +2Ensemble Learning Ordinal Regression
Department of Clinical-Epidemiological Research in Primary Care, IMIBIC, Avda. Menéndez Pidal S/N, Córdoba, 14004, Spain · Programa de doctorado en Computación Avanzada, Energía y Plasmas, Universidad de Córdoba, Campus de Rabanales, Ctra. N-IVa, Km. 396, Córdoba, 14071, Spain · Departmento de Ciencia de la Computación e Inteligencia Artificial, Universidad de Córdoba, Campus de Rabanales, Ctra. N-IVa, Km. 396, Córdoba, 14071. +2
Deep Neural Networks with Ordinal Loss for Medical Applications
Jun 24, 2026Tal Dvora, Rotem Haba, Gonen SingerCost-Sensitive Learning Ordinal Regression
Bar-Ilan University Ramat Gan, Israel
Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score
Jun 23, 2026Stefan Haas, Luca Killmaier, Alireza Javanmardi +1Conformal Prediction Ordinal Regression
Institute of Informatics, LMU Munich, Munich, Germany · BMW Group, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany +1

Jul 23, 2026J/K move · Enter open · S save

ADABORD: a novel AdaBoost approach for ordinal classification

Rafael Ayllón-Gavilán, Francisco José Martínez-Estudillo, David Guijo-Rubio, César Hervás-Martínez+1

Department of Clinical-Epidemiological Research in Primary Care, IMIBIC, Avda. Menéndez Pidal S/N, Córdoba, 14004, Spain · Programa de doctorado en Computación Avanzada, Energía y Plasmas, Universidad de Córdoba, Campus de Rabanales, Ctra. N-IVa, Km. 396, Córdoba, 14071, Spain · Departmento de Ciencia de la Computación e Inteligencia Artificial, Universidad de Córdoba, Campus de Rabanales, Ctra. N-IVa, Km. 396, Córdoba, 14071.+2

Ensemble Learning Ordinal Regression

Open paper AskPDF arXiv

Ordinal Classification (OC) deals with classification tasks where the classes follow a natural order. Despite the progress in OC, many existing approaches fail to fully leverage the ordinal information, treating the problem as nominal classification and thereby losing performance potential. In this work, ADABORD, an AdaBoost framework specifically designed for ordinal classification problems, is introduced. The ordinal nature of the classes is incorporated into two key components of the well-known AdaBoost algorithm: 1) the base estimator, where decision trees with the ordinal Gini splitting criterion are proposed; 2) the error function used to update sample weights at each stage and the weights of the classifier in the final ensemble model, given by the absolute ranked probability score, a measure that accounts for both the ordering and the distance between classes. ADABORD is extensively compared against seven state-of-the-art methods on the TOC-UCO repository, the largest benchmark collection for OC to date. The experimental results, supported by statistical analysis, show that ADABORD significantly outperforms competing methods, particularly on datasets with five or more classes, where the ordinal structure becomes more pronounced. Source code, along with all experimental protocols, is publicly available to ensure reproducibility and facilitate future research in OC.

Abstract

Ordinal Regression

Explore similar work

ADABORD: a novel AdaBoost approach for ordinal classification

Deep Neural Networks with Ordinal Loss for Medical Applications

Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score