CAIRO: Decoupling Order from Scale in Regression

Harri Vanhems; Yue Zhao; Peng Shi; Archer Y. Yang

stat.MEFeb 16, 2026

CAIRO: Decoupling Order from Scale in Regression

Authors: Harri Vanhems, Yue Zhao, Peng Shi, Archer Y. Yang

Organizations: McGill University and Mila · University of York · University of Wisconsin

Abstract

Standard regression methods typically optimize a single pointwise objective, such as mean squared error, which conflates the learning of ordering with the learning of scale. This coupling renders models vulnerable to outliers and heavy-tailed noise. We propose CAIRO (Calibrate After Initial Rank Ordering), a framework that decouples regression into two distinct stages. In the first stage, we learn a scoring function by minimizing a scale-invariant ranking loss; in the second, we recover the target scale via isotonic regression. We theoretically characterize a class of "Optimal-in-Rank-Order" objectives -- including variants of RankNet and Gini covariance -- and prove that they recover the ordering of the true conditional mean under mild assumptions. We further show that subsequent monotone calibration recovers the true regression function at the population level and mathematically guarantees that finite-sample predictions are strictly auto-calibrated. Empirically, CAIRO combines the representation learning of neural networks with the robustness of rank-based statistics. It matches the performance of state-of-the-art tree ensembles on tabular benchmarks and significantly outperforms standard regression objectives in regimes with heavy-tailed or heteroskedastic noise.

Figures & tables

Figure 1: CAIRO-RankNet across Data Regimes. Top Row (Normal Regime): Stage 1 (a) learns a linear ranking, and Stage 2 (b) learns a linear calibration map, resulting in a standard regression fit (c). Bottom Row (Heteroskedastic and Heavy Tail Regime): In the presence of heavy-tailed noise, Stage 1 (d) learns a non-linear ranking to preserve order. Stage 2 (e) learns a non-parametric step function to correct this warp. The final model (f) successfully recovers the underlying signal (blue line) despite the large variance in the raw data (gray cloud).

	Model	Spearman’s $\rho$	Kendall’s $\tau$	RMSE
Scenario	Normal ( $\sigma=1.0$ )
	CAIRO–RankNet	$0.594\pm 0.074$	$0.435\pm 0.059$	$1.093\pm 0.026$
	CAIRO–RankNet-GiniW	$0.588\pm 0.074$	$0.431\pm 0.056$	$1.095\pm 0.025$
	CAIRO–GiniNet-SoftRank	$0.654\pm 0.051$	$\mathbf{0.484\pm 0.041}$	$1.031\pm 0.010$
	NN-MSE	$\mathbf{0.667\pm 0.045}$	$0.481\pm 0.037$	$\mathbf{1.004\pm 0.008}$
	Random Forest	$0.642\pm 0.045$	$0.459\pm 0.037$	$1.039\pm 0.020$

Table 1: Synthetic data performance across noise models: mean (95% CI) over

5

repetitions. Best value per scenario/metric is in bold.

Dataset	No. of data points ( $n$ )	No. of predictors ( $p$ )	No. of unique response values	Source
AUTO	392	7	127	UCI
CCRI	1994	102	98	UCI
COMP	209	6	116	UCI
CSLU	103	7	90	UCI
ABALONE	4177	7	29	UCI

Table 2: Characteristics of the real datasets used in our study.

AUTO (Auto MPG)
Model	Spearman’s $\rho$	Kendall’s $\tau$	RMSE
CAIRO –GiniNet-SoftRank	$0.870\pm 0.022$	$0.719\pm 0.022$	$4.244\pm 0.294$
CAIRO –RankNet	$0.933\pm 0.011$	$0.795\pm 0.020$	$3.188\pm 0.204$
CAIRO –RankNet-GiniW	$0.923\pm 0.025$	$0.787\pm 0.027$	$3.405\pm 0.455$
NN-MSE	$0.050\pm 0.578$	$0.026\pm 0.434$	$12.321\pm 4.540$
Random Forest	$\mathbf{0.937\pm 0.015}$	$\mathbf{0.798\pm 0.017}$	$\mathbf{2.994\pm 0.239}$

Table 3: Real data performance on five UCI regression tasks. Entries are mean (95% CI) over five random 70/30 train–test splits. Best value per dataset/metric is in bold.

Robust Regression

Robust regression develops methods that predict reliably despite outliers, heavy-tailed noise, corrupted data, or shifts in the data distribution.

See all 28 papers

Jul 29, 2026

An analysis of binary isotonic regression: degrees of freedom and implications for calibration

Isotonic regression is a canonical tool for estimating monotone functions and calibrating probabilistic predictors. We provide a fully sharp finite-sample characterization of its worst-case degrees of freedom on binary samples. Specifically, we identify the binary sequences that maximize the number of distinct fitted values produced by isotonic regression. We develop a sharp bound on the degrees of freedom with a leading term of

\frac{3}{(4π^2)^{1/3}} n^{2/3}

using analytic number theory, improving on previous bounds. We then apply this result to calibration. Calibration is a central requirement for probabilistic prediction, and isotonic regression is a widely used post-processing method for improving calibration. Building on deterministic degrees-of-freedom bounds, we derive, to our knowledge, the first nontrivial distribution-free guarantee on the Expected Calibration Error (ECE) of isotonic regression. This ECE bound is fully model-free and distribution-free, only assuming

Y \in \{0,1\}

Raphael Rossellini, Rina Foygel Barber, Zhimei Ren +1

Department of Statistics, University of Chicago · Department of Statistics and Data Science, University of Pennsylvania · Department of Statistics, University of Michigan

Probability Calibration Model Calibration Statistical Learning Theory Isotonic Regression

Ask PDF

Feb 21, 2026

CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization

Ad ranking systems must simultaneously optimize multiple objectives including click-through rate (CTR), conversion rate (CVR), revenue, and user experience metrics. However, production systems face critical challenges: score scale inconsistency across traffic segments undermines threshold transferability, and position bias in click logs causes offline-online metric discrepancies. We propose CaliCausalRank, a unified framework that integrates training-time scale calibration, constraint-based multi-objective optimization, and robust counterfactual utility estimation. Our approach treats score calibration as a first-class training objective rather than post-hoc processing, employs Lagrangian relaxation for constraint satisfaction, and utilizes variance-reduced counterfactual estimators for reliable offline evaluation. Experiments on the Criteo and Avazu datasets demonstrate that CaliCausalRank achieves 1.1% relative AUC improvement, 31.6% calibration error reduction, and 3.2% utility gain compared to the best baseline (PairRank) while maintaining consistent performance across different traffic segments.

Xikai Yang, Sebastian Sun, Yilin Li +3

Columbia University, New York, USA · University of Wisconsin-Madison, Madison, USA · Carnegie Mellon University, Pittsburgh, USA +3

Learning to Rank Multi-Objective Optimization CTR Prediction

Ask PDF

Jul 9, 2026

Contrastive Order Learning: A General Framework for Ordinal Regression

We propose contrastive order learning (ConOrd), a contrastive learning framework for ordinal regression that integrates the strengths of contrastive learning and order learning. While contrastive learning effectively leverages all samples in a batch, it typically ignores the inherent ordering among rank labels. Conversely, order learning explicitly models label ordinality but often relies on local, margin-based comparisons, limiting its ability to capture global ordinal structure. ConOrd addresses these limitations by introducing a contrastive order loss with soft affinity and disparity weights based on rank differences, enabling fine-grained modeling of ordinal relationships across all sample pairs within a batch. Extensive experiments on a range of ordinal regression tasks, including facial age estimation, blind image quality assessment, and blind video quality assessment, demonstrate that ConOrd consistently achieves state-of-the-art performance and generalizes well across diverse ordinal regression scenarios. The source code is available at https://github.com/cwlee00/ConOrd.

Chaewon Lee, BeomJun Shim, Kwang Pyo Choi +1

School of Electrical Engineering, Korea University, Seoul, Korea · Samsung Electronics, Seoul, Korea

Contrastive Learning Ordinal Regression

Ask PDF

Abstract

Robust Regression

Explore similar work

An analysis of binary isotonic regression: degrees of freedom and implications for calibration

CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization

Contrastive Order Learning: A General Framework for Ordinal Regression