LOCUS: A Distribution-Free Loss-Quantile Score for Risk-Aware Predictions

Matheus Barreto; Mário de Castro; Thiago R. Ramos; Denis Valle; Rafael Izbicki

stat.MLMar 2, 2026

LOCUS: A Distribution-Free Loss-Quantile Score for Risk-Aware Predictions

Authors: Matheus Barreto, Mário de Castro, Thiago R. Ramos, Denis Valle, Rafael Izbicki

Organizations: Department of Statistics, Federal University of S˜ao Carlos, S˜ao Carlos, S˜ao Paulo, Brazil · Institute of Mathematics and Computer Science, University of S˜ao Paulo, S˜ao Carlos, S˜ao Paulo, Brazil · School of Forest, Fisheries, and Geomatics Sciences, University of Florida, Gainesville, Florida, United States of America

Abstract

Modern machine learning models can be accurate on average yet still make mistakes that dominate deployment cost. We introduce Locus, a distribution-free wrapper that produces a per-input loss-scale reliability score for a fixed prediction function. Rather than quantifying uncertainty about the label, Locus models the realized loss of the prediction function using any engine that outputs a predictive distribution for the loss given an input. A simple split-calibration step turns this function into a distribution-free interpretable score that is comparable across inputs and can be read as an upper loss level. The score is useful on its own for ranking, and it can optionally be thresholded to obtain a transparent flagging rule with distribution-free control of large-loss events. Experiments across 13 regression benchmarks show that Locus yields effective risk ranking and reduces large-loss frequency compared to standard heuristics.

Figures & tables

Figure 1: Loss-centric reliability versus variance. We fit two models to the same data: linear regression (left) and $k$ NN regression (right). (a) Data with the true mean $f(x)$ (black) and fitted prediction function $g(x)$ (red). (b) True conditional standard deviation $\sigma(x)$ (aleatoric proxy). (c) Realized absolute loss $Z=|g(X)-Y|$ (points) and our Locus score $U_{\alpha}(x)$ (curve), a calibrated estimate of the $(1-\alpha)$ loss quantile. (d) The induced accept/flag rule for this specific prediction function: accept if $U_{\alpha}(x)\leq\tau$ (blue) and flag otherwise (red). $U_{\alpha}$ is interpretable in loss units and captures failures in the predicted values (e.g., linear misfit in low-variance regions) that $\sigma(x)$ can miss.

Flagged as unreliable by Locus ?	Flagged as outlier by IF?	Outcomes (thousands of US$)			Locus (thousands of US$)		Other
		Estimated price $\hat{y}$	True price $y$	$Z=\|\hat{y}-y\|$	$U_{0.5}(x)$	$U_{0.1}(x)$	$\widehat{sd}(Y\mid X=x)$
yes	yes	2358.64	3567.00	1208.36	294.06	1276.52	104.75
yes	no	1648.52	2200.00	551.48	203.00	554.59	104.07
yes	yes	337.83	150.00	187.83	38.64	188.92	17.15
no	yes	217.85	215.00	2.85	17.94	67.93	76.95
no	yes	208.04	219.95	11.91	17.27	43.58	25.10

Table 1: Example home-price predictions. Six test instances from house price predictions; all monetary quantities are in thousands of US dollars. We report

\hat{y}=g(x)

y

, and

Z=|\hat{y}-y|

, alongside Locus bounds

U_{0.5}(x)

and

U_{0.1}(x)

(approx. conditional median and 90th percentile of the loss at each

x

). The Locus

\alpha=10\%

flag uses

\tau=\

70{,}000

(flag iff

U_{0.1}(x)>\tau

); IF denotes an Isolation Forest outlier flag;

\widehat{sd}(Y\mid X=x)

is a label-uncertainty baseline.

U_{\alpha}(x)$ is interpretable in dollars and can indicate high-loss cases missed by variance or OOD-style scores.

	Baselines (no Locus )		Locus-Tuned (loss-quantile score)
Dataset	IFlag (IF outlier)	VARNet (label var.)	Locus -BART	Locus -BART ( $\gamma$ -infl.)	Locus -MC Dropout	Locus -MC Dropout ( $\gamma$ -infl.)
airfoil	31.0 (27.4; 35.3)	27.5 (23.0; 30.7)	24.7 (19.6; 28.0)	23.1 (18.4; 28.2)	25.0 (18.6; 28.1)	24.3 (19.4; 27.5)
bike	30.1 (30.1; 30.1)	28.4 (26.1; 29.2)	20.1 (19.4; 21.6)	20.0 (19.1; 20.8)	22.6 (21.8; 23.7)	22.9 (21.7; 24.0)
concrete	33.5 (27.2; 38.6)	27.7 (23.0; 31.2)	25.3 (18.7; 29.3)	26.2 (20.5; 30.4)	27.7 (21.5; 31.1)	27.4 (22.7; 32.4)
cycle	28.9 (27.2; 30.0)	29.7 (28.5; 30.2)	27.5 (26.4; 28.6)	28.0 (26.6; 28.8)	28.7 (26.8; 30.4)	29.2 (27.9; 30.1)
electric	31.5 (30.1; 32.1)	29.4 (28.1; 30.7)	24.2 (22.3; 25.4)	24.0 (22.5; 26.2)	24.6 (22.3; 26.4)	24.6 (22.9; 25.8)

Table 2: Tuned conditional large-loss rate at matched acceptance. Conditional exceedance among accepted points,

\Prbb(Z>\tau\mid X\in A_{\lambda})

(%). For each dataset and method, the threshold

\lambda

is selected on the validation split to achieve an acceptance rate of

\approx 70\%

; test performance is then reported. Locus-Tuned methods use the Locus score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses; lower is better. Bold marks the lowest value in each row.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$n$	$p$
Airfoil	1503	5
Bike	10885	12
Concrete	1030	8
Cycle	9568	4
Homes	21613	17
Electric	10000	12

Appendix

Table 3: Dataset summary (sizes and feature counts).

Dataset	BART (%)	BART ( $\gamma$ ) (%)	MC Dropout (%)	MC Dropout( $\gamma$ ) (%)
airfoil	89.4 (80.4; 93.1)	90.7 (84.1; 95.4)	88.7 (81.1; 93.4)	91.1 (80.9; 94.8)
bike	90.6 (89.6; 91.4)	91.3 (90.0; 91.9)	91.1 (90.5; 92.2)	92.8 (92.0; 93.6)
concrete	90.8 (84.5; 94.7)	91.3 (84.4; 96.1)	89.3 (84.9; 95.2)	91.7 (83.0; 97.1)
cycle	90.3 (88.5; 91.6)	90.9 (89.4; 92.1)	90.0 (88.5; 91.8)	91.1 (89.3; 92.3)
electric	90.3 (88.8; 92.0)	90.5 (89.2; 92.4)	90.0 (88.1; 91.3)	91.3 (89.8; 92.7)
homes	90.5 (89.0; 91.3)	90.9 (90.0; 91.9)	90.3 (89.2; 91.1)	92.4 (91.1; 93.5)

Appendix

Table 4: Marginal coverage check for Locus . Reports the coverage probability,

\Prbb(Z<U_{\alpha}(x))

, which is expected to be approximately

1-\alpha

Dataset	BART (%)	BART ( $\gamma$ ) (%)	MC Dropout (%)	MC Dropout( $\gamma$ ) (%)
airfoil	6.3 (0.0; 26.2)	9.9 (0.7; 23.8)	2.6 (0.0; 13.1)	1.0 (0.0; 23.2)
bike	28.0 (25.3; 29.0)	26.5 (25.2; 28.2)	15.5 (12.5; 19.2)	13.8 (11.8; 16.8)
concrete	0.5 (0.0; 5.8)	0.0 (0.0; 1.9)	0.0 (0.0; 0.0)	0.0 (0.0; 1.0)
cycle	0.0 (0.0; 0.5)	0.0 (0.0; 0.1)	0.0 (0.0; 0.4)	0.0 (0.0; 0.0)
electric	1.8 (0.3; 3.2)	2.8 (1.0; 4.4)	0.7 (0.1; 2.5)	0.3 (0.0; 2.7)
homes	15.8 (12.1; 18.6)	16.1 (12.7; 18.2)	14.0 (9.8; 18.0)	11.7 (9.3; 14.6)

Appendix

Table 5: Acceptance rate for ( Locus ). Acceptance probability,

\Prbb(X\in A_{\lambda})

(%), over the test split. Locus uses the score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses.

Dataset	BART (%)	BART ( $\gamma$ ) (%)	MC Dropout (%)	MC Dropout( $\gamma$ ) (%)
airfoil	0.0 (0.0; 11.5)	0.0 (0.0; 18.7)	7.1 (0.0; 30.7)	6.9 (0.0; 15.2)
bike	1.6 (0.7; 2.4)	1.0 (0.4; 1.7)	2.2 (0.7; 4.4)	1.9 (0.7; 4.1)
concrete	0.0 (0.0; 65.0)	0.0 (0.0; 50.0)	100.0 (100.0; 100.0)	0.0 (0.0; 90.0)
cycle	33.3 (2.5; 86.7)	100.0 (10.0; 100.0)	0.0 (0.0; 0.0)	– (–; –)
electric	4.2 (0.0; 20.5)	4.8 (0.0; 15.9)	16.7 (0.0; 37.3)	0.0 (0.0; 31.2)
homes	5.0 (3.5; 7.0)	5.6 (3.3; 6.6)	4.9 (2.9; 6.9)	4.0 (2.0; 5.5)

Appendix

Table 6: Conditional large-loss rate. Conditional exceedance among accepted points,

\Prbb(Z>\tau\mid X\in A_{\lambda})

(%) over the test split. Locus uses the score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses; lower is better. Bold marks the lowest value in each row.

	Baselines (no Locus )		Locus-Tuned (loss-quantile score)
Dataset	IFlag	VARNet	Locus -BART	Locus -BART ( $\gamma$ )	Locus -MC	Locus -MC ( $\gamma$ )
airfoil	73.2 (63.8; 80.8)	82.8 (69.1; 95.9)	72.8 (65.1; 78.8)	71.2 (61.5; 78.9)	70.5 (61.2; 77.2)	71.5 (62.5; 77.2)
bike	73.0 (73.0; 73.0)	88.2 (74.8; 94.3)	72.2 (70.9; 74.6)	71.9 (70.5; 74.3)	73.3 (71.9; 75.1)	72.5 (71.0; 73.7)
concrete	72.3 (66.0; 80.1)	80.6 (73.8; 89.9)	72.3 (61.6; 83.2)	72.3 (61.0; 85.9)	72.8 (64.5; 86.1)	74.3 (64.1; 80.7)
cycle	72.4 (67.9; 76.2)	92.2 (74.8; 98.9)	73.8 (69.2; 78.6)	73.1 (68.4; 78.9)	75.0 (70.8; 81.9)	74.0 (68.8; 80.5)
electric	71.5 (67.5; 75.9)	79.0 (71.5; 83.9)	71.7 (68.6; 74.9)	71.5 (69.1; 75.4)	71.0 (68.2; 74.7)	71.4 (69.1; 75.1)

Appendix

Table 7: Tuned acceptance rate at matched target coverage ( Locus-Tuned ). Acceptance probability,

\Prbb(X\in A_{\lambda})

(%). For each dataset and method, the threshold

\lambda

is selected on the validation split to target an acceptance rate of

\approx 70\%

; test performance is then reported. Locus-Tuned methods use the Locus score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses.

	Baselines (no Locus )		Locus- $\alpha$ (loss-quantile score)
Dataset	IFlag	VARNet	Locus -BART	Locus -BART ( $\gamma$ )	Locus -MC	Locus -MC ( $\gamma$ )
airfoil	73.2 (63.8; 80.8)	82.8 (69.1; 95.9)	71.5 (64.5; 78.1)	71.5 (62.5; 79.2)	70.5 (60.3; 77.5)	71.5 (63.5; 82.3)
bike	73.0 (73.0; 73.0)	88.2 (74.8; 94.3)	71.4 (69.5; 73.3)	71.7 (70.3; 73.6)	72.3 (70.9; 74.0)	70.9 (69.2; 72.8)
concrete	72.3 (66.0; 80.1)	80.6 (73.8; 89.9)	71.8 (62.4; 83.7)	71.8 (62.5; 84.2)	73.8 (62.0; 83.6)	72.8 (64.5; 79.2)
cycle	72.4 (67.9; 76.2)	92.2 (74.8; 98.9)	71.6 (68.5; 75.1)	71.6 (67.4; 75.8)	72.6 (68.2; 79.1)	75.8 (68.8; 85.3)
electric	71.5 (67.5; 75.9)	79.0 (71.5; 83.9)	71.6 (68.2; 74.2)	70.8 (67.8; 73.4)	71.0 (67.7; 74.0)	71.1 (68.5; 75.2)

Appendix

Table 8: Tuned acceptance rate at matched target coverage ( Locus-

\alpha

). Acceptance probability,

\Prbb(X\in A_{\lambda})

(%). For each dataset and method, the threshold

\lambda

is selected on the validation split to target an acceptance rate of

\approx 70\%

; test performance is then reported. Locus-

\alpha

methods use the Locus score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses.

	Baselines (no Locus )		Locus- $\alpha$ (loss-quantile score)
Dataset	IFlag	VARNet	Locus -BART	Locus -BART ( $\gamma$ )	Locus -MC	Locus -MC ( $\gamma$ )
airfoil	31.0 (27.4; 35.3)	27.5 (23.0; 30.7)	23.3 (19.6; 26.7)	22.2 (18.4; 28.3)	24.5 (19.2; 29.3)	24.7 (19.1; 27.4)
bike	30.1 (30.1; 30.1)	28.4 (26.1; 29.2)	19.0 (18.3; 20.8)	19.4 (18.3; 20.6)	22.1 (21.3; 22.7)	21.8 (20.6; 23.1)
concrete	33.5 (27.2; 38.6)	27.7 (23.0; 31.2)	23.7 (19.0; 28.0)	25.2 (20.1; 29.1)	27.6 (20.0; 31.9)	27.4 (21.8; 32.9)
cycle	28.9 (27.2; 30.0)	29.7 (28.5; 30.2)	26.7 (25.2; 27.8)	27.1 (25.5; 28.2)	28.4 (26.7; 29.6)	28.9 (27.5; 30.1)
electric	31.5 (30.1; 32.1)	29.4 (28.1; 30.7)	24.1 (22.2; 25.4)	24.1 (22.4; 25.8)	24.7 (22.9; 26.0)	24.6 (23.2; 26.4)

Appendix

Table 9: Tuned conditional large-loss rate at matched acceptance ( Locus-

\alpha

). Conditional exceedance among accepted points,

\Prbb(Z>\tau\mid X\in A_{\lambda})

(%). For each dataset and method, the threshold

\lambda

is selected on the validation split to achieve an acceptance rate of

\approx 70\%

; test performance is then reported. Locus-

\alpha

methods use the Locus score

U_{\alpha}

with fixed calibration level

\alpha=10\%

. Entries are the median over 30 runs, with the 5th and 95th percentiles in parentheses; lower is better. Bold marks the lowest value in each row.

Uncertainty Calibration

Uncertainty calibration aligns model confidence and predictive intervals with observed correctness and outcome frequencies.

See all 264 papers

ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules
Mar 31, 2026Jonas Landsgesell, Pascal Knoll, Tizian WenzelTabular Regression Tabular Foundation Models
University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)
Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty
Mar 16, 2026Arthur Charpentier, Agathe Fernandes MachadoProbability Calibration Uncertainty Decomposition
UQAM, Montréal, Canada · Kyoto University, Japan
Gradient Boosted Risk Scores
May 4, 2026Costa Georgantas, Jonas RichiardiInterpretable ML Clinical Prediction
†Department of Radiology, Lausanne University Hospital and University of Lausanne, Switzerland

Mar 31, 2026J/K move · Enter open · S save

ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

Jonas Landsgesell, Pascal Knoll, Tizian Wenzel

University of Stuttgart (Stuttgart, Germany) · Ludwig Maximilian University of Munich (Munich, Germany) · Munich Center for Machine Learning (Munich, Germany)

Tabular Regression Tabular Foundation Models Proper Scoring Rules

Open paper AskPDF arXiv

Tabular foundation models such as TabPFN, TabICL, ExaOneTabular, and Nori already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics such as RMSE. This overlooks the distributional information these models are designed to provide. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including (weighted) CRPS, energy scores and interval scores - alongside standard point metrics. ScoringBench covers 101 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving

z

-score ranking approach. Our evaluation of various model types, including (fine-tuned) tabular foundation models, gradient-boosted trees, and MLPs, shows that model rankings shift depending on the chosen scoring rule. Models that perform well on point-estimate metrics can rank lower on probabilistic ones. These findings demonstrate that evaluation metric selection is a core modeling decision and a domain-specific requirement for applications where certain types of errors carry higher consequences.

Abstract

Uncertainty Calibration

Explore similar work

ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

Decomposing Probabilistic Scores: Reliability, Information Loss and Uncertainty

Gradient Boosted Risk Scores