Benchmarking uncertainty quantification for engineering regression under graded distribution shift

Ahmad Lawal; Rahim Pasbanigoloojeh; Asmau Lawal Bilbis

cs.LGOct 26, 2026

Benchmarking uncertainty quantification for engineering regression under graded distribution shift

Authors: Ahmad Lawal, Rahim Pasbanigoloojeh, Asmau Lawal Bilbis

Organizations: School of Computer Science and Informatics, De Montfort University, Leicester, UK

Abstract

Machine learning models in engineering are often used beyond their training conditions, where reliable uncertainty quantification (UQ) matters most. Existing comparisons of regression UQ methods rely on random splits, which assess calibration only in distribution. We introduce a three-rung shift ladder (in distribution, soft shift and hard shift) built along physically meaningful shift variables, and evaluate 12 methods from five families on seven public engineering datasets: five extrapolation settings, an unseen turbofan fault mode and a winter air-quality sensor deployment. All tuning uses source data only. In distribution, conformal methods cover at least 0.946 of test points on every dataset. Under hard shift, mean coverage falls to 0.40-0.75 across the 11 methods that always return an interval, and no systematic difference in method ranks is detected (Friedman p = 0.63). Weighted conformalised quantile regression stays near nominal on the points it answers, but abstains on 85% of hard-shift points on average and covers only 0.67 of the few Concrete points it answers. Source-validation gains predict improvement in distribution and under soft shift, but not under hard shift (Spearman's rho = 0.04). Engineering evaluations should split calibration data by deployment unit, report subgroup coverage and sharpness, and treat abstention as an explicit operational outcome.

Figures & tables

Fig. 1: The shift ladder on CCPP (seed 0). Soft shift:

P(\text{source}\mid v)=\sigma(-1.5\,z(v))

. Every dataset uses the same construction on its own shift variable (Table 1 ).

Dataset	Rows	Feat.	Target	Shift variable $v$	Hard shift: source $\to$ target	Target rows	Groups for $\mathrm{PICP}_{\min}$	Deployment scenario
Concrete	1,030	8	Compressive strength (MPa)	Curing age (log days)	age $\leq$ 28 d $\to$ age $>$ 28 d	281	4 age quantile bands	Predicting long-term strength from early-age tests
Energy	768	8	Heating load (kWh/m 2 )	Glazing area ratio	glazing $\leq$ 25% $\to$ 40%	240	4 relative-compactness bands	Assessing a more glazed design than any simulated
CCPP	9,568	4	Net electrical output (MW)	Ambient temperature (°C)	coolest 75% of hours $\to$ hottest 25%	2,388	5 relative-humidity bands	Plant output during a heatwave
Yacht	308	6	Residuary resistance per unit weight	Froude number	Fr $\leq$ 0.35 $\to$ Fr $>$ 0.35	88	3 Froude bands	Hull resistance at higher speed than tank-tested
Naval	11,934	14	Gas-turbine (GT) compressor decay coefficient	Ship speed (knots)	$\leq$ 21 kn $\to$ 24–27 kn	2,652	3 speed bands	Condition monitoring at top-speed regimes
C-MAPSS	45,351	15	RUL, capped at 125 cycles	Fault mode	FD001 (HPC fault) $\to$ FD003 (HPC + fan faults)	24,720	5 RUL bands (0–25 … 100–125)	Prognostics under an unseen fault mode

Table 1: Datasets, shift definitions and the grouping used for worst-group coverage.

Method	Family	Uncertainty represented	Interval	Fixed settings; search space (default first)
GPR	Bayesian	Epistemic + homoscedastic aleatoric	$\mu\pm 1.96\sigma$	Type-II maximum likelihood (GPy); exact up to 2,000 rows, else sparse with 200 $k$ -means inducing points. Automatic relevance determination (ARD) kernel $\in$ {radial basis function (RBF), Matérn-5/2, Matérn-3/2}
BNN (VI)	Bayesian	Epistemic (weights) + heteroscedastic aleatoric	$\mu\pm 1.96\sigma$ , moment-matched over 100 samples	Mean-field Gaussian posterior, Bayes-by-backprop, up to 600 epochs, patience 40. Width $\in$ {64, 32, 128}; prior sd $\in$ {1, 0.1, 0.5}; learning rate $\in$ { $10^{-3}$ , $3\cdot 10^{-3}$ }
MC Dropout	Bayesian (approx.)	Epistemic + heteroscedastic aleatoric	As BNN	100 stochastic passes, patience 30. Dropout $\in$ {0.1, 0.05, 0.2, 0.3}; width $\in$ {64, 128}; weight decay $\in$ { $10^{-5}$ , $10^{-4}$ }; learning rate $\in$ { $10^{-3}$ , $3\cdot 10^{-3}$ }
Deep Ensemble	Ensemble	Member disagreement + heteroscedastic aleatoric	Gaussian mixture, moment-matched	5 members with Gaussian negative log-likelihood heads, patience 30. Width $\in$ {64, 32, 128}; learning rate $\in$ { $10^{-3}$ , $3\cdot 10^{-3}$ }; weight decay $\in$ {0, $10^{-4}$ }
NGBoost	Distributional	Heteroscedastic aleatoric	$\mu\pm 1.96\sigma$	Normal distribution, up to 1,000 trees, early stopping after 50 rounds. Learning rate $\in$ {0.03, 0.01, 0.1}; tree depth $\in$ {3, 2, 4, 5}; minibatch fraction $\in$ {1, 0.5}
Quantile RF	Quantile	Conditional quantiles (aleatoric)	Empirical 2.5% and 97.5% quantiles	300 trees. Minimum leaf $\in$ {5, 1, 10, 20, 40}; features per split $\in$ {all, half, $\sqrt{d}$ }

Table 2: Methods, the uncertainty each represents and their settings. The last column gives the search space; the first value listed is the default, used in the fixed-setting run. Conformal methods inherit their base model’s tuned configuration (Section 4.3 ).

		PICP				PINAW	NRMSE		Winkler
Method	Family	IID	Soft	Min, soft	Hard	Hard	IID	Hard	rank, hard
GPR	Bayesian	0.93	0.87	0.79	0.70	0.56	0.044	0.209	5.7
BNN (VI)	Bayesian	0.96	0.90	0.81	0.54	0.29	0.061	0.303	7.0
MC Dropout	Bayesian	0.96	0.93	0.85	0.65	0.31	0.052	0.229	4.9
Deep Ensemble	Ensemble	0.96	0.91	0.81	0.60	0.41	0.051	0.362	6.0
NGBoost	Distributional	0.88	0.81	0.72	0.40	0.15	0.050	0.185	7.9

Table 3: Cross-dataset summary (tuned run; mean over 7 datasets of 5-seed means).

\mathrm{PICP}_{\min}

: worst-subgroup coverage. NRMSE: root-mean-square error divided by the target range; Weighted CQR shares CQR’s point model. Winkler rank: mean rank among the 11 non-selective methods (1 = best). a On answered points, over the four datasets where Weighted CQR answers any.

Fig. 2: Mean coverage by rung, sorted by hard-shift coverage. Weighted CQR: coverage on answered points.

Fig. 3: Coverage per method and dataset: (a) IID, (b) soft shift, (c) hard shift. Mean over 5 seeds; red = under-coverage, blue = over-coverage. ‘ab.’: Weighted CQR abstention rate; its PICP is on answered points.

Fig. 4: Gap between mean and worst-subgroup coverage under soft shift. For Concrete, Yacht and Naval the subgroups are bands of the shift variable. Violet: the two shift-responsive methods. Weighted CQR: coverage on answered points.

Fig. 5: (a) NRMSE by rung. (b) IID-to-hard coverage drop against NRMSE inflation; one point per method and dataset (77, mean over 5 seeds). Weighted CQR omitted.

Fig. 6: Air Quality deployment (seed 0, tuned settings, hard rung; training before 1 December 2004). (a) Hourly reference CO and two methods’ intervals over one winter week. (b) Monthly coverage: dotted = held-out calibration-period hours, solid = deployment hours. April 2005 covers 1–4 April only.

Fig. 7: Hard-shift predictions on Energy (seed 0, tuned settings, trained on glazing

\leq 0.25

). Partial dependence: each training building is assigned every glazing value, and curves and dots average over buildings. Labels give values at 40% glazing.

Fig. 8: GP interval width and coverage under hard shift (mean over 5 seeds). Arrows: fixed RBF kernel

\to

kernel chosen by source validation. CGPR: Conformal GPR.

Fig. 9: Width–error Spearman correlation against coverage under hard shift (tuned run).

x

: median over datasets (

\rho

is undefined on Naval for constant-width methods);

y

: mean coverage, as in Table 3 .

	IID			Soft shift			Hard shift
Dataset	ESS	Abstain	PICP	ESS	Abstain	PICP	ESS	Abstain	PICP
Concrete	73	1%	0.95	38	26%	0.95	60	97%	0.67
Energy	64	0%	0.98	12	54%	0.94	47	100%	–
CCPP	593	0%	0.95	172	0%	0.97	46	95%	0.93
Yacht	19	31%	0.99	7	76%	0.90	10	100%	–
Naval	594	0%	0.95	130	0%	0.94	86	100%	–

Table 4: Weighted CQR per dataset and rung (tuned run, mean over 5 seeds). ESS is the effective calibration sample size; PICP is on the points it answers; “–” means it abstains on every point.

Fig. 10: (a) Weighted CQR abstention against effective sample size; one point per dataset, seed and rung (140 test sets, tuned run). (b) Abstention with exact versus estimated weights on the soft rung, mean over 5 seeds.

	PICP		$\mathrm{PICP}_{\min}$		PINAW
Method	Engine	Row	Engine	Row	Engine	Row	$p$
Split CP	0.96	0.90	0.90	0.78	0.66	0.50	0.002
CQR	0.95	0.93	0.85	0.81	0.54	0.48	0.016
Conformal GPR	0.96	0.91	0.91	0.80	0.74	0.52	0.004
Weighted CQR	0.95	0.94	0.86	0.81	0.54	0.49	0.027

Table 5: C-MAPSS, IID rung: conformal methods with engine-level versus row-level calibration (5 seeds

\times

2 ladder modes).

\mathrm{PICP}_{\min}

: worst remaining-useful-life band.

p

: paired Wilcoxon test on PICP. Fig. S2 shows all three rungs.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 663 papers

Audited Conformal Prediction for Classification under Unknown Distribution Shift
Jun 12, 2026Yanfei Zhou, Rizal Fathony, Nam H. Nguyen +1Distribution Shift Robustness Uncertainty Quantification
Department of Data Sciences and Operations University of Southern California Los Angeles, CA 90089 · AI Foundations, Capital One New York, NY 10017 · Department of Data Sciences and Operations Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90089
A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification
Aug 30, 2026Kehan Long, Yiqi Zhao, Pol Mestres +3Distribution Shift Robustness Uncertainty Quantification
Contextual Robotics Institute, University of California San Diego, La Jolla, CA, USA · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles, CA, USA · Department of Mechanical and Civil Engineering, California Institute of Technology, Pasadena, CA, USA +1
On the QUEST for Uncertainty Quantification via Highest Density Regions
Jun 17, 2026Sam Goring, Tom Kuipers, Nicola Paoletti +1Uncertainty Quantification Aleatoric Uncertainty
King’s College London · Northeastern University London

Jun 12, 2026J/K move · Enter open · S save

Audited Conformal Prediction for Classification under Unknown Distribution Shift

Yanfei Zhou, Rizal Fathony, Nam H. Nguyen, Matteo Sesia

Department of Data Sciences and Operations University of Southern California Los Angeles, CA 90089 · AI Foundations, Capital One New York, NY 10017 · Department of Data Sciences and Operations Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA 90089

Distribution Shift Robustness Uncertainty Quantification Group-Conditional Conformal Prediction Conformal Prediction

Open paper AskPDF arXiv

We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.

Abstract

Uncertainty Quantification

Explore similar work

Audited Conformal Prediction for Classification under Unknown Distribution Shift

A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification

On the QUEST for Uncertainty Quantification via Highest Density Regions