Many high-stakes regression decisions ask three things of a model at once: a point estimate to act on, an interval that bounds the risk of acting, and the option to abstain when neither can be trusted. Deep learning provides these in isolation: prediction-interval methods return an interval, and at best a value estimate inside it, for every input, while selective-prediction methods learn when to abstain, but only for point predictions and with no statement of how far the truth may lie from the estimate. We introduce SelPI, which trains interval, value and abstention heads jointly under a single loss with an explicit acceptance-rate budget. A natural objection is that interval width already tells us when to abstain. We show that it does not: across 18 tabular benchmarks, width predicts large point errors well (AUROC 0.72) but is anti-correlated with interval misses (AUROC 0.38), so abstaining on wide intervals discards precisely the inputs the interval got right. SelPI therefore selects with two gates, one trained on the interval loss and one on the value loss, coupled by a Fr'echet penalty that ties their separate budgets to the joint acceptance rate they deliver. With the base predictor held fixed, the learned gate beats width thresholding on a proper interval score on all 18 tabular datasets, closing a median 87% of the gap to an oracle selector, and it also beats selection by deep-ensemble disagreement on every dataset. The advantage grows with the heteroscedasticity of the task, and we report the settings in which it disappears.
Figures & tables
Figure 1: SelPI in one picture. Left: one backbone feeds the interval and value heads, two selection gates, and auxiliary heads that are trained on all inputs and dropped at inference; a prediction is released only if both gates accept. Right: the cases a width threshold confuses (bar: predicted interval; dot: point estimate y^ ; star: true y ).
Question (origin)
Comparator
Metric
Q1
Does learning to abstain damage the interval model? ( PI field )
IPIV, all inputs answered, IPIV’s protocol and datasets
PICP , MPIW , RMSE at ϕ=1
Q2
Is it a competitive selective regressor? ( selective-prediction field )
SelectiveNet-regression, its own grid and settings
selective RMSE / MAE vs. ϕ
Q3
Given an interval model, does a learned gate beat the model’s own uncertainty? ( new )
same predictor, ranked by gate / by width U−L / by an oracle; a deep ensemble ranked by member disagreement
selective interval score and RMSE vs. ϕ ; selection efficiency
Table 1: Evaluation design. SelPI has no like-for-like competitor, so each question is asked on the home ground of the field it comes from, with that field’s comparator and metric.
IPIV
SelPI 2G RMSE cut vs. IPIV
sel. RMSE at ϕ=0.7
SelPI 2G sel. PICP at ϕ≈0.7
SelPI 2G sel. ISα at ϕ≈0.7
Dataset
ϕ=1
ϕ≈0.55
ϕ≈0.75
Sel.Net
SelPI 2G
Δ
gate
width
gate
width
yacht
1.04
63.3%
56.1%
1.62
0.35
+78.3%
0.975
0.893
0.10
0.45
boston
2.64
10.5%
7.5%
2.42
2.39
+1.3%
0.934
0.869
1.72
2.16
energy
1.63
59.4%
50.5%
1.17
0.82
+30.2%
0.945
0.855
0.45
1.41
concrete
6.17
33.5%
22.2%
4.66
4.59
+1.5%
0.850
0.809
1.66
2.03
wine
0.66
16.2%
10.1%
0.56
0.56
0.0%
0.812
0.701
5.39
9.29
Table 2: Per-dataset results , 18 tabular datasets, 1−α=0.95 , SelPI 2G median search trial throughout. Left (Q2): RMSE of IPIV answering every query; RMSE reduction of SelPI 2G at two realized acceptance levels; SelectiveNet-regression vs. SelPI 2G at ϕ=0.7 ( Δ>0 favors SelPI). Right (Q3): learned gate vs. width threshold on the same predictor at the realized acceptance nearest 0.7 ; lower ISα is better.
Figure 2: Q3: selective RMSE vs. acceptance rate on a fixed predictor (single-gate SelPI 1G , one model per acceptance rate) for Concrete, NYC Taxi and bone age (illustrative only, § 5 ). Curves: learned gate, width threshold, oracle, and deep-ensemble point disagreement on its own five-model predictor; bands: ±1 s.d. over runs; lower is better.
AUROC
selection efficiency
Signal
cov. ↑
val. ↑
sel. RMSE
ISα
Width U−L
0.378
0.717
0 (reference)
gPI
0.790
0.633
0.59
0.87
gval
0.710
0.647
Pt. disagr.
0.584
0.606
−0.34
−0.62
Bd. disagr.
0.591
0.581
−0.37
−0.55
Table 3: What each signal sees, and how well it selects. AUROC for coverage failure (cov.) and value failure (val.; captured, but error above the captured-set median), and median SE against each selector’s own predictor’s width threshold and oracle ( <0 : worse than width). The SE on the gPI / gval rows is that of the Q3 gate (SelPI 1G ); Pt./Bd. disagr. is the deep-ensemble baseline on its own five-model predictor. Medians over 18 tabular datasets.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
n
d
Test/split
Epochs
Reps
Split protocol
(A) UCI PI suite — the standard benchmark of QD and IPIV
Yacht
308
6
31
2000
5
90/10 random
Boston
506
13
51
300
5
90/10 random
Energy
768
8
77
1200
5
90/10 random
Concrete
1,030
8
103
800
5
90/10 random
Wine
1,599
11
160
1000
5
90/10 random
Appendix
Table 4: The 20 datasets. n and d are read directly from the data files as loaded by our pipeline. “Test/split” is the held-out count per repetition under the protocol in the last column. Epochs are the per-dataset budgets inherited from IPIV’s published configuration for the UCI sets and sized by timing probes for the larger ones. Every tabular dataset uses the same 10-point acceptance grid ϕ∈{0.30,0.40,0.50,0.60,0.70,0.75,0.80,0.85,0.90,0.95} , one model trained per point.
Dataset
Hidden
λPI
σinit
Learning rate
Decay
Boston
50
15.0
0.100
5×10−3
0.990
Concrete
50
15.0
0.200
5×10−3
0.990
Energy
50
17.0
0.350
5×10−3
0.990
Kin8nm
50
12.0
0.436
5×10−3
0.990
Naval
50
4.0
0.100
5×10−3
0.990
Power-plant
50
15.7
0.257
5×10−3
0.990
Appendix
Table 5: Per-dataset settings. Everything except the learning rate is inherited unchanged from IPIV’s published configuration for that dataset. The learning rate is capped at 5×10−3 because IPIV’s larger values destabilize the multi-term selective objective (§ 4.4 ); where the cap binds, the decay rate is raised to compensate.
Target ϕ
Realized ϕ
PICPsel
MPIWsel
MAE sel
bone age, single gate — IPIV at ϕ=1 : 0.915, 2.189, 20.71
0.30
0.000
—
—
—
0.40
0.070
0.929
1.589
10.901
0.50
0.075
0.933
1.729
16.505
0.60
0.190
1.000
2.063
14.099
0.70
0.225
0.889
1.889
15.371
Appendix
Table 6: Full image-regression grids , every trained point rather than the three buckets of Table 7 . Top : bone age, single gate, one run over 200 test images — note the severe drift between target and realized acceptance rate, which is why Table 7 is keyed by the realized rate. Middle : IMDB-WIKI, single gate then the symmetric dual-gate grid, 5-fold CV at 90 epochs. Bottom : SelectiveNet-regression on IMDB-WIKI over its own published coverage grid.
Figure 3: All four comparisons on the two image tasks , at 1−α=0.95 : single-gate SelPI (orange), dual-gate SelPI (blue: every Optuna trial faintly, best-per-bucket as the line), plain IPIV at full acceptance (gray dotted), and SelectiveNet-regression (red). Left, bone age : both SelPI variants beat IPIV everywhere and the dual gate beats SelectiveNet at every comparable acceptance rate. Right, IMDB-WIKI : the dual gate beats IPIV over the 0.5 – 0.8 range but SelectiveNet wins clearly and consistently , by 0.7 – 1.6 years of MAE. IMDB-WIKI is the largest dataset in the study and this is the paper’s clearest negative result (§ 6 ).
Figure 4: Risk–coverage curves for the two image tasks : selective PICP , MPIW and MAE against target acceptance rate, with plain IPIV at full acceptance as the hollow diamond. Note the contrast in calibration transfer: IMDB-WIKI’s realized acceptance rate tracks its target almost exactly ( 0.70→0.700 , 0.95→0.959 ), whereas bone age drifts severely (target 0.90→ realized 0.68 ; target 0.30→ realized 0.00 ). The cause is sample size — thresholds calibrated on validation transfer cleanly to IMDB-WIKI’s ∼34 k-sample folds and poorly to bone age’s 200-image test set. Bone age’s “target coverage” axis should therefore be read as a training knob, not as a description of what fraction of test inputs was accepted.
RSNA bone age
IMDB-WIKI
Realized acceptance rate
[0.5,0.6)
[0.7,0.8)
[0.8,0.9)
[0.5,0.6)
[0.7,0.8)
[0.8,0.9)
IPIV ( ϕ=1 , no selection)
20.71
7.68
SelectiveNet-regression
14.96
15.94
17.01
—
6.29
6.76
SelPI 1G single gate
14.53
—
18.03
7.83
7.80
8.42
SelPI 2G dual gate (median trial)
15.07
16.23
17.83
7.92
8.16
8.21
SelPI 2G dual gate (best trial)
13.41
15.11
15.50
7.36
7.59
7.94
Appendix
Table 7: Image regression , selective MAE at 1−α=0.95 (bone age in months, IMDB-WIKI in years), grouped by realized acceptance rate. Plain IPIV at full acceptance is 20.71 (bone age) and 7.68 (IMDB-WIKI). For the dual gate we give the best and the median trial in each bucket, because its budgets are tuned by search while SelectiveNet is run once at its published settings — the median is the fair comparison. Bone age : every SelPI variant beats IPIV by a wide margin, and the dual gate’s best trial beats SelectiveNet in all three buckets. IMDB-WIKI : the dual gate beats IPIV in the two lower buckets, but SelectiveNet wins decisively wherever the two overlap , by 1.3–1.5 years. Rows are keyed by realized rather than target acceptance rate, which matters most for bone age, whose drift is severe (target 0.90 gives realized 0.68 ) because a threshold calibrated on validation transfers poorly to a 200-image test set; IMDB-WIKI has no such problem and is the more trustworthy of the two.
Dataset
[0.5,0.6)
[0.6,0.7)
[0.7,0.8)
[0.8,0.9)
yacht
63.3%
61.6%
56.1%
36.5%
boston
10.5%
8.5%
7.5%
4.2%
energy
59.4%
51.0%
50.5%
42.3%
concrete
33.5%
25.5%
22.2%
16.6%
wine
16.2%
14.8%
10.1%
8.1%
kin8nm
11.0%
7.1%
1.5%
-6.0%
Appendix
Table 8: Dual gate vs. plain IPIV by acceptance bucket , median search trial, reduction in selective RMSE; buckets are the realized joint acceptance rate. The gain decays as acceptance rises on every dataset, as it must: as ϕ→1 the accepted set converges on the full test set and there is nothing left to select.
IPIV
SelPI 1G selective RMSE at ϕ=
Cut at
Dataset
( ϕ=1 )
0.3
0.5
0.7
0.9
ϕ=0.5
yacht
1.04
0.45
0.44
0.55
1.07
57.7%
boston
2.64
2.23
2.18
2.63
2.55
17.5%
energy
1.63
0.61
0.91
0.84
1.10
44.4%
concrete
6.17
3.81
4.31
5.03
5.60
30.1%
wine
0.66
0.47
0.52
0.54
0.60
21.2%
Appendix
Table 9: What abstention buys. Single-gate SelPI 1G , one model trained per acceptance rate (not a searched trial population). Selective RMSE at four acceptance rates, against plain IPIV answering every query; 18 tabular datasets, 1−α=0.95 . The last column is the reduction at ϕ=0.5 . Naval’s full-acceptance error is 0.001 , so its percentage is a small-denominator artifact rather than a meaningful loss.
Dataset
SelectiveNet
SelPI 2G best
SelPI 2G med.
Best
Median
yacht
1.62
0.27
0.35
83.1%
78.3%
boston
2.42
2.20
2.39
9.2%
1.3%
energy
1.17
0.63
0.82
46.2%
30.2%
concrete
4.66
4.24
4.59
9.1%
1.5%
wine
0.56
0.53
0.56
5.6%
0.0%
kin8nm
0.07
0.08
0.08
-9.0%
-15.5%
Appendix
Table 10: Dual gate vs. SelectiveNet-regression , selective RMSE at 70% acceptance. “Best” is the strongest configuration the search found, “median” the typical one; the median column is the fair comparison, because SelectiveNet is run once at its published settings while the dual gate is tuned (§ 5 ). Positive percentages favor SelPI. Naval and kin8nm are the two clear losses, and both are near-deterministic tasks with little uncertainty for an interval method to characterize.
Figure 5: Gate specialization, all 18 tabular datasets. For each dataset, the two bars show how over-represented each failure type is among the inputs that only gPI rejects and the inputs that only gval rejects, relative to that dataset’s own base rate (dashed line at 1). The signature of successful specialization is a tall green bar on the left pair and a tall orange bar on the right pair. gval shows it on 17 of 18; gPI on a clear majority, with the exceptions discussed in § 7 .
rejected by gPI only
rejected by gval only
Dataset
n
cov. lift
val. lift
n
val. lift
cov. lift
yacht
6
3.91
0.42
21
1.69
0.22
boston
16
1.20
1.10
13
0.78
1.11
energy
24
2.38
0.42
77
1.51
0.43
concrete
3
1.70
1.66
116
1.29
0.88
wine
15
2.81
0.51
253
1.40
0.78
Appendix
Table 11: Each gate’s unique rejections are enriched for the failure mode it was trained on. For inputs rejected by only one gate, how over-represented each failure type is relative to that dataset’s base rate; 1.0 means no enrichment. gval ’s unique rejections are enriched for value failure on 17 of 18 datasets; gPI ’s are enriched for coverage failure on 13 of the 17 where it has any unique rejections at all (on naval it has none, its rejection set being a subset of gval ’s). The value gate is therefore the more consistently load-bearing of the two, which the aggregate results do not reveal.
Figure 6: The two-budget frontier , Concrete. Each dot is one dual-gate configuration, colored by the joint acceptance rate it actually realized; the orange line is the single-gate model traced over its acceptance grid. Down and to the right is better (lower selective error, higher selective interval coverage). The dual-gate cloud sits below and to the right of the single-gate curve at matched acceptance — the trade-off a single scalar gate cannot express. Per-dataset versions for all 18 tabular datasets are in Figure 13 .
Figure 7: When joint selection helps. Per-dataset selection efficiency of the learned gate against (left) the full-acceptance miscoverage rate, (middle) a heteroscedasticity statistic, (right) target excess kurtosis. Each point is one of 20 datasets (18 for the left panel, where the two image tasks lack a full-acceptance miscoverage figure); Spearman ρ and its p -value are in each panel title, with a least-squares line for orientation only. Only the heteroscedasticity relationship is distinguishable from noise at this sample size.
Variant
Med.
W/B
Range (%)
no auxiliary heads
+2.0%
14/4
−37 , +41
frozen IPIV, post-hoc gate
+5.6%
15/3
−20 , +54
n for Sg
+0.2%
9/9
−12 , +39
Appendix
Table 13: Ablations : change in selective RMSE at ϕ=0.5 vs. the full recipe (positive: ablation is worse); 18 tabular datasets. W/B: datasets worse/better.
Figure 8: Ablations on Yacht , the dataset where the effects are largest: end-to-end SelPI against a gate trained on a frozen IPIV model, and against removing the auxiliary head. Selective PICP , MPIW and RMSE against target acceptance rate; shaded bands are ±1 s.d. over 5 runs. End-to-end training dominates on both interval width and selective error across the whole grid. Equivalent figures exist for all 18 tabular datasets.
Figure 9: Selection efficiency per dataset (single-gate SelPI 1G ): the share of the gap between the same-predictor width threshold ( SE=0 ) and the oracle ( SE=1 ) that the learned gate closes, on selective RMSE and on the selective interval score. The gate beats the width threshold on every dataset on both metrics, and by more on the interval score on all 20. The two image tasks are illustrative (§ 5 ); SE is scale-free, so their real-unit widths do not affect comparability.
Figure 10: Risk–coverage curves : selective PICP , MPIW and RMSE against target acceptance rate, with plain IPIV at full acceptance as the hollow diamond. Bands are ±1 s.d. over runs.
Figure 11: Selector comparison : the learned gate against IPIV + width threshold and against an oracle ranking by realized error. The persistent oracle gap is the headroom that remains on every dataset.
Figure 12: Dual gate vs. SelectiveNet-regression : point accuracy, then the interval reliability and sharpness SelectiveNet cannot produce at all. Shading between best and median trial is the tuning advantage of § 5 .
Figure 13: Two-budget frontier : every dual-gate configuration, colored by realized joint acceptance rate, against the single-gate curve. Down and to the right is better.
Figure 14: Ablation: end-to-end vs. frozen-gate training , with the no-auxiliary-head variant overlaid. Frozen-gate is worse on 15 of 18 datasets (Table 13 ). This overlay also contains the no-auxiliary-head comparison shown on its own in Figure 15 below.
Figure 15: Ablation: with vs. without the auxiliary head , isolated into its own panel (also visible as part of the overlay in Figure 14 ).
Figure 16: Ablation: Sg vs. n penalty normalization. The curves track each other closely on most datasets, which is the visual form of the null result in Table 13 .
RMSE
MPIW
Dataset
DE
QD
QD+
IPIV
repro
Ens.
DE
QD
QD+
IPIV
repro
Ens.
Boston
2.87
3.39
6.12
3.13
2.64
2.70
0.87
1.15
4.98
1.09
0.88
1.09
Concrete
5.21
5.88
11.87
5.43
6.17
5.79
1.01
1.08
3.43
1.02
0.86
1.08
Energy
1.68
2.28
7.05
1.65
1.63
1.70
0.49
0.45
1.52
0.42
0.35
0.42
Kin8nm
0.08
0.08
0.19
0.07
0.08
0.08
1.14
1.18
2.64
1.10
0.88
1.10
Naval
0.00
0.00
0.01
0.00
0.00
0.00
0.31
0.27
2.85
0.24
0.21
0.27
Appendix
Table 14: Full-acceptance ( ϕ=1 ) results as published in the IPIV paper, with our own reproduction of IPIV alongside (the repro columns — this is plain IPIV retrained in our environment, not SelPI). QD, QD+ and DE are that paper’s own baselines. Ens. is our own M=5 -model IPIV ensemble (§ 5 ), the base predictor behind the deep-ensemble selector of § 6 . Published figures use 20 splits and 5-model ensembles; ours use 5 splits (and, for Ens. , 5 members per split), so small differences are expected and the comparison is one of fidelity, not of ranking. RMSE/MAE are on the original target scale; MPIW is in standardized units.
SE, selective RMSE
SE, interval score
AUROC, point disagr.
AUROC, bound disagr.
Dataset
gate
point
bound
gate
point
bound
cov.
val.
cov.
val.
yacht
0.833
−0.961
−1.062
0.868
–
–
0.761
0.735
0.840
0.747
boston
0.456
0.108
−0.001
0.766
0.225
0.170
0.603
0.575
0.509
0.572
energy
0.732
−1.456
−1.471
0.951
−1.060
−0.552
0.681
0.795
0.674
0.837
concrete
0.572
−0.331
−0.269
0.841
−0.403
−0.235
0.605
0.519
0.598
0.527
wine
0.423
−0.088
−0.158
0.795
0.050
0.047
0.548
0.577
0.583
0.494
Appendix
Table 15: Per-dataset selection efficiency (SE) on selective RMSE and on the interval score for the learned gate (SelPI 1G ) and for the two deep-ensemble disagreement selectors, and AUROC of each disagreement signal for coverage failure (cov.) and value failure (val.). Each SE is computed against its own predictor’s width threshold and oracle; “–” marks an undefined SE (width and oracle AURC coincide).
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
Interval prediction aims to achieve a target coverage level while producing intervals that are as short as possible. Many conformal regression pipelines first predict an uncertainty surrogate and then convert it into an interval through calibration or selection. This separation supports coverage calibration, but post hoc rules largely determine the final interval and do not fully use the learned output distribution. We observe that the resulting intervals have inherently hierarchical geometry: an interval can be recursively refined into nested subintervals, and binary trees naturally represent this structure. We formulate this hierarchy as next-interval prediction and propose PATH, which learns how probability mass flows from each interval to its next nested subintervals. PATH predicts a base leaf distribution and uses an autoregressive decoder to refine branch probabilities. Matching the distribution to the interval hierarchy aligns learning with extraction: PATH accumulates probability over adjacent output intervals and returns the shortest contiguous range reaching a selected mass. We compare PATH with 24 baselines for interval prediction on PATHBench, comprising 56 OpenML regression datasets. PATH substantially shortens the resulting intervals, achieving the lowest mean normalized length, 0.1473, while maintaining mean coverage of 0.9144. These results establish hierarchical output modeling as an effective approach for compact interval prediction on tabular data. Code is publicly available at https://github.com/pxcai/PATH.
Pengxiang Cai, Wanchen Lian, Chenyang Liu +4
The Hong Kong University of Science and Technology (Guangzhou) · Ant Group
Deep learning methods have achieved state-of-the-art in time series forecasting, yet their accuracy varies considerably across samples, as some instances remain inherently difficult to predict. Reject option mechanisms, which allow models to abstain from high-risk predictions, are well established in classification and regression but underexplored in forecasting. Existing abstention strategies typically rely on proxies, such as the width of the prediction interval or learned confidence scores derived from forecasts. However, these approaches are inherently tied to the training domain, limiting their ability to generalize. We propose a selective forecasting framework that addresses this limitation by modeling the empirical percentile of forecasting errors, that is, a scale-invariant statistic, based on structural characteristics extracted from recent lags via metalearning. By decoupling the rejection decision from the forecast itself and grounding it in domain-agnostic features, the framework enables effective abstention transfer across heterogeneous time series. Experiments in both in-domain and transfer learning settings show that rejecting samples predicted as challenging consistently improves forecasting accuracy across coverage levels.
A classical question in statistics is which observable quantities to condition on when drawing inferences about unobservable targets. For conformal prediction in nonparametric uncertainty quantification, standard marginal validity offers limited resolution at the prediction values on which decisions are based, and fully conditional guarantees with respect to the covariates are provably unattainable. We address this gap by introducing a prediction-based conditioning framework that we refer to as Prediction-Interval-Conditional Prediction Intervals (PICPIs). Formally, a PICPI is an interval I satisfying a self-consistency condition: E[Y∣p(X)∈I]∈I, for predictive model p, contextual covariate X, and outcome Y. Thus, an interval simultaneously defines a stratum of prediction values and certifies that the mean outcome in that stratum lies in the same interval. This self-consistency condition yields data-adaptive strata without altering the original prediction. Such intervals can be constructed using practical algorithms. Under regularity of the prediction distribution, the constructed intervals cover all but an arbitrarily small fraction of prediction values and have widths that decrease at rate n−1/3, up to logarithmic factors and the prediction error. Moreover, identifying these locally calibrated intervals can, in turn, inform downstream decision-making. We derive inference procedures for PICPIs in probabilistic prediction and multi-class classification, accompanied by theoretical guarantees. Empirical results are provided that compare PICPIs with existing interval-based baselines.
Xuelin Yang, Baihe Huang, Yilong Hou +2
University of California, Berkeley · Stanford University · Inria Paris