Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting

Abhinaba Basu

cs.LGMar 9, 2026

Cross-Domain Uncertainty Quantification for Selective Prediction: A Comprehensive Bound Ablation with Transfer-Informed Betting

Authors: Abhinaba Basu

Abstract

We present a comprehensive ablation of nine finite-sample bound families for selective prediction with risk control, combining concentration inequalities (Hoeffding, Empirical Bernstein, Clopper-Pearson, Wasserstein DRO, CVaR) with multiple-testing corrections (union bound, Learn Then Test fixed-sequence) and betting-based confidence sequences (WSR). Our main theoretical contribution is Transfer-Informed Betting (TIB), which warm-starts the WSR wealth process using a source domain's risk profile, achieving tighter bounds in data-scarce settings with a formal dominance guarantee. We prove that the TIB wealth process remains a valid supermartingale under all source-target divergences, that TIB dominates standard WSR when domains match, and that no data-independent warm-start can achieve better convergence. The combination of betting-based confidence sequences, LTT monotone testing, and cross-domain transfer is, to our knowledge, a three-way novelty not present in the literature. We evaluate all nine bound families on four benchmarks-MASSIVE (n=1,102), NyayaBench (n=280), CLINC-150 (n=22.5K), and Banking77 (n=13K)-across 18 (alpha, delta) configurations. On MASSIVE at alpha=0.10, LTT eliminates the ln(K) union-bound penalty, achieving 94.0% guaranteed coverage versus 73.8% for Hoeffding-a 27% relative improvement. On NyayaBench, where the small calibration set makes Hoeffding-family bounds infeasible below alpha=0.20, Transfer-Informed Betting achieves 18.5% coverage at alpha=0.10, a 5.4x improvement over LTT + Hoeffding. We additionally compare with split-conformal prediction, showing that conformal methods produce prediction sets (avg. 1.67 classes) whereas selective prediction provides single-prediction risk guarantees. We apply these methods to agentic caching systems, formalizing a progressive trust model where the guarantee determines when cached responses can be served autonomously.

Figures & tables

Figure 1: Reliability diagrams for SetFit confidence calibration. (a) MASSIVE: bars show pre-calibration binned accuracy; dots show post-temperature-scaling. (b) NyayaBench v2: higher ECE reflects the harder 20-class task.

Method	$\alpha{=}0.01$	$\alpha{=}0.02$	$\alpha{=}0.05$	$\alpha{=}0.10$	$\alpha{=}0.15$	$\alpha{=}0.20$
Hoeffding + Union	—	—	—	73.8	99.8	100.0
Emp. Bernstein + Union	—	—	6.0	79.6	96.0	100.0
LTT + Hoeffding	—	—	51.5	94.0	100.0	100.0
LTT + Emp. Bernstein	—	6.0	67.3	94.0	100.0	100.0
$\dagger$ Clopper-Pearson + LTT	48.5	59.1	83.9	100.0	100.0	100.0
$\dagger$ WSR Betting + LTT	6.0	62.7	83.9	96.0	100.0	100.0

Table 1: Guaranteed test coverage (%) on MASSIVE (

n_{\text{cal}}{=}549

\delta{=}0.10

). Higher is better. “—” indicates no feasible

\tau^{*}

. All feasible entries have zero guarantee violations on the test set. New methods (this work) marked with

\dagger

Method	$\alpha{=}0.01$	$\alpha{=}0.02$	$\alpha{=}0.05$	$\alpha{=}0.10$	$\alpha{=}0.15$	$\alpha{=}0.20$
Hoeffding + Union	—	—	—	—	—	14.4
Emp. Bernstein + Union	—	—	—	—	—	3.4
LTT + Hoeffding	—	—	—	3.4	18.5	26.0
LTT + Emp. Bernstein	—	—	—	3.4	14.4	23.3
$\dagger$ Clopper-Pearson + LTT	—	—	5.5	18.5	23.3	32.9
$\dagger$ WSR Betting + LTT	—	—	5.5	18.5	26.0	41.1

Table 2: Guaranteed test coverage (%) on NyayaBench v2 (

n_{\text{cal}}{=}134

\delta{=}0.10

). Transfer methods use MASSIVE risk profile. All entries with zero guarantee violations. New methods marked with

\dagger

Method	$\alpha{=}0.01$	$\alpha{=}0.02$	$\alpha{=}0.05$	$\alpha{=}0.10$	$\alpha{=}0.15$	$\alpha{=}0.20$
Hoeffding + Union	—	19.1	74.1	93.2	99.4	100.0
LTT + Hoeffding	—	48.9	79.0	94.3	100.0	100.0
$\dagger$ Clopper-Pearson + LTT	44.8	61.9	82.2	95.2	100.0	100.0
$\dagger$ WSR Betting + LTT	42.7	60.2	85.7	95.5	100.0	100.0
Wasserstein ( $\varepsilon{=}0.01$ )	—	—	66.8	91.3	98.3	100.0
CVaR ( $\beta{=}0.20$ )	—	—	—	19.1	53.0	66.8

Table 3: Guaranteed test coverage (%) on CLINC-150 (

n_{\text{cal}}{=}11{,}250

\delta{=}0.10

, simulated confidence scores). Higher is better. “—” indicates no feasible

\tau^{*}

. New methods marked with

\dagger

Method	$\alpha{=}0.01$	$\alpha{=}0.02$	$\alpha{=}0.05$	$\alpha{=}0.10$	$\alpha{=}0.15$	$\alpha{=}0.20$
Hoeffding + Union	—	—	52.2	77.3	87.8	95.7
LTT + Hoeffding	—	23.4	59.2	80.2	89.7	96.5
$\dagger$ Clopper-Pearson + LTT	25.2	43.4	64.0	81.8	90.6	96.8
$\dagger$ WSR Betting + LTT	28.8	47.2	65.5	81.2	90.1	97.6
Wasserstein ( $\varepsilon{=}0.01$ )	—	—	43.4	73.7	86.6	94.2
CVaR ( $\beta{=}0.20$ )	—	—	—	—	23.4	43.4

Table 4: Guaranteed test coverage (%) on Banking77 (

n_{\text{cal}}{=}6{,}468

\delta{=}0.10

, simulated confidence scores). Higher is better. “—” indicates no feasible

\tau^{*}

. New methods marked with

\dagger

Figure 2: Test coverage as a function of risk tolerance

\alpha

(

\delta{=}0.10

). (a) MASSIVE: LTT + Emp. Bernstein (teal) dominates Hoeffding (orange) at all

\alpha

. (b) NyayaBench v2: PAC-Bayes with transfer (red) is the only method achieving meaningful coverage below

\alpha{=}0.15

Figure 3: Guaranteed coverage as a function of calibration set size (

\alpha{=}0.10

\delta{=}0.10

). (a) MASSIVE: LTT + Hoeffding (teal) reaches 62% coverage at

n{=}150

and 94% at

n{=}549

; Hoeffding + union (orange) remains infeasible until

n{=}400

. (b) NyayaBench v2: PAC-Bayes transfer (red) is the only method achieving coverage at any

n

, stabilizing at

{\approx}14\%

from

n{=}50

onward. Shaded regions show

\pm 1

standard deviation across 20 subsamples.

Figure 4: Correction term

C(n,\delta{=}0.10)

as a function of calibration set size. The dashed line marks

\alpha{=}0.10

; a method achieves feasibility when its correction falls below this line (assuming

\hat{R}(\tau)\approx 0

). LTT + Hoeffding crosses at

n\approx 120

; Hoeffding + union bound requires

n\approx 350

Figure 5: Per-intent guaranteed coverage (

\alpha{=}0.10

\delta{=}0.10

). (a) MASSIVE: only check_calendar (

n_{\text{cal}}{=}167

, 92.2% accuracy) achieves subgroup feasibility with LTT + Bernstein, at 60.7% coverage. All other intents have insufficient per-class calibration data. (b) NyayaBench v2: no intent achieves per-class feasibility (largest class has

n_{\text{cal}}{=}16

	MASSIVE ( $n_{\text{cal}}{=}549$ , 8 classes)				NyayaBench v2 ( $n_{\text{cal}}{=}134$ , 20 classes)
$\alpha$	Conf. cov.	Set size	Sel. cov.	Sel. risk	Conf. cov.	Set size	Sel. cov.	Sel. risk
0.01	97.8%	1.15	—	—	100.0%	1.00	—	—
0.02	96.4%	1.25	—	—	100.0%	1.00	—	—
0.05	94.8%	1.37	51.5%	0.2%	98.6%	1.26	—	—
0.10	90.4%	1.67	94.0%	6.3%	95.2%	1.91	3.4%	0.0%
0.15	85.9%	1.99	100.0%	9.0%	91.8%	2.56	18.5%	2.1%

Table 5: Conformal prediction vs. selective prediction (RCPS) on MASSIVE and NyayaBench v2 (

\delta{=}0.10

). Conformal guarantees true class

\in

prediction set; selective guarantees risk

\leq\alpha

on cached single predictions. “Set size” is the average number of classes in the conformal prediction set. “Sel. cov.” uses LTT + Hoeffding.

Figure 6: Conformal prediction vs. selective prediction across

\alpha

levels. (a) MASSIVE: conformal achieves high coverage but with growing prediction sets (right axis); selective prediction provides single-prediction guarantees. (b) NyayaBench v2: prediction sets grow rapidly with 20 classes, reaching 4.77 at

\alpha{=}0.20

Selective Prediction

Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.

See all 286 papers

Jun 7, 2026

A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control

Selective predictors answer on confident inputs and abstain elsewhere; deploying one safely needs a single finite-sample certificate that simultaneously upper-bounds the selected risk, lower-bounds the acceptance probability

\pacc

above a floor

\pmin

, and lower-bounds the deployment utility. This certificate must be valid under adaptive threshold selection from a finite grid of

m

pairs on

\ncert

samples. We give such a certificate for bounded, possibly non-monotone losses by treating the selected risk directly as a ratio rather than through a Hoeffding-style range bound. The construction couples three confidence bounds: a variance-adaptive empirical-Bernstein bound on the ratio risk, a Clopper--Pearson bound on acceptance, and a two-sided closeness bound on utility. Together they lower-bound the certified policy's utility absolutely and to within

2\gammau

of the best over the certified set, both non-vacuous whenever feasible; a regime-scoped third leg matches an external oracle, informative only where the risk margin

\gammar < α

and vacuous at the headline operating points. Relative to the range-only Hoeffding-ratio construction this sharpens the acceptance-floor dependence from

1/\pmin

1/\sqrt{\pmin}

, and a closed-form corollary identifies a per-pair regime in which our risk bound dominates a Hoeffding conformal risk control (Hoeffding--CRC) selective bound. Empirically, on ImageNet (three ResNets) and COCO val 2017 panoptic, the certificate opens a

+22

pp certified-acceptance frontier over Hoeffding--CRC and is

{\approx}10{\times}

tighter than a non-vacuous matched-valid baseline; these gains are regime-scoped, not universal, and absent on ADE20K. The certifier runs in

O(\ncert m)

time.

Xiaoli Yu, Jiamiao Liu

School of Cyber Security and Information Law Chongqing University of Posts and Telecommunications Chongqing 400065, China · Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China

Selective Prediction Conformal Risk Control Generalization Bounds

Ask PDF

Mar 25, 2026

Conformal Selective Prediction with General Risk Control

In deploying artificial intelligence (AI) models, selective prediction offers the option to abstain from making a prediction when uncertain about model quality. To fulfill its promise, it is crucial to enforce strict and precise error control over cases where the model is trusted. We propose Selective Conformal Risk control with E-values (SCoRE), a new framework for deriving such decisions for any trained model and any user-defined, bounded and continuously-valued risk. SCoRE offers two types of guarantees on the risk among ``positive'' cases in which the system opts to trust the model. Built upon conformal inference and hypothesis testing ideas, SCoRE first constructs a class of (generalized) e-values, which are non-negative random variables whose product with the unknown risk has expectation no greater than one. Such a property is ensured by data exchangeability without requiring any modeling assumptions. Passing these e-values on to hypothesis testing procedures, we yield the binary trust decisions with finite-sample error control. SCoRE avoids the need of uniform concentration, and can be readily extended to settings with distribution shifts. We evaluate the proposed methods with simulations and demonstrate their efficacy through applications to error management in drug discovery, health risk prediction, and large language models.

Tian Bai, Ying Jin

Department of Statistics, Stanford University · Department of Statistics and Data Science, University of Pennsylvania

Selective Prediction Conformal Risk Control Conformal Prediction E-Values

Ask PDF

Sep 18, 2026

Available Guardrails: Certifying Selective Prediction across ML Systems

A selective predictor acts as a safety gate: it returns an output only when the prediction appears sufficiently trustworthy. Deployments increasingly require this reliability to be certified at a target precision for every reporting unit of interest, such as a tool, policy label, or patient subgroup. The main difficulty is often not whether a granted certificate is valid, but whether finite calibration data can produce one at all. As the gate becomes safer or more fine-grained, some units may receive too little evidence to certify. We make this notion of availability computable through classical exact-binomial inversion and formulate reporting-partition selection, under a fixed group order, as a dynamic program that exposes the trade-off among safety, granularity, and served traffic. The resulting frontier reveals a large population opportunity that finite-sample estimation nearly erases: a truth-informed planner gains

0.157

mean coverage over support balancing, whereas a naive estimator recovers only

0.005

, making recovery from finite data the central challenge. Constructing candidate partitions on one planning split and selecting among them on another recovers part of this gap, improving mean coverage over support balancing by

0.060

, with the direction reproduced in

59

60

model effects across three intent-routing datasets and two architectures. A complementary validity-preserving lever, reallocating the familywise error budget across reporting units, recovers additional coverage both with population quantities and noisy estimates. The same frontier recurs, with predictor-specific ceilings, across LLM tool-calling, content moderation, lesion classification, and recommendation. Certified availability is therefore a plannable deployment resource that determines when a safety gate can be certified, at what granularity, and over how much traffic.

Parivesh Priye, Yufeng Wang, Haibin Ling +1

Rivian and Volkswagen Group Technologies · Stony Brook University · Westlake University

Selective Prediction Language Model Safety Evaluation

Ask PDF

Abstract

Selective Prediction

Explore similar work

A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control

Conformal Selective Prediction with General Risk Control

Available Guardrails: Certifying Selective Prediction across ML Systems