We present a comprehensive ablation of nine finite-sample bound families for selective prediction with risk control, combining concentration inequalities (Hoeffding, Empirical Bernstein, Clopper-Pearson, Wasserstein DRO, CVaR) with multiple-testing corrections (union bound, Learn Then Test fixed-sequence) and betting-based confidence sequences (WSR). Our main theoretical contribution is Transfer-Informed Betting (TIB), which warm-starts the WSR wealth process using a source domain's risk profile, achieving tighter bounds in data-scarce settings with a formal dominance guarantee. We prove that the TIB wealth process remains a valid supermartingale under all source-target divergences, that TIB dominates standard WSR when domains match, and that no data-independent warm-start can achieve better convergence. The combination of betting-based confidence sequences, LTT monotone testing, and cross-domain transfer is, to our knowledge, a three-way novelty not present in the literature. We evaluate all nine bound families on four benchmarks-MASSIVE (n=1,102), NyayaBench (n=280), CLINC-150 (n=22.5K), and Banking77 (n=13K)-across 18 (alpha, delta) configurations. On MASSIVE at alpha=0.10, LTT eliminates the ln(K) union-bound penalty, achieving 94.0% guaranteed coverage versus 73.8% for Hoeffding-a 27% relative improvement. On NyayaBench, where the small calibration set makes Hoeffding-family bounds infeasible below alpha=0.20, Transfer-Informed Betting achieves 18.5% coverage at alpha=0.10, a 5.4x improvement over LTT + Hoeffding. We additionally compare with split-conformal prediction, showing that conformal methods produce prediction sets (avg. 1.67 classes) whereas selective prediction provides single-prediction risk guarantees. We apply these methods to agentic caching systems, formalizing a progressive trust model where the guarantee determines when cached responses can be served autonomously.
Figures & tables
Figure 1: Reliability diagrams for SetFit confidence calibration. (a) MASSIVE: bars show pre-calibration binned accuracy; dots show post-temperature-scaling. (b) NyayaBench v2: higher ECE reflects the harder 20-class task.
Method
α=0.01
α=0.02
α=0.05
α=0.10
α=0.15
α=0.20
Hoeffding + Union
—
—
—
73.8
99.8
100.0
Emp. Bernstein + Union
—
—
6.0
79.6
96.0
100.0
LTT + Hoeffding
—
—
51.5
94.0
100.0
100.0
LTT + Emp. Bernstein
—
6.0
67.3
94.0
100.0
100.0
† Clopper-Pearson + LTT
48.5
59.1
83.9
100.0
100.0
100.0
† WSR Betting + LTT
6.0
62.7
83.9
96.0
100.0
100.0
Table 1: Guaranteed test coverage (%) on MASSIVE ( ncal=549 , δ=0.10 ). Higher is better. “—” indicates no feasible τ∗ . All feasible entries have zero guarantee violations on the test set. New methods (this work) marked with † .
Method
α=0.01
α=0.02
α=0.05
α=0.10
α=0.15
α=0.20
Hoeffding + Union
—
—
—
—
—
14.4
Emp. Bernstein + Union
—
—
—
—
—
3.4
LTT + Hoeffding
—
—
—
3.4
18.5
26.0
LTT + Emp. Bernstein
—
—
—
3.4
14.4
23.3
† Clopper-Pearson + LTT
—
—
5.5
18.5
23.3
32.9
† WSR Betting + LTT
—
—
5.5
18.5
26.0
41.1
Table 2: Guaranteed test coverage (%) on NyayaBench v2 ( ncal=134 , δ=0.10 ). Transfer methods use MASSIVE risk profile. All entries with zero guarantee violations. New methods marked with † .
Method
α=0.01
α=0.02
α=0.05
α=0.10
α=0.15
α=0.20
Hoeffding + Union
—
19.1
74.1
93.2
99.4
100.0
LTT + Hoeffding
—
48.9
79.0
94.3
100.0
100.0
† Clopper-Pearson + LTT
44.8
61.9
82.2
95.2
100.0
100.0
† WSR Betting + LTT
42.7
60.2
85.7
95.5
100.0
100.0
Wasserstein ( ε=0.01 )
—
—
66.8
91.3
98.3
100.0
CVaR ( β=0.20 )
—
—
—
19.1
53.0
66.8
Table 3: Guaranteed test coverage (%) on CLINC-150 ( ncal=11,250 , δ=0.10 , simulated confidence scores). Higher is better. “—” indicates no feasible τ∗ . New methods marked with † .
Method
α=0.01
α=0.02
α=0.05
α=0.10
α=0.15
α=0.20
Hoeffding + Union
—
—
52.2
77.3
87.8
95.7
LTT + Hoeffding
—
23.4
59.2
80.2
89.7
96.5
† Clopper-Pearson + LTT
25.2
43.4
64.0
81.8
90.6
96.8
† WSR Betting + LTT
28.8
47.2
65.5
81.2
90.1
97.6
Wasserstein ( ε=0.01 )
—
—
43.4
73.7
86.6
94.2
CVaR ( β=0.20 )
—
—
—
—
23.4
43.4
Table 4: Guaranteed test coverage (%) on Banking77 ( ncal=6,468 , δ=0.10 , simulated confidence scores). Higher is better. “—” indicates no feasible τ∗ . New methods marked with † .
Figure 2: Test coverage as a function of risk tolerance α ( δ=0.10 ). (a) MASSIVE: LTT + Emp. Bernstein (teal) dominates Hoeffding (orange) at all α . (b) NyayaBench v2: PAC-Bayes with transfer (red) is the only method achieving meaningful coverage below α=0.15 .
Figure 3: Guaranteed coverage as a function of calibration set size ( α=0.10 , δ=0.10 ). (a) MASSIVE: LTT + Hoeffding (teal) reaches 62% coverage at n=150 and 94% at n=549 ; Hoeffding + union (orange) remains infeasible until n=400 . (b) NyayaBench v2: PAC-Bayes transfer (red) is the only method achieving coverage at any n , stabilizing at ≈14% from n=50 onward. Shaded regions show ±1 standard deviation across 20 subsamples.
Figure 4: Correction term C(n,δ=0.10) as a function of calibration set size. The dashed line marks α=0.10 ; a method achieves feasibility when its correction falls below this line (assuming R^(τ)≈0 ). LTT + Hoeffding crosses at n≈120 ; Hoeffding + union bound requires n≈350 .
Figure 5: Per-intent guaranteed coverage ( α=0.10 , δ=0.10 ). (a) MASSIVE: only check_calendar ( ncal=167 , 92.2% accuracy) achieves subgroup feasibility with LTT + Bernstein, at 60.7% coverage. All other intents have insufficient per-class calibration data. (b) NyayaBench v2: no intent achieves per-class feasibility (largest class has ncal=16 ).
MASSIVE ( ncal=549 , 8 classes)
NyayaBench v2 ( ncal=134 , 20 classes)
α
Conf. cov.
Set size
Sel. cov.
Sel. risk
Conf. cov.
Set size
Sel. cov.
Sel. risk
0.01
97.8%
1.15
—
—
100.0%
1.00
—
—
0.02
96.4%
1.25
—
—
100.0%
1.00
—
—
0.05
94.8%
1.37
51.5%
0.2%
98.6%
1.26
—
—
0.10
90.4%
1.67
94.0%
6.3%
95.2%
1.91
3.4%
0.0%
0.15
85.9%
1.99
100.0%
9.0%
91.8%
2.56
18.5%
2.1%
Table 5: Conformal prediction vs. selective prediction (RCPS) on MASSIVE and NyayaBench v2 ( δ=0.10 ). Conformal guarantees true class ∈ prediction set; selective guarantees risk ≤α on cached single predictions. “Set size” is the average number of classes in the conformal prediction set. “Sel. cov.” uses LTT + Hoeffding.
Figure 6: Conformal prediction vs. selective prediction across α levels. (a) MASSIVE: conformal achieves high coverage but with growing prediction sets (right axis); selective prediction provides single-prediction guarantees. (b) NyayaBench v2: prediction sets grow rapidly with 20 classes, reaching 4.77 at α=0.20 .
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
School of Cyber Security and Information Law Chongqing University of Posts and Telecommunications Chongqing 400065, China · Department of Information, Xinqiao Hospital Army Medical University (Third Military Medical University) Chongqing 400037, China