Stability-Compensated Threshold Calibration for Selection-Conditioned Risk Control in Large Language Models
Authors: Zhiyuan Wang, Yuanchang Ye, Jiahao Yuan, Jinhao Duan, Xiaoshuang Shi, Weijun Xie, Ye Wei, Kaidi Xu
Organizations: City University of Hong Kong · National University of Singapore · East China Normal University · University of North Carolina at Chapel Hill · University of Electronic Science and Technology of China · The Hong Kong Institute of AI for Science
Large language models often produce hallucinated answers, motivating selective prediction policies that abstain on uncertain queries. However, heuristic uncertainty thresholds alone do not provide statistical control of errors among accepted predictions. Linear Expectation Constraints (LEC) reduce selection-conditioned error rate (SCER) calibration to a nonpositive expected linear-loss constraint, but the associated loss is non-monotonic in the acceptance threshold. We construct an exact counterexample under i.i.d. sampling demonstrating that LEC's empirical threshold-selection rule can violate the population SCER target. To address this risk-transfer gap, we introduce Stability-Compensated Linear Expectation Control (SC-LEC), which uses the negative linear risk of a theoretical full-data reference to compensate for threshold-selection instability. For local expansion beyond a safe anchor, we derive two finite-sample certificates: a stability-based certificate exploiting rank crossings and a direct certificate using the structure of threshold expansion to reduce conservatism. With a valid population anchor margin, our certificate conditions ensure finite-sample marginal SCER control, and independent margin certification further provides a 1−δ high-probability guarantee for this calibration-averaged control. Experiments on CommonsenseQA, ARC-Challenge, and TriviaQA show useful retention gains beyond the same certified anchor, while revealing conservative behavior in small-margin regimes.
Figures & tables
Figure 1: Overview of SC-LEC. A safe anchor supports a one-sided local rank search, certified through stability compensation (SC-S) or direct analysis (SC-D). The rule scans the anchor and up to h subsequent observed scores, selecting the largest empirically feasible candidate or abstaining if none exists. Known valid margins yield finite-sample marginal SCER control at positive retention. Independently certified margins provide an upstream 1−δ certificate of this calibration-averaged guarantee.
Figure 2: Mean CommonsenseQA retention across five target risks and all 100 paired repetitions, including failures. Anchor, SC-S, and SC-D use the margin/KL design; marker shapes and line styles identify methods. LEC is an empirical reference.
Empirical
Certified anchored
Dataset
Model
LEC
Anchor
SC-S
SC-D
Qwen2.5-7B
99.77 / 19.43
58.88 / 5.59
64.06 / 7.09
67.23 / 8.03
Qwen2.5-32B
99.96 / 16.00
67.25 / 5.38
74.46 / 7.00
77.52 / 7.76
Qwen3-8B
94.67 / 20.04
54.30 / 7.12
57.45 / 7.93
60.73 / 8.77
CommonsenseQA
Qwen3-32B
99.94 / 18.85
62.27 / 6.43
67.47 / 7.64
70.64 / 8.49
Qwen2.5-7B
99.83 / 10.42
80.56 / 4.13
87.13 / 5.95
92.84 / 7.83
Table 1: Retention / empirical SCER (%) at α=0.20 , aggregated over all 100 repetitions, including failed and unavailable designs. SC methods use the margin/KL design. A dash denotes undefined SCER at zero retention. SC-S is unavailable on TriviaQA at this target. Boldface indicates the highest retention among Anchor, SC-S, and SC-D, including ties at the displayed precision.
Figure 3: Certified margin and search radius at α=0.20 (margin/KL design). Labels identify Qwen family and size. Points average available repetitions within each branch; segments connect branches with identical upstream cohorts. Gray crosses in the status strip mark unavailable SC-S designs.
Figure 4: Exact four-band i.i.d. counterexample with n=25 , c=1 , and α=0.20 . Left: population linear risk by threshold. Right: the signed stability gap exceeds the reserve, giving R25≈0.002078>0 and marginal SCER ≈0.2080>0.20 . Appendix B.10 derives these quantities by exact enumeration.
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Model
Rows
Acc. (%)
Unique (%)
Max tie
12dp (%)
12dp tie
Qwen2.5-7B
10956
80.39
99.93
2
85.04
14
Qwen2.5-32B
10956
83.99
100.00
1
56.95
1507
Qwen3-8B
10956
78.07
99.94
2
72.98
411
CommonsenseQA
Qwen3-32B
10956
81.14
100.00
1
99.80
2
Qwen2.5-7B
2590
89.61
99.42
2
86.87
6
Qwen2.5-32B
2590
95.29
99.34
2
35.48
815
Appendix
Table 2: Prediction accuracy and uncertainty-score ties before question deduplication. Unique reports the percentage of distinct scores, and Max tie is the largest tie group. The 12dp columns repeat these statistics after rounding scores to 12 decimal places.
Figure A1: Retention across all five target risks and twelve dataset–model pairs under the margin/KL design with primary uncertainty scores. Results average all 100 repetitions, including failed and unavailable designs. Zero retention corresponds to undefined SCER.
Figure A2: Effects of anchor selection and concentration bounds at α=0.20 . The left panel reports SC-D retention, and the right panel reports the percentage of repetitions with an available SC-S design. Values are aggregated over all 100 repetitions. Numbers and horizontal bars use the same 0–100% scale. The Hoeffding variant replaces both margin-certification and anchor-feasibility bounds.
Figure A3: Sensitivity of certified expansion for Qwen2.5-32B at α=0.20 . Retention includes all failed and unavailable designs, with undefined SCER at zero retention. Panels vary the certification error budget, candidate count, sample allocation, calibration size, SC-D radius, and concentration bounds. Panel (e) changes SC-D only. Panel (f) compares KL bounds with Hoeffding replacements for the tail alone or for both certification and tail bounds. Marker fill identifies the dataset, while color and shape identify the method. Small horizontal offsets separate coincident points. Allocation changes may also change the test set.
Figure A4: Retention changes relative to the same anchor at α=0.20 under the margin/KL design. Positive bars show additional accepted answers, hatched negative bars show lost anchor acceptance, and black ticks show the net change. Results include all 100 repetitions and are reported in percentage points.
Figure A5: Separate question-disjoint evaluation of paired retention gains over the same margin/KL anchor at α=0.20 . Pointwise 95% intervals jointly bootstrap test questions while holding the 100 fitted policies fixed. They quantify question-level variability conditional on these policies, excluding refitting and certification uncertainty. Unavailable designs are included, giving negative TriviaQA SC-S gains. Dataset panels use their labeled horizontal ranges.
Figure A6: Separate question-disjoint evaluation at α=0.20 , margin/KL, over all 100 fixed policies. Colors and shapes identify methods, and each panel represents a dataset/model pair. Horizontal and vertical bars are pointwise 95% intervals from jointly bootstrapping test questions, conditional on these policies and excluding refitting and certification uncertainty. The vertical line marks the target. TriviaQA SC-S has zero retention and undefined SCER, shown by an annotation. LEC is an empirical reference.
Dataset
Model
AUROC
100η
Pass
SC-S
hD
ΔS
ΔZ
qnew
100Δg
Qwen2.5-7B
0.827
6.72
100
100
230
8.35
2.11
25.27
0.44
Qwen2.5-32B
0.846
8.16
100
100
279
10.28
2.40
23.37
0.35
Qwen3-8B
0.799
5.19
100
100
177
6.43
1.46
22.66
0.17
CSQA
Qwen3-32B
0.829
6.73
100
100
230
8.37
1.99
23.83
0.32
Qwen2.5-7B
0.876
9.89
100
100
79
12.28
3.94
32.10
1.49
Qwen2.5-32B
0.921
13.96
100
100
112
6.56
2.29
34.88
0.98
Appendix
Table 3: Anchor margins, certificate availability, and incremental errors at α=0.20 , margin/KL. AUROC uses the complete deduplicated benchmark pool (higher score predicts error). η is the median certified margin conditional on passing. Pass and SC-S give anchor passes and available SC-S designs out of 100. hD is the median radius conditional on an available direct design. ΔS and ΔZ are additional SC-D answers and errors per 100 test exposures. qnew=100∑ΔZ/∑ΔS is undefined when no answers are added. The final column is the change in mean linear loss per 100 exposures. All failed designs are included. SC-D loses no anchor-accepted answers in these runs.
Figure A7: Reserve and stability bounds for unrestricted LEC with c=1 under the distribution defined in Appendix B.7 . Left: the integrated population linear-risk curve. Right: mean reserve, signed gap, directional bound, and absolute bound across 500 full-data/leave-one-out evaluations of the same rule. The directional bound is 27.3% smaller than the absolute bound, but neither certifies the unrestricted rule.
Setting
Pass (%)
Misses
Upper (%)
Default
100.00
0
0.30
m=100
0.00
0
0.30
m=300
89.80
1
0.47
m=1000
100.00
0
0.30
L=1
93.40
0
0.30
L=5
100.00
2
0.63
Appendix
Table 4: Known-law simultaneous margin coverage, 1,000 independent draws per setting. Default: m=3000,L=20,δ=0.05,πcore=0.02 . Misses check every candidate, including nonpassing ones. Upper is a pointwise one-sided binomial 95% bound, separate from the certificate budget and not simultaneous across settings. Rules are prespecified.
πcore
Method
Available
Ret. (%)
R(H)>0
R(H)≤0
Unresolved
0.02
SC-S
200
51.38
0
200
0
SC-D
200
56.47
0
200
0
0.10
SC-S
0
0.00
0
200
0
SC-D
200
50.58
0
200
0
0.19
SC-S
0
0.00
0
200
0
SC-D
0
0.00
0
200
0
Appendix
Table 5: Known-law nested evaluation: 200 fixed designs per branch and 500 fresh calibrations per design, with exact fresh-test integration. Risk classifications retain simultaneous inner Monte Carlo uncertainty. The risk columns denote confirmed positive, confirmed nonpositive, and unresolved conditional marginal risk. Availability is shown separately. Unavailable designs have zero risk through abstention and undefined SCER, not useful retention.
Figure A8: Known-distribution evaluation with a small calibration sample. Left: available and abstaining designs. Right: conditional marginal linear risk for the 199 available SC-D designs, sorted by the estimate and shown with simultaneous Monte Carlo bounds. The remaining SC-D design and all SC-S designs abstain, giving zero linear risk and undefined SCER. Table 5 reports the separate strong-core results.
n
Sort
Scan
Total
Cert.
100
0.005
0.022
0.024
2.979
1000
0.078
0.030
0.103
3.142
10000
1.230
0.097
1.515
4.424
100000
17.254
0.803
18.399
21.241
Appendix
Table 6: Single-process CPU medians (ms). Total time includes sorting and scanning. Separately measured component medians need not sum to the total median. Model inference and leave-one-out diagnostics are excluded. These timings exclude the final conservative numerical checks.
Figure A9: Prespecified routing pairs, α=0.20 , margin/KL, over all 100 repetitions including unavailable designs. Colors encode selectors and shapes also encode methods. Filled/open markers encode the Qwen2.5/Qwen3 pairs. No-escalation rate (cheap-model answers) and overall hybrid error are empirical offline-replay quantities. The cheap-branch SCER certificate does not transfer to the plotted whole-system error.
Method
Ret. (%)
SCER (%)
Linear risk
LEC
25.9713
20.8001
0.002078
Anchor
10
0
−0.02
SC-S
0
–
0
SC-D
10
0
−0.02
Appendix
Table 7: Exact-law comparison with a known valid margin and the same final-calibration size. SC-S’s zero risk comes from fallback abstention, with undefined SCER. The anchor and SC-D use population margin information. The comparison therefore does not match the information available to each method.
Task
Cells
Empirical exceedances
Lower CI above α
CommonsenseQA
20
6
0
ARC-Challenge
20
2
0
TriviaQA
20
13
4
Appendix
Table 8: Descriptive LEC target attainment over the complete primary grid. Each task has four models and five targets. The last column counts pointwise split-bootstrap intervals whose lower endpoint exceeds the target. All primary configurations enter these counts; this is not a simultaneous population-validity test.
Quantity
SC-S
SC-D
λ0 : score
≈6.25×10−5
η
≈0.0806
Certification m
3834
Calibration n
2738
L , δ
20, 0.05
c
≈154.39
0
Appendix
Table 9: Example calibrated designs from the first repetition on CommonsenseQA Qwen2.5-32B, primary entropy, α=0.20 , margin/KL. The branches share the certified anchor. Parameters are rounded for display. Ties use the prespecified key.
Candidate rank
Prefix linear loss
r0
B0−αr0
r0+1
Cr0+e1−α
r0+k
Cr0+∑s=1kes−αk
r0+h′
Cr0+∑s=1h′es−αh′
Appendix
Table 10: Symbolic prefix scan, with h′=min{h,n−r0} . Each row tests Cj≤−c . The corresponding threshold is λ0 for j=r0 and U(j) otherwise. Positive-expansion rows require h′>0 .
Dataset
Cells
Higher
Equal
Lower
CommonsenseQA
20
20
0
0
ARC-Challenge
20
14
0
6
TriviaQA
20
17
3
0
All
60
51
3
6
Appendix
Table 11: Retention relative to margin/KL SC-D over all primary cells and 100 paired repetitions. LTT-cal and LTT-pool give the same counts. All three ties have zero acceptance and undefined SCER. LTT-cal uses n labels. LTT-pool and SC-D use m+n in different roles, with different guarantee objects. Counts summarize observed retention differences across configurations.
Model
α
LEC
Anchor
SC-S
SC-D
LTT-cal
LTT-pool
Qwen2.5-7B
0.05
56.33 / 5.01
11.45 / 1.91
0.00 / –
11.45 / 1.91
44.01 / 3.06
48.49 / 3.60
0.10
73.59 / 10.00
47.44 / 3.48
0.49 / 3.56
49.28 / 3.73
64.72 / 7.31
66.96 / 7.99
0.15
87.54 / 14.89
54.83 / 4.74
56.81 / 5.15
59.58 / 5.85
79.64 / 12.12
81.49 / 12.75
0.20
99.77 / 19.43
58.88 / 5.59
64.06 / 7.09
67.23 / 8.03
93.02 / 16.88
94.90 / 17.50
0.30
99.96 / 19.53
69.35 / 8.76
83.68 / 13.56
87.58 / 14.96
99.90 / 19.49
99.90 / 19.49
Qwen2.5-32B
0.05
65.75 / 4.98
40.12 / 1.63
0.00 / –
40.18 / 1.63
55.17 / 3.18
58.60 / 3.55
Appendix
Table 12: Complete CommonsenseQA primary comparison: retention / empirical SCER (%) at five target risks, aggregated over all 100 paired repetitions. Anchor and SC methods use the prespecified margin/KL design. Failed and unavailable designs remain included; a dash denotes undefined SCER at zero acceptance. LTT-cal and LTT-pool use different fitting-label budgets, as described in the text.
Model
α
LEC
Anchor
SC-S
SC-D
LTT-cal
LTT-pool
Qwen2.5-7B
0.05
84.81 / 5.08
19.88 / 2.23
0.00 / –
19.88 / 2.23
66.87 / 2.36
74.37 / 2.90
0.10
98.11 / 9.79
72.06 / 2.69
0.74 / 3.50
74.95 / 3.09
87.60 / 5.93
91.34 / 7.20
0.15
99.83 / 10.42
75.26 / 3.13
77.31 / 3.53
82.44 / 4.63
98.55 / 9.93
99.58 / 10.34
0.20
99.83 / 10.42
80.56 / 4.13
87.13 / 5.95
92.84 / 7.83
99.58 / 10.34
99.58 / 10.34
0.30
99.83 / 10.42
85.71 / 5.30
99.80 / 10.42
99.83 / 10.42
99.58 / 10.34
99.58 / 10.34
Qwen2.5-32B
0.05
99.55 / 4.51
77.47 / 0.78
0.00 / –
78.89 / 0.88
91.94 / 2.38
94.75 / 2.76
Appendix
Table 13: Complete ARC-Challenge primary comparison: retention / empirical SCER (%) at five target risks, aggregated over all 100 paired repetitions. Anchor and SC methods use the prespecified margin/KL design. Failed and unavailable designs remain included; a dash denotes undefined SCER at zero acceptance. LTT-cal and LTT-pool use different fitting-label budgets, as described in the text.
Model
α
LEC
Anchor
SC-S
SC-D
LTT-cal
LTT-pool
Qwen2.5-7B
0.05
2.58 / 7.33
0.00 / –
0.00 / –
0.00 / –
0.00 / –
0.00 / –
0.10
17.85 / 9.97
0.00 / –
0.00 / –
0.00 / –
0.52 / 7.46
3.08 / 8.03
0.15
26.99 / 14.86
0.00 / –
0.00 / –
0.00 / –
15.86 / 9.93
20.08 / 10.76
0.20
34.69 / 19.89
0.40 / 12.19
0.00 / –
0.40 / 12.19
25.61 / 14.11
27.93 / 15.64
0.30
48.71 / 29.85
25.05 / 13.68
0.00 / –
26.64 / 14.74
41.10 / 24.40
42.82 / 25.59
Qwen2.5-32B
0.05
8.43 / 6.25
0.00 / –
0.00 / –
0.00 / –
0.00 / –
0.00 / –
Appendix
Table 14: Complete TriviaQA primary comparison: retention / empirical SCER (%) at five target risks, aggregated over all 100 paired repetitions. Anchor and SC methods use the prespecified margin/KL design. Failed and unavailable designs remain included; a dash denotes undefined SCER at zero acceptance. LTT-cal and LTT-pool use different fitting-label budgets, as described in the text.
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.