Figure 26: The violin plots for the ablation study over varying
α on datasets under stochastic environments. The violin plots are drawn with randomly chosen
30K samples over 30 random trials. Here, converged SRs mostly go along with target risk
α ( i.e., violin plots are below of the diagonal line), which empirically demonstrates that our algorithm finds efficient conformal abstainers with SR guarantees. Note that
ErrT refers to the error rate on a given dataset, i.e.,
ErrT:=∑t=1T\mathbbm1(G(xt)=Eyt)/T and the SRs of conformal abstainers should be lower than
ErrT to be meaningful in controlling the SRs. In (a), the SR exceeds
α for small
α due to the poor calibration of
fcon , not a failure of the guarantee. For NQ with GPT-5.4,
44.5% of the samples have the maximum score
fcon≈1 (the largest floating-point value below
1 , as
ft∈[0,1) ), and
14.8% of them are incorrect. As no threshold can separate these tied samples, every
τ<1 answers all of them, so for
α below
14.8% , ExAUL is driven toward the always-abstaining threshold
τ=1 . The SR violation
VT in ( 1 ), which our theory bounds, then remains controlled since abstention incurs no violation, whereas the empirical
SRT is a ratio over only a few answered samples and thus becomes large. This is consistent with our SR bound in Section 3.4 , which depends on
1−IneffT , and thus becomes vacuous when ExAUL rarely answers (see Appendix I.1 for further discussion).