Efficient Conformal Calibration for Paired Recommendation Decisions

Xinyu Qiao; Yichen Lin; Zhewen Chen; Xue Wang; Tao Yao

cs.LGOct 31, 2026

Efficient Conformal Calibration for Paired Recommendation Decisions

Authors: Xinyu Qiao, Yichen Lin, Zhewen Chen, Xue Wang, Tao Yao

Organizations: Antai College of Economics and Management Shanghai Jiao Tong University, Shanghai, China · Meta

Abstract

Risk control can suppress useful changes to a frozen recommendation rule. We study how to recover valid activation from the same observed fractional losses. Two specified internal calibrators can lose efficiency by compressing losses into their sum or clipping auxiliary inverse counts before averaging. We propose Averaged Conformal Risk Calibration, or ACRC, which averages before clipping. A product identity proves expected-risk validity, while pointwise bounds characterize the additional activation. For fixed populations sampled without replacement, we connect activation to signed utility under heterogeneous users and switching constraints. Near the budgets, positive population mean gain and the stated nondegeneracy conditions give positive leading-order utility increments despite sampling dependence. Across six datasets, changing only calibration improves discounted net utility by $0.010226$ over raw and $0.008337$ over RB. A separately frozen Jester confirmation gives $0.011025 \pm 0.001082$ over raw with the rule and filter fixed, using a five-seed approximate 95 percent paired interval. Against information-matched SCoRE selecting risk or reward scores, the mean utility difference is $+0.015934 \pm 0.001440$ . These gains use more mean realized risk within the declared budgets. Upstream failures delimit the benefit: efficient calibration releases useful modifications when the proposals themselves have value.

Figures & tables

Set.	Calibrator	$U\uparrow$	$D_{U}\downarrow$	$D_{R}\downarrow$
A	Raw	$0.1724$	$0.01113$	$0.00448$
	RB	$0.1743$	$0.01129$	$0.00462$
	ACRC (ours)	$0.1826$	$0.01249$	$0.00498$
B	ACRC (ours)	$0.1826$	$0.01249$	$0.00498$
	SCoRE, risk prediction	$0.1657$	$0.00941$	$0.00320$
	SCoRE, reward ratio	$0.1672$	$0.00942$	$0.00328$

Table 1: Calibration for the same selected proposal. Five-seed means average forty splits per model, then equally weight six domains and three backbones. Risk limits are

0.02/0.01

. Decimal places are consistent within each column. A uses default budgets, B selects configurations on

S

. Red and blue mark the best and second-best means within each setting, not significance.

Dataset	Model	Top20	PerK [ 15 ]	UMC [ 16 ]	SCoRE/PerK [ 6 ]	SCoRE/UMC [ 6 ]	ACRC (ours)
KuaiRec	LG	$0.2233$ $\pm$ 0.0067	$0.2242$ $\pm$ 0.0066	$0.2763$ $\pm$ 0.0471	$0.2242$ $\pm$ 0.0066	$0.2278$ $\pm$ 0.0074	$0.5391$ $\pm$ 0.0085
	X	$0.2532$ $\pm$ 0.0086	$0.2344$ $\pm$ 0.0114	$0.1766$ $\pm$ 0.1438	$0.2464$ $\pm$ 0.0088	$0.2499$ $\pm$ 0.0118	$0.5221$ $\pm$ 0.0122
	SIG	$0.6812$ $\pm$ 0.0162	$0.6812$ $\pm$ 0.0162	$0.6626$ $\pm$ 0.0220	$0.6812$ $\pm$ 0.0162	$0.6791$ $\pm$ 0.0148	$0.7543$ $\pm$ 0.0104
KuaiRand	LG	$-0.4960$ $\pm$ 0.0006	$-0.2069$ $\pm$ 0.0008	$-0.5933$ $\pm$ 0.0260	$-0.4883$ $\pm$ 0.0013	$-0.4977$ $\pm$ 0.0007	$-0.4846$ $\pm$ 0.0007
	X	$-0.5096$ $\pm$ 0.0014	$-0.1584$ $\pm$ 0.0022	$-0.6102$ $\pm$ 0.0264	$-0.5008$ $\pm$ 0.0022	$-0.5113$ $\pm$ 0.0015	$-0.4977$ $\pm$ 0.0015
	SIG	$-0.4675$ $\pm$ 0.0003	$-0.2127$ $\pm$ 0.0005	$-0.5644$ $\pm$ 0.0380	$-0.4608$ $\pm$ 0.0005	$-0.4695$ $\pm$ 0.0003	$-0.4555$ $\pm$ 0.0005

Table 2: Complete-procedure utility, higher is better. All six columns are ranked jointly within each row, with best means in red and second-best in blue. Displayed ties share a color. Values are five-seed means

\pm

95%

t_{4}

interval half-widths. LG, X, SIG denote LightGCN [ 24 ] , XSimGCL [ 25 ] , SIGformer [ 22 ] . Native objectives are unchanged. SCoRE paths and ACRC share risk budgets, not realized risks. Jester uses common-support users without filtering. Risks and MIR results appear in Appendix C .

Figure 1: ACRC minus SCoRE with feature and score families selected on

S

. Proposals and fitted information are shared. Bars are pointwise paired five-seed 95% intervals. 18 of eighteen mean differences are positive and 16 intervals exclude zero in favor of ACRC. Panel scales differ.

Panel of Figure 2 — Figure 2: Activation and its limits. Left, for risk-prediction SCoRE, blue gives $(p-\bar{m})\overline{g}_{T}$ , yellow adds $-\operatorname{Cov}_{T}(m,g)$ , and diamonds show their sum with paired five-seed 95% intervals. Right, percentages of splits by candidate-loss budget region, each row totaling 100%. Near means within one sampling standard deviation. Regions use $C\cup T$ labels only for post-hoc description, not selection or verification of asymptotic assumptions. Details appear in Appendix E .

Panel of Figure 3 — Figure 3: Calibration sensitivity on fifteen development settings. Points show macro utility increments over raw, with pointwise five-seed 95% intervals. All configurations are retained. Resolution $m$ uses $\lceil mL\rceil/m$ , changing both fractionality and mean loss. Protocols and budgets appear in Appendix E .

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Finite-sample increments and theoretical limits under two boundary constraints. Bars are approximate 95% Monte Carlo intervals, often smaller than the markers. They quantify simulation error, not uncertainty across training seeds. The right panel compares minimizing constraint indices, breaking ties by the first index.

Cohort	Model	Observed $R_{U}$	Expected $R_{U}$ bound	Observed $R_{R}$	Expected $R_{R}$ bound
shared	LightGCN	0.02060	0.01994	0.007015	0.008328
	XSimGCL	0.01935	0.01948	0.006632	0.007360
	SIGformer	0.01762	0.01898	0.006378	0.007226
full	LightGCN	0.02024	0.01996	0.007067	0.008644
	XSimGCL	0.01964	0.01971	0.006855	0.007853
	SIGformer	0.01964	0.01955	0.006795	0.007463

Appendix

Table 3: Empirical risk and conditional expected-risk upper bounds, all lower is better. Budgets are

0.02

and

0.01

. Bounds use the complete remaining pool for each fixed

S

, then average five seeds and forty assignments. The averaged bound is not a confidence bound for the reported finite empirical average.

Domain	Source version	Positive label	Interaction split and duplicate handling	Minimum
KuaiRec	KuaiRec 2.0 small matrix	Watch ratio $\geq 1$	Keep the earliest observation per pair. A label-blind global permutation assigns 10% to training, 10% to validation, and 80% to held-out feedback.	20
KuaiRand-random	KuaiRand-Pure	Official click label	Split the April 8–21 standard log globally in time into 80% training and 20% validation. Use the April 22–May 8 random log for held-out feedback. Keep the earliest pair in each log and remove all historical pairs from the random log.	20
Music	SIGformer-released Music benchmark	Rating $\geq 4$	Retain the released training, validation, and test splits of 34,915, 4,986, and 9,974 pairs. No within-split duplicate or cross-split overlap is present.	2
Epinions	SIGformer-released Epinions benchmark	Rating $\geq 4$	Retain the released training, validation, and test splits of 289,645, 41,376, and 82,753 pairs. No within-split duplicate or cross-split overlap is present.	2
MovieLens-25M	Complete official ML-25M release	Rating $\geq 4.0$	Use all 25,000,095 ratings. Unique pairs are sorted within each user by timestamp, with item-ID tie breaking, and split chronologically into 80% training, 10% validation, and 10% held-out feedback.	40

Appendix

Table 4: Data protocol for the five development domains. The final column is the minimum number of held-out observed candidates required before fitting users are sampled.

Data	Training users	Items	Eligible	Common	$\|T\|$
KuaiRec	1411	3185	1411	1211	611
KuaiRand	24231	7004	12626	10213	9613
Music	3472	2498	2066	1145	545
Epinions	17894	17660	13206	9382	8782
ML-25M	162205	35015	14041	13618	13018

Appendix

Table 5: Frozen populations. Eligible users include

F=200

. Common users and

T

exclude

F

S=400

and

C=200

are removed from the common population to obtain

T

Dataset	$\Delta Q^{+}\uparrow$	$-\Delta Q^{-}\uparrow$	$\Delta U_{1}\uparrow$	$\Delta\|a\|$
KuaiRec	$+0.017711\pm 0.000294$	$+0.017931\pm 0.000327$	$+0.035643\pm 0.000621$	$-0.006606$
KuaiRand	$-0.000096\pm 0.000012$	$+0.000544\pm 0.000049$	$+0.000448\pm 0.000051$	$-0.016436$
Music	$+0.000027\pm 0.000015$	$+0.000081\pm 0.000053$	$+0.000108\pm 0.000068$	$-0.000993$
Epinions	$+0.000806\pm 0.000245$	$+0.000924\pm 0.000267$	$+0.001730\pm 0.000511$	$-0.002525$
ML-25M	$+0.005934\pm 0.000649$	$+0.005934\pm 0.000649$	$+0.011867\pm 0.001297$	$0.000000$
Jester	$+0.005765\pm 0.000623$	$+0.005796\pm 0.000624$	$+0.011561\pm 0.001247$	$-0.000794$

Appendix

Table 6: Net-utility decomposition of ACRC minus raw with the proposal fixed, on the common-support population and default budgets of Table 1 . The first three columns give paired five-seed means and approximate 95% interval half-widths, all higher is better. Length change is descriptive. Backbones are equally weighted within seed, followed by equal domain weighting for the last row. Jester here is supplementary, not the frozen confirmation. Columns are accounting components, not a ranking of methods.

Dataset	Model	$\Delta$ NDCG@20	$\Delta$ Recall@20	$R_{U}\downarrow$	$R_{R}\downarrow$	Length
KuaiRec	LG	$+0.15723\pm 0.00421$	$+0.00370\pm 0.00010$	$0.01667$	$0.00021$	$19.961$
	X	$+0.13322\pm 0.00496$	$+0.00331\pm 0.00010$	$0.01655$	$0.00019$	$19.926$
	SIG	$+0.03649\pm 0.00504$	$+0.00117\pm 0.00023$	$0.01501$	$0.00017$	$19.999$
KuaiRand	LG	$-0.00521\pm 0.00026$	$-0.00967\pm 0.00024$	$0.00758$	$0.00986$	$19.519$
	X	$-0.00534\pm 0.00074$	$-0.00953\pm 0.00050$	$0.00924$	$0.00994$	$19.534$
	SIG	$-0.00442\pm 0.00019$	$-0.00964\pm 0.00008$	$0.01225$	$0.00964$	$19.594$

Appendix

Table 7: Quality changes relative to Top20, higher is better, and mean deployed downside magnitudes, lower is better. Quality differences show paired seed means and standard deviations. Length is descriptive. The target mean downsides are

0.02

for utility and

0.01

for Recall. These are not violation probabilities or per-split guarantees.

Dataset	Model	SCoRE/PerK			SCoRE/UMC			ACRC (ours)
		$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$	$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$	$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$
KuaiRec	LG	0.2242	0.000827	0.000093	0.2278	0.010583	0.000122	0.5391	0.016667	0.000206
	X	0.2464	0.015507	0.001013	0.2499	0.011773	0.000136	0.5221	0.016547	0.000195
	SIG	0.6812	0.000055	0.000012	0.6791	0.010530	0.000123	0.7543	0.015014	0.000172
KuaiRand	LG	-0.4883	0.000552	0.005961	-0.4977	0.003047	0.002622	-0.4846	0.007583	0.009855
	X	-0.5008	0.000487	0.006042	-0.5113	0.003403	0.003096	-0.4977	0.009244	0.009944

Appendix

Table 8: Original risk-control comparison at common budgets. Five seed means, each averaged over forty shared assignments. Colors rank each metric among the three risk-control methods.

Dataset	Backbone	Method	$U\uparrow$	$D_{U}\downarrow$	$D_{R}\downarrow$	NDCG $\uparrow$	Recall $\uparrow$
ML-25M	LightGCN	MIR [ 17 ]	$0.1882\,{\scriptstyle\pm\,0.0014}$	$0.04465$	$0.01927$	$0.6540$	$0.4864$
	LightGCN	ACRC (ours)	$0.1559\,{\scriptstyle\pm\,0.0025}$	$0.01690$	$0.00519$	$0.6370$	$0.4672$
	XSimGCL	MIR [ 17 ]	$0.1914\,{\scriptstyle\pm\,0.0009}$	$0.03901$	$0.01688$	$0.6565$	$0.4887$
	XSimGCL	ACRC (ours)	$0.1616\,{\scriptstyle\pm\,0.0115}$	$0.01385$	$0.00962$	$0.6411$	$0.4774$
	SIGformer	MIR [ 17 ]	$0.1928\,{\scriptstyle\pm\,0.0015}$	$0.02532$	$0.01290$	$0.6554$	$0.4856$
	SIGformer	ACRC (ours)	$0.0407\,{\scriptstyle\pm\,0.0151}$	$0.01744$	$0.00794$	$0.5674$	$0.4173$

Appendix

Table 9: Original MIR versus the complete ACRC recommendation procedure on common-support users. ACRC is independent of MIR. Both downside losses use backbone Top20 as reference. Utility reports five-seed means

\pm

approximate 95% interval half-widths. Other entries are means. Within each dataset and backbone, red and blue mark the best and second-best displayed means. Colors imply neither equal risk constraints nor statistical significance.

Dataset	Backbone	Method	$U\uparrow$	$D_{U}\downarrow$	$D_{R}\downarrow$	NDCG $\uparrow$	Recall $\uparrow$	Length
ML-25M	LightGCN	Raw	$0.1978\,{\scriptstyle\pm\,0.0029}$	$0.01241$	$0.00720$	$0.6573$	$0.4878$	$19.896$
		RB	$0.1985\,{\scriptstyle\pm\,0.0032}$	$0.01316$	$0.00764$	$0.6575$	$0.4880$	$19.891$
		ACRC (ours)	$0.1996\,{\scriptstyle\pm\,0.0038}$	$0.01458$	$0.00836$	$0.6581$	$0.4885$	$19.890$
	XSimGCL	Raw	$0.2030\,{\scriptstyle\pm\,0.0036}$	$0.01295$	$0.00715$	$0.6617$	$0.4920$	$19.943$
		RB	$0.2038\,{\scriptstyle\pm\,0.0039}$	$0.01389$	$0.00766$	$0.6621$	$0.4923$	$19.941$
		ACRC (ours)	$0.2054\,{\scriptstyle\pm\,0.0044}$	$0.01563$	$0.00857$	$0.6630$	$0.4930$	$19.940$

Appendix

Table 10: Calibration comparison with frozen MIR20 as reference. Fitted heads, proposals, and the selected modification are identical within each setting. Only the calibrator changes. Utility reports five-seed means

\pm

approximate 95% interval half-widths. Other entries are means. The utility and Recall downside budgets are

0.02

and

0.01

. Within each dataset and backbone, red and blue mark the best and second-best displayed means, with ties sharing a color. Length is descriptive. This comparison isolates calibration and does not replace the main procedure.

Dataset	Backbone	Combination vs. MIR	$w_{+}$ vs. raw	$w_{+}$ vs. RB
ML-25M	LG	$+0.01145\,{\scriptstyle\pm\,0.00469}$	$+0.001776\,{\scriptstyle\pm\,0.001046}$	$+0.001149\,{\scriptstyle\pm\,0.000665}$
	X	$+0.01401\,{\scriptstyle\pm\,0.00381}$	$+0.002444\,{\scriptstyle\pm\,0.000783}$	$+0.001586\,{\scriptstyle\pm\,0.000526}$
	SIG	$+0.00682\,{\scriptstyle\pm\,0.00191}$	$+0.000299\,{\scriptstyle\pm\,0.000049}$	$+0.000173\,{\scriptstyle\pm\,0.000077}$
KuaiRand	LG	$+0.02281\,{\scriptstyle\pm\,0.00638}$	$+0.000965\,{\scriptstyle\pm\,0.000806}$	$+0.000388\,{\scriptstyle\pm\,0.000422}$
	X	$+0.02461\,{\scriptstyle\pm\,0.00645}$	$+0.001193\,{\scriptstyle\pm\,0.000852}$	$+0.000434\,{\scriptstyle\pm\,0.000457}$
	SIG	$+0.02395\,{\scriptstyle\pm\,0.00341}$	$+0.001271\,{\scriptstyle\pm\,0.000432}$	$+0.000465\,{\scriptstyle\pm\,0.000246}$

Appendix

Table 11: Paired utility differences in the MIR supplement, reported as five-seed means and approximate 95% interval half-widths. The first contrast includes the complete modification procedure. The last two fix the selected modification and change only calibration. Columns have different comparators and are not ranked against one another.

Panel of Figure 5 — Figure 5: Risk and utility on KuaiRand. The horizontal axis is maximum normalized downside, linear to one and logarithmic above one. Gain is relative to Top20. Lines join eleven predefined budgets, stars mark our default, and open circles mark best feasible original-path or native-anchor states. The scan is descriptive and introduces no interpolated algorithm.

Dataset	Model	Comparator	$s$	ACRC (ours) $U\uparrow$	External $U\uparrow$	$\Delta U\uparrow$	External $R_{U}\downarrow$	External $R_{R}\downarrow$
KuaiRec	LG	SCoRE/U	1.3167	0.5391	0.2309	+0.308189	0.016655	0.000191
	X	Top20	fixed	0.5221	0.2532	+0.268899	0.000000	0.000000
	SIG	PerK	fixed	0.7543	0.6812	+0.073020	0.000055	0.000012
KuaiRand	LG	SCoRE/P	1.3551	-0.4846	-0.4836	-0.000992	0.000766	0.009852
	X	SCoRE/P	1.4101	-0.4977	-0.4954	-0.002220	0.000694	0.009937
	SIG	SCoRE/P	1.3282	-0.4555	-0.4573	+0.001813	0.000762	0.009632

Appendix

Table 12: Default-setting comparisons. Both external empirical risks are no larger than ours, whose default risks appear in Table 8 . Fixed denotes a native anchor. Displayed multipliers are rounded, while exact rational budgets and states are retained. Differences subtract external utility from ours. Colors compare utility only and do not imply continuous-frontier dominance.

Proposal	Metric	ACRC (ours)	SCoRE
			Original features	Augmented default	Augmented tuned	Feature selection
Selected proposal	$U\uparrow$	$0.18265$	$0.16470$	$0.16426$	$0.16562$	$0.16572$
	$D_{U}\downarrow$	$0.01249$	$0.00929$	$0.00941$	$0.00948$	$0.00941$
	$D_{R}\downarrow$	$0.00498$	$0.00319$	$0.00324$	$0.00324$	$0.00320$
Within-Top20 reranking	$U\uparrow$	$0.14210$	$0.14074$	$0.14078$	$0.14086$	$0.14081$
	$D_{U}\downarrow$	$0.01516$	$0.01370$	$0.01426$	$0.01364$	$0.01369$

Appendix

Table 13: Macro comparison across five fixed proposal classes using risk-prediction SCoRE. Both

w_{+}

and original-feature SCoRE are tuned only on

S

. The table includes all three configurations. Colors rank the five calibration methods within each row. Lower risk does not by itself imply higher utility.

Dataset	Model	$U_{w_{+}}\uparrow$	$U_{\rm SCoRE}\uparrow$	$\Delta U\uparrow$	$w_{+}$ risks	SCoRE risks
KuaiRec	LightGCN	$0.5391$	$0.4721$	$+0.066984\,{\scriptstyle\pm\,0.016319}$	$0.01667\,/\,0.00021$	$0.01134\,/\,0.00015$
	XSimGCL	$0.5221$	$0.4571$	$+0.064932\,{\scriptstyle\pm\,0.004587}$	$0.01655\,/\,0.00020$	$0.01148\,/\,0.00014$
	SIGformer	$0.7543$	$0.7477$	$+0.006598\,{\scriptstyle\pm\,0.002882}$	$0.01501\,/\,0.00017$	$0.01279\,/\,0.00015$
KuaiRand	LightGCN	$-0.4846$	$-0.4867$	$+0.002078\,{\scriptstyle\pm\,0.000918}$	$0.00758\,/\,0.00986$	$0.00614\,/\,0.00584$
	XSimGCL	$-0.4977$	$-0.4986$	$+0.000952\,{\scriptstyle\pm\,0.000741}$	$0.00924\,/\,0.00994$	$0.00770\,/\,0.00630$
	SIGformer	$-0.4555$	$-0.4569$	$+0.001464\,{\scriptstyle\pm\,0.000202}$	$0.01225\,/\,0.00964$	$0.01037\,/\,0.00684$

Appendix

Table 14:

S

-selected feature family.

\Delta U=U_{w_{+}}-U_{\mathrm{SCoRE}}

, with means and paired five-seed intervals. Risk columns report

D_{U}/D_{R}

. Values are means

\pm

95% CI half-widths.

Calibrator	$U\uparrow$	$D_{U}\downarrow$	$D_{R}\downarrow$
ACRC (ours)	$0.1826$	$0.01249$	$0.00498$
SCoRE risk prediction	$0.1657$	$0.00941$	$0.00320$
SCoRE reward ratio, default	$0.1660$	$0.00941$	$0.00330$
SCoRE reward ratio, $S$	$0.1672$	$0.00942$	$0.00328$
SCoRE both score families, $S$	$0.1667$	$0.00940$	$0.00323$

Appendix

Table 15: Same-proposal macro results with both SCoRE score families. All choices precede final calibration. Means equally weight six domains and three backbones within five training seeds.

Dataset	Control	LightGCN	XSimGCL	SIGformer
KuaiRec	raw	$+0.05545\,{\scriptstyle\pm\,0.00223}$	$+0.04606\,{\scriptstyle\pm\,0.00290}$	$+0.00541\,{\scriptstyle\pm\,0.00055}$
KuaiRec	RB	$+0.04499\,{\scriptstyle\pm\,0.00247}$	$+0.03996\,{\scriptstyle\pm\,0.00241}$	$+0.00588\,{\scriptstyle\pm\,0.00173}$
KuaiRand	raw	$+0.00046\,{\scriptstyle\pm\,0.00005}$	$+0.00043\,{\scriptstyle\pm\,0.00013}$	$+0.00045\,{\scriptstyle\pm\,0.00005}$
KuaiRand	RB	$+0.00010\,{\scriptstyle\pm\,0.00008}$	$+0.00013\,{\scriptstyle\pm\,0.00012}$	$+0.00016\,{\scriptstyle\pm\,0.00004}$
ML-25M	raw	$+0.00393\,{\scriptstyle\pm\,0.00049}$	$+0.00984\,{\scriptstyle\pm\,0.00301}$	$+0.02183\,{\scriptstyle\pm\,0.00488}$
ML-25M	RB	$+0.00549\,{\scriptstyle\pm\,0.00047}$	$+0.00525\,{\scriptstyle\pm\,0.00277}$	$+0.01002\,{\scriptstyle\pm\,0.00291}$

Appendix

Table 16: All eighteen same-proposal settings at default budgets. Entries are

w_{+}

minus raw or RB, with paired five-seed approximate 95% intervals. Forty assignments are averaged within each seed. Negative and zero-crossing intervals are retained. No multiplicity correction is applied. Values are means

\pm

95% CI half-widths. All three backbone columns give utility increments, higher is better.

Domain	Both below	One near	Both near	Any above
Jester	$19.50$	$41.00$	$13.67$	$25.83$
KuaiRand	$0.00$	$0.00$	$0.00$	$100.00$
KuaiRec	$22.33$	$70.00$	$0.00$	$7.67$
MovieLens	$2.33$	$37.67$	$22.00$	$38.00$
Music	$95.00$	$4.17$	$0.00$	$0.00$
Epinions	$95.50$	$4.50$	$0.00$	$0.00$

Appendix

Table 17: Budget regions in the full remaining population. Entries are percentages of role splits. One near means that the other constraint is below budget. Music additionally has

0.83\%

of splits with a zero-variance constraint. Forty splits are averaged within each model before equal weighting within common training seeds.

Dataset	$\bar{g}_{S}$	$D_{U,S}/\beta_{U}$	$D_{R,S}/\beta_{R}$	$1-p_{0}$	$\widehat{\Delta U}_{S}$	$\Delta U_{T}$
KuaiRec	0.249	0.857	0.021	0.201	$0.035798\,{\scriptstyle\pm\,0.001353}$	$0.035643\,{\scriptstyle\pm\,0.000621}$
KuaiRand	0.068	0.983	5.475	0.599	$0.000429\,{\scriptstyle\pm\,0.000056}$	$0.000448\,{\scriptstyle\pm\,0.000051}$
Music	0.009	0.358	0.189	0.008	$0.000041\,{\scriptstyle\pm\,0.000038}$	$0.000108\,{\scriptstyle\pm\,0.000068}$
Epinions	0.036	0.524	0.205	0.028	$0.000950\,{\scriptstyle\pm\,0.000244}$	$0.001730\,{\scriptstyle\pm\,0.000511}$
ML-25M	0.100	1.074	1.037	0.334	$0.011702\,{\scriptstyle\pm\,0.001215}$	$0.011867\,{\scriptstyle\pm\,0.001297}$
Jester	0.096	0.866	0.859	0.234	$0.011550\,{\scriptstyle\pm\,0.001300}$	$0.011561\,{\scriptstyle\pm\,0.001247}$

Appendix

Table 18: Information available on

S

and the corresponding observed calibration increment on

T

. Risk ratios compare candidate downside with the budget. Headroom is

1-p_{0}

on pseudo-calibration halves. Effect columns report paired five-seed means

\pm

95% interval half-widths.

Method	Risk	Mean	Exceedance %	99th percentile	Maximum
raw	$D_{U}$	0.01113	2.08	0.02097	0.02566
raw	$D_{R}$	0.00448	6.19	0.01119	0.01352
RB	$D_{U}$	0.01129	2.11	0.02107	0.02571
RB	$D_{R}$	0.00462	7.58	0.01162	0.01545
ACRC (ours)	$D_{U}$	0.01249	6.64	0.02374	0.02951
ACRC (ours)	$D_{R}$	0.00498	10.72	0.01236	0.01602

Appendix

Table 19: Realized-risk distribution over correlated role splits. Budgets are

0.02

and

0.01

. Lower values indicate less realized downside, not higher utility. The three SCoRE variants select configurations only on

S

Proposal	$U_{+}-U_{\rm SCoRE}$	$(p-\bar{m})\overline{g}_{T}$	$\operatorname{Cov}_{T}(m,g)$
Selected proposal	$+0.016928\,{\scriptstyle\pm\,0.001081}$	$+0.017707\,{\scriptstyle\pm\,0.001054}$	$+0.000778\,{\scriptstyle\pm\,0.000257}$
Within-Top20 reranking	$+0.001293\,{\scriptstyle\pm\,0.000310}$	$+0.001555\,{\scriptstyle\pm\,0.000328}$	$+0.000263\,{\scriptstyle\pm\,0.000099}$
Pool reranking	$+0.019470\,{\scriptstyle\pm\,0.002052}$	$+0.020483\,{\scriptstyle\pm\,0.001980}$	$+0.001013\,{\scriptstyle\pm\,0.000240}$
PerK proposal	$-0.000566\,{\scriptstyle\pm\,0.000105}$	$-0.004605\,{\scriptstyle\pm\,0.000326}$	$-0.004039\,{\scriptstyle\pm\,0.000281}$
UMC proposal	$-0.002023\,{\scriptstyle\pm\,0.002228}$	$+0.000019\,{\scriptstyle\pm\,0.001897}$	$+0.002042\,{\scriptstyle\pm\,0.000594}$

Appendix

Table 20: Utility decomposition for information-matched risk-prediction SCoRE, with feature family selected using

S

only. Entries show paired five-seed intervals. The final two columns are the activation term and the covariance to subtract. Values are means

\pm

95% CI half-widths.

Calibration sample size			Risk budgets			Loss resolution
$n$	$\Delta U\uparrow$	$+/=/-$	$\epsilon$	$\Delta U\uparrow$	$+/=/-$	$m$	$\Delta U\uparrow$	$+/=/-$
$50$	$+0.004147$	$15/0/0$	$(0.0050,0.0025)$	$+0.003553$	$15/0/0$	$1$	$0.000000$	$0/15/0$
$100$	$+0.008892$	$15/0/0$	$(0.0100,0.0050)$	$+0.007100$	$15/0/0$	$4$	$+0.002469$	$15/0/0$
$200$	$+0.009841$	$15/0/0$	$(0.0500,0.0200)$	$+0.000450$	$7/8/0$	$16$	$+0.008978$	$15/0/0$
$400$	$+0.006352$	$14/1/0$				$64$	$+0.009952$	$15/0/0$

Appendix

Table 21: Frozen-rule mechanism study.

\Delta U

is the macro mean of

w_{+}

minus raw, higher is better. Counts are positive/equal/negative group means with tolerance

10^{-12}

. Default budgets are

(0.02,0.01)

, and non-size variants use

n=200

. The resolution panel replaces each loss by

\lceil mL\rceil/m

Interval	Dataset	LightGCN	XSimGCL	SIGformer
A Unadjusted	KuaiRec	$+0.056385\,{\scriptstyle\pm\,0.005515}$	$+0.043870\,{\scriptstyle\pm\,0.005005}$	$+0.005452\,{\scriptstyle\pm\,0.002182}$
	KuaiRand	$+0.000396\,{\scriptstyle\pm\,0.000044}$	$+0.000403\,{\scriptstyle\pm\,0.000068}$	$+0.000371\,{\scriptstyle\pm\,0.000021}$
	Music	$+0.000135\,{\scriptstyle\pm\,0.000038}$	$+0.000219\,{\scriptstyle\pm\,0.000263}$	$+0.000008\,{\scriptstyle\pm\,0.000021}$
	Epinions	$+0.001683\,{\scriptstyle\pm\,0.000508}$	$+0.001018\,{\scriptstyle\pm\,0.000468}$	$+0.002200\,{\scriptstyle\pm\,0.001318}$
	ML-25M	$+0.004476\,{\scriptstyle\pm\,0.001040}$	$+0.008837\,{\scriptstyle\pm\,0.002389}$	$+0.022159\,{\scriptstyle\pm\,0.005400}$
B Bonferroni	KuaiRec	$+0.056385\,{\scriptstyle\pm\,0.023284}$	$+0.043870\,{\scriptstyle\pm\,0.021131}$	$+0.005452\,{\scriptstyle\pm\,0.009211}$

Appendix

Table 22: Utility increments and intervals under the new reserved-pool protocol with

n=200

, higher is better. A reports unadjusted 95% intervals. B reports Bonferroni-adjusted intervals for 165 mechanism comparisons. Intervals use approximate

t_{4}

inference over five training-seed means. Both interval groups use the same utility-increment means and report each mean

\pm

the corresponding interval half-width.

Dataset	Model	Ungated $U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$	ACRC (ours) $U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$
KuaiRec	LG	0.5814	0.018759	0.000233	0.5391	0.016667	0.000206
	X	0.5561	0.018546	0.000219	0.5221	0.016547	0.000195
	SIG	0.7621	0.017043	0.000191	0.7543	0.015014	0.000172
KuaiRand	LG	-0.4182	0.020338	0.066605	-0.4846	0.007583	0.009855
	X	-0.4360	0.021081	0.061804	-0.4977	0.009244	0.009944
	SIG	-0.4200	0.020456	0.041176	-0.4555	0.012247	0.009640

Appendix

Table 23: Same-rule ungated ablation. Only activation is set to one, retaining heads, rules, the internal action coin and test users. Higher utility may violate the risk budgets. Colors do not imply joint dominance.

$\lambda$	Pool	$\Delta U_{+0}\uparrow$	Higher/tie/lower	$\Delta U_{+\pi_{0}}\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$
0.25	80	+0.002220	14/1/0	+0.04796	0.008987	0.002876
0.50	80	+0.004828	14/1/0	+0.05685	0.010754	0.003821
1.00	160	+0.010821	15/0/0	+0.07856	0.011896	0.004462
	40	+0.007526	15/0/0	+0.05933	0.012104	0.004410
	80	+0.009959	15/0/0	+0.06914	0.012042	0.004444
2.00	80	+0.009301	15/0/0	+0.07793	0.013235	0.005351

Appendix

Table 24: Macro means over fifteen settings. Counts compare new versus raw utility on the same selected rule, using tolerance

10^{-12}

. Selection and calibration are repeated for each condition.

Dataset	Zero gains	$\overline{d}\uparrow$	$\Delta U\uparrow$	$\frac{\sum_{t}d_{t}\overline{g}_{T,t}}{\sum_{t}d_{t}}$
KuaiRec	128/600	0.12143	0.035643	0.29354
KuaiRand-random	0/600	0.01811	0.000448	0.02473
Amazon-Music	495/600	0.01620	0.000108	0.00665
Epinions	319/600	0.04958	0.001730	0.03489
MovieLens-25M	44/600	0.11281	0.011867	0.10520

Appendix

Table 25: Mechanism decomposition on the existing shared support. Each row contains three backbones, five seeds, and forty role splits.

d

denotes the executed activation increment over raw. The final column weights candidate utility gains by that increment. All entries are descriptive.

$s$	Method	S CPU	C CPU	T CPU/1000	S wall	C wall	T wall/1000
1	ACRC (ours)	68.005	11.264	0.182	68.018	11.264	0.182
	SCoRE/P	476.288	7.163	25.615	476.457	7.165	25.625
	SCoRE/U	1653.562	8.617	30.457	1653.925	8.615	30.460
8	ACRC (ours)	66.932	11.302	0.181	66.940	11.302	0.181
	SCoRE/P	475.630	7.174	25.596	475.704	7.173	25.597
	SCoRE/U	1653.568	8.606	30.539	1653.935	8.605	30.553

Appendix

Table 26: Additional times after frozen outputs, in milliseconds, lower is better. Each entry averages repeated case medians across fifteen real settings.

S=400

C=200

, with application normalized to one thousand users.

Method	LightGCN			XSimGCL			SIGformer
	$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$	$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$	$U\uparrow$	$R_{U}\downarrow$	$R_{R}\downarrow$
Top20	0.2850	0.00000	0.00000	0.2876	0.00000	0.00000	0.1662	0.00000	0.00000
PerK-PDCG	0.2720	0.02626	0.07102	0.2703	0.04676	0.12140	0.1598	0.01043	0.02897
UMC-top20	0.2402	0.12743	0.03786	0.1852	0.15663	0.04963	0.1774	0.08436	0.02729
raw	0.3249	0.01778	0.00607	0.3343	0.01671	0.00571	0.2447	0.01526	0.00551
RB	0.3275	0.01886	0.00644	0.3353	0.01705	0.00583	0.2458	0.01550	0.00560

Appendix

Table 27: New-feedback utility and both actual downside risks. Native recommenders retain their objectives. The LightGCN empirical utility downside exceeds

0.02

, illustrating why the expectation guarantee is not a guarantee for a finite empirical average. Red and blue mark the best and second-best distinct mean for each metric within each backbone group. Ties share a color. Colors do not indicate statistical significance or joint superiority.

Comparator	LightGCN	XSimGCL	SIGformer
SCoRE/PerK UR	$+0.04632\,{\scriptstyle\pm\,0.00331}$	$+0.05489\,{\scriptstyle\pm\,0.01187}$	$+0.09163\,{\scriptstyle\pm\,0.00821}$
RB	$+0.00372\,{\scriptstyle\pm\,0.00086}$	$+0.00659\,{\scriptstyle\pm\,0.00278}$	$+0.01090\,{\scriptstyle\pm\,0.00258}$
SCoRE/UMC UR	$+0.04907\,{\scriptstyle\pm\,0.00422}$	$+0.05649\,{\scriptstyle\pm\,0.01070}$	$+0.08867\,{\scriptstyle\pm\,0.00927}$
Top20	$+0.04620\,{\scriptstyle\pm\,0.00349}$	$+0.05424\,{\scriptstyle\pm\,0.01068}$	$+0.09054\,{\scriptstyle\pm\,0.00817}$
raw	$+0.00630\,{\scriptstyle\pm\,0.00082}$	$+0.00756\,{\scriptstyle\pm\,0.00182}$	$+0.01204\,{\scriptstyle\pm\,0.00173}$

Appendix

Table 28: Paired seed differences on new feedback. Intervals use five training-seed differences and are approximate conditional descriptions, not distribution-free utility certificates for arbitrary populations. Values are means

\pm

95% CI half-widths. Each backbone column reports

\Delta U\uparrow

Panel of Figure 6 — Figure 6: Jester confirmation on all eligible users. Paired five-seed 95% intervals separate calibration at fixed filtering from filtering at fixed calibration. The first row is primary. Contrasts are not additive.

Contrast	LightGCN	XSimGCL	SIGformer
Calibration with filter	$+0.009405\,{\scriptstyle\pm\,0.001445}$	$+0.007430\,{\scriptstyle\pm\,0.000523}$	$+0.016240\,{\scriptstyle\pm\,0.002571}$
Calibration without filter	$+0.009499\,{\scriptstyle\pm\,0.001608}$	$+0.007825\,{\scriptstyle\pm\,0.000313}$	$+0.015798\,{\scriptstyle\pm\,0.001691}$
Filter at new calibrator	$+0.000699\,{\scriptstyle\pm\,0.000644}$	$+0.001065\,{\scriptstyle\pm\,0.000477}$	$+0.004084\,{\scriptstyle\pm\,0.003703}$

Appendix

Table 29: Prespecified confirmation and secondary paired effects on all eligible Jester users. Approximate 95% intervals use five seed means, with forty splits averaged within seed. Values are means

\pm

95% CI half-widths. Each backbone column reports

\Delta U\uparrow

Metric	Calibrator	LightGCN	XSimGCL	SIGformer
$R_{U}\downarrow$	raw	$0.012164\,{\scriptstyle\pm\,0.000581}$	$0.011597\,{\scriptstyle\pm\,0.000256}$	$0.013532\,{\scriptstyle\pm\,0.000750}$
$R_{U}\downarrow$	ACRC (ours)	$0.014300\,{\scriptstyle\pm\,0.000420}$	$0.013399\,{\scriptstyle\pm\,0.000275}$	$0.015838\,{\scriptstyle\pm\,0.000601}$
$R_{R}\downarrow$	raw	$0.006513\,{\scriptstyle\pm\,0.000147}$	$0.006141\,{\scriptstyle\pm\,0.000090}$	$0.006743\,{\scriptstyle\pm\,0.000296}$
$R_{R}\downarrow$	ACRC (ours)	$0.007682\,{\scriptstyle\pm\,0.000221}$	$0.007089\,{\scriptstyle\pm\,0.000075}$	$0.007917\,{\scriptstyle\pm\,0.000398}$

Appendix

Table 30: Risk-budget use for the same filtered rule. Lower risks are better. Intervals describe training-seed variation and do not replace the expected-risk theorem. Values are means

\pm

95% CI half-widths. Red and blue mark the best and second-best distinct mean for each metric within each backbone group. Ties share a color. Colors do not indicate statistical significance or joint superiority.

Panel of Figure 7 — Figure 7: Complete-pipeline utility comparisons on the Jester cohort supported by original PerK. Differences subtract each comparator from the default new-calibrator procedure without the optional filter. Points and bars show five-seed means and approximate pointwise 95% $t_{4}$ intervals, with forty splits averaged within seed. The panels share one linear scale and retain all seven comparators. These supplementary comparisons include differences in prediction heads and list rules, so the gains cannot be attributed solely to calibration. They do not compare matched empirical risks or constitute new prespecified confirmation tests. SCoRE U uses one utility-risk call and UR intersects two unmodified calls.

Comparator	LightGCN	XSimGCL	SIGformer
Top20	$+0.06367\,{\scriptstyle\pm\,0.00192}$	$+0.05505\,{\scriptstyle\pm\,0.00182}$	$+0.10924\,{\scriptstyle\pm\,0.01395}$
PerK-PDCG	$+0.13470\,{\scriptstyle\pm\,0.00174}$	$+0.12270\,{\scriptstyle\pm\,0.00222}$	$+0.18794\,{\scriptstyle\pm\,0.01380}$
UMC-20	$+0.12442\,{\scriptstyle\pm\,0.01136}$	$+0.02839\,{\scriptstyle\pm\,0.00430}$	$+0.07691\,{\scriptstyle\pm\,0.07082}$
SCoRE / PerK U	$+0.06504\,{\scriptstyle\pm\,0.00233}$	$+0.04618\,{\scriptstyle\pm\,0.00166}$	$+0.11642\,{\scriptstyle\pm\,0.01293}$
SCoRE / PerK UR	$+0.06462\,{\scriptstyle\pm\,0.00201}$	$+0.05588\,{\scriptstyle\pm\,0.00177}$	$+0.11054\,{\scriptstyle\pm\,0.01412}$
SCoRE / UMC U	$+0.06972\,{\scriptstyle\pm\,0.00171}$	$+0.04997\,{\scriptstyle\pm\,0.00244}$	$+0.09784\,{\scriptstyle\pm\,0.01482}$

Appendix

Table 31: Paired utility differences between the default unfiltered new-calibrator procedure and external methods on the supported cohort. Approximate five-seed intervals are supplementary pointwise summaries, not new prespecified confirmation tests. Values are means

\pm

95% CI half-widths. All five seeds have positive differences in each of the twenty-one comparisons. Each backbone column reports

\Delta U\uparrow

Method	$U\uparrow$	NDCG $\uparrow$	Recall $\uparrow$	F1 $\uparrow$	Length	$R_{U}\downarrow$	$R_{R}\downarrow$
Top20	${\color[rgb]{0,0,1}0.2608}\pm 0.0010$	${\color[rgb]{0,0,1}0.7619}\pm 0.0007$	${\color[rgb]{0,0,1}0.8136}\pm 0.0002$	${\color[rgb]{0,0,1}0.6705}$	$20.000$	${\color[rgb]{1,0,0}0.0000}$	${\color[rgb]{1,0,0}0.0000}$
PerK-PDCG	$0.1897\pm 0.0017$	$0.4930\pm 0.0021$	$0.4627\pm 0.0021$	$0.4501$	$10.903$	$0.1179$	$0.3509$
UMC-20	$0.2000\pm 0.0098$	$0.7201\pm 0.0059$	$0.7971\pm 0.0050$	$0.6568$	$20.000$	$0.1084$	$0.0430$
ACRC (ours)	${\color[rgb]{1,0,0}0.3244}\pm 0.0012$	${\color[rgb]{1,0,0}0.8030}\pm 0.0010$	${\color[rgb]{1,0,0}0.8388}\pm 0.0016$	${\color[rgb]{1,0,0}0.6911}$	$19.989$	$0.0144$	$0.0078$
SCoRE / PerK UR	$0.2598\pm 0.0010$	$0.7587\pm 0.0007$	$0.8085\pm 0.0002$	$0.6686$	$19.853$	${\color[rgb]{0,0,1}0.0022}$	$0.0052$
SCoRE / UMC UR	$0.2583\pm 0.0008$	$0.7602\pm 0.0006$	$0.8131\pm 0.0003$	$0.6700$	$20.000$	$0.0046$	${\color[rgb]{0,0,1}0.0017}$

Appendix

Table 32: Jester, LightGCN, PerK-supported cohort. The first three metrics show seed mean and standard deviation, the rest show means. Red and blue rank reported means at the displayed precision, not significance or certification.

Conformal Risk Control

Conformal risk control uses calibration data to choose prediction or action thresholds with finite-sample guarantees on expected error or other risks.

See all 64 papers

Conformal Tradeoffs: Operational Profiles Beyond Coverage
Feb 20, 2026Petrus H. ZwartConformal Prediction Selective Prediction
Center for Advanced Mathematics in Energy Research Applications, Berkeley Synchrotron Infrared Structural Biology Program, & Molecular Biophysics and Integrated Bioimaging Division, 1 Cyclotron Road, Berkeley, CA 94720, USA
Replicable Conformal Prediction
Aug 23, 2026Marios Papamichalis, Regina Ruane, Theofanis PapamichalisConformal Prediction ML Reproducibility
Human Nature Lab, Yale University, New Haven, CT 06511, USA. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. · Department of Economics, Yale University, New Haven, CT, USA.
Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics
May 3, 2026Fernando Martin-Maroto, Nabil Abderrahaman, Gonzalo G. de PolaviejaProbability Calibration Uncertainty Calibration
Mathematics of Behavior and Intelligence Lab. Champalimaud Research. Avenida de Brasília, Lisbon · Algebraic AI Calle Castelló, 117, Madrid

Feb 20, 2026J/K move · Enter open · S save

Conformal Tradeoffs: Operational Profiles Beyond Coverage

Petrus H. Zwart

Center for Advanced Mathematics in Energy Research Applications, Berkeley Synchrotron Infrared Structural Biology Program, & Molecular Biophysics and Integrated Bioimaging Division, 1 Cyclotron Road, Berkeley, CA 94720, USA

Conformal Prediction Selective Prediction

Open paper AskPDF arXiv

Conformal prediction gives exact finite-sample coverage guarantees under exchangeability, but deployed systems are judged by more than coverage alone. For a fixed calibrated rule reused over a finite operational window, stakeholders also care about deployment-facing quantities such as commitment frequency, deferral, and decisive error exposure. These are not determined by coverage: calibration choices with similar coverage can still induce materially different operational profiles. We study this characterization gap in a scoped setting: binary split conformal prediction under exchangeability with a fixed deployed rule. We introduce the Small-Sample Beta Correction (SSBC) which gives finite-sample coverage semantics for the deployed rule: it inverts the Beta/Beta--Binomial law governing calibration-conditional coverage to map a user request

(α^\star,δ)

to the least conservative calibration grid point with calibration-conditional PAC semantics for the realized deployed rule. Calibrate-and-Audit then fixes the rule by calibration and uses an independent audit split to estimate the induced region--class label table, a reusable summary from which deployment-facing Key Performance Indicators (KPIs) follow by projection. Under this design, fixed operational rates admit exact finite-sample Binomial inference, while Beta--Binomial envelopes serve as practical predictive summaries for future windows. The induced partition also exposes regime boundaries, Pareto-relevant tradeoffs, and inverse-pricing questions for fixed downstream conventions. Simulations validate the SSBC semantics and compare audit-based summaries with leave-one-out planning proxies; molecular toxicity data provide an audit-based empirical example, and a solubility case study illustrates scenario planning once coverage semantics are fixed.

Abstract

Conformal Risk Control

Explore similar work

Conformal Tradeoffs: Operational Profiles Beyond Coverage

Replicable Conformal Prediction

Beyond ECE: Calibrated Size Ratio, Risk Assessment, and Confidence-Weighted Metrics