RACER: Role-Aligned Competence Estimation for Human-AI Routing

Joshua Strong; Emma Sun; Alexander Capstick; Pramit Saha; Cheng Ouyang; J. Alison Noble

cs.LGSep 18, 2026

RACER: Role-Aligned Competence Estimation for Human-AI Routing

Authors: Joshua Strong, Emma Sun, Alexander Capstick, Pramit Saha, Cheng Ouyang, J. Alison Noble

Organizations: Department of Engineering Science, University of Oxford, UK

Abstract

Learning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this problem to unseen experts using a small context set of expert behavior. Neural context encoders such as L2D-Pop can be query-dependent, but may learn routing shortcuts tied to absolute class coordinates. Identity-Free Deferral (IFD) removes such shortcuts through role-indexed classwise competence profiles, but its estimates are constant within each class and cannot capture instance-level expert specialization. We propose RACER---Role-Aligned Competence Estimation for Routing---a role-relative framework for estimating an unseen expert's competence from context. RACER estimates the posterior-predictive probability that the expert is correct on a query under each candidate class role, then combines these estimates with the model posterior to obtain the Bayes-relevant expert-correctness probability. Nonparametric and neural kernel-pooling estimators use candidate-role relations, shared aggregation, and symmetric summaries, excluding absolute class-identity channels. We prove coherent class-relabelling invariance, derive a Bayes-aligned deferral surrogate, and give a plug-in regret bound relating routing regret to classifier and competence-estimation error. On controlled synthetic benchmarks, including a PathMNIST histopathology context-scaling study with simulated experts, RACER benefits from additional context under hidden subtype dependence and gives the strongest aggregate performance on a separately sampled unseen-expert split in the CIFAR-100 synthetic experiments. On the radiologist and human--AI chest-radiography benchmarks (VinDr-CXR and CheXpert), the RACER family is competitive or best in budget-swept deferral, with calibration results varying across metrics and datasets.

Figures & tables

Method	Context interface	Learned or estimated object	Strength and limitation
L2D-Pop QI/QC (softmax) [ 22 ]	Latent embedding $\psi(C_{e})$ or $\psi(x,C_{e})$	Routing margin $g_{\perp}(x,\psi)-\max_{y}g_{y}(x)$	Strong one-stage routing and, for QC, instance dependence; not explicitly calibrated as expert correctness and not generally identity-free
IFD [ 19 ]	Bayesian classwise profile read through role-indexed summaries	$\Gamma_{\mathrm{IFD}}(x,y,C_{e})=\theta_{e,y}$	Explicit probability-scale competence and coherent invariance; query-independent within class
Racer (Ours)	Role-relative query–context competence estimator	$\Gamma_{\textsc{Racer}{}}(x,y,C_{e})\approx\mathbb{P}(M_{E}=y\mid X=x,Y=y,C_{E}=C_{e})$	Instance-dependent, probability-scale, and identity-free; requires informative context geometry and sufficient support

Table 1: Main conceptual comparison. L2D-Pop is expressive but learns a latent routing margin; IFD is role-aligned but classwise. Racer estimates the query-conditioned role competence object whose marginalization gives expert correctness.

Figure 1: PathMNIST context-scaling utility in the hardest locally complementary synthetic setting (

\rho=1

\lambda_{\mathrm{id}}=1

). The y-axis shows AURSAC gain over the classifier-confidence router; positive values indicate better budget-swept routing than classifier uncertainty alone. The x-axis is normalized context support

B/K

with

K=9

. Shaded bands for RACER methods show one standard deviation over three seeds. We partition the results into (i) unseen OOD experts, (ii) unseen ID experts, and (iii) overall experts.

Figure 2: PathMNIST context-scaling calibration. Mean ECE and Brier score are reported for the plotted probability-output methods. RACER bands show one standard deviation over three seeds. Lower is better. The learned Racer -kernel head has the best Brier score among these plotted methods, while classwise IFD variants have slightly lower final ECE but weaker routing utility.

$K$	Profile	Model acc.	Expert acc.	Oracle
20	Weak	$0.8609\pm 0.0085$	$0.6465\pm 0.0113$	$0.9005\pm 0.0049$
20	Strong	$0.8629\pm 0.0124$	$0.6696\pm 0.0164$	$0.9095\pm 0.0078$
100	Weak	$0.6560\pm 0.0043$	$0.6438\pm 0.0072$	$0.8255\pm 0.0025$
100	Strong	$0.6548\pm 0.0067$	$0.6709\pm 0.0171$	$0.8362\pm 0.0070$

Table 2: OOD reference quantities for the synthetic CIFAR-100 experiments. The oracle is an upper reference for routing and is not a learned method. Variability is the standard deviation over four settings and three seeds.

Figure 3: OOD AURSAC gain over the classifier-confidence router. Across both label-space sizes and both expert-strength profiles, Racer -kernel is the only method with a positive aggregate OOD gain over the classifier-confidence baseline. Bars show means over four settings and three seeds, without inferential error bars. “OOD” denotes the recorded synthetic split.

$K$	Method	Weak OOD	Strong OOD	$\Delta_{\mathrm{OOD}}$	Weak overall	Strong overall
20	Racer -kernel (Ours)	$\mathbf{0.8087}\pm 0.0086$	$\mathbf{0.8199}\pm 0.0110$	$+0.0112$	$\mathbf{0.8072}\pm 0.0032$	$\mathbf{0.8222}\pm 0.0105$
	Classifier-conf. (Baseline)	$0.8007\pm 0.0063$	$0.8130\pm 0.0094$	$+0.0122$	$0.7996\pm 0.0025$	$0.8151\pm 0.0085$
	L2D-Pop (QI) [ 22 ]	$0.7986\pm 0.0071$	$0.8096\pm 0.0091$	$+0.0109$	$0.7973\pm 0.0021$	$0.8124\pm 0.0089$
	L2D-Pop (QC) [ 22 ]	$0.7994\pm 0.0083$	$0.8090\pm 0.0105$	$+0.0096$	$0.7983\pm 0.0032$	$0.8114\pm 0.0094$
	Racer -KNN (Ours)	$0.7995\pm 0.0075$	$0.8082\pm 0.0133$	$+0.0087$	$0.7969\pm 0.0032$	$0.8104\pm 0.0132$
	IFD-MLP [ 19 ]	$0.7936\pm 0.0093$	$0.8063\pm 0.0099$	$+0.0127$	$0.7922\pm 0.0055$	$0.8078\pm 0.0090$

Table 3: Synthetic CIFAR-100 routing utility. Entries are mean

\pm

standard deviation AURSAC over

4

stress-test settings and

3

seeds. The “Classifier-conf.” baseline sweeps a classifier-confidence deferral rule and uses no expert context.

\Delta_{\mathrm{OOD}}

is strong-minus-weak OOD AURSAC.

$K$	Expert profile	Cells	Mean $\Delta_{\mathrm{OOD}}$
20	Weak	12	$+0.0080$
20	Strong	12	$+0.0070$
100	Weak	12	$+0.0046$
100	Strong	12	$+0.0072$

Table 4: Descriptive paired CIFAR-100 nominal OOD AURSAC differences for Racer -kernel versus the classifier-confidence router.

\Delta_{\mathrm{OOD}}

is AURSAC( Racer -kernel ) minus AURSAC(classifier) on unseen-OOD experts. Each block contains four settings and three seeds. The cell count describes the experimental grid, not a count of independent replications.

$K$	Profile	IFD-score		Racer -KNN		Racer -kernel
$K$	Profile	Brier	ECE	Brier	ECE	Brier	ECE
20	Weak	$0.2472$	$0.1109$	$0.2788$	$0.1725$	$\mathbf{0.2281}$	$\mathbf{0.0209}$
20	Strong	$0.2397$	$0.1106$	$0.2727$	$0.1753$	$\mathbf{0.2206}$	$\mathbf{0.0191}$
100	Weak	$0.2504$	$0.1126$	$0.3274$	$0.2500$	$\mathbf{0.2293}$	$\mathbf{0.0135}$
100	Strong	$0.2476$	$0.1357$	$0.3137$	$0.2382$	$\mathbf{0.2205}$	$\mathbf{0.0133}$

Table 5: OOD calibration of expert-correctness probabilities. Lower is better for both Brier score and 15-bin ECE. Values are averaged over all

(\rho,\lambda_{\mathrm{id}})

settings and seeds.

Figure 4: Real-expert chest-radiography results. Panels (i)–(ii) show macro balanced accuracy as the deferral budget increases; curves are means and shaded bands denote standard errors over matched runs. Panels (iii)–(iv) summarize expert-correctness calibration for methods with probability-scale outputs.

Method	Pooled	Seen	ID	OOD	Brier $\downarrow$	ECE $\downarrow$	Ctx.
Racer -C	$0.7019\pm 0.0016$	$0.7373\pm 0.0180$	$0.6359\pm 0.0363$	$0.6841\pm 0.0040$	$0.0297\pm 0.0004$	$0.0177\pm 0.0106$	$0.0542\pm 0.0250$
Racer -kernel	$0.6973\pm 0.0041$	$0.7436\pm 0.0155$	$0.6811\pm 0.0413$	$0.6901\pm 0.0057$	$0.0301\pm 0.0003$	$0.0243\pm 0.0055$	$0.0000\pm 0.0000$
Classifier-conf.	$0.6878\pm 0.0078$	$0.7057\pm 0.0152$	$0.7083\pm 0.0466$	$0.6898\pm 0.0077$	–	–	$0.0000\pm 0.0000$
L2D-QC [ 22 ]	$0.6839\pm 0.0128$	$0.7005\pm 0.0605$	$0.6806\pm 0.0330$	$0.6787\pm 0.0110$	–	–	$0.0000\pm 0.0000$
IFD-MLP [ 19 ]	$0.6761\pm 0.0098$	$0.6749\pm 0.0516$	$0.6868\pm 0.0629$	$0.6832\pm 0.0082$	$0.0324\pm 0.0010$	$0.0378\pm 0.0186$	$0.0000\pm 0.0000$
L2D-QI [ 22 ]	$0.6685\pm 0.0174$	$0.6788\pm 0.0580$	$0.6942\pm 0.0478$	$0.6805\pm 0.0054$	–	–	$0.0000\pm 0.0000$

Table 6: VinDr-CXR real-radiologist deferral. Overall AURSBAC is pooled over all valid rows and therefore corresponds to one global deferral budget. Seen, unseen-ID, and unseen-OOD columns compute group-conditional AURSBAC. Brier and ECE evaluate expert-correctness probabilities and are omitted for methods that only output routing scores. Ctx. gain is the label-NLL improvement of

\pi_{\rho}(y\mid x,C_{e})

over the image-only posterior.

Method	Seen+unseen AURSBAC	Seen AURSBAC	Brier $\downarrow$	ECE $\downarrow$
Racer -C	$0.7243\pm 0.0341$	$0.7244\pm 0.0328$	$0.0947\pm 0.0071$	$0.0418\pm 0.0134$
Racer -kernel	$0.7241\pm 0.0259$	$0.7276\pm 0.0224$	$0.0916\pm 0.0068$	$0.0366\pm 0.0098$
IFD-MLP [ 19 ]	$0.7105\pm 0.0405$	$0.7139\pm 0.0349$	$0.0861\pm 0.0061$	$0.0247\pm 0.0075$
IFD-score [ 19 ]	$0.7013\pm 0.0444$	$0.7092\pm 0.0431$	$0.0849\pm 0.0052$	$0.0293\pm 0.0067$
L2D-QC [ 22 ]	$0.7007\pm 0.0219$	$0.7026\pm 0.0202$	–	–
L2D-QI [ 22 ]	$0.6986\pm 0.0265$	$0.7029\pm 0.0224$	–	–

Table 7: CheXpert multi-expert human–AI deferral. Values are mean

\pm

standard deviation over 27 matched cells. Seen+unseen AURSBAC is the primary metric; Brier and ECE are reported only for probability-output methods and concern each method’s selected expert, not common expert–query pairs.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

$B$	9	25	50	100	200	500	1000
$B/K$	1.0	2.8	5.6	11	22	56	111

Appendix

Table 8: PathMNIST context sizes used in the context-scaling ablation.

Method	$\alpha=1.00$	$\alpha=0.75$	$\alpha=0.50$	$\alpha=0.25$	$\alpha=0.00$	Random
Classifier-conf.	$0.8134\pm 0.0099$	$0.8193\pm 0.0032$	$0.8137\pm 0.0117$	$0.8172\pm 0.0035$	$0.8167\pm 0.0044$	$0.8231\pm 0.0011$
IFD-score [ 19 ]	$0.8295\pm 0.0034$	$0.8357\pm 0.0017$	$0.8345\pm 0.0067$	$0.8324\pm 0.0028$	$0.8342\pm 0.0054$	$0.8081\pm 0.0024$
IFD-MLP [ 19 ]	$0.8114\pm 0.0048$	$0.8179\pm 0.0053$	$0.8151\pm 0.0074$	$0.8156\pm 0.0056$	$0.8193\pm 0.0091$	$0.8089\pm 0.0076$
L2D-Pop (QC) [ 22 ]	$0.8124\pm 0.0038$	$0.8104\pm 0.0201$	$0.8132\pm 0.0036$	$0.8106\pm 0.0230$	$0.8173\pm 0.0040$	$0.8122\pm 0.0021$
Racer -KNN (Ours)	$0.8313\pm 0.0052$	$0.8342\pm 0.0064$	$0.8453\pm 0.0076$	$0.8330\pm 0.0019$	$0.8397\pm 0.0101$	$0.8121\pm 0.0009$
Racer -kernel (Ours)	$0.8432\pm 0.0030$	$0.8583\pm 0.0045$	$0.8563\pm 0.0052$	$0.8573\pm 0.0025$	$0.8593\pm 0.0068$	$0.8136\pm 0.0010$

Appendix

Table 9: PathMNIST representation-geometry ablation. Entries are overall AURSAC mean

\pm

standard deviation over three seeds at

B=500

K=9

\rho=1

, and

\lambda_{\mathrm{id}}=1

. The

\alpha

columns generate hidden subtypes from classifier/auxiliary feature mixtures using ( 15 ). The random column assigns hidden subtypes independently within class. Bold denotes the best method in each column.

Subtype source	Racer -kernel gain	Same-role purity	Brier $\downarrow$	ECE $\downarrow$
$\alpha=1.00$	$+0.0298$	$0.6013$	$0.2017$	$0.0234$
$\alpha=0.75$	$+0.0390$	$0.7609$	$0.1850$	$0.0207$
$\alpha=0.50$	$+0.0426$	$0.7888$	$0.1800$	$0.0184$
$\alpha=0.25$	$+0.0401$	$0.7819$	$0.1838$	$0.0259$
$\alpha=0.00$	$+0.0425$	$0.7853$	$0.1787$	$0.0142$
Random	$-0.0095$	$0.3338$	$0.2247$	$0.0122$

Appendix

Table 10: Mechanism diagnostics for Racer -kernel in the representation-geometry ablation. Gain is AURSAC improvement over the classifier-confidence router. Same-role purity is the fraction of the top-

k

same-true-role context neighbors that share the query’s hidden subtype under the geometry used by Racer -kernel . With three hidden subtypes, chance purity is

1/3

$\lambda$	Kernel AURSAC	Racer -C AURSAC	Kernel Brier	Racer -C Brier	$\Delta$ NLL
0.00	$0.8149\pm 0.0034$	$0.8157\pm 0.0043$	$0.2254\pm 0.0014$	$0.2253\pm 0.0015$	$0.0050\pm 0.0056$
0.25	$0.8173\pm 0.0007$	$0.8160\pm 0.0033$	$0.2252\pm 0.0003$	$0.2251\pm 0.0007$	$-0.0032\pm 0.0170$
0.50	$0.8187\pm 0.0058$	$0.8229\pm 0.0039$	$0.2239\pm 0.0005$	$0.2244\pm 0.0012$	$-0.0212\pm 0.0058$
1.00	$0.8211\pm 0.0071$	$0.8342\pm 0.0040$	$0.2222\pm 0.0018$	$0.2213\pm 0.0008$	$-0.1154\pm 0.0018$

Appendix

Table 11: Context-posterior stress test.

\Delta

label NLL is

\mathrm{NLL}(\pi_{\rho})-\mathrm{NLL}(p_{\omega})

, so negative values indicate that the context-conditioned posterior improves held-out label-posterior fit. At small violation strengths Racer -C is tied with Racer -kernel ; when the context strongly carries case-mix information, Racer -C improves both label fit and routing.

Setting	Fine classes	Train	Val.	Test
$K=20$	20	9,000	1,000	2,000
$K=100$	100	45,000	5,000	10,000

Appendix

Table 12: Synthetic CIFAR-100 label-space splits. The canonical training split is stratified into train/validation subsets.

Superclass	Selected fine class	Superclass	Selected fine class
aquatic_mammals	beaver	fish	shark
flowers	poppy	food_containers	bottle
fruit_and_vegetables	mushroom	household_electrical	clock
household_furniture	chair	insects	beetle
large_carnivores	wolf	large_manmade_outdoor	bridge
large_natural_outdoor	forest	large_omnivores_herbivores	elephant

Appendix

Table 13: The seeded superclass-balanced

K=20

subset. Exactly one fine class is selected from each CIFAR-100 superclass.

Method	Training objective	Score swept at inference
Classifier-conf.	$L_{\mathrm{CE}}$	$-p_{\max}$
IFD-score	$L_{\mathrm{CE}}$	$\widehat{q}-p_{\max}$
IFD-MLP	$L_{\mathrm{aug}}(\mathrm{LCB}_{y}\mathds{1}\{y=k_{\mathrm{best}}\})$	$d-\max_{y}f_{y}$
L2D-Pop QI/QC	$L_{\mathrm{aug}}(\mathds{1}\{m=y\})$	$d-\max_{y}f_{y}$
Racer -KNN	$L_{\mathrm{aug}}(\operatorname{sg}[\Gamma_{y}])$	$\widehat{q}-p_{\max}$
Racer -kernel / DeepSets	$L_{\mathrm{aug}}(\operatorname{sg}[\Gamma_{y}])+L_{\mathrm{comp}}$	$\widehat{q}-p_{\max}$

Appendix

Table 14: Implemented objectives and budget-ranking scores. The auxiliary RACER routing head is trained but its logit is not the primary RACER budget score. Multi-label losses are averaged over pathologies with valid targets.

Protocol	Context source and separation	Selection and evaluation
Synthetic	Training contexts exclude the current minibatch. Evaluation samples labelled context within the validation/test split and removes those record indices from its query set.	Separate training for each $B$ ; checkpoint selection uses unseen-ID validation AURSAC. Test queries are scored using their images and expert contexts.
VinDr-CXR	Disjoint image splits; each expert’s evaluation context and queries are disjoint subsets of images they annotated. Up to 64 context images are used, leaving at least one query.	Leave-one-reader-out agreement target; disagreement between the other readers is masked. R8/R9/R10 define a co-annotation-cluster holdout.
CheXpert	Disjoint record splits; training contexts contain 64 examples. Validation/test contexts occupy half their respective splits, with disjoint query remainders.	One candidate human plus AI experts; the held-out expert is an AI system. Calibration uses the expert selected by each method.

Appendix

Table 15: Information and split protocol in the supplied implementations. Contexts contain historical image labels and expert responses. Query outcomes are used to score predictions, not as competence-head inputs at evaluation.

Group	Classifier acc.	Expert acc.	Expert bal. acc.	Classifier AURSBAC
Overall	$0.9026\pm 0.0004$	$0.9700\pm 0.0003$	$0.8365\pm 0.0032$	$0.6878\pm 0.0078$
Seen	$0.9860\pm 0.0015$	$0.9942\pm 0.0010$	$0.8314\pm 0.0108$	$0.7057\pm 0.0152$
Unseen-ID	$0.9628\pm 0.0060$	$0.9862\pm 0.0041$	$0.8323\pm 0.0739$	$0.7083\pm 0.0466$
Unseen-OOD	$0.8780\pm 0.0006$	$0.9628\pm 0.0004$	$0.8414\pm 0.0014$	$0.6898\pm 0.0077$

Appendix

Table 16: VinDr-CXR expert-strength diagnostics for the classifier-confidence baseline. Values are mean

\pm

standard deviation over the five final seeds. The deferred radiologist is substantially more accurate than the image-only classifier, especially on unseen-OOD cases, which explains why a simple classifier-confidence deferral rule is competitive. Expert balanced accuracy is reported because raw multi-label accuracy is inflated by frequent negative pathology labels.

Top triples		Top pairs
Experts	Count	Experts	Count
R8/R9/R10	5501	R8/R10	5931
R2/R3/R5	115	R8/R9	5621
R1/R2/R3	104	R9/R10	5529
R2/R3/R6	100	R2/R3	1042
R12/R13/R16	99	R1/R2	911

Appendix

Table 17: VinDr-CXR co-annotation structure used to define the inferred R8/R9/R10 OOD expert cluster.

Method	Seen+unseen	Seen	AURSAC	Best acc.	Best budget	Model acc.	Expert acc.	Brier	ECE
Racer -C	$0.7243\pm 0.0341$	$0.7244\pm 0.0328$	$0.9008\pm 0.0068$	$0.9128\pm 0.0082$	$0.1259\pm 0.1305$	$0.9042\pm 0.0127$	$0.8896\pm 0.0084$	$0.0947\pm 0.0071$	$0.0418\pm 0.0134$
Racer -kernel	$0.7241\pm 0.0259$	$0.7276\pm 0.0224$	$0.9030\pm 0.0075$	$0.9149\pm 0.0080$	$0.2041\pm 0.2126$	$0.9029\pm 0.0090$	$0.8907\pm 0.0092$	$0.0916\pm 0.0068$	$0.0366\pm 0.0098$
IFD-MLP [ 19 ]	$0.7105\pm 0.0405$	$0.7139\pm 0.0349$	$0.9012\pm 0.0093$	$0.9083\pm 0.0095$	$0.3133\pm 0.3320$	$0.9011\pm 0.0109$	$0.8947\pm 0.0083$	$0.0861\pm 0.0061$	$0.0247\pm 0.0075$
IFD-score [ 19 ]	$0.7013\pm 0.0444$	$0.7092\pm 0.0431$	$0.9057\pm 0.0060$	$0.9111\pm 0.0062$	$0.2930\pm 0.2901$	$0.8967\pm 0.0102$	$0.8993\pm 0.0065$	$0.0849\pm 0.0052$	$0.0293\pm 0.0067$
L2D-QC [ 22 ]	$0.7007\pm 0.0219$	$0.7026\pm 0.0202$	$0.9014\pm 0.0089$	$0.9119\pm 0.0100$	$0.2200\pm 0.2297$	$0.9030\pm 0.0107$	$0.8861\pm 0.0080$	–	–
L2D-QI [ 22 ]	$0.6986\pm 0.0265$	$0.7029\pm 0.0224$	$0.9035\pm 0.0098$	$0.9118\pm 0.0098$	$0.2174\pm 0.1581$	$0.9012\pm 0.0100$	$0.8905\pm 0.0095$	–	–

Appendix

Table 18: Extended CheXpert multi-expert metrics corresponding to Table 7 . The main text reports the primary balanced-accuracy and calibration columns; secondary accuracy and budget summaries are included here for completeness.

Method	$B=9$	$B=25$	$B=50$	$B=100$	$B=200$	$B=500$	$B=1000$
IFD-score	$-0.0084\pm 0.0023$	$-0.0119\pm 0.0072$	$+0.0000\pm 0.0081$	$-0.0055\pm 0.0078$	$-0.0012\pm 0.0047$	$+0.0025\pm 0.0045$	$+0.0044\pm 0.0052$
IFD-MLP	$-0.0143\pm 0.0056$	$-0.0113\pm 0.0036$	$-0.0045\pm 0.0037$	$-0.0068\pm 0.0020$	$-0.0043\pm 0.0096$	$-0.0020\pm 0.0049$	$+0.0003\pm 0.0062$
L2D-Pop (QI)	$-0.0099\pm 0.0045$	$-0.0191\pm 0.0116$	$-0.0100\pm 0.0023$	$-0.0192\pm 0.0162$	$-0.0112\pm 0.0117$	$-0.0049\pm 0.0042$	$-0.0061\pm 0.0040$
L2D-Pop (QC)	$-0.0060\pm 0.0013$	$-0.0146\pm 0.0088$	$-0.0076\pm 0.0056$	$-0.0061\pm 0.0023$	$-0.0050\pm 0.0013$	$-0.0042\pm 0.0038$	$-0.0092\pm 0.0070$
Racer -KNN	$-0.0086\pm 0.0034$	$-0.0128\pm 0.0081$	$-0.0049\pm 0.0083$	$-0.0039\pm 0.0065$	$-0.0008\pm 0.0102$	$+0.0058\pm 0.0124$	$+0.0147\pm 0.0064$
Racer -kernel	$-0.0080\pm 0.0034$	$\mathbf{-0.0006}\pm 0.0036$	$\mathbf{+0.0098}\pm 0.0144$	$\mathbf{+0.0129}\pm 0.0025$	$\mathbf{+0.0186}\pm 0.0037$	$\mathbf{+0.0251}\pm 0.0068$	$\mathbf{+0.0273}\pm 0.0051$

Appendix

Table 19: PathMNIST context-scaling AURSAC gain over the classifier-confidence router. Entries are mean

\pm

standard deviation over three seeds, paired by seed against classifier confidence at the same context size. The classifier baseline is omitted because its gain is zero by definition. Bold denotes the best non-classifier method at each context size.

Method	Metric	$B=9$	$B=25$	$B=50$	$B=100$	$B=200$	$B=500$	$B=1000$
IFD-score	Brier $\downarrow$	$0.2560\pm 0.0016$	$0.2519\pm 0.0013$	$0.2416\pm 0.0004$	$0.2339\pm 0.0010$	$0.2284\pm 0.0004$	$0.2237\pm 0.0007$	$0.2224\pm 0.0006$
IFD-score	ECE $\downarrow$	$0.1054\pm 0.0046$	$0.1358\pm 0.0027$	$0.1064\pm 0.0003$	$0.0807\pm 0.0059$	$0.0567\pm 0.0015$	$0.0328\pm 0.0003$	$0.0225\pm 0.0018$
IFD-MLP	Brier $\downarrow$	$0.2563\pm 0.0016$	$0.2516\pm 0.0019$	$0.2417\pm 0.0007$	$0.2339\pm 0.0012$	$0.2283\pm 0.0005$	$0.2236\pm 0.0008$	$0.2222\pm 0.0004$
IFD-MLP	ECE $\downarrow$	$0.1055\pm 0.0047$	$0.1351\pm 0.0040$	$0.1067\pm 0.0009$	$0.0808\pm 0.0054$	$0.0567\pm 0.0029$	$0.0320\pm 0.0007$	$\mathbf{0.0214}\pm 0.0026$
Racer -KNN	Brier $\downarrow$	$0.4302\pm 0.0062$	$0.3013\pm 0.0035$	$0.2572\pm 0.0002$	$0.2387\pm 0.0009$	$0.2306\pm 0.0005$	$0.2253\pm 0.0019$	$0.2221\pm 0.0013$
Racer -KNN	ECE $\downarrow$	$0.4252\pm 0.0068$	$0.2171\pm 0.0059$	$0.1441\pm 0.0023$	$0.0978\pm 0.0051$	$0.0732\pm 0.0015$	$0.0633\pm 0.0053$	$0.0572\pm 0.0025$

Appendix

Table 20: PathMNIST context-scaling expert-correctness calibration. Entries are mean

\pm

standard deviation over three seeds. Brier score and 15-bin ECE are reported only for methods that output probability-scale expert-correctness estimates. Bold denotes the best value at each context size for the corresponding metric.

Learning to Defer

Learning to defer trains AI systems to decide when to predict autonomously and when to route cases to human or machine experts.

See all 26 papers

May 8, 2026

MPD $^2$ -Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosis

Learning-to-defer (L2D) can make glaucoma screening safer by routing difficult/uncertain cases to humans, yet standard formulations overlook expert availability, heterogeneous readers behavior, workload imbalance, asymmetric diagnostic harm, case difficulty from morphology and deployment shift. We introduce MPD

^2

-Router, a mask-aware multi-expert deferral framework that recasts ophthalmic triage as constrained human--AI routing: whether to defer and to which available expert. It couples a dual-head deferral/allocation policy with mask-aware Gumbel--sigmoid gating that strictly enforces per-sample availability, and fuses uncertainty, morphology, image-quality, and OOD signals. Training uses an asymmetric cost-sensitive objective with an augmented-Lagrangian deferral budget, a group-specific distribution prior, and a rank-majorization JS regularizer that jointly prevent expert collapse without forcing uniform allocation. Across three cross-national glaucoma cohorts (REFUGE, CHAKSU, ORIGA) with a frozen REFUGE-trained backbone, MPD

^2

-Router substantially lowers clinical cost and improves MCC over AI-only at a moderate deferral rate. It is Pareto-optimal in F1--MCC--cost, robust under cross-domain shift, and yields balanced expert utilization.

Wenxin Zhan

Baruch college

AI-Assisted Decision Making Learning to Defer Medical Diagnosis Decision-Focused Learning Glaucoma Detection Ophthalmology

Ask PDF

Mar 15, 2026

Consistent Learning-to-Defer with Expert-Conditional Advice

We introduce learning-to-defer with advice, a new setting in which a system learns both which expert should handle an input and which information that expert should receive. The policy acts before observing advice and minimizes prediction loss plus expert and acquisition fees. Training may reveal all advice sources or only a subset on each record. We show that composing routing and advice losses can be inconsistent and that augmented training can incur avoidable estimation variance. We propose reference conditional classification, which jointly learns a fixed expert's error probability and each action's error conditional on that expert's success or failure. The reference outcome is observed on every labeled record, so records with missing advice still inform the estimates. We prove a consistency bound for bounded task losses, show that limited training preserves the full population objective, and establish variance reductions for the analyzed fits. On FEVER and Sensitive, full training has the lowest mean dev loss at every evaluated fee, while limited training improves on full training at most matched acquisition budgets.

Yannis Montreuil, Leïna Montreuil, Axel Carlier +2

School of Computing National University of Singapore Singapore, 117418 · Département de Mathématiques Sorbonne University Paris, 75005, France · Fédération ENAC ISAE-SUPAERO ONERA Université de Toulouse Toulouse, 31555, France +1

Learning to Defer Cost-Sensitive Learning

Ask PDF

Feb 20, 2026

RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

Efficiently routing queries to the optimal large language model (LLM) is crucial for optimizing the cost-performance trade-off in multi-model systems. However, most existing routers rely on single-model selection, making them susceptible to misrouting. In this work, we formulate LLM routing as the

α

-VOR problem to minimize expected set size while controlling the misrouting risk, and propose a novel method -- RACER, extending base routers to output model sets that can be subsequently aggregated for improved output. In particular, RACER constructs nested model sets via augmented scoring and utilizes finite-sample concentration bounds to calibrate a threshold that allows for both variable set sizes and abstention. We theoretically prove that RACER achieves rigorous distribution-free risk control on unseen test data in a post-hoc and model-agnostic manner. Extensive experiments verify our theoretical guarantees and demonstrate that RACER consistently enhances downstream accuracy across a wide range of benchmarks.

Sai Hao, Hao Zeng, Hongxin Wei +1

Southern University of Science and Technology, China · The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

Set-Valued Prediction LLM Routing Adaptive Model Routing

Ask PDF

Abstract

Learning to Defer

Explore similar work

MPD2^22-Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosis

Consistent Learning-to-Defer with Expert-Conditional Advice

RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

MPD $^2$ -Router: Mask-aware Multi-expert Prior-regularized Dual-head Deferral Router in Glaucoma Screening and Diagnosis