Model routing determines whether to use an accurate black-box model or a simpler surrogate that approximates it at lower cost or greater interpretability. In deployment settings, practitioners often wish to restrict surrogate use to inputs where its degradation relative to a reference model is controlled. We study proactive (input-based) routing, in which a lightweight gate selects the model before either runs, enabling distribution-free control of the fraction of routed inputs whose degradation exceeds a tolerance τ. The gate is trained to distinguish safe from unsafe inputs, and a routing threshold is chosen via Clopper-Pearson conformal calibration on a held-out set, guaranteeing that the routed-set violation rate is at most α with probability 1-δ. We derive a feasibility condition linking safe routing to the base safe rate π and risk budget α, along with sufficient AUC thresholds ensuring that feasible routing exists. Across 35 OpenML datasets and multiple black-box model families, gate-based conformal routing maintains controlled violation while achieving substantially higher coverage than regression conformal and naive baselines. We further show that probabilistic calibration primarily affects routing efficiency rather than distribution-free validity.
Figures & tables
Figure 1: Feasibility landscape: critical AUC Φc(π,α) as a function of base safe rate π and risk budget α . Darker regions require higher AUC for guaranteed feasibility. Stars mark our six τ operating points at α=0.2 , colored by empirical outcome: green if routing achieves positive coverage, red if coverage is zero. At τ≤0 (low π ), the gate falls in the hard region and correctly abstains. At τ=0.5 and τ=1.0 , routing succeeds despite AUC falling below Φc , illustrating that the sufficient condition is conservative. At τ=2.0 , AUC exceeds Φc and feasibility is guaranteed.
τ
π
AUC
ECE
Φc
Φc∗
Conf. ( α=0.2 )
(med.)
Cov.
Viol.
−1.5
0.13
0.71
0.07
1.00
0.92
0.000
—
−1.0
0.16
0.70
0.09
1.00
0.95
0.000
—
0.0
0.43
0.54
0.25
1.00
0.88
0.003
0.03
0.5
0.66
0.57
0.20
1.00
0.76
0.243
0.15
1.0
0.74
0.58
0.17
0.70
0.65
0.446
0.14
Table 1: Gate properties across tolerance levels. π : base safe rate; AUC: gate ranking quality (median [IQR]); ECE: calibration error. Φc / Φc∗ : sufficient AUC thresholds from Theorems 1 / 2 . Coverage and mean violation at α=0.2 averaged over 35 datasets.
α
Gate Conf.
Reg. Conf.
Naive
0.10
7/34 0 (21%)
31/50 0 (62%)
151/172 (88%)
0.20
8/66 0 (12%)
57/102 (56%)
128/172 (74%)
0.30
6/82 00 (7%)
80/144 (56%)
106/172 (62%)
0.50
8/112 0 (7%)
64/181 (35%)
0 62/172 (36%)
Table 2: Fraction of (dataset, τ ) pairs where the realized violation rate exceeds target α . Gate conformal is broadly consistent with δ=0.10 . Baselines violate substantially more often.
Figure 2: Coverage vs. violation rate at α=0.2 . Each point corresponds to a τ value averaged over 35 datasets. Gate conformal remains below α . Regression conformal and naive operate above it.
Figure 3: Left: gate ECE varies from 0.07 to 0.28 across τ . Right: despite high ECE, conformal violation remains below α=0.1 . Naive violations are 4–8 × higher.
τ
α
#DS
Higher Cov.
Violates α
Gate
Reg
Gate
Reg
0.5
0.2
23
10
11
2
15
0.5
0.3
32
16
14
2
18
1.0
0.2
29
18
7
4
11
1.0
0.3
32
20
7
1
12
2.0
0.2
32
18
6
2
7
Table 3: Per-dataset comparison at practical operating points. Gate conformal achieves higher coverage on more datasets while violating α on far fewer.
Figure 4: Fraction of (dataset, τ ) pairs violating target α across the full α range. Gate conformal remains near the δ=0.10 line. Baselines converge only at very permissive α .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
τ
α
Gate Cov.
Gate Viol.
Reg Cov.
Reg Viol.
0.5
0.10
0.096
0.021
0.073
0.098
0.5
0.20
0.243
0.059
0.175
0.187
0.5
0.30
0.431
0.134
0.335
0.320
1.0
0.10
0.132
0.015
0.122
0.064
1.0
0.20
0.446
0.091
0.320
0.160
1.0
0.30
0.638
0.145
0.550
0.253
Appendix
Table 4: Gate conformal vs. regression conformal: mean coverage and violation across 35 datasets at representative operating points.
Comparison
τ , α
Diff.
p -value
n
Gate vs. Naive (violation rate):
0.5, 0.2
− 0.076
0.007
14
1.0, 0.2
− 0.060
0.002
23
2.0, 0.2
− 0.025
0.009
28
Gate vs. Reg. Conformal (coverage):
0.5, 0.2
+ 0.256
0.005
13
Appendix
Table 5: Significance tests (Wilcoxon signed-rank, two-sided). To evaluate operational tradeoffs, tests are conditioned on problem instances where both methods successfully found a valid routing threshold (coverage ¿ 0). Under these conditions, Gate Conformal achieves significantly lower violation rates than the naive baseline and significantly higher coverage than regression conformal.
τ
π
AUC (median)
ECE
-1.5
0.135
0.50
0.137
-1.0
0.173
0.50
0.170
0.0
0.459
0.51
0.302
0.5
0.639
0.58
0.213
1.0
0.717
0.56
0.160
2.0
0.799
0.57
0.136
Appendix
Table 6: Gate properties for XGBoost vs. Decision Tree across τ . AUC is reported as median across datasets.
τ
π
AUC (median)
ECE
-1.5
0.300
0.688
0.122
-1.0
0.368
0.627
0.154
0.0
0.629
0.544
0.221
0.5
0.723
0.570
0.194
1.0
0.770
0.557
0.153
2.0
0.830
0.535
0.112
Appendix
Table 7: Gate properties for MLP vs. Decision Tree.
Calibration points routed ( n )
k unsafe
50
100
200
300
500
0
0.045
0.023
0.011
0.008
0.005
1
0.076
0.038
0.019
0.013
0.008
5
0.178
0.091
0.046
0.031
0.018
10
0.291
0.150
0.076
0.051
0.031
20
—
0.261
0.133
0.089
0.054
Appendix
Table 8: Clopper–Pearson upper bounds UCB0.10(k,n) for selected calibration set sizes. With n≈300 , the bound is sufficiently tight for practical risk control.
τ
Recalibration
ECE
Coverage
Violation
0.5
None (Platt)
0.215
0.243
0.059
0.5
Beta
0.066
0.243
0.059
0.5
Temperature
0.077
0.243
0.059
0.5
Isotonic
0.050
0.216
0.043
1.0
None (Platt)
0.184
0.446
0.091
1.0
Beta
0.060
0.421
0.085
Appendix
Table 9: Effect of post-hoc recalibration at α=0.2 , averaged over 35 datasets (RF vs. decision tree). Recalibration substantially reduces ECE but does not increase coverage, indicating the efficiency gap is dominated by ranking limitations.
α
Method
Violations
Mean Viol.
Mean Cov.
0.10
Gate Conf.
7/34 (21%)
0.071
0.595
HGBR Conf.
93/119 (78%)
0.385
0.164
Naive 0.5
151/172 (88%)
0.404
0.692
0.20
Gate Conf.
8/66 (12%)
0.129
0.717
HGBR Conf.
128/177 (72%)
0.406
0.249
Naive 0.5
128/172 (74%)
0.404
0.692
Appendix
Table 10: Guarantee check across (dataset,τ) pairs when regression-conformal uses HistGradientBoostingRegressor . For each target α , we report the fraction of pairs whose realized violation rate exceeds α , along with mean violation rate and mean coverage.
AUC
Conf. ( α=0.2 )
τ
π
(med.)
ECE
Cov.
Viol.
−1.5
0.18
0.53
0.06
0.000
—
−1.0
0.19
0.52
0.05
0.000
—
0.0
0.43
0.52
0.16
0.048
0.20
0.5
0.69
0.52
0.14
0.452
0.05
1.0
0.73
0.51
0.11
0.462
0.04
Appendix
Table 11: Gate properties and conformal routing performance across tolerance levels on 22 general-purpose OpenML datasets. Coverage and violation at α=0.2 averaged over datasets with positive coverage.
α
Gate Conf.
Reg. Conf.
Naive
0.10
0/35 (0.0%)
20/62 (32.3%)
64/91 (70.3%)
0.20
1/38 (2.6%)
48/88 (54.5%)
58/91 (63.7%)
0.30
2/43 (4.7%)
50/105 (47.6%)
53/91 (58.2%)
0.50
4/66 (6.1%)
50/112 (44.6%)
39/91 (42.9%)
Appendix
Table 12: Fraction of (dataset, τ ) pairs where the realized violation rate exceeds target α on general-purpose OpenML datasets. Gate conformal maintains strong safety control. Baselines violate substantially more often.
High-fidelity simulations are essential to scientific and engineering design, but can be expensive to run repeatedly. Learned surrogates offer a faster alternative, yet their higher errors may alter downstream decisions. This accuracy-speed tradeoff creates a need to determine whether a surrogate can be used or the full simulator remains necessary. We study decisions determined by whether a scalar quantity of interest lies above or below a fixed threshold. For each input, we use the surrogate when its conformal interval lies entirely on one side of the threshold and route the input to simulation when the interval intersects it. Standard conformal prediction constructs intervals without reference to the downstream decision threshold: even a narrow interval near the threshold can cross it and trigger simulation, whereas a wider interval farther away can remain entirely on one side and require no simulation. We introduce Threshold-Aware Conformal Routing (TACR), which learns an input-dependent scale using a threshold-aware objective that concentrates interval tightness near the decision boundary. Exact split-conformal calibration on held-out data preserves distribution-free marginal coverage, which also upper-bounds the probability of an incorrect threshold decision that is not routed. Across various scientific and engineering datasets, TACR reduces simulator deferrals by 14-75% relative to standard conformal prediction at the same coverage target. Against a variant without threshold-local weighting but with similar predictor accuracy, TACR further reduces deferrals by 10-24% on four datasets. These results show that optimizing interval allocation for routing can reduce simulator calls without weakening the standard conformal guarantee.
Set-based language model routing aims to increase the coverage of the event where at least one model answers a query correctly while controlling the number of inference calls made. Existing risk-controlled routing policies bound the all-miss risk (the probability that every model in the selected set fails) by calibrating a single conformal threshold and applying it to a nested family of model sets. Because each model is scored in isolation, these policies ignore the dependence structure of model failures. Models that share training data or architecture tend to fail on the same queries, so two individually strong models can be jointly redundant while a weaker but complementary model covers the queries that the others miss. To account for this dependence, which existing risk-controlled policies ignore, we propose CopRoute, which orders models by how many new queries each one answers correctly beyond those already selected, decides per query how far down that order to go, and then picks the smallest cutoff that still meets the target risk. We show that CopRoute's ordering attains, on the design data, the best guarantee any efficient fixed-ordering algorithm can give for the average number of calls to the first correct model, and that a separate calibration step certifies the all-miss risk of the deployed sets. On RouterBench and RouterEval, CopRoute attains the fewest certified calls on nine of ten datasets at a target risk of 0.05 and on eight of ten at 0.10, making up to 79 percent fewer calls (median 24 percent) than the state-of-the-art risk-controlled policy at the same certified all-miss risk.
Efficiently routing queries to the optimal large language model (LLM) is crucial for optimizing the cost-performance trade-off in multi-model systems. However, most existing routers rely on single-model selection, making them susceptible to misrouting. In this work, we formulate LLM routing as the α-VOR problem to minimize expected set size while controlling the misrouting risk, and propose a novel method -- RACER, extending base routers to output model sets that can be subsequently aggregated for improved output. In particular, RACER constructs nested model sets via augmented scoring and utilizes finite-sample concentration bounds to calibrate a threshold that allows for both variable set sizes and abstention. We theoretically prove that RACER achieves rigorous distribution-free risk control on unseen test data in a post-hoc and model-agnostic manner. Extensive experiments verify our theoretical guarantees and demonstrate that RACER consistently enhances downstream accuracy across a wide range of benchmarks.
Sai Hao, Hao Zeng, Hongxin Wei +1
Southern University of Science and Technology, China · The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China