RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

Sai Hao; Hao Zeng; Hongxin Wei; Bingyi Jing

cs.LGFeb 20, 2026

RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models

Authors: Sai Hao, Hao Zeng, Hongxin Wei, Bingyi Jing

Organizations: Southern University of Science and Technology, China · The Chinese University of Hong Kong, Shenzhen, China · Shenzhen Loop Area Institute, China

Abstract

Efficiently routing queries to the optimal large language model (LLM) is crucial for optimizing the cost-performance trade-off in multi-model systems. However, most existing routers rely on single-model selection, making them susceptible to misrouting. In this work, we formulate LLM routing as the $α$ -VOR problem to minimize expected set size while controlling the misrouting risk, and propose a novel method -- RACER, extending base routers to output model sets that can be subsequently aggregated for improved output. In particular, RACER constructs nested model sets via augmented scoring and utilizes finite-sample concentration bounds to calibrate a threshold that allows for both variable set sizes and abstention. We theoretically prove that RACER achieves rigorous distribution-free risk control on unseen test data in a post-hoc and model-agnostic manner. Extensive experiments verify our theoretical guarantees and demonstrate that RACER consistently enhances downstream accuracy across a wide range of benchmarks.

Figures & tables

Figure 1: Overview of the RACER paradigm. RACER operates in two phases. Risk Calibration (Left): The calibration module uses a labeled dataset $\mathcal{D}_{\mathrm{cal}}$ and a user-specified risk level $\alpha$ . It augments the standard model space $\mathcal{M}$ with a null model $m_{\emptyset}$ to construct augmented ground truth set $G^{\prime}$ . The threshold $\hat{\lambda}$ is then computed to guarantee risk control. Model Routing (Right): During inference, the paradigm applies the calibrated $\hat{\lambda}$ to the augmented scores of a test query $\bm{x}$ . This generates a prediction set $C_{\hat{\lambda}}(\bm{x})$ . If the set contains only the null model, the system triggers abstention; otherwise, it proceeds to Response Aggregation , where the outputs of the selected standard LLMs are combined via majority voting or weighted aggregation to produce the final prediction $\hat{y}$ .

Panel of Figure 2 — Figure 2: Distributions of risk and size for RACER on CMMLU over 100 independent trials with a target risk level $\alpha=0.1$ . Left : The distribution of risk , where the black dashed line represents the user-specified risk level. Results demonstrate that RACER consistently maintains the risk below the target $\alpha$ for all base routers and non-conformity scores. Right : The distribution of prediction set size . The green and orange boxes represent the router score-gap and inverse irobability non-conformity scores, respectively.

Candidate Models
Method	GSM8K	MMLU	C-MMLU	ARC-C	Average
Chinese-Mistral-7B-v0.1	42.7 $\pm$ 1.4	57.5 $\pm$ 0.8	49.1 $\pm$ 1.0	45.3 $\pm$ 2.4	48.7
Dolphin-2.6-Mistral-7b	54.7 $\pm$ 1.5	60.7 $\pm$ 0.8	44.2 $\pm$ 0.9	53.7 $\pm$ 2.1	53.3
Dolphin-2.9-Llama3-8b	75.0 $\pm$ 1.2	59.6 $\pm$ 0.8	43.9 $\pm$ 0.9	49.8 $\pm$ 2.2	57.1
Meta-Llama-3-8B	47.1 $\pm$ 1.5	64.6 $\pm$ 0.8	51.2 $\pm$ 0.9	50.5 $\pm$ 2.2	53.4
MetaMath-Mistral-7B	75.1 $\pm$ 1.2	59.9 $\pm$ 0.8	44.4 $\pm$ 1.0	48.7 $\pm$ 2.3	57.0

Table 1: Performance comparison on diverse benchmarks. We report the mean and standard deviation (

\pm

std) of accuracy across 100 independent trials . The “Average” column represents the average accuracy across all four benchmarks. “Base” refers to the base routers, while “+ RACER” indicates our proposed paradigm with aggregation (w/ Agg.). RACER-G and RACER-P denote two RACER variants with different non-conformity score definitions, based on the router score-gap and inverse irobability , respectively. The best results in each column are bolded , and the second best are underlined .

Figure 3: Trade-off between computational efficiency and performance. The scatter plot illustrates the reduction in inference overhead (Model Calls Saved) versus the absolute improvement in test accuracy (Accuracy Gain) for RACER relative to the full model ensemble. The concentration of data points in the upper-right region indicates that RACER effectively filters noise to achieve significant savings (up to 58.6%) while simultaneously improving accuracy (up to +4.49%) across various dataset-router configurations.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The prompt template used for extracting model confidence. The placeholders {question} and {answer} are replaced by the input query $\bm{x}_{n+1}$ and the model’s generated response $a_{m}$ , respectively.

Panel of Figure 5 — Figure 5: Distributions of empirical Risk and Size over 100 independent trials with a target risk level $\alpha=0.1$ . The top row displays the risk distribution, where the vertical black dashed line indicates the target risk level $\alpha$ . The bottom row illustrates the distribution of prediction set sizes. The green bars represent the router score-gap non-conformity score, while the orange bars represent the inverse probability non-conformity score. The results empirically demonstrate that RACER strictly controls the risk around the target level across different base routers and benchmarks.

Figure 6: Heatmap of optimal hyperparameters selected on the validation set. The figure displays the grid-search results for the target risk level $\alpha$ (indicated by the cell value and color intensity) and the aggregation method (text annotation) that achieved the highest accuracy on the validation set. The left panel shows configurations for RACER-G (router score-gap), and the right panel for RACER-P (inverse probability). These optimal configurations were applied to the test set to produce the main results in Table 1 .

Figure 7: Performance evaluation of RACER versus full model aggregation. The figure presents the comparison of RACER-G/P (w/ Agg.) against the full model aggregation baseline across varying dataset-router pairs. RACER-P demonstrates superior performance in 10/12 cases, and RACER-G in 8/12 cases, thereby confirming that effectively filtering noisy models leads to more robust predictions than simply aggregating all available models.

Dataset	Method	$\alpha$
Dataset	Method	0.01	0.05	0.10	0.15	0.20	0.30	0.40	0.50
GSM8K	KNNR+RACER-G	0.010	0.048	0.098	0.147	0.197	0.297	0.394	0.495
	KNNR+RACER-P	0.009	0.048	0.098	0.147	0.196	0.296	0.395	0.495
	MLPR+RACER-G	0.010	0.048	0.096	0.145	0.195	0.297	0.396	0.497
	MLPR+RACER-P	0.010	0.048	0.096	0.145	0.195	0.296	0.396	0.497
	RouterDC+RACER-G	0.009	0.048	0.099	0.146	0.195	0.296	0.394	0.493

Appendix

Table 2: Risk across varying target risk levels (

\alpha

). We report the empirical risk achieved by different RACER configurations (combining three base routers with two non-conformity scores) over 100 independent trials. The results span four benchmarks (GSM8K, MMLU, CMMLU, ARC-Challenge) and a range of user-specified risk level

\alpha\in[0.01,0.50]

. The close alignment between the risk (values in the table) and the target

\alpha

(column headers) confirms the validity of the RACER calibration procedure across a wide range of risk levels.

Figure 8: Evolution of average selected model set size with respect to the target risk level $\alpha$ . The plots display the mean set size computed over 100 independent trials across four diverse benchmarks (GSM8K, MMLU, CMMLU, and ARC-Challenge) as the user-specified risk level varies within the range $\alpha\in[0.01,0.3]$ . Different colors distinguish the base routers: Blue for KNNR, Orange for MLPR, and Green for RouterDC. Line styles differentiate the non-conformity scores: solid lines correspond to RACER-G (router score-gap) and dashed lines to RACER-P (inverse probability). Shaded regions indicate the standard deviation. The plots illustrate a monotonic trade-off where higher risk allows for more compact prediction sets, with RouterDC consistently demonstrating superior efficiency (smallest sizes) across a wide range of risk levels.

Explore similar work

WISERouter: LLM Routing with Workload Budget Constraint
Jul 26, 2026Yifei Li, Zihui Gao, Laks V. S. LakshmananLLM Routing Contextual Bandits
The University of British Columbia
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval
Jul 20, 2026Yu-Chien Tang, Jun-Chen Hung, Wen-Chih Peng +1LLM Inference Efficiency Adaptive Model Routing
National Yang Ming Chiao Tung University · Department of Computer Science Hsinchu, Taiwan

Jul 26, 2026J/K move · Enter open · S save

WISERouter: LLM Routing with Workload Budget Constraint

Yifei Li, Zihui Gao, Laks V. S. Lakshmanan

The University of British Columbia

LLM Routing Contextual Bandits

Open paper AskPDF arXiv

Large language models (LLMs) achieve impressive performance across multiple domains, but using the most capable model for every query is prohibitive at scale. LLM routing exploits diversity in model capability and cost by assigning each query to a suitable model to balance utility and budget. Current methods have two limitations: (i) they either use heuristics that do not always enforce the budget constraint or impose a fixed per-query budget that cannot adapt across the workload and leads to suboptimal performance; (ii) they require supervised learning on a dense dataset with statistics for every query-model pair, which is expensive to collect. To address these challenges, we formulate LLM routing as a constrained contextual multi-armed bandit problem and introduce WISERouter (WR for short), a framework that supports offline learning from historical interactions as well as online learning with exploration. We further prove that WR-Online achieves a sublinear regret bound of

O(\sqrt{T})

over a time horizon

T

. Empirical results on RouterBench and SWE-Bench demonstrate that (i) WR-Offline surpasses existing baselines in performance under a fixed budget and adheres more closely to budget constraints, and (ii) WR-Online achieves comparable performance to the baselines, while using substantially less exploration data.

Abstract

Explore similar work

WISERouter: LLM Routing with Workload Budget Constraint

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

VDAR-Router: Adaptive LLMs Routing via Verbalized Query Difficulty Analysis Retrieval