A Unified Approach to Routing and Cascading for LLMs

Jasper Dekoninck; Maximilian Baader; Martin Vechev

cs.CLOct 14, 2024

A Unified Approach to Routing and Cascading for LLMs

Authors: Jasper Dekoninck, Maximilian Baader, Martin Vechev

Organizations: Department of Computer Science, ETH Zurich, Switzerland.

Abstract

The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of three model selection strategies. Routing selects a single model for a query, cascading processes queries through a sequence of models, and cascade routing generalizes both.

	Three Models			Five Models			Eleven Models
	Low	Med	High	Low	Med	High	Low	Med	High
Linear Interp.	$69.619\,211$	$69.619\,211$	$69.619\,211$	$69.223\,137$	$69.223\,137$	$69.223\,137$	$70.506\,406$	$70.506\,406$	$70.506\,406$
Routing	$79.726\,787$	$74.965\,568$	$71.812\,637$	$81.235\,991$	$74.432\,665$	$71.328\,369$	$83.245\,409$	$74.632\,066$	$72.670\,410$
Cascade (Baseline)	$80.863\,873$	$74.641\,304$	$72.475\,432$	$82.332\,691$	$73.029\,091$	$69.526\,597$	$84.480\,300$	$73.643\,526$	$69.788\,751$
Cascade (Ours)	$81.087\,637$	$76.163\,850$	$72.668\,429$	$83.057\,026$	$75.169\,221$	$70.179\,556$	$84.468\,958$	$75.095\,880$	$70.255\,078$
Cascade Routing (Ours)	82.364416	76.553922	73.216015	84.329598	76.313683	72.745103	87.243456	77.568917	74.404521

Table 1: AUC scores in

\%

for different strategies on RouterBench across model and noise levels. All baselines are always worse than the

95\%

confidence intervals of cascade routing. For a discussion on confidence intervals, we refer to App. E .

Panel of Figure 2 — Figure 2: Difference in AUC performance between cascade routing and baseline strategies on RouterBench for various noise values. Red indicates cascade routing is much better, while blue indicates it is only a bit better.

	SWE-Bench		Math+Code	Classification		Open-Form
	10 Models	5 Models	Qwen	Llama	Gemma	Llama	Gemma
Linear Interp.	$40.505\,392$	$38.637\,494$	$39.626\,901$	$74.282\,600$	$61.683\,007$	$79.113\,000$	$54.097\,636$
Routing	$40.470\,854$	$39.395\,796$	$47.463\,307$	$74.922\,474$	$64.44$	$79.318\,910$	$58.395\,591$
Cascade (Baseline)	$38.519\,802$	$45.890\,315$	$37.677\,355$	$74.807\,582$	$54.319\,901$	$79.232\,862$	$56.175\,065$
Cascade (Ours)	$53.20$	$50.94$	$46.763\,934$	$75.46$	$62.786\,118$	$79.224\,978$	$56.184\,755$
Cascade Routing (Ours)	54.121852	51.090586	48.549854	75.516728	64.835861	79.877363	59.656503

Table 2: AUC scores on practical benchmarks. On the left, resp. right, side we show the benchmarks with good, resp. poor, quality estimates. The highest numbers are bolded, and underlined numbers are within the

95\%

confidence intervals of the highest number. For a discussion on confidence intervals, refer to App. E . In App. G , we present benchmark-specific AUC values for results averaged over several benchmarks.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Quality		Cost
	$\sigma_{\text{before}}$	$\sigma_{\text{after}}$	$\sigma_{\text{before}}$	$\sigma_{\text{after}}$
Low	0.6	0.3	0.0002	0.00005
Medium	1.6	0.8	0.0004	0.0001
High	2.4	1.2	100	100

Appendix

Table 3: Standard deviations of the noise levels on the RouterBench dataset.

	Three Models			Five Models			Eleven Models
	Low	Med	High	Low	Med	High	Low	Med	High
Cascade Routing (Ours)	$82.37^{+0.31}_{-0.32}$	$76.57^{+0.34}_{-0.35}$	$73.23^{+0.37}_{-0.38}$	$84.33^{+0.29}_{-0.29}$	$76.32^{+0.34}_{-0.37}$	$72.75^{+0.36}_{-0.40}$	$87.24^{+0.23}_{-0.26}$	$77.58^{+0.30}_{-0.33}$	$74.41^{+0.33}_{-0.36}$
$-$ Routing	$2.64^{+0.15}_{-0.16}$	$1.59^{+0.13}_{-0.15}$	$1.40^{+0.15}_{-0.17}$	$3.10^{+0.17}_{-0.15}$	$1.88^{+0.17}_{-0.16}$	$1.41^{+0.17}_{-0.17}$	$4.00^{+0.17}_{-0.21}$	$2.94^{+0.20}_{-0.21}$	$1.73^{+0.19}_{-0.19}$
$-$ Cascade (Baseline)	$1.50^{+0.12}_{-0.12}$	$1.91^{+0.18}_{-0.19}$	$0.74^{+0.19}_{-0.18}$	$2.00^{+0.17}_{-0.15}$	$3.29^{+0.26}_{-0.27}$	$3.22^{+0.24}_{-0.24}$	$2.76^{+0.14}_{-0.14}$	$3.92^{+0.28}_{-0.28}$	$4.61^{+0.28}_{-0.28}$
$-$ Cascade (Ours)	$1.28^{+0.12}_{-0.11}$	$0.39^{+0.15}_{-0.14}$	$0.54^{+0.18}_{-0.17}$	$1.27^{+0.10}_{-0.11}$	$1.14^{+0.20}_{-0.21}$	$2.57^{+0.21}_{-0.26}$	$2.77^{+0.13}_{-0.13}$	$2.46^{+0.22}_{-0.24}$	$4.14^{+0.25}_{-0.27}$

Appendix

Table 4: AUC scores in

\%

for different strategies on RouterBench in the

0

-shot setting with

2\sigma

confidence intervals.

	Three Models			Five Models			Eleven Models
	Low	Med	High	Low	Med	High	Low	Med	High
Cascade Routing (Ours) $83.79^{+0.3}_{-0.33}$	$78.85^{+0.29}_{-0.34}$	$77.1^{+0.32}_{-0.35}$	$85.5^{+0.26}_{-0.26}$	$78.77^{+0.32}_{-0.33}$	$76.74^{+0.34}_{-0.36}$	$88.78^{+0.22}_{-0.22}$	$80.89^{+0.28}_{-0.29}$	$78.03^{+0.3}_{-0.31}$
$-$ Routing	$2.3^{+0.14}_{-0.13}$	$1.64^{+0.15}_{-0.15}$	$1.1^{+0.13}_{-0.14}$	$3.08^{+0.16}_{-0.14}$	$1.94^{+0.16}_{-0.14}$	$1.21^{+0.14}_{-0.15}$	$3.43^{+0.15}_{-0.17}$	$3.13^{+0.19}_{-0.2}$	$1.6^{+0.17}_{-0.16}$
$-$ Cascade (Baseline)	$-0.64^{+0.11}_{-0.1}$	$0.28^{+0.13}_{-0.14}$	$0.22^{+0.16}_{-0.16}$	$1.23^{+0.12}_{-0.12}$	$2.19^{+0.21}_{-0.21}$	$2.83^{+0.24}_{-0.24}$	$1.64^{+0.12}_{-0.13}$	$2.29^{+0.24}_{-0.24}$	$3.09^{+0.27}_{-0.26}$
$-$ Cascade (Ours)	$1.02^{+0.1}_{-0.09}$	$\bf{0.09^{+0.11}_{-0.11}}$	$\bf{0.1^{+0.14}_{-0.14}}$	$1.25^{+0.1}_{-0.09}$	$1.59^{+0.17}_{-0.17}$	$2.45^{+0.21}_{-0.21}$	$2.06^{+0.1}_{-0.1}$	$2.22^{+0.21}_{-0.19}$	$2.95^{+0.23}_{-0.24}$

Appendix

Table 5: AUC scores in

\%

for different strategies on RouterBench in the

5

-shot setting across model and noise levels with

2\sigma

confidence intervals. Bold numbers indicate that the confidence interval contains zero.

	SWE-Bench		Math+Code	Classification			Open-Form
	10 Models	5 Models	Qwen	Llama	Gemma	Mistral	Llama	Gemma	Mistral
Cascade Routing (Ours)	$54.21^{+7.49}_{-7.17}$	$51.20^{+7.44}_{-7.22}$	$48.51^{+2.95}_{-2.99}$	$75.56^{+1.22}_{-1.16}$	$64.89^{+1.36}_{-1.42}$	$65.02^{+1.40}_{-1.24}$	$79.95^{+1.28}_{-1.33}$	$59.70^{+1.62}_{-1.61}$	$58.77^{+1.42}_{-1.50}$
$-$ Routing	$13.65^{+4.50}_{-4.32}$	$11.71^{+4.39}_{-4.14}$	$1.11^{+0.81}_{-0.80}$	$0.60^{+0.32}_{-0.34}$	$\bf{0.39^{+0.50}_{-0.47}}$	$\bf{0.08^{+0.12}_{-0.10}}$	$0.56^{+0.38}_{-0.42}$	$1.26^{+0.57}_{-0.55}$	$\bf{0.02^{+0.05}_{-0.05}}$
$-$ Cascade (Baseline)	$15.36^{+3.90}_{-3.22}$	$5.17^{+1.69}_{-1.63}$	$10.77^{+1.71}_{-1.72}$	$0.71^{+0.30}_{-0.28}$	$10.51^{+0.62}_{-0.61}$	$3.79^{+0.73}_{-0.77}$	$0.65^{+0.23}_{-0.27}$	$3.47^{+0.37}_{-0.40}$	$10.45^{+1.15}_{-1.06}$
$-$ Cascade (Ours)	$\bf{0.83^{+1.90}_{-1.50}}$	$\bf{0.11^{+1.05}_{-1.04}}$	$1.82^{+0.58}_{-0.55}$	$\bf{0.06^{+0.15}_{-0.17}}$	$2.04^{+0.33}_{-0.31}$	$1.67^{+0.41}_{-0.42}$	$0.20^{+0.19}_{-0.19}$	$2.00^{+0.25}_{-0.25}$	$3.06^{+0.65}_{-0.63}$

Appendix

Table 6: AUC scores on the realistic benchmarks with

2\sigma

confidence intervals. Bold numbers indicate that the confidence interval contains zero.

	Low-Noise		Medium-Noise		High-Noise
	AUC (%)	Time (ms)	AUC (%)	Time (ms)	AUC (%)	Time (ms)
Cascade Routing	$87.288\,663$	$15.263\,219$	$77.606\,260$	$9.525\,337$	$74.408\,804$	$13.684\,285$
Slow	$87.299\,149$	$78.883\,860$	$77.608\,258$	$87.722\,173$	$74.395\,604$	$88.757\,837$
Greedy	$85.930\,465$	$1.387\,429$	$77.166\,754$	$1.173\,892$	$74.350\,349$	$0.885\,655$
No-Expect	$85.977\,675$	$4.784\,659$	$77.111\,513$	$2.492\,010$	$74.347\,163$	$2.080\,314$

Appendix

Table 7: AUC scores and average runtime for variations of cascade routing on RouterBench when using all eleven models.

Figure 3: Runtime of cascade routing variants for different numbers of models.

	Llama			Gemma			Mistral
	MMLU	ARC	MixEval	MMLU	ARC	MixEval	MMLU	ARC	MixEval
Linear Interp.	$53.82$	$93.15$	$82.86$	$39.40$	$82.28$	$70.97$	$39.76$	$85.39$	$73.03$
Routing	$55.32$	$93.12$	$82.86$	$40.01$	$83.13$	$73.12$	$40.61$	$85.64$	$74.28$
Cascade (Baseline)	$54.80$	$94.08$	$84.15$	$36.43$	$77.53$	$66.10$	$36.99$	$83.88$	$72.73$
Cascade (Ours)	$55.05$	$94.16$	$84.00$	$37.68$	$79.80$	$70.57$	$37.03$	$86.27$	$74.42$
Cascade Routing (Ours)	$55.40$	$93.90$	$83.91$	$39.93$	$83.74$	$73.16$	$40.56$	$86.52$	$74.64$

Appendix

Table 8: Classification AUC values for each benchmark separately for the experiment performed in Section 5.2 .

	Llama		Gemma		Mistral
	MMLU	GSM8k	MMLU	GSM8k	MMLU	GSM8k
Linear Interp.	$65.64$	$94.43$	$36.52$	$73.86$	$41.40$	$67.84$
Routing	$65.75$	$94.15$	$38.08$	$75.01$	$43.03$	$68.00$
Cascade (Baseline)	$66.07$	$95.17$	$35.76$	$68.44$	$38.88$	$60.82$
Cascade (Ours)	$66.25$	$94.94$	$38.16$	$71.10$	$40.76$	$64.53$
Cascade Routing (Ours)	$66.60$	$94.69$	$40.43$	$75.25$	$42.93$	$68.30$

Appendix

Table 9: Open-form AUC values for each benchmark separately for the experiment performed in Section 5.2 .

Panel of Figure 4 — Figure 4: Quality-cost tradeoff curves for several benchmarks.

	Three Models			Five Models			Eleven Models
	Low	Med	High	Low	Med	High	Low	Med	High
Linear Interp.	$74.210\,281$	$74.210\,281$	$74.210\,281$	$73.823\,577$	$73.823\,577$	$73.823\,577$	$75.160\,194$	$75.160\,194$	$75.160\,194$
Routing	$81.495\,843$	$77.223\,977$	$76.006\,404$	$82.425\,209$	$76.843\,640$	$75.542\,914$	$85.341\,982$	$77.767\,974$	$76.435\,299$
Cascade (Baseline)	$83.163\,363$	$78.58$	$76.89$	$84.271\,977$	$76.593\,376$	$73.918\,097$	$87.143\,259$	$78.603\,027$	$74.942\,672$
Cascade (Ours)	$82.781\,413$	$78.77$	$77.01$	$84.259\,067$	$77.194\,329$	$74.298\,901$	$86.722\,471$	$78.671\,225$	$75.081\,530$
Cascade Routing (Ours)	83.798999	78.858862	77.105493	85.503886	78.781568	76.750747	88.780025	80.900848	78.035322

Appendix

Table 10: AUC scores in

\%

for different strategies on RouterBench across model and noise levels for five-shot evaluation. Highest numbers are bolded, underlined numbers are within the

95\%

confidence intervals of the highest number. For a discussion on confidence intervals, we refer to App. E .

	Classification	Open-Form
Linear Interp.	$63.389\,435$	$53.859\,567$
Routing	$64.89$	$58.71$
Cascade (Baseline)	$61.202\,081$	$48.288\,646$
Cascade (Ours)	$63.311\,541$	$55.507\,380$
Cascade Routing (Ours)	64.970634	58.728561

Appendix

Table 11: AUC scores on several benchmarks for the Mistral model family. Highest numbers are bolded, underlined numbers are within the

95\%

confidence intervals of the highest number. For confidence intervals, see App. E .

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 342 papers

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning
Aug 16, 2026Shihong Huang, Shengjie Wang, Hong Ma +1LLM Routing Language Model Cascading
Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China
Is Escalation Worth It? On the Depth of LLM Cascades
May 7, 2026Dylan BouchardCost-Aware Inference LLM Routing
Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey
Feb 23, 2026Yasmin Moslem, John D. KelleherLLM Routing Efficient Language Model Inference
ADAPT Centre School of Computer Science & Statistics Trinity College Dublin Dublin, Ireland

Aug 16, 2026J/K move · Enter open · S save

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Shihong Huang, Shengjie Wang, Hong Ma, Zhou Xu

Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China

LLM Routing Language Model Cascading Cascaded Inference LLM Inference Efficiency

Open paper AskPDF arXiv

The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a

stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising

stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.

Abstract

LLM Routing

Explore similar work

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Is Escalation Worth It? On the Depth of LLM Cascades

Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey