RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning
Authors: Shihong Huang, Shengjie Wang, Hong Ma, Zhou Xu
Organizations: Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China
The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.
Figures & tables
Figure 1: Comparison of LLM model-selection structures: (a) one-shot routing, (b) fixed-order cascading with quality-based stopping, (c) estimator-based cascade routing, and (d) RLCascadeRouter, which jointly learns “stop” and model actions through a unified MDP policy.
Figure 2: Illustration of the prediction–decision mismatch. A lower prediction loss can reverse the correct decision, whereas a higher prediction loss can preserve it.
Figure 3: Cascade Policy Network. (a) The Complementarity Encoder (CE) contextualizes state-conditioned action embeddings and scores the remaining models. (b) The Value-Aware Stopper (VAS) aggregates the action context to estimate the STOP/CONTINUE preference and state value. Their outputs form the unified masked routing policy.
Mathematics
Code
Knowledge
IF
Tool
Overall
Method
AIME
LMB
LCB
SWE
GPQA
HLE
MMLU-Pro
SimpleQA
ArenaHard
Tau2
Avg. ↑
Cost ↓
Single-model baselines
GPT-5
83.33
78.38
84.54
16.00
88.33
25.97
87.22
48.00
70.35
69.05
65.12
124.81
Gemini-2.5-Pro
77.78
40.54
76.97
36.00
85.00
21.02
87.67
54.01
77.21
39.29
59.55
233.97
Qwen3-235B-Thinking
72.22
48.65
75.71
20.00
80.00
8.19
80.56
49.31
78.10
50.00
56.27
36.84
DeepSeek-R1-0528
72.22
72.97
76.03
25.33
78.33
17.00
84.67
28.66
65.93
34.52
55.57
60.75
Table 1: Performance comparison of individual LLMs and routing methods in the performance-cost setting of LLMRouterBench. Bold and gray-shaded values denote the best and second-best results in each column.
Figure 4: The non-dominated operating points of RLCascadeRouter and Avengers-Pro, together with the individual LLMs.
Setting
AIME
LMB
LCB
SWE
GPQA
HLE
MMLU-Pro
SimpleQA
ArenaHard
Avg.
Cost
Unseen Selection
A: Initial pool
87.50
75.00
93.42
23.53
90.57
26.08
88.16
51.93
77.11
59.32
82.74
0.00
B: Unseen replacement
75.00
64.29
81.58
21.01
66.04
11.42
85.51
50.46
74.67
53.97
72.76
98.18
C: Replacement training
87.50
71.43
93.42
35.29
86.79
19.75
88.65
57.47
76.00
60.61
174.76
97.59
Table 2: Generalization to unseen models.
Variant
α=1.0
α=0.5
α=0.2
Perf. ↑
Cost ↓
Perf. ↑
Cost ↓
Perf. ↑
Cost ↓
Full
68.81
180.22
61.91
28.06
58.02
10.13
w/o CE
68.30
152.60
60.32
21.05
56.81
7.23
w/o VAS
67.43
191.56
60.67
15.80
58.62
9.97
QE Stop
67.70
135.76
61.23
27.43
58.02
10.13
Table 3: Policy ablations and evaluator-based stopping control under three performance-cost preferences. Bold denotes the best result in each column.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Gemini 2.5 Flash is Google’s efficiency-oriented multimodal thinking model for low-latency and high-volume workloads. It handles text, images, audio, video, and long-context inputs, and supports configurable thinking, function calling, code execution, and grounded retrieval. The model is suitable for general question answering, document understanding, routine reasoning, and agentic tasks where throughput and cost efficiency are important.
Appendix
Table 4: Description of Gemini-2.5-Flash .
GPT-5 Chat is OpenAI’s non-reasoning GPT-5 model previously used for conversational interaction in ChatGPT. It is designed for direct, responsive instruction following, general knowledge assistance, writing, and multi-turn dialogue, with support for function calling and structured outputs. The model is suitable for conversational and general-purpose tasks that benefit from strong language understanding without the additional computation of extended reasoning.
Appendix
Table 5: Description of GPT-5-Chat .
GPT-5 Medium is OpenAI’s GPT-5 reasoning model configured with medium reasoning effort. It is designed for complex reasoning, software coding, mathematics, instruction following, and agentic tool use across domains, and supports long-context inputs, function calling, and structured outputs. The medium setting balances solution quality and inference effort, making it suitable for challenging tasks that require deliberate reasoning without the maximum reasoning budget.
Appendix
Table 6: Description of GPT-5 .
Qwen3-235B-A22B-Thinking-2507 is Qwen’s mixture-of-experts reasoning model with 235 billion total parameters and 22 billion activated parameters, operating exclusively in thinking mode. It uses an extended reasoning process for logical reasoning, mathematics, science, coding, academic problems, instruction following, and tool use. The model is suitable for highly complex tasks where deeper reasoning and long-context understanding justify greater generation length and inference cost.
Appendix
Table 7: Description of Qwen3-235B-Thinking .
DeepSeek-V3-0324 is DeepSeek’s general-purpose mixture-of-experts instruction model operating in a direct, non-thinking mode. It is designed for general question answering, reasoning, software coding, front-end development, instruction following, and tool use. The model is suitable for broad language and coding workloads, especially tasks that require capable and efficient direct generation rather than the longer explicit reasoning process of a dedicated reasoning model.
Appendix
Table 8: Description of DeepSeek-V3 .
DeepSeek-V3.1-Terminus is DeepSeek’s hybrid inference model supporting both thinking and non-thinking behavior. It combines general language and reasoning capabilities with tool use, multi-step search, software coding, and agentic task execution, while improving language consistency and output reliability. The model is suitable for general assistants, code and search agents, and workflows that require flexible reasoning depth with a long context window.
Appendix
Table 9: Description of DeepSeek-V3.1-Terminus .
DeepSeek-R1-0528 is DeepSeek’s dedicated reasoning model that generates an explicit reasoning process prior to producing its final answer. It is designed for difficult mathematics, logic, science, software coding, and other multi-step analytical tasks, with support for function calling and structured JSON output. The model is suitable when reasoning depth and solution quality are more important than latency, output length, or inference cost.
Appendix
Table 10: Description of DeepSeek-R1 .
GLM-4.6 is Z.ai’s general-purpose reasoning and agentic model for coding, long-context processing, search, writing, and tool-based workflows. It supports tool use during inference and a 200K-token context window, with a particular emphasis on software engineering, front-end generation, search agents, and readable long-form writing. The model is suitable for complex multi-step tasks that combine reasoning, code, external tools, and substantial context.
Appendix
Table 11: Description of GLM-4.6 .
Kimi-K2-Instruct-0905 is Moonshot AI’s mixture-of-experts instruction model with one trillion total parameters and 32 billion activated parameters, operating as a direct-response model without extended thinking. It is optimized for agentic coding, tool calling, front-end development, general chat, and autonomous problem-solving, with a 256K-token context window. The model is suitable for long-horizon coding and agent workflows that require capable, responsive tool use.
Appendix
Table 12: Description of Kimi-K2 .
Intern-S1 is InternLM’s multimodal mixture-of-experts reasoning model specialized for scientific problem-solving while retaining strong general language and vision capabilities. It is designed for scientific reasoning across chemistry, materials science, life science, and earth science, including molecular formulas, chemical structures, protein sequences, synthesis planning, and physical signals. The model is suitable for research-oriented questions and multimodal scientific tasks requiring specialized domain knowledge.
Appendix
Table 13: Description of Intern-S1 .
Gemini 2.5 Pro is Google’s high-capability multimodal thinking model for complex problems in mathematics, science, coding, and analytical reasoning. It handles text, images, audio, video, documents, and extended contexts, and supports function calling, code execution, and grounded retrieval. The model is suitable for difficult multi-step tasks, large document or codebase analysis, and workloads that prioritize answer quality over latency.
Appendix
Table 14: Description of Gemini-2.5-Pro .
Claude Sonnet 4 is Anthropic’s hybrid reasoning model with both fast-response and extended-thinking modes. It is designed for precise instruction following, complex reasoning, software coding, agentic workflows, and tool use. The model balances capability and efficiency, making it suitable for general assistants, multi-step problem-solving, codebase navigation, and high-volume tasks that may benefit from deeper reasoning.
Appendix
Table 15: Description of Claude-Sonnet-4 .
Qwen3-235B-A22B-Instruct-2507 is Qwen’s mixture-of-experts instruction model with 235 billion total parameters and 22 billion activated parameters, operating exclusively in non-thinking mode. It provides multilingual instruction following, text comprehension, logical reasoning, mathematics, science, coding, tool use, and long-context understanding. The model is suitable for broad general-purpose and open-ended tasks that require strong direct responses without an extended explicit reasoning trace.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.
Jasper Dekoninck, Maximilian Baader, Martin Vechev
Department of Computer Science, ETH Zurich, Switzerland.
Large language model (LLM) cascades answer easy requests with a small model and escalate selected requests to a larger model. Most routers prioritize examples on which the small model appears uncertain or likely to be wrong. This proxy ignores a decisive fact: escalation is useful only when the large model corrects the small model, and it is harmful when the large model replaces a correct answer with an incorrect one. We introduce Signed Rescue Routing (SRR), a budgeted routing method that predicts these two events separately and ranks requests by their difference. We show that this signed conditional gain is the Bayes-optimal routing score under a fixed escalation budget. SRR requires only the small model's output statistics at deployment and adds a lightweight two-head router. We evaluate SRR with Qwen3-4B and Qwen3-8B on TBD examples from MMLU, HellaSwag, and ARC-Challenge. Across the accuracy-compute curve, SRR reaches an area of TBD, compared with TBD for a learned small-model error predictor and TBD for entropy routing. These results show that predicting incremental value, rather than model uncertainty, is a simple and effective objective for efficient LLM cascades.
Zheyuan Wang, Siyu Li, Peiqiao Song +3
School of Artificial Intelligence, Beijing Normal University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Dalian University of Technology +1
LLM cascades, in which a cheap model defers to an expensive one on low-confidence queries, are widely used to reduce inference cost. Given a pool of models, a practitioner must decide how many models to include and where to set each deferral threshold. We derive first-order optimality conditions showing that, at an optimum, the ratio of expected accuracy gain to expected downstream cost is equal across deferral boundaries. A local search based on these conditions closely matches exhaustive search. We also derive an identity that decomposes the accuracy gain of score-based escalation over random escalation into two AUROC terms. Across five benchmarks and nine deferral scores, with model sequences and thresholds optimized from a pool of eight models, two-model cascades improve mean test-set accuracy over single-model selection by 2.1 to 8.2 percentage points. However, allowing more than two models does not improve mean test-set accuracy in 118 of 135 comparisons across scorers, datasets, and depth caps, and adds at most 0.43 percentage points. To understand the role of deferral scores in depth gains, we conduct counterfactual experiments with simulated confidence scores. When these scores have high AUROC and reflect only whether the current model answered correctly, allowing more than two models improves test-set accuracy on four of five benchmarks. However, these gains do not persist when the scores also reflect query difficulty shared across models, even at the same AUROC. These results suggest that gains from additional depth depend on how well the confidence score separates correct from incorrect answers for the current model compared with later models.