RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Shihong Huang; Shengjie Wang; Hong Ma; Zhou Xu

cs.AIAug 16, 2026

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Authors: Shihong Huang, Shengjie Wang, Hong Ma, Zhou Xu

Organizations: Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China

Abstract

The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.

Figures & tables

Figure 1: Comparison of LLM model-selection structures: (a) one-shot routing, (b) fixed-order cascading with quality-based stopping, (c) estimator-based cascade routing, and (d) RLCascadeRouter, which jointly learns “stop” and model actions through a unified MDP policy.

Figure 2: Illustration of the prediction–decision mismatch. A lower prediction loss can reverse the correct decision, whereas a higher prediction loss can preserve it.

Figure 3: Cascade Policy Network. (a) The Complementarity Encoder (CE) contextualizes state-conditioned action embeddings and scores the remaining models. (b) The Value-Aware Stopper (VAS) aggregates the action context to estimate the STOP/CONTINUE preference and state value. Their outputs form the unified masked routing policy.

Single-model baselines
	Mathematics		Code		Knowledge				IF	Tool	Overall
Method	AIME	LMB	LCB	SWE	GPQA	HLE	MMLU-Pro	SimpleQA	ArenaHard	Tau2	Avg. $\uparrow$	Cost $\downarrow$
GPT-5	83.33	78.38	84.54	16.00	88.33	25.97	87.22	48.00	70.35	69.05	65.12	124.81
Gemini-2.5-Pro	77.78	40.54	76.97	36.00	85.00	21.02	87.67	54.01	77.21	39.29	59.55	233.97
Qwen3-235B-Thinking	72.22	48.65	75.71	20.00	80.00	8.19	80.56	49.31	78.10	50.00	56.27	36.84
DeepSeek-R1-0528	72.22	72.97	76.03	25.33	78.33	17.00	84.67	28.66	65.93	34.52	55.57	60.75

Table 1: Performance comparison of individual LLMs and routing methods in the performance-cost setting of LLMRouterBench. Bold and gray-shaded values denote the best and second-best results in each column.

Figure 4: The non-dominated operating points of RLCascadeRouter and Avengers-Pro, together with the individual LLMs.

Setting	AIME	LMB	LCB	SWE	GPQA	HLE	MMLU-Pro	SimpleQA	ArenaHard	Avg.	Cost	Unseen Selection
A: Initial pool	87.50	75.00	93.42	23.53	90.57	26.08	88.16	51.93	77.11	59.32	82.74	0.00
B: Unseen replacement	75.00	64.29	81.58	21.01	66.04	11.42	85.51	50.46	74.67	53.97	72.76	98.18
C: Replacement training	87.50	71.43	93.42	35.29	86.79	19.75	88.65	57.47	76.00	60.61	174.76	97.59

Table 2: Generalization to unseen models.

Variant	$\alpha=1.0$		$\alpha=0.5$		$\alpha=0.2$
Variant	Perf. $\uparrow$	Cost $\downarrow$	Perf. $\uparrow$	Cost $\downarrow$	Perf. $\uparrow$	Cost $\downarrow$
Full	68.81	180.22	61.91	28.06	58.02	10.13
w/o CE	68.30	152.60	60.32	21.05	56.81	7.23
w/o VAS	67.43	191.56	60.67	15.80	58.62	9.97
QE Stop	67.70	135.76	61.23	27.43	58.02	10.13

Table 3: Policy ablations and evaluator-based stopping control under three performance-cost preferences. Bold denotes the best result in each column.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Gemini 2.5 Flash is Google’s efficiency-oriented multimodal thinking model for low-latency and high-volume workloads. It handles text, images, audio, video, and long-context inputs, and supports configurable thinking, function calling, code execution, and grounded retrieval. The model is suitable for general question answering, document understanding, routine reasoning, and agentic tasks where throughput and cost efficiency are important.

Appendix

Table 4: Description of Gemini-2.5-Flash .

GPT-5 Chat is OpenAI’s non-reasoning GPT-5 model previously used for conversational interaction in ChatGPT. It is designed for direct, responsive instruction following, general knowledge assistance, writing, and multi-turn dialogue, with support for function calling and structured outputs. The model is suitable for conversational and general-purpose tasks that benefit from strong language understanding without the additional computation of extended reasoning.

Appendix

Table 5: Description of GPT-5-Chat .

GPT-5 Medium is OpenAI’s GPT-5 reasoning model configured with medium reasoning effort. It is designed for complex reasoning, software coding, mathematics, instruction following, and agentic tool use across domains, and supports long-context inputs, function calling, and structured outputs. The medium setting balances solution quality and inference effort, making it suitable for challenging tasks that require deliberate reasoning without the maximum reasoning budget.

Appendix

Table 6: Description of GPT-5 .

Qwen3-235B-A22B-Thinking-2507 is Qwen’s mixture-of-experts reasoning model with 235 billion total parameters and 22 billion activated parameters, operating exclusively in thinking mode. It uses an extended reasoning process for logical reasoning, mathematics, science, coding, academic problems, instruction following, and tool use. The model is suitable for highly complex tasks where deeper reasoning and long-context understanding justify greater generation length and inference cost.

Appendix

Table 7: Description of Qwen3-235B-Thinking .

DeepSeek-V3-0324 is DeepSeek’s general-purpose mixture-of-experts instruction model operating in a direct, non-thinking mode. It is designed for general question answering, reasoning, software coding, front-end development, instruction following, and tool use. The model is suitable for broad language and coding workloads, especially tasks that require capable and efficient direct generation rather than the longer explicit reasoning process of a dedicated reasoning model.

Appendix

Table 8: Description of DeepSeek-V3 .

DeepSeek-V3.1-Terminus is DeepSeek’s hybrid inference model supporting both thinking and non-thinking behavior. It combines general language and reasoning capabilities with tool use, multi-step search, software coding, and agentic task execution, while improving language consistency and output reliability. The model is suitable for general assistants, code and search agents, and workflows that require flexible reasoning depth with a long context window.

Appendix

Table 9: Description of DeepSeek-V3.1-Terminus .

DeepSeek-R1-0528 is DeepSeek’s dedicated reasoning model that generates an explicit reasoning process prior to producing its final answer. It is designed for difficult mathematics, logic, science, software coding, and other multi-step analytical tasks, with support for function calling and structured JSON output. The model is suitable when reasoning depth and solution quality are more important than latency, output length, or inference cost.

Appendix

Table 10: Description of DeepSeek-R1 .

GLM-4.6 is Z.ai’s general-purpose reasoning and agentic model for coding, long-context processing, search, writing, and tool-based workflows. It supports tool use during inference and a 200K-token context window, with a particular emphasis on software engineering, front-end generation, search agents, and readable long-form writing. The model is suitable for complex multi-step tasks that combine reasoning, code, external tools, and substantial context.

Appendix

Table 11: Description of GLM-4.6 .

Kimi-K2-Instruct-0905 is Moonshot AI’s mixture-of-experts instruction model with one trillion total parameters and 32 billion activated parameters, operating as a direct-response model without extended thinking. It is optimized for agentic coding, tool calling, front-end development, general chat, and autonomous problem-solving, with a 256K-token context window. The model is suitable for long-horizon coding and agent workflows that require capable, responsive tool use.

Appendix

Table 12: Description of Kimi-K2 .

Intern-S1 is InternLM’s multimodal mixture-of-experts reasoning model specialized for scientific problem-solving while retaining strong general language and vision capabilities. It is designed for scientific reasoning across chemistry, materials science, life science, and earth science, including molecular formulas, chemical structures, protein sequences, synthesis planning, and physical signals. The model is suitable for research-oriented questions and multimodal scientific tasks requiring specialized domain knowledge.

Appendix

Table 13: Description of Intern-S1 .

Gemini 2.5 Pro is Google’s high-capability multimodal thinking model for complex problems in mathematics, science, coding, and analytical reasoning. It handles text, images, audio, video, documents, and extended contexts, and supports function calling, code execution, and grounded retrieval. The model is suitable for difficult multi-step tasks, large document or codebase analysis, and workloads that prioritize answer quality over latency.

Appendix

Table 14: Description of Gemini-2.5-Pro .

Claude Sonnet 4 is Anthropic’s hybrid reasoning model with both fast-response and extended-thinking modes. It is designed for precise instruction following, complex reasoning, software coding, agentic workflows, and tool use. The model balances capability and efficiency, making it suitable for general assistants, multi-step problem-solving, codebase navigation, and high-volume tasks that may benefit from deeper reasoning.

Appendix

Table 15: Description of Claude-Sonnet-4 .

Qwen3-235B-A22B-Instruct-2507 is Qwen’s mixture-of-experts instruction model with 235 billion total parameters and 22 billion activated parameters, operating exclusively in non-thinking mode. It provides multilingual instruction following, text comprehension, logical reasoning, mathematics, science, coding, tool use, and long-context understanding. The model is suitable for broad general-purpose and open-ended tasks that require strong direct responses without an extended explicit reasoning trace.

Appendix

Table 16: Description of Qwen3-235B .

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 342 papers

A Unified Approach to Routing and Cascading for LLMs
Oct 14, 2024Jasper Dekoninck, Maximilian Baader, Martin VechevLLM Routing Language Model Cascading
Department of Computer Science, ETH Zurich, Switzerland.
Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference
Sep 7, 2026Zheyuan Wang, Siyu Li, Peiqiao Song +3LLM Inference Efficiency LLM Routing
School of Artificial Intelligence, Beijing Normal University · School of Information Science and Engineering, Chongqing Jiaotong University · School of Computer Science and Technology, Dalian University of Technology +1
Is Escalation Worth It? On the Depth of LLM Cascades
May 7, 2026Dylan BouchardCost-Aware Inference LLM Routing

Oct 14, 2024J/K move · Enter open · S save

A Unified Approach to Routing and Cascading for LLMs

Jasper Dekoninck, Maximilian Baader, Martin Vechev

Department of Computer Science, ETH Zurich, Switzerland.

LLM Routing Language Model Cascading

Open paper AskPDF arXiv

The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.

Abstract

LLM Routing

Explore similar work

A Unified Approach to Routing and Cascading for LLMs

Signed Rescue Routing: Harm-Aware Cascades for Efficient LLM Inference

Is Escalation Worth It? On the Depth of LLM Cascades