Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.
Figures & tables
Figure 1: Predictors have a larger advantage over self-confidence on more suboptimal models. Each point is a released (orange) or finetuned (blue) model. Empirical suboptimality is the gap to the best observed answer log loss on each task. Excess predictability E is the reduction in correctness log loss achieved by a separately trained 4B predictor over self-confidence. Colors identify training stage; shapes identify the release type. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided p -value for positive correlation. See Section 3.4.1 for experimental details.
Figure 2: Finetuning reduces excess predictability in 55 of 59 model-task pairs. Each point compares one released model with its selected finetuned version; both axes show released minus finetuned values. Δδ is the reduction in answer log loss, and ΔE is the reduction in excess predictability. Points above zero have lower excess predictability after finetuning; counts report these pairs. Suboptimality decreases in all 59 pairs. Shapes identify the release type; lines are least-squares fits; r is Pearson correlation and p+ its one-sided p -value for positive correlation.
Figure 3: Answer and correctness losses during finetuning of Llama-3.1-8B on CommonsenseQA . Shading shows excess predictability E(f) ; predictor loss averages three seeds.
Figure 4: More suboptimal models have larger routing advantages. Each point is one released or finetuned weak-model version. Empirical suboptimality is the gap to the best observed answer log loss on each task. Routing advantage 100D is the cascade accuracy gain from a 4B predictor over self-confidence routing to Qwen3.5-122B-A10B , averaged over equal escalation rates and expressed in percentage points. Colors identify training stage; shapes identify the release type. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided p -value for positive correlation.
Figure 5: Finetuning reduces routing advantage in 53 of 59 model-task pairs. Each point compares one weak model before and after finetuning; both axes show released minus finetuned values. Δδ is the reduction in answer log loss. 100ΔD is the reduction in cascading advantage. Points above zero have lower routing advantage after finetuning; counts report these pairs. Shapes identify the release type; lines are least-squares fits; r is Pearson correlation and p+ its one-sided p -value for positive correlation.
Figure 6: Llama-3.1-8B cascades on CommonsenseQA before and after finetuning: predictor (solid), confidence (dashed), and random (dotted) routing. Shading shows routing advantage D .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Base release
Instruction-tuned release
Qwen3.5-9B-Base
Qwen3.5-9B
Gemma-3-12B-PT
Gemma-3-12B-IT
Llama-3.1-8B
Llama-3.1-8B-Instruct
Apertus-8B-2509
Apertus-8B-Instruct-2509
OLMo-3-1025-7B
OLMo-3-7B-Instruct
Falcon3-7B-Base
Falcon3-7B-Instruct
Appendix
Table 1: Base and instruction-tuned releases from eight model families, giving 16 models per task: Qwen3.5 ( Qwen Team, 2026 ) , Gemma 3 ( Gemma Team, 2025 ) , Llama 3.1 ( Grattafiori et al., 2024 ) , Apertus ( Hernández-Cano et al., 2025 ) , OLMo 3 ( Team Olmo et al., 2025 ) , Falcon3 ( Falcon-LLM Team, 2024 ) , Mistral 7B ( Jiang et al., 2023 ) , and Mistral-NeMo-Minitron ( Sreenivas et al., 2024 ) .
Development split
Task
Finetuning
Development
Fit (85%)
Select (15%)
Test
WinoGrande
32,313
8,080
6,868
1,212
1,267
CommonsenseQA
6,394
3,342
2,840
502
1,221
GSM8K
5,088
2,241
1,904
337
1,319
MMLU-Pro
5,233
3,000
2,550
450
3,032
Appendix
Table 2: Questions used at each stage, excluding demonstrations. Predictor fitting and selection partition the development set. For GSM8K and MMLU-Pro , finetuning counts include only questions with an accepted teacher solution.
Task
Loss (nats)
Finetuned model attaining the reference
WinoGrande
0.3229
Mistral-NeMo-Minitron-8B-Base
CommonsenseQA
0.3390
Qwen3.5-9B-Base
GSM8K
0.1378
Qwen3.5-9B
MMLU-Pro
0.7247
Qwen3.5-9B-Base
Appendix
Table 3: Reference answer losses for empirical suboptimality, computed over the released and finetuned models retained on each task.
Task
Steps
Token limit
Checkpoint selection
WinoGrande
2,000
256
Lowest development answer loss
CommonsenseQA
1,200
512
Lowest development answer loss
GSM8K
1,500
4,096
Highest development greedy accuracy
MMLU-Pro
1,200
6,144
Highest development greedy accuracy
Appendix
Table 4: Finetuning settings for the weak models. We save four checkpoints per seed, one at each quarter of training, and select among the 12 candidates from three seeds.
WinoGrande
CommonsenseQA
GSM8K
MMLU-Pro
Model
Rel.
FT
Rel.
FT
Rel.
FT
Rel.
FT
Qwen3.5-9B-Base
88.6
93.3
81.5
83.7
94.3
96.0
66.8
76.7
Qwen3.5-9B
81.9
93.7
81.5
83.6
95.9
96.1
78.9 †
77.3 †
Gemma-3-12B-PT
89.5
94.2
76.5
82.4
87.1
92.7
53.5
63.0
Gemma-3-12B-IT
82.1
93.1
76.2
80.4
93.4
95.4
64.6 †
62.6 †
Llama-3.1-8B
88.3
93.4
73.1
78.8
70.6
82.7
41.6
50.2
Appendix
Table 5: Development accuracy (%) of released models (Rel.) and their selected finetuned checkpoints (FT). † marks a model whose finetuned checkpoint’s development accuracy does not exceed that of its released model. Marked pairs are excluded from all reported comparisons, including Figure 9 .
Comparison
Task
Pearson r
p+
Spearman rs
p+
δ vs. E
WinoGrande
0.98
<0.001
0.73
<0.001
CommonsenseQA
0.98
<0.001
0.94
<0.001
GSM8K
0.93
<0.001
0.94
<0.001
MMLU-Pro
0.65
<0.001
0.74
<0.001
Δδ vs. ΔE
WinoGrande
0.99
<0.001
0.90
<0.001
CommonsenseQA
0.99
<0.001
0.97
<0.001
Appendix
Table 6: Correlations for the main 4B comparisons. p+ tests a positive association using the tests described in Appendix C.1 . Endpoint comparisons contain 32, 32, 28, and 26 model versions in task order; paired changes contain 16, 16, 14, and 13 model pairs.
Figure 7: Excess predictability of the 4B predictor, with released and finetuned models shown separately. Each column is a task; the top row shows released models and the bottom row their finetuned versions, using the same points as Figure 1 . Excess predictability E is the predictor’s correctness-log-loss gain over self-confidence; empirical suboptimality δ is the gap to the best observed answer log loss on each task. Circles denote base models and triangles instruction-tuned models. Lines are least-squares fits within each panel; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation. Correlations and p -values are in Table 7 . Axis ranges differ across panels.
Figure 8: Routing advantage of the 4B predictor, with released and finetuned models shown separately. Routing advantage 100D (percentage points) versus empirical suboptimality δ (nats), using the same points as Figure 4 . 100D is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates; δ is the gap to the best observed answer log loss on each task. Columns are tasks; orange (top) denotes released models and blue (bottom) their finetuned versions. Circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation. Axis ranges differ across panels.
Comparison
Task
Pearson r
p+
Spearman rs
p+
δ vs. E (released)
WinoGrande
1.00
<0.001
0.92
<0.001
CommonsenseQA
0.98
<0.001
0.97
<0.001
GSM8K
0.89
<0.001
0.90
<0.001
MMLU-Pro
0.49
0.045
0.60
0.015
δ vs. E (finetuned)
WinoGrande
0.14
0.301
-0.03
0.547
CommonsenseQA
0.65
0.003
0.64
0.004
Appendix
Table 7: Within-stage correlations for 4B predictors. p+ tests a positive association using the tests described in Appendix C.1 . Each stage contains 16, 16, 14, and 13 models in task order.
Figure 9: Predictor gains over self-confidence and Platt calibration. Mean baseline loss minus predictor loss over the retained models on each task; positive values favor the predictor. Columns show tasks; rows compare against raw and Platt-calibrated confidence. Orange solid lines with circles: released models; blue dashed lines with squares: finetuned models. Predictor size is on the horizontal axis; vertical scales are shared within each row. Lines connect the three evaluated sizes.
Figure 10: Excess predictability with the 0.6B predictor. E is the predictor’s correctness-log-loss gain over self-confidence; δ is the gap to the best observed answer log loss on each task. Orange denotes released models and blue finetuned models; circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation.
Figure 11: Routing advantage with the 0.6B predictor. 100D is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates, in percentage points; δ is the gap to the best observed answer log loss on each task. Orange denotes released models and blue finetuned models; circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation.
Figure 12: Excess predictability with the 8B predictor. E is the predictor’s correctness-log-loss gain over self-confidence; δ is the gap to the best observed answer log loss on each task. Orange denotes released models and blue finetuned models; circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation.
Figure 13: Routing advantage with the 8B predictor. 100D is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates, in percentage points; δ is the gap to the best observed answer log loss on each task. Orange denotes released models and blue finetuned models; circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits; r is Pearson correlation and p+ its one-sided, unadjusted p -value for positive correlation.
Figure 14: Cascade accuracy across all four tasks. We pair Llama-3.1-8B base with Qwen3.5-122B-A10B , finetuned on WinoGrande and CommonsenseQA and released on the other two tasks. Orange denotes the released weak model and blue its finetuned version. Solid curves use the selected 4B predictor g , dashed curves use self-confidence f1,Z , and dotted lines show random routing. Shading shows the accuracy difference between the two routers.
Task
Models
100D
Weak-error term
Strong-error term
WinoGrande
Released
1.38
1.80
0.42
Finetuned
0.18
0.21
0.03
CommonsenseQA
Released
1.09
1.26
0.17
Finetuned
0.18
0.17
-0.01
GSM8K
Released
0.52
0.59
0.07
Finetuned
0.20
0.24
0.03
Appendix
Table 8: Routing advantage split into two contributions. Means over retained models with 4B predictors, in percentage points averaged over escalation rates. Both error terms are differences from self-confidence routing; 100D equals the weak-error term minus the strong-error term, up to rounding.
Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.
The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.
Shihong Huang, Shengjie Wang, Hong Ma +1
Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China
The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.
Jasper Dekoninck, Maximilian Baader, Martin Vechev
Department of Computer Science, ETH Zurich, Switzerland.
Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model's suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction--based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt's category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool.
Kenan Hasanaliyev, Silas Alberti, Jenny Hamer +5
Stanford University · Inception Labs · Cognition Labs +2