Predictable performance is suboptimal, and what it means for model routing

Guanhua Zhang; Moritz Hardt

cs.LGOct 22, 2026

Predictable performance is suboptimal, and what it means for model routing

Authors: Guanhua Zhang, Moritz Hardt

Organizations: Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center

Abstract

Large language models (LLMs) differ in capabilities and costs, motivating routing methods that adaptively choose a model for each query. These methods rely on performance prediction: given a query, estimate whether a model will answer it correctly. Most routing methods train separate predictors of candidate models' performance to guide model selection. Our work shows that performance prediction only beats a simple baseline when the candidate model is suboptimal on the task or the performance predictor adds significant capacity relative to the candidate model. Specifically, we take a model's self-confidence, the probability it assigns to its chosen answer, as a baseline and study how much a performance predictor can improve on it. First, we show that any performance predictor that outperforms a model's self-confidence can be used to improve the model's loss: the predictor's advantage transfers exactly in log loss through recalibration. Second, accounting for the predictor's additional parameters, we bound its advantage by the model's suboptimality plus the gain from those parameters. Third, we extend our theoretical results to two-model cascades, which route queries to a more capable model when the first model is predicted to fail. Experiments with 16 language models on four benchmarks show that predictors gain a larger advantage over self-confidence on more suboptimal models. This advantage shrinks as models improve through finetuning. Cascading exhibits the same two patterns: predictor-based routing offers larger gains over confidence-based routing for more suboptimal models, and these gains diminish after finetuning.

Figures & tables

Figure 1: Predictors have a larger advantage over self-confidence on more suboptimal models. Each point is a released (orange) or finetuned (blue) model. Empirical suboptimality is the gap to the best observed answer log loss on each task. Excess predictability

E

is the reduction in correctness log loss achieved by a separately trained 4B predictor over self-confidence. Colors identify training stage; shapes identify the release type. Lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided

p

-value for positive correlation. See Section 3.4.1 for experimental details.

Figure 2: Finetuning reduces excess predictability in 55 of 59 model-task pairs. Each point compares one released model with its selected finetuned version; both axes show released minus finetuned values.

\Delta\widehat{\delta}

is the reduction in answer log loss, and

\Delta E

is the reduction in excess predictability. Points above zero have lower excess predictability after finetuning; counts report these pairs. Suboptimality decreases in all 59 pairs. Shapes identify the release type; lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided

p

-value for positive correlation.

Figure 3: Answer and correctness losses during finetuning of Llama-3.1-8B on CommonsenseQA . Shading shows excess predictability

E(f)

; predictor loss averages three seeds.

Figure 4: More suboptimal models have larger routing advantages. Each point is one released or finetuned weak-model version. Empirical suboptimality is the gap to the best observed answer log loss on each task. Routing advantage

100D

is the cascade accuracy gain from a 4B predictor over self-confidence routing to Qwen3.5-122B-A10B , averaged over equal escalation rates and expressed in percentage points. Colors identify training stage; shapes identify the release type. Lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided

p

-value for positive correlation.

Figure 5: Finetuning reduces routing advantage in 53 of 59 model-task pairs. Each point compares one weak model before and after finetuning; both axes show released minus finetuned values.

\Delta\widehat{\delta}

is the reduction in answer log loss.

100\Delta D

is the reduction in cascading advantage. Points above zero have lower routing advantage after finetuning; counts report these pairs. Shapes identify the release type; lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided

p

-value for positive correlation.

Figure 6: Llama-3.1-8B cascades on CommonsenseQA before and after finetuning: predictor (solid), confidence (dashed), and random (dotted) routing. Shading shows routing advantage

D

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Base release	Instruction-tuned release
Qwen3.5-9B-Base	Qwen3.5-9B
Gemma-3-12B-PT	Gemma-3-12B-IT
Llama-3.1-8B	Llama-3.1-8B-Instruct
Apertus-8B-2509	Apertus-8B-Instruct-2509
OLMo-3-1025-7B	OLMo-3-7B-Instruct
Falcon3-7B-Base	Falcon3-7B-Instruct

Appendix

Table 1: Base and instruction-tuned releases from eight model families, giving 16 models per task: Qwen3.5 ( Qwen Team, 2026 ) , Gemma 3 ( Gemma Team, 2025 ) , Llama 3.1 ( Grattafiori et al., 2024 ) , Apertus ( Hernández-Cano et al., 2025 ) , OLMo 3 ( Team Olmo et al., 2025 ) , Falcon3 ( Falcon-LLM Team, 2024 ) , Mistral 7B ( Jiang et al., 2023 ) , and Mistral-NeMo-Minitron ( Sreenivas et al., 2024 ) .

Task	Finetuning	Development	Fit (85%)	Select (15%)	Test
			Development split
WinoGrande	32,313	8,080	6,868	1,212	1,267
CommonsenseQA	6,394	3,342	2,840	502	1,221
GSM8K	5,088	2,241	1,904	337	1,319
MMLU-Pro	5,233	3,000	2,550	450	3,032

Appendix

Table 2: Questions used at each stage, excluding demonstrations. Predictor fitting and selection partition the development set. For GSM8K and MMLU-Pro , finetuning counts include only questions with an accepted teacher solution.

Task	Loss (nats)	Finetuned model attaining the reference
WinoGrande	0.3229	Mistral-NeMo-Minitron-8B-Base
CommonsenseQA	0.3390	Qwen3.5-9B-Base
GSM8K	0.1378	Qwen3.5-9B
MMLU-Pro	0.7247	Qwen3.5-9B-Base

Appendix

Table 3: Reference answer losses for empirical suboptimality, computed over the released and finetuned models retained on each task.

Task	Steps	Token limit	Checkpoint selection
WinoGrande	2,000	256	Lowest development answer loss
CommonsenseQA	1,200	512	Lowest development answer loss
GSM8K	1,500	4,096	Highest development greedy accuracy
MMLU-Pro	1,200	6,144	Highest development greedy accuracy

Appendix

Table 4: Finetuning settings for the weak models. We save four checkpoints per seed, one at each quarter of training, and select among the 12 candidates from three seeds.

	WinoGrande		CommonsenseQA		GSM8K		MMLU-Pro
Model	Rel.	FT	Rel.	FT	Rel.	FT	Rel.	FT
Qwen3.5-9B-Base	88.6	93.3	81.5	83.7	94.3	96.0	66.8	76.7
Qwen3.5-9B	81.9	93.7	81.5	83.6	95.9	96.1	78.9 †	77.3 †
Gemma-3-12B-PT	89.5	94.2	76.5	82.4	87.1	92.7	53.5	63.0
Gemma-3-12B-IT	82.1	93.1	76.2	80.4	93.4	95.4	64.6 †	62.6 †
Llama-3.1-8B	88.3	93.4	73.1	78.8	70.6	82.7	41.6	50.2

Appendix

Table 5: Development accuracy (%) of released models (Rel.) and their selected finetuned checkpoints (FT).

\dagger

marks a model whose finetuned checkpoint’s development accuracy does not exceed that of its released model. Marked pairs are excluded from all reported comparisons, including Figure 9 .

Comparison	Task	Pearson $r$	$p_{+}$	Spearman $r_{s}$	$p_{+}$
$\widehat{\delta}$ vs. $E$	WinoGrande	0.98	$<0.001$	0.73	$<0.001$
	CommonsenseQA	0.98	$<0.001$	0.94	$<0.001$
	GSM8K	0.93	$<0.001$	0.94	$<0.001$
	MMLU-Pro	0.65	$<0.001$	0.74	$<0.001$
$\Delta\widehat{\delta}$ vs. $\Delta E$	WinoGrande	0.99	$<0.001$	0.90	$<0.001$
	CommonsenseQA	0.99	$<0.001$	0.97	$<0.001$

Appendix

Table 6: Correlations for the main 4B comparisons.

p_{+}

tests a positive association using the tests described in Appendix C.1 . Endpoint comparisons contain 32, 32, 28, and 26 model versions in task order; paired changes contain 16, 16, 14, and 13 model pairs.

Figure 7: Excess predictability of the 4B predictor, with released and finetuned models shown separately. Each column is a task; the top row shows released models and the bottom row their finetuned versions, using the same points as Figure 1 . Excess predictability

E

is the predictor’s correctness-log-loss gain over self-confidence; empirical suboptimality

\widehat{\delta}

is the gap to the best observed answer log loss on each task. Circles denote base models and triangles instruction-tuned models. Lines are least-squares fits within each panel;

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation. Correlations and

p

-values are in Table 7 . Axis ranges differ across panels.

Figure 8: Routing advantage of the 4B predictor, with released and finetuned models shown separately. Routing advantage

100D

(percentage points) versus empirical suboptimality

\widehat{\delta}

(nats), using the same points as Figure 4 .

100D

is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates;

\widehat{\delta}

is the gap to the best observed answer log loss on each task. Columns are tasks; orange (top) denotes released models and blue (bottom) their finetuned versions. Circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation. Axis ranges differ across panels.

Comparison	Task	Pearson $r$	$p_{+}$	Spearman $r_{s}$	$p_{+}$
$\widehat{\delta}$ vs. $E$ (released)	WinoGrande	1.00	$<0.001$	0.92	$<0.001$
	CommonsenseQA	0.98	$<0.001$	0.97	$<0.001$
	GSM8K	0.89	$<0.001$	0.90	$<0.001$
	MMLU-Pro	0.49	0.045	0.60	0.015
$\widehat{\delta}$ vs. $E$ (finetuned)	WinoGrande	0.14	0.301	-0.03	0.547
	CommonsenseQA	0.65	0.003	0.64	0.004

Appendix

Table 7: Within-stage correlations for 4B predictors.

p_{+}

tests a positive association using the tests described in Appendix C.1 . Each stage contains 16, 16, 14, and 13 models in task order.

Figure 9: Predictor gains over self-confidence and Platt calibration. Mean baseline loss minus predictor loss over the retained models on each task; positive values favor the predictor. Columns show tasks; rows compare against raw and Platt-calibrated confidence. Orange solid lines with circles: released models; blue dashed lines with squares: finetuned models. Predictor size is on the horizontal axis; vertical scales are shared within each row. Lines connect the three evaluated sizes.

Figure 10: Excess predictability with the 0.6B predictor.

E

is the predictor’s correctness-log-loss gain over self-confidence;

\widehat{\delta}

is the gap to the best observed answer log loss on each task. Orange denotes released models and blue finetuned models; circles denote base releases and triangles instruction-tuned releases. Lines are least-squares fits;

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation.

Figure 11: Routing advantage with the 0.6B predictor.

100D

is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates, in percentage points;

\widehat{\delta}

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation.

Figure 12: Excess predictability with the 8B predictor.

E

is the predictor’s correctness-log-loss gain over self-confidence;

\widehat{\delta}

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation.

Figure 13: Routing advantage with the 8B predictor.

100D

is the cascade accuracy gain over self-confidence routing, averaged over equal escalation rates, in percentage points;

\widehat{\delta}

r

is Pearson correlation and

p_{+}

its one-sided, unadjusted

p

-value for positive correlation.

Figure 14: Cascade accuracy across all four tasks. We pair Llama-3.1-8B base with Qwen3.5-122B-A10B , finetuned on WinoGrande and CommonsenseQA and released on the other two tasks. Orange denotes the released weak model and blue its finetuned version. Solid curves use the selected 4B predictor

g

, dashed curves use self-confidence

f_{1,Z}

, and dotted lines show random routing. Shading shows the accuracy difference between the two routers.

Task	Models	$100D$	Weak-error term	Strong-error term
WinoGrande	Released	1.38	1.80	0.42
	Finetuned	0.18	0.21	0.03
CommonsenseQA	Released	1.09	1.26	0.17
	Finetuned	0.18	0.17	-0.01
GSM8K	Released	0.52	0.59	0.07
	Finetuned	0.20	0.24	0.03

Appendix

Table 8: Routing advantage split into two contributions. Means over retained models with 4B predictors, in percentage points averaged over escalation rates. Both error terms are differences from self-confidence routing;

100D

equals the weak-error term minus the strong-error term, up to rounding.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 351 papers

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning
Aug 16, 2026Shihong Huang, Shengjie Wang, Hong Ma +1LLM Routing Language Model Cascading
Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China
A Unified Approach to Routing and Cascading for LLMs
Oct 14, 2024Jasper Dekoninck, Maximilian Baader, Martin VechevLLM Routing Language Model Cascading
Department of Computer Science, ETH Zurich, Switzerland.
Expected Reward Prediction, with Applications to Model Routing
Mar 3, 2026Kenan Hasanaliyev, Silas Alberti, Jenny Hamer +5LLM Routing Reward Modeling
Stanford University · Inception Labs · Cognition Labs +2

Aug 16, 2026J/K move · Enter open · S save

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

Shihong Huang, Shengjie Wang, Hong Ma, Zhou Xu

Polytechnic Institute, Zhejiang University, Hangzhou, China · Department of Logistics and Maritime Studies, Faculty of Business, The Hong Kong Polytechnic University, Hong Kong, China

LLM Routing Language Model Cascading Cascaded Inference LLM Inference Efficiency

Open paper AskPDF arXiv

The growing ecosystem of large language models (LLMs) offers huge potential to optimize performance-cost trade-offs. However, their heterogeneous capabilities and inference costs make efficiently routing queries a significant challenge. Existing paradigms are inflexible: one-shot routers commit before observing responses, whereas conventional cascades stop adaptively but follow a fixed model order. Cascade routing removes both restrictions by reconsidering whether to stop or invoke another model after each response. Current methods use a predict-then-optimize pipeline estimating response quality and future model utility. However, prediction loss for quality or utility is not equivalent to routing-decision loss. A lower prediction error does not necessarily yield a better action; a small boundary-crossing error can reverse a

stop'' or model-selection decision. Therefore, we propose RLCascadeRouter, a quality-estimator-free framework that formulates cascade routing as a Markov decision process with actions comprising

stop'' and model selection. It uses trajectory returns and advantages to directly optimize the performance-cost objective. Its Cascade Policy Network models candidate complementarity for model selection and remaining-action value for stopping, eliminating independent post-hoc response-quality estimators. Evaluated across ten LLMRouterBench benchmarks with thirteen LLMs, RLCascadeRouter outperforms strong baselines and achieves superior performance-cost trade-offs. It incorporates unseen models without retraining, and ablation studies validate both policy components.

Abstract

LLM Routing

Explore similar work

RLCascadeRouter: Quality-Estimator-Free Cascade Routing via Reinforcement Learning

A Unified Approach to Routing and Cascading for LLMs

Expected Reward Prediction, with Applications to Model Routing