cs.AIOct 2, 2026
SaveDynamic LLM Routers are Often Misguided
Organizations: Fastino Labs
Abstract
Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.
Figures & tables
Figure 1: Four patterns of current LLM routers. (a) Difficulty blindness: a strong model’s (Opus 5 (high)) advantage over a weak one (gpt-oss-20b (low)) peaks at moderately hard queries and shrinks at the expensive end, so routers under-escalate hard queries. (b) Length reversal: routing by difficulty (blue) spends more on queries with longer answers, while some commercial routers (orange) spend less. (c) Semantic matching: routers often send most of each data source to one model regardless of query difficulty (OpenRouter auto (medium) is shown as a representative example). (d) Roster suboptimality: only a small number of models are at the cost-accuracy frontier, yet many commercial routers spread queries across many non-frontier models.
| Accuracy | Difficulty blindness | Length reversal | Semantic matching | ||
| Router | (pp) | Source AMI | Within-source | ||
| OR auto (med) | |||||
| OR auto (high) | |||||
| OR auto (xhigh) | |||||
| OR auto- (med) | |||||
| OR auto- (high) | |||||
Table 1: Comparison between commercial routers and a random {Gemini 3.7 Flash, Opus 5 (high)} router at matched cost (§ 3 ), with our router (§ 7 ) below. is the router’s accuracy minus the random router’s, with a 95% paired Wald interval (which is narrower for Nadir because both it and the random baseline route many queries to Opus 5). is Goodman-Kruskal’s gamma between query difficulty and the chosen model’s ability (§ F ); is between expected answer length and the chosen model’s mean cost; Source AMI is the adjusted mutual information between source and routing choice, the share of routing decisions explained by the source (§ B ); Within-source is the mean per-source . These intervals are 95% jackknife intervals over queries. Our router is fit out of distribution (§ A ), and Table 7 gives in-distribution values. A bold point estimate indicates a “pass”: not significantly below zero for and , and significantly above zero for , globally or within source. AMI is reported for context but not graded.
Figure 2: Accuracy against cost when escalating a 20% band of queries from the cheap to expensive model based on different metrics. (a) escalates based on the difficulty of the query based on model pass rates across the whole model set, while (b) escalates based on output length, also averaged over the model set. In (b), we mark the reference point corresponding to hardest-20% escalation. 95% half-widths do not exceed 3.5 pp, and in (a), the 60–80 band outperforms the 80–100 band by pp. The same trends hold across other datasets (Figure 13 ).
Figure 3: Accuracy at each maximum budget using three routing methods. Random is random routing between the two frontier models adjacent to the budget. Difficulty is a router with perfect knowledge of query difficulty, choosing its roster of up to five models and its thresholds on the evaluation set itself (§ 6 ). Source maps each data source to a single model, chosen on the training set. In (a), each query is charged what the answer actually cost, while in (b), each query is charged that model’s average inference cost. Since a map chosen on the training set costs a different amount on the evaluation set, we fit source maps on a dense grid of training budgets and use the highest-budget map that stays under each evaluation budget. Half-widths for the gaps between difficulty and random routing are at most 1.6 pp, and those between difficulty and source routing are at most 2.1 pp.
Figure 4: For each budget, the highest accuracy a router can achieve with perfect knowledge of query difficulty, choosing the roster and thresholds on the evaluation set itself, for rosters of models, where harder queries are sent to more capable models. Accuracies above the strongest standalone model reflect overfitting to the evaluation set, since stronger models typically outperform weaker ones across all difficulty bands (Figure 11 ).
Figure 5: Our router on {Gemini 3.7 Flash, Opus 5 (high)} evaluated using our revised methodology from § 5 , where all costs are per-model mean costs and the hard band contains queries with at least a 50% failure rate. Random is random routing between the two models. Ours (ID) and (OOD) are our router fit in and out of distribution (§ A ). Difficulty oracle uses the true difficulty scores to route. On (a), 95% confidence interval half-widths for the gap with random do not exceed 1.2 pp, with all deviation from random routing remaining within noise; on (b), they do not exceed 7.5 pp, with our router surpassing random routing on hard-band escalation outside noise across 42–73/1k and $121–139/1k queries OOD.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Samples | ||||
| Category | Source | Description | Train | Eval |
| Coding | HumanEval ( Chen et al., 2021 ) | Python function synthesis with unit tests | ||
| HumanEvalFix ( Muennighoff et al., 2024 ) | Bug-fix coding tasks with unit tests | |||
| MBPP ( Odena et al., 2021 ) | Short Python programming tasks | |||
| BigCodeBench ( Zhuo et al., 2025 ) | Harder practical coding scenarios | |||
| Instruction | IFEval ( Zhou et al., 2023 ) | Instruction-following and formatting constraints | ||
Table 2: Data sources in our training and evaluation datasets, with the number of training and evaluation samples from each source.
| ID | OOD | |||
| Method | Global | Source | Global | Source |
| Linear probe | ||||
| Difference of means | ||||
| RankNet | ||||
| PCA logistic | ||||
| SharedTrunkNet | ||||
Table 3: Comparison between difficulty prediction methods by global and mean per-source , the Goodman-Kruskal correlation between predicted and true query difficulty. We consider both the in-distribution (ID) and out-of-distribution (OOD) fits, described in Appendix A . Confidence intervals are 95% jackknife intervals over evaluation queries. Bold marks the highest point estimate in each column.
| Models | ||||
| Router | Cost ($/1k q.) | Acc. (%) | Chosen | Roster |
| OR auto (med) | ||||
| OR auto (high) | ||||
| OR auto (xhigh) | ||||
| OR auto- (med) | ||||
| OR auto- (high) | ||||
Table 4: Costs and accuracies of commercial routers on our evaluation dataset. Chosen is the number of distinct models the router chose, and Roster is the number of models in its roster. OR: OpenRouter; ND: Not Diamond; SR: Semantic Router.
| vs. random router (pp) | ||||
| Price point | Random acc. (%) | Ours (ID) | Ours (OOD) | Difficulty oracle |
| Gemini 3.7 Flash Claude Opus 5 (high) | ||||
| $10/1k q. | ||||
| $20/1k q. | ||||
| $40/1k q. | ||||
| gpt-oss-20b (low) Claude Opus 5 (high) | ||||
Table 5: Our router evaluated as in Table 1 , using accuracy over the entire evaluation set with realized generation costs. ID and OOD are the in-distribution and out-of-distribution fits. Difficulty oracle is a router with perfect knowledge of query difficulty that escalates the hardest queries first. Intervals are 95% paired Wald intervals. A bold point estimate indicates that the router is not significantly below the random router.
| vs. random router (pp) | ||||
| Router | Cost ($/1k q.) | Acc. (%) | Luna / Sol | Flash / Opus |
| OR auto (med) | ||||
| OR auto (high) | ||||
| OR auto (xhigh) | ||||
| OR auto- (med) | ||||
| OR auto- (high) | ||||
Table 6: Comparison between commercial routers and random routers at matched costs, as in Table 1 , but also matched to models available at the time of each commercial router’s last update. Every router’s roster includes the GPT 5.6x series or models released after, and some include Gemini 3.7 Flash, which was released later. Commercial routers whose rosters do not contain Gemini 3.7 Flash at the time of access are compared against {GPT 5.6 Luna (high), GPT 5.6 Sol (xhigh)}, while those that contain it are compared against {Gemini 3.7 Flash, Opus 5 (high)}. The shaded cell in each row is the comparison matched to that router’s roster. A bold point estimate indicates that the router is not significantly below the random router. Five routers are within noise of a random router using models available at the time, and all other routers underperform.
| Difficulty blindness | Length reversal | Semantic matching | ||
| Router | Source AMI | Within-source | ||
| Ours, ID ($10/1k q.) | ||||
| Ours, ID ($20/1k q.) | ||||
| Ours, ID ($40/1k q.) | ||||
| Difficulty oracle ($10/1k q.) | ||||
| Difficulty oracle ($20/1k q.) | ||||
Table 7: Pattern metrics for our router fit in distribution and for a difficulty oracle that escalates the hardest queries by true difficulty, at the same three price points as Table 1 . The oracle’s is 1 by construction. Intervals are 95% jackknife intervals over evaluation queries. A bold point estimate indicates a “pass,” as in Table 1 .
Figure 6: Negative log-likelihood (NLL) of the MIRT model as the number of interaction dimensions grows. The fits use observed pass rates ( outcomes ), Qwen3.5-27B activations ( activations ), or a combination of these ( both ). is the Rasch model. 2 standard error bars do not exceed 0.02, and the lines for outcomes and both are identical.
Figure 7: Standalone costs and accuracies of each model in the 26-model set. The gray line is the cost-accuracy frontier, which linearly interpolates between models on the Pareto frontier. gpt-oss-20b (low) is marked in pink, Gemini 3.7 Flash in yellow, and Opus 5 (high) in green. Error bars are 95% confidence intervals over queries.
Figure 8: Histogram of average answer length in characters for each dataset. Most of RouterBench’s queries are multiple-choice questions. Our dataset spans the widest range of answer lengths, which may be why no previous evaluation reported length reversal.
Figure 9: (a) is a replication of Figure 5 (a) with {gpt-oss-20b (low), Opus 5 (high)} as the roster. (b) shows the improvement of difficulty-based routing methods against the random routing baseline on the easy band. Note that the hard-band escalation against easy-band cost figure is the same shape as in Figure 5 (b), since we are using the same router, just with a different roster. On (a), 95% half-widths do not exceed 3.6 pp; on (b), 95% half-widths do not exceed 2.5 pp, and both ours (ID) and difficulty oracle achieve statistically significant accuracy gaps when easy-band costs range from 11–64/1k queries.
Figure 10: Scaling of predictions using the SharedTrunkNet architecture as a function of the training size. We plot both global and source correlations ID and OOD, with 95% jackknife confidence intervals. For points trained on less than the full training set, we average over three seeds.
Figure 11: Observed versus predicted pass rates for each model using the Rasch model. We group evaluation queries into ten bins by fitted difficulty, so each point represents a difficulty band. Error bars are 95% bootstrapped confidence intervals. Because all models are evaluated on the same queries with the same difficulty predictions, each predicted line is linearly offset in logit space. The fact that each predicted line closely matches observed outcomes indicates that the Rasch model fits outcomes well.
Figure 12: Replication of Figures 7 and 4 for the 20-model set, RouterBench, SPROUT, and LLMRouterBench, demonstrating that small roster sizes suffice. Note that some models in LLMRouterBench are listed with $0.00 inference costs. Half-widths for roster accuracies do not exceed 3.5 pp (20-model set), 0.5 pp (RouterBench), 0.4 pp (SPROUT), and 1.5 pp (LLMRouterBench).
Figure 13: Replication of Figure 2 for all other datasets. We pair the cheapest model with the top-scoring model so accuracy and cost differences are most visible. For RouterBench, we route between Mistral 7B and GPT-4, and we do not include the endpoint stars because they are too far from the area of focus. The trends discussed in Section 5 hold across datasets except for RouterBench on length-based escalation, where query outputs differ little in length so there is less headroom for accuracy hacking (Figure 8 ). The maximum half-widths for each dataset are 3.5 pp (20-model set), 0.9 pp (RouterBench), 0.4 pp (SPROUT), and 1.7 pp (LLMRouterBench). Escalating the 60–80 difficulty band significantly exceeds 80–100 on accuracy on every dataset except RouterBench.
Figure 14: Replication of Figure 3 for all other datasets. For the three benchmarks, we split the dataset into training and evaluation halves. 95% half-widths do not exceed 3.2 pp (20-model set), 0.7 pp (RouterBench), 0.4 pp (SPROUT), and 2.1 pp (LLMRouterBench). On the 20-model set, difficulty routing underperforms random at $2.50/1k queries because many of the most difficult queries also have the longest outputs, and this pattern reverses using uniform costs, an example of accuracy hacking. On SPROUT, source-based routing outperforms difficulty-based routing, suggesting that its model set exhibits some noticeable specialization.
Figure 15: Heatmaps of each router’s choices over each data source. We compare all commercial router tiers, our routers at three price points ID and OOD, and a difficulty oracle using two models at the same price points. Models are ranked in increasing order of ability . Note that router choices can only be meaningfully compared over identical rosters.
| Model | Effort | AA entry used | AAII | Match |
|---|---|---|---|---|
| Claude Opus 5 (high) | high | Claude Opus 5 (high) | Exact | |
| Grok 4.6 (high) | high | Grok 4.6 (high) | Exact | |
| GPT 5.6 Sol (xhigh) | xhigh | GPT-5.6 Sol (xhigh) | Exact | |
| Claude Opus 4.8 | high | Claude Opus 4.8 (max) | Only published effort | |
| Gemini 3.7 Flash | none | Gemini 3.7 Flash (high) | Default effort | |
| GLM-5.2 | none (behaves like high) | GLM-5.2 (max) | Only published effort |
Table 8: AAII scores for the 26 models in our model set, sorted by AAII. Effort is the reasoning setting we sent, AA entry used is the matched Artificial Analysis listing, and Match is the matching method.
| Model ID | Queries | AA entry used | AAII | Match |
|---|---|---|---|---|
| claude-fable-5 | Claude Fable 5 | Default effort | ||
| glm-5.3 | GLM-5.3 (max) | Default effort | ||
| kimi-k3 | Kimi K3 (max) | Default effort | ||
| qwen3.8-max | Qwen3.8 Max | Exact | ||
| gpt-5.6-sol-high | GPT-5.6 Sol (high) | Exact | ||
| muse-spark-1.2 | Muse Spark 1.2 (xhigh) | Default effort |
Table 9: AAII scores for each model outside our model set chosen by a commercial router, sorted by AAII. Model ID is the ID returned by the router, Queries is the number of queries the model answered across all commercial routers, AA entry used is the matched Artificial Analysis listing, and Match is the matching method.