Most of the LLM Routing Gap Is Task Type

Janghoon Lee

cs.CLAug 24, 2026

Most of the LLM Routing Gap Is Task Type

Authors: Janghoon Lee

Organizations: Redrob

Abstract

An LLM router picks which model should answer each query. The appeal is that models fail on different questions. Whatever single model is best overall still gets some wrong, and another model in the pool gets many of those right. Getting that choice right every time is the ceiling, and a router is an attempt to approach it. However, recent work reports that routers do not get close. Across 21 routing methods on five benchmarks, sharply different designs land within a fraction of a point of each other, and all of them stay far below that ceiling. Learned routers often fail to beat simply always calling the strongest model. We ask what those missed questions have in common. We set fourteen models to answer all 294 questions, with 7 task types across 3 languages: Korean, English and Hindi. We ran the whole matrix twice, changing nothing, but 5.37% of the 4,116 model-question pairs came out scored differently anyway. Run-to-run movement like that is normal, and we argue that a small win does not show that routing did anything, ours or anyone else's. Counting an answer correct only when the model got it right in both runs, 29 questions on this matrix can be improved with routing. Every correct-answer count here is on that rule. Task type accounts for most of them: assigning each task type one model in advance, chosen once and never updated, improves 21 of the 29. Splitting each task type by language improves 2 more and leaves 6 of 294 unoptimized. That handful is what a learned router would have been built for, and it is smaller than the run-to-run movement above, which is a share of pairs rather than of questions. The static table we adopted answers 262 of 294 questions at $3.33 per run, against the best single model's 245 at $7.69. All of this is fitted and scored on the same 294 questions with no holdout.

Figures & tables

short name	vendor	model id	in	out
gpt_5_6_sol	OpenAI	openai/gpt-5.6-sol	5.00	30.00
gpt_5_6_terra	OpenAI	openai/gpt-5.6-terra	2.00	12.00
gpt_5_6_luna	OpenAI	openai/gpt-5.6-luna	0.20	1.20
claude_fable_5	Anthropic	anthropic/claude-fable-5	10.00	50.00
claude_opus_5	Anthropic	anthropic/claude-opus-5	5.00	25.00
claude_sonnet_5	Anthropic	anthropic/claude-sonnet-5	2.00	10.00

Table 1: The 14-model pool. The short names are the identifiers the configuration and the artifacts use; prose names each model in full, and the other tables in this paper key on the short names. Prices are USD per million tokens from the 2026-08-21 catalog snapshot and are configuration only; the cost axis is billed cost ( Section 2.5 ).

task type	source	revision	license	scorer	translated field
knowledge	MMLU-ProX ( li-lab/MMLU-ProX )	8e6106a6…	MIT	mcq	none (parallel source)
math	MGSM ( juletxara/mgsm , config en )	b2f13d42…	CC BY-SA 4.0	exact	statement
instruction	IFEval ( google/IFEval )	966cd895…	Apache-2.0	ifeval	prompt (instruction arguments frozen)
extraction	in-house	n/a	Apache-2.0	json_match	document body
toolcall	BFCL v4 ( ShishirPatil/gorilla )	6ea57973…	Apache-2.0	ast	utterance
abstention	BFCL v4 irrelevance (same repo)	6ea57973…	Apache-2.0	abstain	utterance

Table 2: The seven task types, their sources, and how each is scored. One source per task type, which is the confound Section 7.2 states.

Panel of Figure 1 — Figure 1: Where the distance between the oracle and the best single model sits, under each scoring rule. Under either rule most of it is recovered by giving each task type its own best model, and the remainder needs a different model inside a task type. The percentage on each bar is the between-type share of that bar’s total, and the count past its end is the total. The two components are distinguished by fill pattern as well as by shade.

role	variant	oracle	best single model	gap
sensitivity	single-run A $\times$ all items	280/294 = 0.9524	claude_fable_5 = claude_opus_5 249/294 = 0.8469	0.1054
sensitivity	single-run B $\times$ all items	277/294 = 0.9422	claude_opus_5 252/294 = 0.8571	0.0850
main	both-run $\times$ all items	274/294 = 0.9320	claude_opus_5 245/294 = 0.8333	0.0986
sensitivity	single-run A $\times$ cap_hit-excluded	280/294 = 0.9524	qwen3_7_plus 248/282 = 0.8794	0.0729
sensitivity	both-run $\times$ cap_hit-excluded	274/294 = 0.9320	qwen3_7_plus 238/280 = 0.8500	0.0820
sensitivity	single-run A $\times$ parse_failed-excluded	280/294 = 0.9524	claude_fable_5 249/290 = 0.8586	0.0938

Table 3: The seven scoring variants, with the main estimate marked. The distance is oracle minus best single model, given as a difference in accuracy, so 0.1054 is 10.54 percentage points. Models are named by the short names of Table 1. The four exclusion variants are the cap_hit-excluded and parse_failed-excluded rows. They score models on per-model denominators, which the exclusion changes, and are not 294-item rankings.

rule	best single model	language-only	task-type-only	task $\times$ language argmax	adopted lookup	oracle
single-run A	249	254	268	271	n/a	280
both-run (A $\cap$ B)	245	248	266	268	262	274

Table 4: What static structure recovers on this matrix, in items out of 294. The adopted lookup column is the Section 6 policy, shown for reference; it is not the in-sample argmax of its row.

variant	best single	runner-up	margin
single-run A $\times$ all items	claude_fable_5 = claude_opus_5 249/294	claude_sonnet_5 = qwen3_7_plus 248/294	tie at the top
single-run B $\times$ all items	claude_opus_5 252/294	claude_fable_5 244/294 (four-way tie)	8 items
both-run $\times$ all items (main)	claude_opus_5 245/294	gemini_3_7_flash 244/294	1 item
single-run A $\times$ cap_hit-excluded	qwen3_7_plus 248/282	claude_sonnet_5 248/290	0.0243
both-run $\times$ cap_hit-excluded	qwen3_7_plus 238/280	claude_opus_5 245/291	0.0081
single-run A $\times$ parse_failed-excluded	claude_fable_5 249/290	claude_opus_5 249/293	0.0088

Table 5: Who is best single model under each scoring variant, and by how much. Margins are in items where the two models share a denominator and in accuracy points where they do not.

reading	domain	both-run accuracy	mean billed
unique-winner (rejected)	only cells with a unique both-run winner at a one-item margin	267/294	$3.368173
cheapest-one-behind (adopted)	every cell whose unsigned margin over the cheapest one-behind carries a price premium	262/294	$3.332910
wide-pool (rejected)	tie pool widened to all one-behind models with unsigned margins	261/294	$3.326753

Table 6: The three readings of the tie rule’s domain, priced on the same matrix before one was adopted. Accuracy is both-run correct items out of 294 and the cost is the policy’s mean billed cost per run over the 294-query workload. The three readings are lettered A, B and C in the artifacts; this section names them, since A and B already name the two runs.

cell	before	after	both-run correct	mean billed before $\rightarrow$ after
math-ko	gpt_5_6_luna	deepseek_v4_flash	14 $\rightarrow$ 13	$0.002497$ \rightarrow0.001020
extraction-hi	gpt_5_6_terra	gpt_5_6_luna	14 $\rightarrow$ 13	$0.008242$ \rightarrow0.001429
toolcall-ko	deepseek_v4_pro	deepseek_v4_flash	13 $\rightarrow$ 12	$0.007751$ \rightarrow0.000829
abstention-ko	gpt_5_6_luna	deepseek_v4_flash	14 $\rightarrow$ 13	$0.007293$ \rightarrow0.001115
abstention-en	grok_4_3	deepseek_v4_flash	14 $\rightarrow$ 13	$0.014978$ \rightarrow0.001105

Table 7: The five task

\times

language cells that uniform application changes, each losing one item of 294.

Panel of Figure 2 — Figure 2: Billed cost against accuracy for all 14 single models and for the two static policies this section names, under the both-run rule. The lookup sits up and to the left of the best single model, which is the dominance the dashed arrow marks. The all-coding-Grok point is the frozen pre-tie-rule version, so it and the lookup are not two points on one frontier and none is drawn ( Section 6.5 ). Series are distinguished by marker shape as well as by shade.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

task type	lang	model	correct	mean billed	basis
knowledge	ko	g37f	13/14	$0.011409	signed
knowledge	en	opus	14/14	$0.059090	signed
knowledge	hi	g37f	13/14	$0.010892	signed
math	ko	dsflash	13/14	$0.001020	unsigned (tie rule)
math	en	dsflash	14/14	$0.000630	signed
math	hi	dsflash	14/14	$0.000969	signed

Appendix

Table 8: The adopted task

\times

language lookup, all 21 cells. Accuracy is both-run correct out of the 14 items in the cell; billed cost is the mean of the two runs.

scoring variant	best single	best	oracle	distance (pp)
single-run A, no exclusion	fable = opus	249/294	280/294	10.5
both-run, no exclusion	opus	245/294	274/294	9.9
single-run A, cap-hits excluded	qwen	248/282	280/294	7.3
both-run, cap-hits excluded	qwen	238/280	274/294	8.2
single-run B, no exclusion	opus	252/294	277/294	8.5
single-run A, parse failures excluded	fable	249/290	280/294	9.4

Appendix

Table 9: The 7 scoring variants. Distances are in percentage points of items. Where a variant excludes items per model, the denominators differ between rows and the distances are differences between ratios.

Explore similar work

Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
The Routing Plateau: Understanding the Accuracy Limits of LLM Routers
May 27, 2026Yifan Lu, Qiyue Zhang, Shenrun Zhang +4LLM Evaluation Adaptive Model Routing
Rice University · Amazon

Oct 2, 2026J/K move · Enter open · S save

Dynamic LLM Routers are Often Misguided

Sam Wang, Julia White, Sahibzada Allahyar, Dhruv Atreja+2

Fastino Labs

LLM Routing Cost-Aware Inference LLM Evaluation

Open paper AskPDF arXiv

Dynamic LLM routers promise to cut inference costs by sending each query to the cheapest model that can answer it correctly. We analyze six commercial routers across 14 settings on a diverse benchmark spanning eight task categories, finding that none of them outperforms a router that randomly selects between two well-chosen models at matched cost. Some underperform by more than 10 percentage points. We trace this gap to four patterns prevalent across routers: difficulty blindness, length reversal, semantic matching, and roster suboptimality. We show that the first three are what the standard objective rewards: cost-accuracy Pareto efficiency on realized costs favors escalating moderately hard queries over the hardest ones, shorter queries over longer ones, and routing by a query's source over its difficulty. We also argue that the two assumptions that would justify large rosters, model granularity and model specialization, do not hold empirically. We propose an alternative evaluation methodology that does not reward these patterns, and as a proof of concept, we design a simple two-model router that avoids all four. Nevertheless, its gain over random routing is limited, because a well-chosen roster leaves little to route.

Abstract

Explore similar work

Dynamic LLM Routers are Often Misguided

LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

The Routing Plateau: Understanding the Accuracy Limits of LLM Routers