Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Aristides Milios; Christopher Pal

cs.AINov 6, 2026

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Authors: Aristides Milios, Christopher Pal

Organizations: Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal · Canada CIFAR AI Chair

Abstract

Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.

Figures & tables

Figure 1: What predicting cost saves, and where the queries go. Left: cost saved relative to median pricing on all nine datasets at each of 12 matched accuracies, from the lowest (light, low in the row) to the highest (dark, high in the row) in the range both routers reach, averaged over ten training sets. Ticks mark our average over the range and diamonds the same average for perfect per-query cost knowledge ( Table 3 gives 95% intervals). Right: share of LiveCodeBench test queries sent to each candidate at 77% accuracy by three routers that price a query by median length, by our cost probes, or by its actual length (perfect knowledge), with the spend per 1,000 queries (one training set). Ours moves queries from gpt-oss-20b at low effort to gpt-oss-120b at medium effort.

Figure 2: Cost saved at matched accuracy, billed prices. Left: cost saved by ours relative to ZeroRouter’s pricing, both with our success predictions and our 4B scout, with 95% intervals; hollow where the interval includes zero ( Table 1 ). Right: LiveCodeBench cost–accuracy frontiers of our cost probe and median pricing (one training set); arrows mark the saving at matched accuracy, and the diamond is deepseek-v4-flash on every query.

Cost saved by ours vs.	LCB	Omni	MMLU-Pro	AIME	APPS	BCB	CC	SuperGPQA	BBEH
External estimators
Median pricing	27.8	31.5	23.4	14.9	16.8	0.4	9.4	33.0	23.5
Mean pricing	27.8	34.3	25.8	15.0	13.3	3.8	9.4	35.8	22.7
Prompt-feature GBM	20.7	25.8	20.0	14.2	9.7	1.4	8.0	19.1	2.1
MixLLM-style embeddings	16.2	24.7	15.5	13.5	16.4	1.7	4.3	15.7	$-$ 1.3
CARROT-style $k$ NN	17.5	23.2	14.3	15.0	15.3	0.4	4.0	20.9	0.3

Table 1: Cost saved (%) by our cost probe relative to each estimator at matched accuracy. Every row uses our success predictions except the second ZeroRouter row, which uses its own. Both ZeroRouter rows predict query difficulty from our 4B scout’s activations ( Section B.5 gives the version as published). Bold : statistically significant (95% paired bootstrap interval excludes zero). Ours is ahead in 69 of the 72 comparisons with external estimators, significantly in 50. Estimators are described in Section B.5 .

	LCB		Omni		MMLU-Pro		AIME		APPS		BCB		CC		SGPQA		BBEH
Candidate	Acc.	$	Acc.	$	Acc.	$	Acc.	$	Acc.	$	Acc.	$	Acc.	$	Acc.	$	Acc.	$
20b, low	51.8	0.13	41.9	0.21	62.1	0.05	57.2	0.26	58.4	0.12	45.3	0.06	49.1	0.14	31.0	0.05	23.3	0.11
20b, medium	68.2	0.65	57.5	1.20	69.4	0.18	88.3	1.18	73.5	0.74	48.3	0.12	70.6	0.85	36.0	0.24	34.8	1.21
deepseek-v4-flash	89.3	0.90	71.3	1.66	86.2	0.33	98.3	0.86	84.1	1.05	52.1	0.11	84.6	1.15	67.9	0.47	64.7	1.54
120b, medium	82.1	0.64	60.3	1.12	77.2	0.23	89.5	1.23	79.3	0.72	53.4	0.18	78.4	0.84	46.0	0.27	45.3	1.45
120b, high	89.5	3.73	66.6	4.57	79.3	0.90	97.0	3.14	85.7	4.48	53.0	0.64	85.9	4.87	47.8	1.48	54.7	5.26

Table 2: Each candidate alone, without routing. Accuracy (Acc., %) and mean cost per call (

, USD per 1,000 calls) when every test problem goes to one candidate (billed costs on Omni-MATH and MMLU-Pro, tokens at billed prices elsewhere). SGPQA: SuperGPQA. Bold: most accurate. Costs differ 11–49

\times$ , and accuracy is not monotone in price: deepseek-v4-flash is the most accurate on five datasets and within 1.6 points of gpt-oss-120b at high effort on the rest, at 17–37% of its cost.

Dataset	Domain	Test $n$	Headroom	Ours vs. median	Capture	Ours vs. length
						from pred. success
LiveCodeBench	code	341	40.1	27.8 [13.9, 35.6]	.69	3.2
Omni-MATH	math	1,000	51.9	31.5 [23.9, 37.7]	.61	$-3.0$
MMLU-Pro	knowledge	6,500	58.0	23.4 [17.1, 29.0]	.40	18.4
AIME 1983–2024	math	281	29.2	14.9 [10.8, 20.1]	.51	4.3
APPS	code	377	34.4	16.8 [4.2, 30.4]	.49	$-2.2$

Table 3: Headroom and capture on every dataset. Cost saved (%) relative to median pricing by perfect per-query cost knowledge (headroom) and by our cost probe, with 95% intervals; capture is their ratio. Last column: cost saved by the cost probe relative to our ablation that takes length from predicted success; bold: interval excludes zero.

Cost estimator, 20 training problems	LCB	Omni-MATH	MMLU-Pro	SuperGPQA	BBEH
Ours (cost probe)	22.3 †	22.5 †	11.9	19.5	14.2
Prompt-feature GBM	$-$ 1.3	$-$ 2.4	2.2	8.8	2.9
MixLLM-style embeddings	$-$ 3.1	2.0	$-$ 0.6	5.9	$-$ 3.2
CARROT-style $k$ NN	1.4	3.8	2.3	10.3	6.2
ZeroRouter pricing	1.2	5.6	0.1	$-$ 1.9	1.1
Our ablation: length from predicted success	22.3	26.7	3.2	7.0	$-$ 10.0

Table 4: Few labels. Cost saved (%) relative to median pricing fitted on the same problems, when each cost estimator is fitted on 20 random training problems, two subsets per training set. † : ours exceeds every external estimator (95% intervals exclude zero). Below the line, for reference: our ablation, which exceeds ours on Omni-MATH, and ours fitted on all training problems, relative to median pricing fitted on all of them.

	Ours,	Ours,	Prefill-router rule,	Best external,
Sources $\to$ target	in-domain	transferred	transferred	transferred
APPS, BCB, CC $\to$ LCB	27.8	26.0	$-$ 9.4	$-$ 3.0 (ZeroRouter)
LCB, BCB, CC $\to$ APPS	16.8	31.5	9.0	23.7 (ZeroRouter)
Omni, AIME, SGPQA, BBEH $\to$ MMLU-Pro	23.4	26.6	1.1	18.1 ( $k$ NN)
Omni, MMLU-Pro, SGPQA, BBEH $\to$ AIME	14.9	14.3	$-$ 1.7	6.8 ( $k$ NN)
Omni, MMLU-Pro, AIME, BBEH $\to$ SuperGPQA	33.0	22.5	$-$ 11.2	17.6 (MixLLM)

Table 5: A router moved to a new benchmark. Cost saved (%) relative to the target’s own median pricing. Ours: success and cost probes trained on the sources, with every choice made on their calibration splits; external estimators transfer the same way, each with its own success model. Prefill-router rule: the transferred success probes with median pricing. Ours is significantly ahead of every alternative in every training set on LCB and MMLU-Pro (9 of 10 on AIME). SGPQA: SuperGPQA; Omni: Omni-MATH’s 500-problem set.

	Cost saved relative to
Router prices each candidate by	one median	median per benchmark	Ours vs. it
Baselines
One median length (the prefill router’s rule)	–	$-$ 13.9	31.8 [26.1, 36.7]
Median length per benchmark	12.2	–	22.4 [16.5, 27.6]
Mean length per benchmark	13.5	1.5	21.2 [15.9, 26.0]
MixLLM-style embeddings	18.9	7.6	16.0 [11.1, 19.9]

Table 6: One router for mixed traffic. All nine benchmarks in one mixture (150 test problems each), with the same five candidates and success probes in every row. Cost saved (%) at matched accuracy, with 95% intervals resampled within each benchmark; every “ours vs. it” interval excludes zero in all ten training sets. Estimators with access to benchmark identity are given each query’s true benchmark (an optimistic baseline); ours is not ( Section B.5 ).

	LiveCodeBench			Omni-MATH			MMLU-Pro
Scout	vs. median	$/1k	Ours vs. it	vs. median	$/1k	Ours vs. it	vs. median	$/1k	Ours vs. it
Qwen3-0.6B	24.1	.005	3.5	27.9	.002	6.5	17.0	.002	4.8
Qwen3-1.7B	27.6	.007	$-$ 1.6	29.0	.003	3.4	19.4	.004	2.5
Qwen3-4B	25.5	.015	2.7	28.1	.006	3.5	20.6	.007	3.2
Qwen3-8B	24.9	.022	3.6	30.7	.008	0.6	20.7	.011	6.2
Qwen3-14B	28.8	.036	7.3	27.9	.014	9.8	20.1	.018	10.5

Table 7: Scout size. Qwen3 scouts from 0.6B to 14B (the original releases, which switch between thinking and non-thinking modes) and ours (4B Instruct-2507), each feeding both probes. vs. median: cost saved (%) relative to median pricing with the row’s own success probes.

/1k: the scout’s forward pass per 1,000 queries on one H100 at

2 per hour. Ours vs. it: cost saved (%) by our router relative to the row’s, each paying for its own scout’s pass, at our spend in the middle of the accuracy band (one training set); bold: 95% interval excludes zero. Scouts from other model families in Table 16 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	problems	(train)	(test)	(%)	per candidate	pct. ratio
	Train / calib. / test	Samples per candidate	Samples per candidate	Test acc.	Median output	90th/10th
LiveCodeBench ( Jain et al., 2024 )	441 / 110 / 341	16/12/8/6/3	16/12/8/6/3	52–90	0.7–7.0k	11–121
Omni-MATH ( Gao et al., 2024 )	275 / 75 / 1,000	4/3/3/2/2	1/1/1/1/1	42–71	1.4–10.5k	10–126
MMLU-Pro ( Wang et al., 2024 )	550 / 150 / 6,500	4/3/3/2/2	1/1/1/1/1	62–86	0.2–1.6k	7–29
AIME 1983–2024 ( Zhang, 2025 )	513 / 139 / 281	4/3/3/2/2	4/3/3/2/2	57–98	1.8–7.7k	4–16
APPS ( Hendrycks et al., 2021 )	489 / 123 / 377	1/1/1/1/1	1/1/1/1/1	58–86	0.6–10.3k	10–91

Appendix

Table 8: Pools. Split sizes (problems with a valid answer from every candidate), median samples per candidate (in Table 2 ’s order), and ranges over candidates of test accuracy, median output length (tokens) and the 90th/10th-percentile length ratio. Omni-MATH and MMLU-Pro train on Omni-MATH-500 and a 1,000-question subject-stratified sample and test on 1,000 and 6,500 further problems; the other datasets use a random held-out split. Billed rates (USD per million input / output tokens): gpt-oss-20b 0.029 / 0.134, deepseek-v4-flash (StreamLake) 0.042 / 0.084, gpt-oss-120b 0.020 / 0.257.

	Headroom		Ours vs. median		Capture
Test set	reasoning	non-reas.	reasoning	non-reas.	reasoning	non-reas.
LiveCodeBench (341)	40.1	21.4	27.8	12.1 (5.4)	.69	.53
Omni-MATH (1,000)	53.1	40.5	31.5	6.8 (1.6)	.59	.17
MMLU-Pro (2,000)	59.8	26.0	25.8	0.8 (2.2)	.43	.03

Appendix

Table 9: Reasoning versus non-reasoning candidates on the same problems. Headroom, our saving relative to median pricing (s.d. over training sets) and capture, for five reasoning and five non-reasoning candidates, on the test problems every candidate of both pools answers validly (MMLU-Pro: a 2,000-problem subset). Here perfect cost knowledge prices each problem at its billed cost, so Omni-MATH’s reasoning headroom differs slightly from Table 3 . The non-reasoning saving excludes zero in every training set on LCB and Omni-MATH and in none on MMLU-Pro.

Cost saved by ours vs.	LCB ( $n$ =341)	Omni-MATH ( $n$ =1,000)	MMLU-Pro ( $n$ =6,500)
Median pricing	27.8 [13.9, 35.6]	31.5 [23.9, 37.7]	23.4 [17.1, 29.0]
Mean pricing	27.8 [19.5, 34.6]	34.3 [26.3, 40.1]	25.8 [20.4, 30.6]
Prompt-feature GBM	20.7 [10.1, 29.0]	25.8 [15.9, 35.0]	20.0 [13.9, 25.9]
MixLLM-style embeddings	16.2 [7.9, 22.6]	24.7 [15.9, 31.7]	15.5 [10.6, 20.5]
CARROT-style $k$ NN	17.5 [7.9, 25.4]	23.2 [14.9, 29.3]	14.3 [8.3, 20.7]
ZeroRouter pricing, our success	16.8 [9.4, 25.3]	21.0 [10.5, 31.6]	22.0 [16.5, 27.9]

Appendix

Table 10: Cost saved at matched accuracy, billed costs. Cost saved (%) by our cost probe relative to each estimator, with 95% intervals. Every row uses our success predictions except ZeroRouter pricing with its own success model; both ZeroRouter rows predict item parameters from our scout’s activations. All intervals exclude zero except against ZeroRouter with its own success model on Omni-MATH (ours ahead in 7 of 10 training sets). Last row: the headroom, of which the cost probe captures 69%, 61% and 40%. Estimators in Section B.5 .

Dataset		median	external	of all external	from pred. success	behind
	Headroom	Ours vs	Ours $-$ best	Splits ahead	Ours $-$ length	Splits ahead /
LiveCodeBench	45.9 $\pm$ 3.0	34.1 $\pm$ 3.5	12.9 $\pm$ 2.9	10	2.2 $\pm$ 2.6	4 / 0
Omni-MATH	41.7 $\pm$ 2.5	19.0 $\pm$ 3.3	12.0 $\pm$ 3.8	9	$-$ 1.4 $\pm$ 2.6	1 / 2
MMLU-Pro	58.6 $\pm$ 1.0	25.8 $\pm$ 2.3	16.3 $\pm$ 2.6	10	12.8 $\pm$ 6.1	10 / 0
AIME	28.5 $\pm$ 2.7	15.3 $\pm$ 2.6	9.5 $\pm$ 2.6	10	2.5 $\pm$ 2.4	3 / 0
APPS	39.5 $\pm$ 6.0	15.6 $\pm$ 4.9	3.9 $\pm$ 2.8	0	$-$ 1.5 $\pm$ 2.4	0 / 0

Appendix

Table 11: Ten random re-splits per dataset , which, unlike the training sets, also reshuffle the test problems (original sizes; every probe and estimator refitted, configurations chosen on calibration). Mean

\pm

s.d. over splits of cost saved (%) relative to median pricing, and of our saving relative to the best external estimator in each split and to length from predicted success. “Splits ahead”: splits in which ours is significantly ahead of every external estimator, and (last column) ahead of / behind length from predicted success. No external estimator is significantly cheaper than ours in any split. Values differ from Table 3 ; the conclusions do not.

Figure 3: Candidate shares at matched accuracy (one training set). Share of test queries (%) sent to each candidate at the middle of the accuracy band that all three pricings reach; right: spend per 1,000 queries. Candidates are ordered by mean cost per query; shares below 10% are unlabelled.

		Without					Pair: gpt-oss-20b low and
Dataset	All five	20b low	20b med.	deepseek	120b med.	120b high	deepseek	120b high
LiveCodeBench	27.8	6.5	28.4	22.4	27.8	28.3	29.0	32.1
Omni-MATH	31.5	16.2	32.6	15.2	31.8	32.0	33.8	22.4
MMLU-Pro	23.4	–	25.5	17.0	24.0	23.2	27.0	25.9
AIME	14.9	$-$ 0.2	14.9	7.0	15.0	14.9	15.0	10.7
APPS	16.8	1.3	15.0	11.5	11.6	19.3	16.3	9.1

Appendix

Table 12: Which candidates the saving needs. Cost saved (%) by our cost probe relative to median pricing, with the router refitted on a subset of the five candidates; bold: 95% interval excludes zero. –: the frontiers do not overlap (MMLU-Pro without gpt-oss-20b at low effort, where deepseek-v4-flash has the lowest median cost per call).

Dataset	20b low	20b med.	deepseek	120b med.	120b high
LiveCodeBench	.74 / .85 / .99	.71 / .96 / .99	.72 / .92 / .99	.73 / .95 / .99	.75 / .94 / .98
Omni-MATH	.68 / .60 / .84	.66 / .86 / .89	.71 / .79 / .95	.72 / .87 / .94	.73 / .88 / .95
MMLU-Pro	.61 / .42 / .71	.52 / .68 / .79	.26 / .62 / .92	.62 / .76 / .88	.45 / .67 / .89
AIME	.41 / .52 / .89	.44 / .85 / .93	.50 / .79 / .97	.49 / .86 / .92	.51 / .87 / .94
APPS	.56 / .69 / –	.54 / .80 / –	.63 / .86 / –	.63 / .88 / –	.68 / .88 / –
BigCodeBench	.69 / .66 / .97	.61 / .86 / .96	.47 / .71 / .95	.71 / .85 / .97	.45 / .67 / .88

Appendix

Table 13: How much of each candidate’s length we predict, and how much depends on the problem. Test

R^{2}

of log mean output length: our probe / a linear map from the other candidates’ actual lengths on the same problem / the share of variance between problems rather than between samples. A dash means the share is not computable, because APPS has one sample per problem.

Figure 4: Length labels needed. Cost saved relative to median pricing fitted on the same problems, when cost is fitted on

n

labelled training problems: our cost probe (blue) versus length from predicted success (red), which needs length labels only to map success logits to length. Two random subsets per training set; bands:

\pm

1 s.d. over training sets.

	LCB	Omni-MATH	MMLU-Pro	SuperGPQA	BBEH
Ours, $k$ =10	19.5	22.7	19.2	26.7	9.7
ZeroRouter onboarding, $k$ =10	7.3	17.4	9.8	13.7	3.4
CARROT-style $k$ NN, $k$ =10	5.2	11.4	9.5	13.8	8.3
Median length + base rate, $k$ =10	$-$ 13.6	7.6	2.1	9.0	1.8
Ours $-$ ZeroRouter, $k$ =10	12.2 [ $-$ 5.9, 26.7]	5.2 [ $-$ 7.9, 18.2]	9.4 [ $-$ 0.9, 19.3]	13.0 [ $-$ 1.8, 25.9]	6.3 [ $-$ 11.5, 29.1]
Ours $-$ ZeroRouter, $k$ =50	9.6 [3.0, 18.7]	5.1 [ $-$ 5.4, 17.7]	8.1 [ $-$ 0.5, 16.5]	12.8 [4.8, 20.8]	4.8 [ $-$ 1.5, 10.5]

Appendix

Table 14: Onboarding a new model from

k

examples. Cost saved (%) relative to the full pool at median pricing, mean over three held-out models (gpt-oss-20b with both efforts, gpt-oss-120b with both efforts, deepseek-v4-flash) and four random choices of the

k

labelled problems per training set (the same problems for every method). Differences in points, with 95% intervals.

	Ours vs. embedding cost probe			Ours vs.	Success AUC
Dataset	Ridge	MixLLM	$k$ NN	text router	text	ours
LiveCodeBench	16.9	19.5	23.1	12.0	.75	.83
Omni-MATH	15.8	18.8	19.6	18.5	.73	.83
MMLU-Pro	15.8	13.3	12.0	12.6	.64	.72
AIME	9.7	10.1	12.2	6.8	.65	.78
APPS	6.5	11.7	9.0	5.0	.75	.84

Appendix

Table 15: A text-embedding model of the scout’s size. Cost saved (%) by our router relative to routers that predict cost from Qwen3-Embedding-4B (our ridge recipe, a MixLLM-style ensemble, CARROT-style

k

NN; all with our success probes) and to a fully text-based router that also predicts success from the embedding. Bold: 95% interval excludes zero. Success AUC: mean over candidates on the test set.

	Omni-MATH			MMLU-Pro
Scout	$R^{2}$	vs. median	Ours vs. it	$R^{2}$	vs. median	Ours vs. it
Qwen3-0.6B	.61	27.9	7.3 [ $-$ 0.6, 15.7] (5)	.33	17.0	8.2 [0.5, 14.0] (8)
Qwen3-1.7B	.65	29.0	4.1 [ $-$ 5.5, 13.4] (4)	.40	19.4	5.2 [ $-$ 1.5, 12.1] (7)
Qwen3-4B (original)	.66	28.1	3.6 [ $-$ 6.5, 12.3] (4)	.44	20.6	3.4 [ $-$ 2.5, 7.4] (8)
Phi-4-mini (3.8B)	.64	28.2	7.6 [ $-$ 2.4, 16.5] (7)	.43	21.4	2.0 [ $-$ 6.2, 6.8] (5)
Granite-3.3 (2.5B)	.62	25.1	5.6 [ $-$ 3.3, 14.6] (3)	.38	18.2	7.1 [2.1, 12.8] (10)

Appendix

Table 16: Other scouts. Each scout’s activations feed both probes, refitted with the paper’s recipes (Omni-MATH and MMLU-Pro, billed costs).

R^{2}

: test log-length

R^{2}

, mean over candidates. vs. median: cost saved (%) relative to median pricing with the row’s own success predictions. Ours vs. it: cost saved by our router relative to the row’s, with 95% intervals (bold: excludes zero) and the number of training sets in which ours is significantly ahead; ours is significantly behind in one training set only (Phi-4-mini, MMLU-Pro).

	Ours vs median,	Best external	Ours $-$ best	Ours $-$ length	Ours: frontier loss
Dataset	deployed	(deployed)	external	from pred. success	/ target miss
LiveCodeBench	31.3 (7.0)	MixLLM 14.9	16.3 (10)	2.7 (2)	2.4 / 2.6
Omni-MATH	35.0 (2.2)	ZeroRouter 13.9	21.5 (10)	$-$ 1.6 (0)	2.0 / 5.3
MMLU-Pro	31.1 (4.8)	$k$ NN 18.1	12.7 (10)	17.3 (10)	1.0 / 2.3
AIME	19.6 (2.1)	ZeroRouter 10.3	9.4 (9)	2.2 (2)	1.3 / 4.6
APPS	23.3 (5.1)	ZeroRouter 15.8	5.2 (1)	$-$ 2.4 (0)	9.4 / 3.6

Appendix

Table 17: Deployable policies. Each estimator picks on the calibration split the cheapest

V

reaching each of ten accuracy targets, applied once to the test set. Cost saved (%) relative to median pricing’s deployed policies at the accuracy reached, mean over targets (in parentheses: s.d. over training sets for our saving; for differences, the number of training sets in which ours is significantly ahead; ours is significantly behind only in one APPS and one Omni-MATH set, against length from predicted success). Last column: extra cost over our own test frontier at the accuracy reached (%) / mean

|

test accuracy

-

target

|

(points). Billed prices.

LLM Output Length Prediction

See all 8 papers

The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More
Mar 25, 2026Lingjiao Chen, Chi Zhang, Yeye He +3Cost-Aware Inference Large Reasoning Models
Stanford University · Microsoft Research · CMU +1
Dynamic LLM Routers are Often Misguided
Oct 2, 2026Sam Wang, Julia White, Sahibzada Allahyar +3LLM Routing Cost-Aware Inference
Fastino Labs
When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing
Nov 3, 2026Teng-Ruei ChenLLM Routing Language Model Reasoning Evaluation
Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Krixvon, Taipei, Taiwan

Mar 25, 2026J/K move · Enter open · S save

The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More

Lingjiao Chen, Chi Zhang, Yeye He, Ion Stoica+2

Stanford University · Microsoft Research · CMU+1

Cost-Aware Inference Large Reasoning Models

Open paper AskPDF arXiv

Developers and consumers increasingly choose reasoning models (RMs) based on their listed API prices. However, how accurately do these prices reflect actual inference costs? We conduct the first systematic study of this question, evaluating 8 frontier RMs across 12 diverse tasks covering competition math, science QA, code generation, and multi-domain agents. We uncover the pricing reversal phenomenon: in 32% of model-pair comparisons, the model with a lower listed price actually incurs a higher total cost, with reversal magnitude reaching up to 28x. For example, Gemini 3 Flash's listed price is 80% cheaper than GPT-5.4's, yet its actual cost across all tasks is 38% higher. We build a formal cost attribution framework based on Shapley value, and leverage it to trace the dominating contributors to vast heterogeneity in thinking token consumption and number of interaction turns: on the same query, one model may use 900% more thinking tokens than another, or 10x more turns of environment interactions. We further show that per-query cost prediction is fundamentally difficult: repeated runs of the same query yield thinking token variation up to 9.7x, establishing an irreducible noise floor for any predictor. Thus, we propose cost distribution prediction as an open challenge. Our findings demonstrate that listed API pricing is an unreliable proxy for actual cost, calling for cost-aware model selection and transparent per-request cost monitoring.

Abstract

LLM Output Length Prediction

Explore similar work

The Price Reversal Phenomenon: When Cheaper Reasoning Models Cost More

Dynamic LLM Routers are Often Misguided

When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing