When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing

Teng-Ruei Chen

cs.LGNov 3, 2026

When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing

Authors: Teng-Ruei Chen

Organizations: Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu, Taiwan · Krixvon, Taipei, Taiwan

Abstract

Routing benchmarks often judge a router against a hindsight oracle that counts a query as solved whenever any model's recorded answer is correct, but under stochastic decoding that oracle mixes two different objects: the clairvoyant ceiling of committing to one model per query and the premium of calling several models and keeping any correct answer. A companion study measured this premium at 3.5 percentage points (pp) for 11 small open models on MATH-500; how large it is once current reasoning models enter the pool was unknown. We measure it with a fixed-protocol, pre-committed generation on a single NVIDIA B300: all 500 MATH-500 problems, 30 seeded samples per problem, Llama-3.3-70B-Instruct and NVIDIA's Nemotron Nano 9B v2 and Nemotron Super 49B v1.5 (a NAS derivative of the same 70B) under vendor sampling, a 65,536-token cap, a pinned Math-Verify scorer, and an append-only attempt ledger, for 45,000 scored responses. The two reasoning models both reach 96.1% accuracy against 76.2% for the 70B baseline. The empirical product-coupling premium is 0.215-0.588 pp across all two- and three-model pools; the empirical plug-in selection headroom over always calling Nano is 1.27 pp; and resampling Nano three times gives practically the same empirical any-correct rate as calling all three models once. Only 31 responses reached the cap, 25 of them with exactly periodic tails, yet the capped responses account for up to 60.2% of a model's output-length-squared proxy, and long generations made a fourth planned model, Llama-3.1-8B, too slow to finish. The premium is far below the companion's 3.5 pp, but models, pool size and protocol changed together; an oracle gap should therefore be reported with the pool, the protocol, the samples per cell and the decoding budget that produced it.

Figures & tables

Fig. 1: Overview. Top: the question; the prior result of the companion study [ 1 ] , a

3.54

pp union premium for 11 small open models on MATH-500 under its own protocol; and the protocol fixed before generation on one NVIDIA B300, planned for four models, of which three completed (the scoring amendment is described in Section IV ). Bottom: the findings. The empirical union premium over the single-commit ceiling is 0.215–0.588 pp across pools. The empirical plug-in selection headroom over always calling Nano is 1.27 pp, an oracle built from one sample per model ranges over 97.4–98.6%, and resampling Nano three times gives practically the same empirical any-correct rate as calling all three models once. Of

45{,}000

responses, 31 hit the cap, 25 of them with exactly periodic tails; the capped responses take 60.2% of the 70B output-length-squared proxy, and the Llama-3.1-8B run was stopped for cost.

Study	Multi-sample cell	Ceiling vs. union	Protocol fixed	Current	Decoding-cost
	success rates	separated	before generation	reasoning models	accounting
RouterBench [ 4 ]	–	–	–	–	–
RouteLLM [ 3 ]	–	–	–	–	–
Repeated sampling [ 11 ]	✓	–	–	–	✓
Routing oracle [ 1 ]	✓	✓	archived tensors	–	–
This report	✓	✓	✓	✓	✓

TABLE I: Positioning against prior evaluations. ✓: addressed; –: not addressed.

Model	Revision	$T$	top- $p$	Reasoning
Llama-3.3-70B-Instruct	6f6073b4	$0.6$	$0.9$	–
Nemotron Nano 9B v2	6533e8de	$0.6$	$0.95$	on
Nemotron Super 49B v1.5	420ba7d2	$0.6$	$0.95$	on
All: BF16 weights and KV cache, cap $65{,}536$ , 30 samples/query.

TABLE II: Models and decoding (pinned Hugging Face revisions).

GPU	1 $\times$ NVIDIA B300 SXM6 AC (Blackwell)
	compute capability 10.3, 148 SMs
GPU memory	275,040 MiB HBM, ECC on
Power limit	1,100 W
Host link	PCIe Gen6 $\times$ 16, KVM pass-through
CPU / RAM	30 vCPU AMD EPYC 9575F / 251 GiB
OS / driver	Ubuntu 24.04.5, Linux 6.8 / 580.178.04

TABLE III: Execution platform.

Model	Acc. (%)	All-30	None-30	Median	$p_{99}$	Capped
	$\bar{\hat{p}}$	correct	correct	tokens	tokens	at $65{,}536$
Llama-3.3-70B	76.19	260	45	463	2,039	6
Nemotron Nano 9B v2	96.13	433	2	1,890	18,096	9
Nemotron Super 49B v1.5	96.09	431	6	2,249	22,804	16

TABLE IV: Per-model results over

500\times 30

samples.

Fig. 2: Per-model results. (a) Distribution of per-problem success rates

\hat{p}_{im}

over 30 samples (log scale); the legend gives the number of problems each model solves in every sample. (b) Accuracy by MATH difficulty level.

Fig. 3: Ceilings and premiums per pool (Q1 = 70B + Super, Q2 = Nano + Super, Q3 = 70B + Nano, Q4 = all three). (a) Single-commit ceiling

\bar{\hat{R}}

(one call, clairvoyant), product union

\bar{\hat{U}}^{\perp}

(all models, gold-based selection) and the premium between them; the grey band is the range, over the 30 sample indices, of an oracle computed from one recorded sample per model. (b) Gain over the best single model, split into empirical plug-in selection headroom (

\bar{\hat{R}}

minus the best model’s accuracy) and union premium (

\bar{\hat{U}}^{\perp}-\bar{\hat{R}}

), in percentage points.

Pool	Members	$k$	$\bar{\hat{R}}$ (%)	$\bar{\hat{U}}^{\perp}$ (%)	$\bar{\hat{\Delta}}^{\perp}$ (pp)	Marginal interval (pp)
Q1	70B + Super	2	96.67	96.88	0.215	$[0,\ 0.400]$
Q2	Nano + Super	2	97.19	97.70	0.518	$[0,\ 0.760]$
Q3	70B + Nano	2	96.69	97.05	0.360	$[0,\ 0.653]$
Q4	70B + Nano + Super	3	97.40	97.99	0.588	$[0,\ 0.827]$

TABLE V: Single-commit ceiling, product-coupling union and premium per pool, with the marginally identified premium interval of [ 1 ] . Means over 500 problems; pp = percentage points.

Fig. 4: Where the premium arises. (a) Per-problem premium against ceiling for the three-model pool; it is bounded by

1-\hat{R}_{i}

and is zero whenever some model solves the problem in every sample. (b) Three-model premium by subject and by difficulty level.

Fig. 5: Sampling. (a) Mean plug-in premium recomputed from the first

k

of the 30 frozen samples per cell; with one sample every

\hat{p}

is 0 or 1 and the plug-in premium is identically zero. (b) Any-correct rate against calls per problem: resampling one model (lines, unbiased pass@

k

) versus calling different models once (markers, product union); gold-based selection in both.

Fig. 6: Runaway generations. (a) Share of responses at least

L

tokens long; the red line is the

65{,}536

-token cap. (b) Share of distinct tokens in the last

4{,}096

tokens of responses that ended normally (grey, length

\geq 4{,}096

) and of responses that hit the cap (

\times

); the dashed line marks a single token repeated.

Fig. 7: Cost of the decoding budget. (a) Share of responses, output tokens and the output-length-squared proxy

\sum L^{2}

taken by capped responses (log scale). (b) Length distributions of correct and wrong responses (shaded, log scale) with medians and 10–90% ranges.

Model	Wall (min)	Output tokens	Tokens/s	Capped
Llama-3.3-70B	74	9.5 M	2,147	6
Nemotron Nano 9B v2	78	46.0 M	9,806	9
Nemotron Super 49B v1.5	368	57.3 M	2,597	16

TABLE VI: Formal-run execution on the B300 (one engine call per model, 15,000 requests). Throughput includes model loading and the long-response tail.

Scorer status	70B	Nano	Super
correct	11429	14420	14413
not equivalent	3507	556	526
length-truncated	6	9	16
reasoning unterminated	0	0	13
reasoning malformed	0	0	1
no boxed answer	0	2	0

TABLE S1: Scorer status counts per model (15,000 responses each).

Pool	Best	$\bar{\hat{R}}$	$\bar{\hat{U}}^{\perp}$	Headroom	Premium	One-sample
	(%)	(%)	(%)	(pp)	(pp)	oracle (%)
Q1	96.09	96.67	96.88	0.58	0.215	95.2–98.0
Q2	96.13	97.19	97.70	1.05	0.518	97.2–98.2
Q3	96.13	96.69	97.05	0.55	0.360	96.4–98.0
Q4	96.13	97.40	97.99	1.27	0.588	97.4–98.6

TABLE S2: From the best single model to the union, per pool (Q1 = 70B + Super, Q2 = Nano + Super, Q3 = 70B + Nano, Q4 = all three). Headroom

=\bar{\hat{R}}

minus the best single model’s accuracy; premium

=\bar{\hat{U}}^{\perp}-\bar{\hat{R}}

; the one-sample oracle range is the minimum and maximum, over the 30 sample indices, of an oracle computed from one recorded sample per model.

Subject / level	$n$	70B	Nano	Super	Premium
Algebra	124	93.6	99.5	99.4	0.25
Counting & Prob.	38	74.4	95.7	97.6	0.69
Geometry	41	54.0	85.9	85.8	2.01
Interm. Algebra	97	59.0	94.5	93.9	0.79
Number Theory	62	80.0	99.6	99.6	0.00
Prealgebra	82	87.9	95.0	94.8	0.48

TABLE S3: Accuracy (%) per model and three-model premium

\bar{\hat{\Delta}}^{\perp}_{Q4}

(pp) by subject and difficulty level.

Comparison	Mean diff. (pp)	Non-zero	$W$	$p$
Super $-$ 70B	+19.89	251	1,266.5	$<10^{-15}$
Nano $-$ Super	+0.05	87	1,714	0.392
Nano $-$ 70B	+19.94	243	1,138.5	$<10^{-15}$
Friedman, 3 models			$\chi^{2}=315.8$	$<10^{-15}$

TABLE S4: Exploratory two-sided Wilcoxon signed-rank tests on per-problem differences in success counts (zero differences dropped, SciPy 1.13.1) and a Friedman test.

Calls	Strategy	Type	Any-correct (%)
2	Nano, 2 samples	resample	97.54
2	Super, 2 samples	resample	97.27
2	Nano + Super (Q2)	reroute	97.70
3	Nano, 3 samples	resample	97.99
3	Super, 3 samples	resample	97.62
3	70B + Nano + Super (Q4)	reroute	97.99

TABLE S5: Resample versus reroute at equal calls (gold-based any-correct; unbiased pass@

k

for resampling, product union for rerouting).

Model	Capped	Periodic	Share taken by capped (%)			Periodic
		tail	responses	tokens	$\sum L^{2}$	$\sum L^{2}$ (%)
Llama 70B	6	3	0.040	4.15	60.2	30.1
Nano 9B	9	8	0.060	1.28	11.4	10.2
Super 49B	16	14	0.107	1.83	12.8	11.2

TABLE S6: Capped responses: count, exactly periodic tails, and their share of responses, output tokens and the output-length-squared proxy

\sum L^{2}

; the last column is the share taken by the periodic-tail subset alone.

Model	Correct lost / tokens saved (%)
	cap $8{,}192$	cap $16{,}384$	cap $32{,}768$
Llama 70B	0.01 / 5.4	0.00 / 4.3	0.00 / 2.6
Nano 9B	4.64 / 11.6	0.51 / 2.8	0.01 / 0.7
Super 49B	7.33 / 16.4	1.19 / 5.3	0.03 / 1.0

TABLE S7: Post hoc: correct answers lost and output tokens saved under tighter caps.

Model	Median tokens		Wrong share (%)		AUROC	GPU-min
	correct	wrong	resp.	tokens	length	per 1k
Llama 70B	400	873	23.8	42.6	0.82	6.4
Nano 9B	1,833	9,924	3.9	14.7	0.89	5.4
Super 49B	2,184	12,265	3.9	15.1	0.88	25.5

TABLE S8: Length of correct and wrong responses, share of wrong responses and of their output tokens, AUROC of length as a wrong-answer signal, and GPU-minutes per 1,000 correct answers.

LLM Routing

Large language model routing selects models, tools, experts, or inference strategies to balance answer quality, cost, latency, and deployment constraints.

See all 368 papers

Nov 6, 2026

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

Routers typically choose a model per query by trading predicted correctness against cost. Most price each model at a fixed cost, as if every query used the same number of output tokens, but the output length of reasoning models varies by an order of magnitude across queries. When is per-query cost worth predicting? What factors drive cost? How cheaply can predictions be made? We predict both the success probability and the output length of candidate models from the prefill activations of a single small frozen LLM, the scout. With reasoning candidates, perfect per-query cost knowledge would save 29-61% of spending on eight of nine datasets, and much less with non-reasoning candidates on the same problems. On those eight datasets, our predicted costs save 9-33% at matched accuracy relative to median-length pricing, up to 69% of what perfect knowledge would save. They are significantly cheaper than each estimator we reimplemented or adapted from published work, including ZeroRouter and a MixLLM-style predictor, on a majority of datasets. Output length depends mostly on the problem. On most math and code datasets, harder problems are longer, so predicted success alone prices cost nearly as well. On knowledge and mixed-format datasets they are not, and predicted success falls well behind a probe trained on length. The probe keeps most of its saving with 20 labelled problems and transfers without labels to four of seven new benchmarks. On a mixture of all nine benchmarks, it saves 22% beyond pricing each benchmark at its own median length. Savings stop growing at about a 2B scout, so above 4B a larger scout costs more than it adds.

Aristides Milios, Christopher Pal

Mila – Quebec AI Institute, Université de Montréal · ServiceNow AI Research · Mila – Quebec AI Institute, Polytechnique Montréal +1

LLM Output Length Prediction LLM Routing Cost-Aware Inference Language Model Performance Prediction Language Model Probing

Ask PDF

Aug 14, 2026

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.

Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang +4

Argonne National Laboratory, Lemont, IL, USA · Oregon State University, Corvallis, OR, USA

LLM Agent Routing Efficient Language Model Reasoning Multi-Agent Collaboration Multi-Agent Systems

Ask PDF

May 7, 2026

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning

Inference-time computation has greatly enhanced the performance of large language models (LLMs) on challenging reasoning tasks, but this strategy can incur high inference costs. One solution is to route intermediate chain-of-thought (CoT) states to language models of different sizes; however, existing approaches rely on handcrafted routing strategies that limit performance, or on training large process reward models that may be infeasible in many applications. We formulate stepwise model routing as a constrained decision-making problem, which we solve by training a small control policy using reinforcement learning in conjunction with threshold calibration to tune the performance-efficiency tradeoff. We validate our method on three math benchmarks (GSM8K, MATH500, and OmniMath) on both open and closed models. Our method consistently improves the accuracy-cost tradeoff compared to handcrafted approaches, while achieving a comparable tradeoff to methods that require training large process reward models.

Wenwen Si, Insup Lee, Osbert Bastani

Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104, USA

Cost-Aware Inference Efficient Language Model Reasoning LLM Routing

Ask PDF

Abstract

LLM Routing

Explore similar work

Scout Before You Route: Predicting Task Success and Token Count from a Small LLM Prefill

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning