Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation

Hamed Khosravi; Xiaoming Huo

cs.LGAug 30, 2026

Which LLM for Which Work? Budgeted Model Allocation under Uncertain Evaluation

Authors: Hamed Khosravi, Xiaoming Huo

Organizations: H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332

Abstract

A company with a fixed artificial intelligence (AI) budget must decide which large language model (LLM) handles each recurring workload. What it lacks is the quality table, how well each model performs on each workload. Given that table, the decision is a multiple-choice knapsack problem and is routine to solve, so estimating it is the difficulty, and that estimation fails in two ways. Models are rarely compared on the same work, and the recorded score is usually a proxy rather than the outcome the company values. Causal and off-policy methods repair the first but condition on the second, while evaluator-validation methods estimate the second but stop short of the decision. Worse, buying more re-evaluation cannot settle the second: randomization governs which requests are scored, not how a score is produced, so the table stays uncertain however much evaluation is purchased. Yet the deployment decision may still be determined even when the table is not. We therefore ask whether one assignment stays optimal across every quality table consistent with the evidence. For the fixed-budget problem, this admits an exact two-solve certificate: solve once at the estimated table and once at a least-favourable table. Agreement certifies the assignment; disagreement identifies the model-workload pairs where further evidence can matter. We propose CASE (causal active sequential experimentation), which targets evaluation to those pairs and repeats the test as evidence accumulates. On a production log, the measurement failure is the larger of the two: correcting assignment exactly still leaves most of the loss, and randomized re-evaluation does not remove it. In our experiments, the available evidence often does not determine the assignment. On paid software tasks, better information about model quality yields more savings than further optimization of the assignment on the same estimates.

Figures & tables

	evaluator = click (D1)		evaluator = target (D0)
Deployed decision	A1 (selective)	A0 (randomized)	A1 (selective)	A0 (randomized)
Trust-the-log	$1.76\pm 0.14$	$1.49\pm 0.07$	$1.76\pm 0.10$	$1.94\pm 0.08$
CASE	$1.57\pm 0.06$	$1.60\pm 0.06$	$1.74\pm 0.06$	$1.72\pm 0.06$
Traffic vote	$1.94\pm 0.04$	—	$1.94\pm 0.04$	—
Cellwise-perfect repair	$2.01\pm 0.02$	$2.02\pm 0.02$	$0.00$	$0.00$

Table 1: Removing assignment bias does not remove measurement loss. Target-value loss is measured in the KuaiRand like outcome (

\times 10^{3}

) with a

35\%

evidence budget, reported as mean

\pm

one standard error over

30

log partitions. The traffic vote assigns each workload to its most-used model and does not use the evaluator. Under randomized assignment there is no most-used model to vote for, so those cells are dashed. Allocations use the quality-constrained rule ( 2 ) with

\delta=0

. Target-outcome scoring is unbiased by construction but noisier because likes are

36

times rarer than clicks.

Figure 1: Randomized evidence repairs selection but leaves the evaluator floor. Target-value loss of ( 1 ) against

\Gamma

on LLMRouterBench [ 19 ] , with a

35\%

evidence budget and

\pm

SE over

30

partitions

\times

6

seeds. Dashed is the unachievable Cellwise-perfect assignment repair .

Corpus	$\Gamma$ / cheapest	median margin	margin $\geq 0.01$	margin $\geq 0.02$
LLMRouterBench	$1.25\times$	$0.0456$	$23/30$	$21/30$
LLMRouterBench	$2\times$	$0.0179$	$20/30$	$14/30$
LLMRouterBench	$8\times$	$0.0026$	$5/30$	$0/30$
RouterBench	$1.25\times$	$0.0132$	$16/30$	$10/30$
RouterBench	$2\times$	$0.0133$	$18/30$	$9/30$
RouterBench	$8\times$	$0.0037$	$6/30$	$1/30$

Table 2: The range is more likely to determine the allocation under a tight serving budget on LLMRouterBench. The decision margin is the largest evaluator spread

\eta

for which the allocation stays optimal over all of

\mathcal{U}

, with counts of the

30

partitions reaching two spreads.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Name	What it is
Primitives (§ 2 )
$w_{j}$ , $p_{kj}$	share, price	workload $j$ ’s traffic share, and model $k$ ’s per-request price on it
$N$ , $\Gamma$	volume, serving budget	requests per budget period, and the spend allowed on them
$\mathcal{K}_{j}$	candidate set	models supported on workload $j$
Tables (§ 2 , § 3.1 )
$\mu^{\mathrm{ref}}_{kj}$	target-outcome table	average designated outcome of model $k$ over all workload- $j$ requests; the unknown

Appendix

Table 3: Notation, in the order this section introduces it. The tables

\mu^{\mathrm{ref}}

M

and

R

differ only in who was measured and what the measurement was . Only

M

and

R

are observable, and only through estimates. The evaluator gap

C

and its parts are introduced in App. B and used only in the appendices.

Figure 2: Illustrative budgeted allocation , not an empirical result. Rows are models with per-thousand-request prices, columns are workloads with monthly volumes, cells report quality, and outlined cells show the allocation, the heavier outline marking the two decisions that differ. The right panel changes only the Small model’s target quality, yet coding and drafting both move.

Correction surface	Deployable?	A1–D0	A1–D1
Raw routed log	yes	$0.0499{\scriptstyle\pm 0.0030}$	$0.0469{\scriptstyle\pm 0.0025}$
CASE row offset	yes	$0.0338{\scriptstyle\pm 0.0020}$	$0.0338{\scriptstyle\pm 0.0017}$
Oracle row offset, no budget limit	no	$0.0318{\scriptstyle\pm 0.0025}$	$0.0337{\scriptstyle\pm 0.0024}$
Cellwise-perfect assignment repair	no	$0.0000$	$0.0030{\scriptstyle\pm 0.0006}$

Appendix

Table 4: Correction-family ceilings at a

35\%

evidence budget over

30

partitions. The cellwise repair removes assignment error everywhere. It reaches zero target-value loss under faithful scoring and leaves

0.0030

under distorted scoring.

Arm	Audit $	A1–D0	A1–D1
Trust the log	$0$	$0.0499$	$0.0469$
Outcome regression	$0$	$0.0360$	$0.0370$
Stabilized IPW, exact propensities	$0$	$0.0327$	$0.0335$
Cross-fitted DR, exact propensities	$0$	$0.0360$	$0.0348$
Stabilized IPW, estimated propensities	$0$	$0.0375$	$0.0384$
Cross-fitted DR, estimated propensities	$0$	$0.0435$	$0.0456$

Appendix

Table 5: Target-value loss of log-only causal estimators under selective routing over

30

partitions. The estimators use no randomized-audit budget but require the logging propensities. The cellwise-perfect repair is an oracle and carries no audit price.

Corpus	$\Gamma$ / cheapest	box	model-level	box	model-level	budget binds
		median margin		margin $\geq 0.02$
LLMRouterBench	$1.25\times$	$0.0456$	$0.0536$	$21/30$	$21/30$	$30/30$
LLMRouterBench	$1.5\times$	$0.0180$	$0.0233$	$15/30$	$17/30$	$30/30$
LLMRouterBench	$2\times$	$0.0179$	$0.0209$	$14/30$	$17/30$	$30/30$
LLMRouterBench	$3\times$	$0.0122$	$0.0129$	$7/30$	$12/30$	$30/30$
LLMRouterBench	$5\times$	$0.0103$	$0.0132$	$5/30$	$9/30$	$30/30$

Appendix

Table 6: The full budget sweep, over the free box of ( 10 ) and over the model-level set of Assumption B.2 . Median margin over

30

partitions and counts with margin at least

0.02

, under

\mathcal{U}

as written and under the row-structured set

\{C:C_{kj}=C_{k}\}

. The last column gives the partitions on which the serving budget still binds.

Contrast, $35\%$ budget	Mean	Partition	Request	Slice clusters
A1–D0, CASE $-$ trust	$-0.0161$	$[-.0219,-.0103]$	$[-.0190,-.0132]$	$[-.0410,-.0083]$
A1–D0, CASE $-$ R-EPIG corr	$+0.0003$	$[-.0019,+.0025]$	$[-.0007,+.0013]$	$[-.0017,+.0007]$
A1–D1, CASE $-$ trust	$-0.0130$	$[-.0179,-.0081]$	$[-.0159,-.0103]$	$[-.0293,-.0087]$
A1–D1, CASE $-$ pop.-operational oracle	$+0.0309$	$[+.0271,+.0346]$	$[+.0278,+.0338]$	$[+.0193,+.0444]$
A1–D1, measurement floor $G_{\mathrm{meas}}$	$+0.0030$	$[+.0017,+.0043]$	$[+.0011,+.0047]$	$\mathbf{[-.0002,+.0259]}$
A0–D0, CASE $-$ trust	$+0.0107$	$[+.0067,+.0147]$	$[+.0083,+.0132]$	$\mathbf{[-.0013,+.0142]}$

Appendix

Table 7:

95\%

intervals for the confirmatory contrasts under three resampling schemes. R-EPIG corr is a decision-information acquisition rule that reveals the top-two contender in the most uncertain column, run on the selection-adjusted surface. Bold marks a slice-cluster interval that includes zero where the partition and request intervals exclude it. A dash marks a contrast without a request- or slice-level analogue.

	evaluator = click (D1)		evaluator = target (D0)
Deployed decision	A1 selective	A0 randomized	A1 selective	A0 randomized
Trust-the-log	$1.76\pm 0.14$	$1.49\pm 0.07$	$1.76\pm 0.10$	$1.94\pm 0.08$
Uniform audit	$1.59\pm 0.06$	$1.59\pm 0.06$	$1.83\pm 0.05$	$1.80\pm 0.07$
CASE	$1.57\pm 0.06$	$1.60\pm 0.06$	$1.74\pm 0.06$	$1.72\pm 0.06$
Traffic vote	$1.94\pm 0.04$	—	$1.94\pm 0.04$	—
Cellwise-perfect repair	$2.01\pm 0.02$	$2.02\pm 0.02$	$0.00$	$0.00$

Appendix

Table 8: The KuaiRand comparison in full with uniform randomized auditing added. Target-value loss is measured in units of the KuaiRand like indicator multiplied by

10^{3}

at the

35\%

evidence budget. Means and standard errors are over

30

partitions. Under the target evaluator the cellwise-perfect repair deploys the reference table itself, zero by construction, and the dash marks the traffic vote on the randomized stream, which has no production traffic to read.

Evaluator	Corpus and target	Items	Agrees	Position-flip	Longer answer
gpt-4o-mini	LMArena, human verdict	$19{,}368$	$0.430$	$0.335$	$+0.243$
gpt-4o-mini	HelpSteer2, human correctness	$2{,}400$	$0.717$	—	$+0.085$

Appendix

Table 9: Randomized reruns do not establish measurement validity. LMArena [ 6 ] and HelpSteer2 [ 30 ] . Agrees is raw agreement, not chance-corrected, position-flip the share of battles whose verdict changes when answers are swapped, and Longer answer the judge’s partial length preference. HelpSteer2 is rated one response at a time, so it has no position to swap and the dash marks what the corpus cannot define. Its agreement compares the responses within each of the

1{,}182

prompts holding more than one, and its length preference the

1{,}160

of those holding two of unequal length.

Setting	Information used for the decision	Target quality	Deployment or total cost
LLMRouterBench	raw production log	$0.4798{\scriptstyle\pm 0.0041}$	$\$ 3{,}109{\scriptstyle\pm 186}$ per 1M
LLMRouterBench	selection-adjusted scores	$0.4924{\scriptstyle\pm 0.0027}$	$\$ 3{,}850{\scriptstyle\pm 91}$ per 1M
LLMRouterBench	designated outcome	$0.5255{\scriptstyle\pm 0.0021}$	$\$ 4{,}386{\scriptstyle\pm 39}$ per 1M
SWE-Lancer	best single model	—	$\$ 71{,}625$ total
SWE-Lancer	observable success estimate	—	$\$ 70{,}625$ total
SWE-Lancer	realized per-task outcome	—	$\$ 35{,}625$ total

Appendix

Table 10: Economic scale of better quality information. The LLMRouterBench block reports target quality and deployment cost per million requests under the budgeted rule. The SWE-Lancer block reports total expected cost over

150

managerial tasks, where lower is better. Quality is not reported for SWE-Lancer because the human fallback completes every task, making the comparison one of cost alone.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Feb 17, 2026

LLM-as-Judge on a Budget

LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pairs. Since LLM judgments are stochastic, practitioners commonly query each pair multiple times to estimate mean scores accurately. This raises a critical challenge: given a fixed computational budget

B

, how to optimally allocate queries across

K

prompt-response pairs to minimize estimation error? We present a principled variance-adaptive approach leveraging multi-armed bandit theory and concentration inequalities. Our method dynamically allocates queries based on estimated score variances, concentrating resources where uncertainty is highest. Further, our algorithm is shown to achieve a worst-case score-estimation error of

\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)

σ_i^2

being the unknown score variance for pair

i \in [K]

with near-optimal budget allocation. Experiments on Summarize-From-Feedback and HelpSteer2 demonstrate that our method significantly outperforms uniform allocation, reducing worst-case estimation error while maintaining identical budgets. Our work establishes a theoretical foundation for efficient LLM evaluation with practical implications for AI safety, model alignment, and automated assessment at scale.

Aadirupa Saha, Aniket Wagde, Branislav Kveton

UIC · Adobe

LLM Evaluation LLM-as-a-Judge Adaptive Sampling

Ask PDF

May 11, 2026

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on every example. Multi-armed bandit (MAB) algorithms can reduce the number of LLM calls by sequentially selecting the next model-example pair to evaluate, thereby avoiding wasted evaluations on clearly underperforming models. Further savings can be achieved by predicting model scores from the partially observed model-example score matrix using low-rank factorization. However, such predictions are not ground truth: they can be biased and may therefore lead to incorrect identification of the best model. In this work, we propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity. Specifically, we derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance. This enables the construction of valid finite-sample confidence intervals in our setting, where models are selected adaptively and examples are sampled without replacement. Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost while accurately identifying the best-performing model.

Elad Tolochinsky, Yaniv Tenzer, Yaniv Romano

Department of Computer Science, Technion – Israel Institute of Technology · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology

Model Selection Multi-Armed Bandits LLM Evaluation Robust Matrix Completion Best-Arm Identification Doubly Robust Estimation

Ask PDF

May 22, 2026

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

Evaluating large language models increasingly relies on LLM-as-a-judge protocols, but such evaluations remain costly: different judges have different prices and reliabilities, and the difficulty of each prompt-response pair can vary substantially. This raises a basic allocation question: under a fixed budget, how should one distribute evaluation queries across heterogeneous judges and instances to obtain the most accurate score estimates? We formalize this question as budgeted heteroskedastic multi-judge estimation. Given

K

prompt-response pairs,

J

judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an

\ell_p

-error. Our first contribution is to analyze the inverse-variance weighted estimator (IVWE) and to derive the oracle allocation that minimizes its error rate. Since this allocation depends on the unknown variances, we then address the practical unknown-variance setting by proposing EST-IVWE, an adaptive algorithm that constructs and leverages optimistically biased variance estimates to stabilize the empirical allocation. We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget. Our second and central theoretical contribution is a matching local minimax lower bound, which establishes the instance-optimality of the proposed algorithms. A key technical insight is that Fano-type high-probability arguments are too coarse for this problem: their packing construction loses the local variance structure that governs the optimal allocation. We instead use an Assouad-type in-expectation argument, based on local perturbations, which preserves this structure and yields the sharp allocation-dependent lower bound. Finally, we numerically validate the superiority of our approach over naïve uniform allocation on synthetic and HelpSteer2 datasets.

Junghyun Lee, Sanghwa Kim, Yassir Jedra +2

KAIST AI · ICL EEE · KTH EECS

Parameter Estimation LLM Evaluation LLM-as-a-Judge Minimax Estimation Adaptive Sampling

Ask PDF

Abstract

LLM Evaluation

Explore similar work

LLM-as-Judge on a Budget

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

Instance-Optimal Estimation with Multiple LLM Judges on a Budget