LLM-as-Judge on a Budget

Aadirupa Saha; Aniket Wagde; Branislav Kveton

cs.LGFeb 17, 2026

LLM-as-Judge on a Budget

Authors: Aadirupa Saha, Aniket Wagde, Branislav Kveton

Organizations: UIC · Adobe

Abstract

LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pairs. Since LLM judgments are stochastic, practitioners commonly query each pair multiple times to estimate mean scores accurately. This raises a critical challenge: given a fixed computational budget $B$ , how to optimally allocate queries across $K$ prompt-response pairs to minimize estimation error? We present a principled variance-adaptive approach leveraging multi-armed bandit theory and concentration inequalities. Our method dynamically allocates queries based on estimated score variances, concentrating resources where uncertainty is highest. Further, our algorithm is shown to achieve a worst-case score-estimation error of $\tilde{O}\left(\sqrt{\frac{\sum_{i=1}^K σ_i^2}{B}}\right)$ , $σ_i^2$ being the unknown score variance for pair $i \in [K]$ with near-optimal budget allocation. Experiments on Summarize-From-Feedback and HelpSteer2 demonstrate that our method significantly outperforms uniform allocation, reducing worst-case estimation error while maintaining identical budgets. Our work establishes a theoretical foundation for efficient LLM evaluation with practical implications for AI safety, model alignment, and automated assessment at scale.

Figures & tables

Prompt	in a chat, can I feed chatGPT informations so it can answer questions better?
Response	Yes, you can feed ChatGPT information to help it answer questions better. This is called "training" the model, and it
	involves providing it with examples of good and bad responses to questions. You can also use tools like GPT-J’s "Teacher"
	mode to provide more detailed feedback on its responses.
Rationale	The response is extremely helpful and clearly explains the concept of training a model like ChatGPT, including the use of
	tools like GPT-J’s "Teacher" mode. It directly addresses the user’s question and provides relevant information, making it
	easy to understand. The response does not miss any essential details, and its content is accurate and up-to-date. Overall,

Table 1: Prompt, response, rationale for its evaluation, and score for one example in the HelpSteer2 dataset Wang et al. (2024) .

\hat{s}_{i}(\mathcal{R}):=\frac{1}{n_{i}({\mathcal{R}})}\sum_{\ell=1}^{B}{\mathbf{1}}(i_{\ell}=i)X_{i_{\ell}},~\forall i\in[K]

Algorithm 1 ROBIN : Resource Optimization via Budget-aware INference.

\hat{s}_{i}({\mathcal{R}}_{\mathcal{H}}):=\frac{1}{n_{i}(B)}\sum_{\ell=1}^{B}{\mathbf{1}}(i_{\ell}=i)X_{\ell},~\forall i\in[K].

Algorithm 2 ROBIN-HOOD : ROBIN with Hidden resources.

Model	Attribute	50k_Uniform	50k_ ROBIN-HOOD	50k_ ROBIN	100k_Uniform	100k_ ROBIN-HOOD	100k_ ROBIN
llama-3-1-8b	complexity	0.385±0.011	0.324±0.010	0.314±0.007	0.266±0.006	0.233±0.009	0.202±0.003
gpt-4.1-nano	complexity	0.437±0.010	0.368±0.011	0.381±0.011	0.320±0.007	0.287±0.013	0.251±0.007
llama-3-1-8b	correctness	0.420±0.015	0.296±0.006	0.288±0.006	0.310±0.012	0.219±0.006	0.195±0.004
gpt-4.1-nano	correctness	0.451±0.012	0.300±0.008	0.273±0.006	0.321±0.007	0.249±0.011	0.187±0.003
llama-3-1-8b	helpfulness	0.383±0.007	0.314±0.006	0.321±0.007	0.265±0.006	0.255±0.009	0.203±0.004
gpt-4.1-nano	helpfulness	0.413±0.016	0.265±0.005	0.263±0.005	0.291±0.012	0.220±0.005	0.187±0.004

Table 2: WCE across different configurations. All values are in mean ± 1 std-deviation errors. Column headings are formatted as “number of queries _ algorithm used"

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: y-axis: Maximum error (WCE). Top row: GPT-4.1 nano, $\delta$ =0.007, warm-up period: 20164 samples. Bottom row: Llama 3.1 8B instruct, $\delta$ =0.007, warm-up period: 20164 samples.

Panel of Figure 9 — Figure 9: y-axis: Correlations with Human Ratings. Top row: GPT-4.1 nano, $\delta$ =0.007, warm-up period: 20164 samples. Bottom row: Llama 3.1 8B instruct, $\delta$ =0.007, warm-up period: 20164 samples. Legend: (Red - Pearson’s coefficient, Blue - Spearman’s coefficient, Uniform - Kendall’s Tau)

Attribute	Attribute text
Helpfulness	Helpfulness: Score 0: The response is not useful or helpful at all. The response completely missed the essence of what the user wanted. Score 1: The response is borderline unhelpful and mostly does not capture what the user was looking for, but it is still usable and helpful in a small way. Score 2: The response is partially helpful but misses the overall goal of the user’s query/input in some way. The response did not fully satisfy what the user was looking for. Score 3: The response is mostly helpful and mainly aligned with what the user was looking for, but there is still some room for improvement. Score 4: The response is extremely helpful and completely aligned with the spirit of what the prompt was asking for.
Correctness	Correctness: Score 0: The response is completely incorrect. All information provided is wrong, false or hallucinated. If the prompt asks the assistant to do a task, the task is not at all attempted, or the wrong task was attempted in the response. The response is completely irrelevant to the prompt. Score 1: The response has some correct elements but is mostly wrong or incomplete. The response may contain multiple instances of hallucinations, false information, misleading information, or irrelevant information. If the prompt asks the assistant to do a task, the task was attempted with a small amount of success. Score 2: The response contains a mix of correct and incorrect information. The response may miss some details, contain misleading information, or minor hallucinations, but is more or less aligned with what the prompt asks for. If the prompt asks the assistant to perform a task, the task is attempted with moderate success but still has clear room for improvement. Score 3: The response is mostly accurate and correct with a small amount of missing information. It contains no misleading information or hallucinations. If the prompt asks the assistant to perform a task, the task is mostly successfully attempted. Score 4: The response is completely correct and accurate to what is requested by the prompt with no necessary details missing and without false, misleading, or hallucinated information. If the prompt asks the assistant to do a task, the task is completely done and addressed in the response.
Complexity	Complexity: Score 0: (Basic) – The response uses very easy to understand language that is clear and completely interpretable by children, adults, and anyone with a functional command of the language. Score 1: (Simple) – The response uses relatively straightforward language and wording, but some schooling through elementary or a middle school in the language might be required to understand the response. Score 2: (Intermediate) – People who have completed up through a high school education will probably be able to understand the vocabulary and sentence structure used, but those at the basic level or children might struggle to understand the response. Score 3: (Advanced) – The response uses a fairly sophisticated vocabulary and terminology. Someone majoring in this subject at a college or university could have written it and would understand the response. An average adult who does not work or study in this area could not have written the response. Score 4: (Expert) – An expert in the field or area could have written the response. It uses specific and technically relevant vocabulary. Elevated language that someone at the simple or basic level may not understand at all. The professional language of a lawyer, scientist, engineer, or doctor falls into this category.
Verbosity	Verbosity: Score 0: (Succinct) – The response is short, to the point, and the most concise it can be. No additional information is provided outside of what is requested by the prompt (regardless of if the information or response itself is incorrect, hallucinated, or misleading. A response that gives an incorrect answer can still be succinct.). Score 1: (Pretty Short) – The response is on the shorter side but could still have words, details, and/or text removed before it’s at a bare minimum of what the response is trying to convey. Score 2: (Average Length) – The response isn’t especially long or short given what the prompt is asking of the model. The length is adequate for conveying a full response but isn’t particularly wordy nor particularly concise. Score 3: (Moderately Long) – The response is on the longer side but could still have more added to it before it is considered fully detailed or rambling. Score 4: (Verbose) – The response is particularly lengthy, wordy, and/or extensive with extra details given what the prompt requested from the assistant model. The response can be verbose regardless of if the length is due to repetition and incoherency or if it is due to rich and insightful detail.

Appendix

Table 3: Attribute text for querying the Judge-LLM

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,320 papers

Instance-Optimal Estimation with Multiple LLM Judges on a Budget
May 22, 2026Junghyun Lee, Sanghwa Kim, Yassir Jedra +2Parameter Estimation LLM Evaluation
KAIST AI · ICL EEE · KTH EECS
On Cost-Effective LLM-as-a-Judge Improvement Techniques
Apr 15, 2026Ryan Lail, Luke MarkhamLLM-as-a-Judge LLM Evaluation
Composo AI.
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
May 11, 2026Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1LLM-as-a-Judge Cost-Aware Inference
Department of Statistics, University of California, Irvine, USA.

May 22, 2026J/K move · Enter open · S save

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

Junghyun Lee, Sanghwa Kim, Yassir Jedra, Alexandre Proutière+1

KAIST AI · ICL EEE · KTH EECS

Parameter Estimation LLM Evaluation LLM-as-a-Judge Minimax Estimation Adaptive Sampling

Open paper AskPDF arXiv

Evaluating large language models increasingly relies on LLM-as-a-judge protocols, but such evaluations remain costly: different judges have different prices and reliabilities, and the difficulty of each prompt-response pair can vary substantially. This raises a basic allocation question: under a fixed budget, how should one distribute evaluation queries across heterogeneous judges and instances to obtain the most accurate score estimates? We formalize this question as budgeted heteroskedastic multi-judge estimation. Given

K

prompt-response pairs,

J

judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an

\ell_p

-error. Our first contribution is to analyze the inverse-variance weighted estimator (IVWE) and to derive the oracle allocation that minimizes its error rate. Since this allocation depends on the unknown variances, we then address the practical unknown-variance setting by proposing EST-IVWE, an adaptive algorithm that constructs and leverages optimistically biased variance estimates to stabilize the empirical allocation. We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget. Our second and central theoretical contribution is a matching local minimax lower bound, which establishes the instance-optimality of the proposed algorithms. A key technical insight is that Fano-type high-probability arguments are too coarse for this problem: their packing construction loses the local variance structure that governs the optimal allocation. We instead use an Assouad-type in-expectation argument, based on local perturbations, which preserves this structure and yields the sharp allocation-dependent lower bound. Finally, we numerically validate the superiority of our approach over naïve uniform allocation on synthetic and HelpSteer2 datasets.

Abstract

LLM Evaluation

Explore similar work

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

On Cost-Effective LLM-as-a-Judge Improvement Techniques

Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge