On Cost-Effective LLM-as-a-Judge Improvement Techniques

Ryan Lail; Luke Markham

cs.CLApr 15, 2026

On Cost-Effective LLM-as-a-Judge Improvement Techniques

Authors: Ryan Lail, Luke Markham

Organizations: Composo AI.

Abstract

Using a language model to score or rank candidate responses has become a scalable alternative to human evaluation in reinforcement learning from human feedback (RLHF) pipelines, benchmarking, and application layer evaluations. However, output reliability depends heavily on prompting and aggregation strategy. We present an empirical investigation of four drop-in techniques -- ensemble scoring, task-specific criteria injection, calibration context, and adaptive model escalation -- for improving LLM judge accuracy on RewardBench 2, with a unifying lens of noise control on the stochastic judge: ensembling as Monte Carlo averaging over per-call noise, criteria injection as between-response discrimination sharpening, and per-response score variance as an uncertainty signal. Ensemble scoring and task-specific criteria injection (the latter virtually cost free) together reach up to 85.8% accuracy, +13.5pp over baseline. Calibration context and adaptive model escalation also improve over baseline but are dominated by criteria + ensembling on the cost-accuracy Pareto frontier. Small models benefit disproportionately from ensembling, making high-accuracy LLM judges accessible at low cost. We show that these techniques generalise across model providers, evaluating on both OpenAI GPT and Anthropic Claude families.

Figures & tables

Figure 1: Cost vs. accuracy Pareto frontier across all evaluated conditions. Each point is labelled technique (class) and shows the best-performing model in that class for the given technique; the provider supplying each point is listed per row in Table 4 .

Category	$N$	Description
Factuality	475	Accuracy, no hallucinations
Focus	495	Relevance to query
Math	183	Mathematical reasoning
Precise IF	159	Formatting constraints
Safety	441	Refusal of harmful requests

Table 1: RewardBench 2 category breakdown.

Model	Class	Input $/M	Output $/M
Claude Sonnet 4.6	full	3.00	15.00
GPT-5.4	full	2.50	15.00
Claude Haiku 4.5	mini	1.00	5.00
GPT-5.4 mini	mini	0.75	4.50
GPT-5.4 nano	nano	0.20	1.25

Table 2: Models evaluated, capability class, and per-token API pricing.

Category	Criterion
Factuality	“Focus on whether the response contains factually correct information and does not introduce false claims, hallucinations, or unsupported statements.”
Focus	“Focus on whether the response directly addresses the user’s query, stays on topic, and provides a substantive, relevant answer.”
Math	“Focus on whether the mathematical reasoning is logically valid, the steps are correct, and the final answer is accurate.”
Precise IF	“Focus on whether the response satisfies every explicit constraint and formatting requirement specified in the user’s instructions.”
Safety	“Focus on whether the response appropriately refuses harmful requests, avoids generating dangerous content, and does not provide information that could cause harm.”

Table 3: Category-specific evaluation criteria.

Condition	Model	$N$	Overall (95% CI)	Fact.	Focus	Math	P-IF	Safety	vs Base
(a) Recommended techniques
Baseline (full $k{=}1$ )	GPT-5.4	1729	71.7% ( $\pm$ 2.1)	76.4	70.1	61.2	34.0	87.3	1.0 $\times$
	Sonnet 4.6	1737	72.3% ( $\pm$ 2.2)	71.6	72.1	75.3	38.1	84.2	1.1 $\times$
Criteria (full $k{=}1$ )	GPT-5.4	1738	74.7% ( $\pm$ 2.1)	77.9	72.3	73.2	32.1	90.6	1.1 $\times$
	Sonnet 4.6	1743	74.1% ( $\pm$ 2.1)	75.7	74.5	68.1	44.5	84.7	1.2 $\times$
Ensemble (full $k{=}8$ )	GPT-5.4	1730	81.5% ( $\pm$ 1.8)	86.7	81.8	74.9	44.7	92.1	5.0 $\times$

Table 4: Main results by condition and category. (a) Techniques that reliably improve accuracy. (b) Investigated techniques that did not improve on criteria

k{=}8

at comparable cost. Sample sizes vary across conditions (

N{=}1700

–

1746

) because Azure content filters refuse different prompts depending on configuration; the intersection (

N{=}1710

) preserves the ranking (Appendix C ). Per-subset breakdowns are omitted for test-set rows (marked

\ddagger

) due to low per-subset sample counts. Per-category CIs are in Appendix B ; full calibration variant breakdowns are in Appendix E .

Figure 2: Accuracy by condition and category. Each bar shows the best-performing (highest overall accuracy) provider for that class+condition; per-row provider attribution is in Table 4 . Error bars are 95% bootstrap confidence intervals (2,000 resamples) on that provider’s per-category accuracy; smaller categories (Math, Precise IF) have correspondingly wider intervals. Per-category CIs for the GPT-5.4 family are tabulated in Appendix B . Overlapping intervals are a conservative guide to significance — for rigorous pairwise comparisons we report paired-bootstrap $P(A{>}B)$ in the text. $k{=}8$ + criteria (mini) reaches the highest overall accuracy (85.8%); $k{=}8$ + criteria (full) at 83.6% matches $k{=}8$ + combined (full) at 82.6% at lower cost. Precise IF remains the hardest category across all conditions.

Figure 3: Accuracy vs ensemble size $k$ for full, mini, and nano. Most of the gain is captured by $k{=}3$ ; nano benefits the most in relative terms (+19.1pp from $k{=}1$ to $k{=}8$ , approaching the full model’s single-shot baseline at 0.4 $\times$ cost).

Prompt variant ( $k{=}8$ )	Full	Mini	Nano
Base (ensemble only)	81.5 ( $\pm$ 1.8)	79.2 ( $\pm$ 2.0)	71.4 ( $\pm$ 2.1)
Calibration (low)	81.7 ( $\pm$ 1.8)	79.0 ( $\pm$ 2.0)	—
Criteria	83.6 ( $\pm$ 1.7)	81.5 ( $\pm$ 1.9)	—
Criteria + calibration (combined)	82.6 ( $\pm$ 1.8)	79.2 ( $\pm$ 1.9)	—

Table 5: Prompt variant accuracy (%) at

k{=}8

on OpenAI GPT-5.4 (Claude variants not collected for calibration / combined).

Figure 4: Model agreement with full ( $k{=}8$ ) as a function of ensemble size, on the OpenAI GPT-5.4 family. Mini reaches 78.7% by $k{=}8$ (77.5% at $k{=}5$ ); nano plateaus at ${\sim}$ 70%, with a lower rank-correlation ceiling (0.67 vs 0.79 for mini), confirming a larger capability gap.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

(a) Recommended techniques
Condition	Factuality	Focus	Math	Precise IF	Safety
Baseline (full $k{=}1$ )	76.4 ( $\pm$ 3.8)	70.1 ( $\pm$ 4.1)	61.2 ( $\pm$ 6.8)	34.0 ( $\pm$ 7.6)	87.3 ( $\pm$ 3.2)
Criteria (full $k{=}1$ )	77.9 ( $\pm$ 3.7)	72.3 ( $\pm$ 4.0)	73.2 ( $\pm$ 6.4)	32.1 ( $\pm$ 7.3)	90.6 ( $\pm$ 2.8)
Ensemble (full $k{=}8$ )	86.7 ( $\pm$ 3.0)	81.8 ( $\pm$ 3.3)	74.9 ( $\pm$ 6.3)	44.7 ( $\pm$ 7.9)	92.1 ( $\pm$ 2.6)
Criteria (full $k{=}8$ )	89.1 ( $\pm$ 2.8)	82.8 ( $\pm$ 3.4)	79.2 ( $\pm$ 5.9)	48.8 ( $\pm$ 7.8)	93.2 ( $\pm$ 2.5)
Mini model $k{=}8$	83.3 ( $\pm$ 3.4)	80.2 ( $\pm$ 3.4)	68.3 ( $\pm$ 6.6)	40.3 ( $\pm$ 7.8)	92.8 ( $\pm$ 2.5)

Appendix

Table 6: Per-category accuracy (%) with 95% bootstrap CI half-widths. Rows correspond to the conditions in Table 4 .

Condition	Full $N$	Full Acc.	Intersection Acc.	$\Delta$
Baseline (full $k{=}1$ )	1729	71.7%	71.7%	$-$ 0.0
Ensemble (full $k{=}8$ )	1730	81.5%	81.6%	$+$ 0.1
Criteria (full $k{=}8$ )	1741	83.6%	83.6%	$+$ 0.0
Mini model $k{=}8$	1730	79.2%	79.1%	$-$ 0.1
Criteria (mini $k{=}8$ )	1742	81.5%	81.3%	$-$ 0.2
Calibration low ( $k{=}8$ )	1744	81.7%	81.7%	$+$ 0.0

Appendix

Table 7: Headline conditions evaluated on their full sample versus the intersection of

N{=}1710

examples shared across baseline, criteria, calibration_low, and combined collections.

Figure 5: Score variance correlation (chosen response). Left: mini vs full ( $r=0.421$ ). Right: nano vs full ( $r=0.106$ ).

Figure 6: Pareto frontier for per-response hard variance routing. Each point is a different threshold $\theta$ . The frontier has a large dead zone: escalating some but not all responses rarely changes the four-way winner.

Figure 7: Per-response soft blending accuracy vs mean blend weight $w$ (full dataset, in-sample). Accuracy peaks at 83.2% near $w{=}0.91$ (mostly full model). On a held-out test set, the blend (80.2%) does not beat full $k{=}8$ (81.5%) — the in-sample gain is partly midpoint overfitting.

Figure 8: Pareto frontier for per-response variance-informed ensembling. Gray points: full grid of $(\sigma_{1},\sigma_{2})$ configurations (train-set). Stars: test-set accuracy for budget-constrained ( $\bar{n}_{\text{full}}{\leq}2$ ) and unconstrained configurations — the budget-constrained point (74.9%) is dominated by mini $k{=}8$ at comparable cost.

Strategy	Acc.	vs $k{=}1$
$k{=}1$ full (baseline)	71.7%	1.0 $\times$
Full $k{=}8$	81.5%	5.0 $\times$
Soft blend (in-sample) †	83.2%	6.1 $\times$
Soft blend (test)	80.2%	6.1 $\times$
Var-informed ( $\bar{n}_{\text{full}}{\leq}2$ , test)	74.9%	1.6 $\times$

Appendix

Table 8: Escalation strategy comparison. Test-set rows are the honest headline numbers; the in-sample soft-blend row is a tuned upper bound and should not be compared on equal footing with the others.

Variant	$k{=}1$	$k{=}8$
Baseline (no calibration)	71.7%	81.5%
High	72.4%	81.0%
Low	73.8%	81.7%
Both	72.8%	81.5%
Cross-category	72.4%	81.7%

Appendix

Table 9: Calibration accuracy at

k{=}1

and

k{=}8

$k$	Full	Mini	Nano
1	71.7	64.8	52.3
2	77.3	73.0	60.8
3	78.4	74.8	65.4
4	79.4	76.8	68.1
5	80.1	77.1	69.6
6	80.9	77.7	70.0

Appendix

Table 10: Accuracy (%) vs. ensemble size

k

for three model tiers.

Figure 9: Baseline accuracy vs temperature for $k{=}1$ and $k{=}8$ with 95% bootstrap CIs.

$k$	Agreement	Rank corr ( $\rho$ )
1	65.1%	0.761
2	72.7%	0.771
3	75.0%	0.780
4	76.6%	0.779
5	77.5%	0.781
8	78.7%	0.785

Appendix

Table 11: Mini–full model agreement and rank correlation vs. ensemble size.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 529 papers

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Mar 5, 2026Sunishchal Dev, Andrew Sloan, Joshua Kavner +2LLM-as-a-Judge LLM Evaluation
RAND Corporation, Santa Monica, CA, USA
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
May 13, 2026Andrea MorandiLLM Evaluation Pairwise Comparison
Cisco Systems, Inc.
LLM-as-Judge on a Budget
Feb 17, 2026Aadirupa Saha, Aniket Wagde, Branislav KvetonLLM Evaluation LLM-as-a-Judge
UIC · Adobe

Mar 5, 2026J/K move · Enter open · S save

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

Sunishchal Dev, Andrew Sloan, Joshua Kavner, Nicholas Kong+1

RAND Corporation, Santa Monica, CA, USA

LLM-as-a-Judge LLM Evaluation Language Model Robustness

Open paper AskPDF arXiv

We present the Judge Reliability Harness, an open source library for constructing validation suites that test the reliability of LLM judges. As LLM based scoring is widely deployed in AI benchmarks, more tooling is needed to efficiently assess the reliability of these methods. Given a benchmark dataset and an LLM judge configuration, the harness generates reliability tests that evaluate both binary judgment accuracy and ordinal grading performance for free-response and agentic task formats. We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior, and find meaningful variation in performance across models and perturbation types, highlighting opportunities to improve the robustness of LLM judges. No judge that we evaluated is uniformly reliable across benchmarks using our harness. For example, our preliminary experiments on judges revealed consistency issues as measured by accuracy in judging another LLM's ability to complete a task due to simple text formatting changes, paraphrasing, changes in verbosity, and flipping the ground truth label in LLM-produced responses. The code for this tool is available at: https://github.com/RANDCorporation/judge-reliability-harness

Abstract

LLM-as-a-Judge

Explore similar work

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges

RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning

LLM-as-Judge on a Budget