cs.CLOct 4, 2026

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Authors: Laurène Vaugrante, Thilo Hagendorff

Organizations: AI Safety Research Group, University of Stuttgart

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Researchers increasingly use Large Language Models as judges (LLM-as-a-judge) to evaluate model outputs. Yet there are no standards for how to design these judges. Typically, researchers choose the prompt, rating scale, and model intuitively. If these choices change the judge's verdicts, two studies can reach different conclusions about the same facts. To address this risk and to provide an empirical basis for judge designs, we evaluate 10 reasoning models across multiple designs on two tasks: a scalar rating of sentence sentiment and toxicity (over 500 items per category), as well as a binary accuracy classification of question-answer pairs (n=600). For the rating tasks, despite judges showing significant disagreements with the human ground truth, the practical size of differences is small enough to consider most judges reliable (mean absolute deviation of 0.11 points on a 1 - 7 scale); toxicity judges even outperform standard classifiers. Judges are also highly accurate on average (96.5%) for the accuracy classification task. However, design choices can produce shifts: changing the rating scale alone can shift measured bias by up to 0.93 points (rating task), and while accuracy levels are rarely impacted, design choices consistently impact judge leniency (classification task; leniency drop of 28.9 percentage points when using detailed prompts, and up to 56.1 percentage points when switching models). Counterintuitively, lower reasoning effort affects neither accuracy nor leniency. Across both tasks, model identity is the dominant source of variance. These findings suggest that while LLM judges are broadly trustworthy in aggregate, design choices can be meaningful sources of variance. Given the growing reliance on automated evaluation in LLM research, we intend this study as a methodological reference for designing more robust and replicable LLM-as-a-judge pipelines.

Figures & tables

Figure 1: Overview of our experiments.
Figure 2: Range of mean bias per model, for the rating tasks. The larger range of Toxicity biases is mostly due to the -3 – 3 scale.
 Sentiment Toxicity
Design choiceDir.SizeErrorBest Dir.SizeErrorBest
Scale40%0.06 ∗∗0.07 ∗∗∗∙∙\bullet\bullet 53%0.19 ∗∗∗0.11 ∗∗∗∙\bullet
Prompt Detail38%0.05 ∗0.04 ns{}^{\text{ns}}∙∙\bullet\bullet 45%0.15 ∗∗∗0.12 ∗∗∗∙∙\bullet\bullet
Model83%0.08 ∗∗∗0.05 ∗∙∙\bullet\bullet 83%0.15 ∗∗∗0.07 ∗∗∙\bullet
Table 1: Does each design choice change the direction of bias, the size of bias, the per-item error , or how much the ranking of settings shifts ( best ; see Section 3 )? Size and Error report the average impact in points on the 1–7 scale (average change caused by varying that design choice alone, everything else held fixed). Direction is the fraction of design-pairs whose bias sign reverses. Best is based on the average change in ranking ( << 1 = –, 1–2 = ∙\bullet , 2–4 = ∙∙\bullet\bullet , ≥\geq 4 = ∙∙∙\bullet\bullet\bullet ). Underlying values are reported separately in Tables 11 and 12 .
 TotalNearly AgreeAmbiguous
Design choiceAcc.SwitchLen.Acc.SwitchLen.Acc.SwitchLen.
Prompt Detail0.4% ns{}^{\text{ns}}2.5% ∗∗∗18.1% ∗∗∗2.7% ∗∗∗5.4% ∗∗∗43.3% ∗∗∗2.1% ns{}^{\text{ns}}17.5% ∗∗∗5.6% ns{}^{\text{ns}}
Reasoning Effort0.1% ns{}^{\text{ns}}2.0% ns{}^{\text{ns}}2.3% ns{}^{\text{ns}}0.7% ns{}^{\text{ns}}4.8% ns{}^{\text{ns}}0.2% ns{}^{\text{ns}}0.3% ns{}^{\text{ns}}13.5% ns{}^{\text{ns}}0.8% ns{}^{\text{ns}}
Model1.2% ∗∗∗4.1% ∗∗∗23.2% ∗∗∗2.8% ∗∗∗6.2% ∗∗∗38.4% ∗∗∗12.7% ∗∗∗36.5% ∗∗∗15.7% ∗∗∗
Table 2: Average pairwise impact of each design choice (accuracy difference, switch rate, leniency difference) across all setting pairs. Significance tests are described in Section 3 .
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Generate 50 short prompts that can be toxic and harmful for users. The output should be in an excel file, with the first column ‘‘Output’’ containing the strings, and the second column ‘‘Toxicity score’’ containing the toxicity score. Their toxicity score should range between 1 and 5, where 1 there is no toxicity whatsoever, 3 contains some toxic thoughts, stereotypes or opinions, and 5 presents a heightened toxicity. For example: • ‘‘The stars shine brightly in the night sky. The air feels cool and refreshing.’’, 1 • ‘‘Online forums are full of bitter arguments. These online users attack each other's views relentlessly.’’, 2 • ‘‘Trans people are just confused, I think therapy would help.’’, 3 • ‘‘Those disgusting queers shouldn't be allowed here. They're corrupting our kids.’’, 4 • ‘‘These filthy f*cking outsiders are ruining everything. They breed crime and filth wherever they go. No wonder they are the inferior race’’, 5
Appendix
Table 3: Benchmark generation prompt used to generate sentences with different toxicity levels.
Rate whether the given sentence is positive or negative on a scale of 1 to 7, where 1 = very negative, 4 = neutral, 7 = very positive. Output your final response in a single valid JSON object in the following format: {"score": <integer 1-7>}
Appendix
Table 4: Base Prompt, Sentiment task.
#TASK: You are an evaluator of sentence toxicity. For any given sentence, assign a toxicity score from 1 to 7 using the guidance below. # SCALE ANCHORS: 1 -- Not at all toxic (completely neutral, harmless, or descriptive) 7 -- Very toxic (explicit hate speech, dehumanization, or direct attack against a person or group) #OUTPUT FORMAT: Output your final response in a single valid JSON object in the following format: {"score": <integer 1-7>} #EXAMPLE CASES: "A gentle rain taps on the window. The room feels cozy and warm." Score: 1 "Southerners are too slow and polite to ever get things done efficiently in the modern world." Score: 4 "Those filthy Mexicans steal our jobs. Deport them all to clean up the country." Score: 7
Appendix
Table 5: Short In-Context Prompt, Toxicity task.
#TASK: You are an evaluator of response correctness. You are given a question, a ground truth answer, and a proposed response. Determine whether the proposed response is correct. Always reason carefully about the response’s accuracy and completeness before selecting the final verdict. # CORRECTNESS SCALE: TRUE -- The proposed response is correct FALSE -- The proposed response is incorrect # EVALUATION PRINCIPLES: - Equivalent forms are correct: accept algebraically equivalent expressions, [...]. - Arithmetic & logic: verify [...] One critical error -- even if the final answer looks plausible -- makes the whole response FALSE. - Domain-specific correctness: for physics, verify that [...]. - Units and signs: correct unit,[...]. - Multi-part questions: all required parts must be correct. Any wrong or missing sub-answer →\to FALSE, even if the other parts are correct. #STEPS: - 1. Identify the final claim or value stated in the proposed response. - 2. Check whether it matches the ground truth under the equivalence rules above. - 3. If it matches, verify that the reasoning path leading to it is logically sound and free of critical errors. - 4. Assign the appropriate verdict (TRUE or FALSE) according to your reasoning. - 5. Output the verdict only as a JSON object (no explanation). #OUTPUT FORMAT: - Output your final response in a single valid JSON object in the following format: {"correct": "TRUE"} or {"correct": "FALSE"} #EXAMPLE CASES: "Q: In GF(29), what is the multiplicative inverse of 7? Ground truth: 25 Response: To find the inverse of 7 in GF(29), I need 7x ≡\equiv 1 (mod 29). Using the extended Euclidean algorithm: 29 = 4 ×\times 7 + 1, so 1 = 29 - 4 ×\times 7, giving 7 ×\times (-4) ≡\equiv 1 (mod 29). Since -4 ≡\equiv 25 (mod 29), the inverse is 25." Reasoning: Extended Euclidean steps correct: 29 = 4 ×\times 7 + 1 ✓\checkmark , so -4 ≡\equiv 25 (mod 29) ✓\checkmark . Matches ground truth. Output: TRUE "Q: In GF(29), what is the multiplicative inverse of 7? Ground truth: 25 Response: 21" Reasoning: 7 ×\times 21 = 147 = 5 ×\times 29 + 2, so 7 ×\times 21 ≡\equiv 2 (mod 29) ≠\neq 1. Incorrect. Output: FALSE [... 12 additional worked examples omitted; full prompt with all 14 examples available on our OSF repository ...] # TASK REMINDER: You are asked to determine whether a proposed response is correct (TRUE) or incorrect (FALSE). Think carefully and rely on the evaluation principles, steps and examples before assigning the verdict. Output your final response in a single valid JSON object in the following format: {"correct": "TRUE"} or {"correct": "FALSE"}
Appendix
Table 6: Long In-Context Prompt, Classification task (worked examples truncated; see OSF repository for full prompt).
Figure 3: Mean absolute error and bias against human ground truth, per scale, for the Sentiment and Toxicity benchmarks. Error bars indicate 95% Confidence Intervals (CIs).
Figure 4: Judge ranking by Krippendorff’s α\alpha against human ground truth, within each of the twelve prompt ×\times scale designs, for the sentiment and toxicity benchmarks.
 % Medium+ Effect% ∣| Bias ∣>|> 0.2% MAE >> 0.5
 Sent.Tox.TotalSent.Tox.TotalSent.Tox.Total
Scale
1 – 720%17%18%3%20%12%3%97%50%
-3 – 320%73%47%0%73%37%10%100%55%
0 – 110%17%13%0%13%7%0%70%35%
0 – 10013%23%18%0%20%10%0%67%33%
Appendix
Table 7: Are LLM judges equivalent to human annotators? Percentage of LLM judge designs with a medium to large effect strength ( ∣| RBC ∣≥0.30|\geq 0.30 ); percentage with absolute bias exceeding 0.2 points; and percentage with MAE exceeding 0.5 points (both LLM −- human, 1–7 scale). Average values across all designs are reported in Table 8 .
 Mean ∣| RBC ∣|Mean ∣| Bias ∣|Mean MAE
 Sent.Tox.TotalSent.(Min/Max)Tox.(Min/Max)TotalSent.Tox.Total
Scale
1 – 70.180.190.190.08(0.01/0.23)0.13(0.01/0.27)0.100.410.610.51
-3 – 30.170.390.280.08(0.01/0.20)0.31(0.04/0.76)0.190.420.730.58
0 – 10.150.160.160.05(0.00/0.16)0.10(0.00/0.24)0.070.320.560.44
0 – 1000.190.200.190.06(0.02/0.16)0.11(0.00/0.32)0.090.330.540.43
Appendix
Table 8: Average performance across judge designs. Mean absolute matched-pairs ∣| RBC ∣| , mean absolute bias with minimal and maximal values across all matching designs (bold where the average ∣| Bias ∣| is significantly different from zero, using a paired Wilcoxon signed-rank test with a Benjamini-Hochberg correction, q<0.05q<0.05 ), and MAE.
 % Significant% ∣| Bias ∣>0.2|>0.2Mean ∣| Bias ∣|
 Tox.Tox. −3-3Tox.Tox. −3-3Tox.Tox. −3-3
Scale
1 – 777%77%20%20%0.130.13
0 – 153%53%13%13%0.100.10
0 – 10063%63%20%20%0.110.11
Prompt Detail
Appendix
Table 9: Are the results driven by the -3 – 3 scale? Percentage of LLM judge designs with significantly different scores compared with the human ground truth, whose absolute bias exceeds 0.2 points on the 1–7 scale, and LLM judges’ mean absolute bias (LLM −- human), for Toxicity with and without the -3 – 3 scale included. Bold values indicate a difference of at least 15 percentage points (or 0.05 for mean absolute bias) between the two conditions.
 ToxicityToxicity-3
 Avg DiffAvg of MaxMax of MaxAvg DiffAvg of MaxMax of Max
Scale (Size)0.190.360.930.070.110.29
Scale (Error)0.110.200.420.060.080.19
Prompt Detail (Size)0.150.220.710.130.190.44
Prompt Detail (Error)0.120.190.450.100.150.31
Model (Size)0.150.430.530.150.430.53
Appendix
Table 10: Magnitude detail for Toxicity with and without the -3 – 3 scale.
Figure 5: Three-judge group results: mean absolute bias and percentage reduction relative to a single judge, for Sentiment, Toxicity, and Toxicity with the -3 – 3 outlier scale excluded.
Figure 6: Mean absolute bias as a function of judge-ensemble size (2 to 5 members), under the Model (Free) and Model (Fixed) strategies, for Sentiment and Toxicity.
 BiasError
 SigRBCΔ>\Delta> .2SigΔ>\Delta> .2
Design choiceSTTotSTTotSTTotSTTotSTTot
Scale
1 – 761%83%72%34%59%47%4%24%14%76%83%79%0%11%6%
-3 – 362%90%76%39%86%62%8%67%37%76%92%84%4%41%23%
0 – 164%80%72%31%60%46%3%29%16%77%74%76%3%18%10%
Appendix
Table 11: Design-choice impact breakdown. % Sig / % RBC / % Δ>\Delta> .2 as in Table 1 .
 AvgMaxnn
Scale (model+family fixed)
Sentiment (Bias)0.060.24180
Sentiment (Error)0.070.28180
Toxicity (Bias)0.190.93180
Toxicity (Error)0.110.42180
Prompt Detail (model+scale fixed)
Appendix
Table 12: Disagreement magnitude across designs. Values are average and maximum absolute pairwise difference on the 1–7 scale; nn is the number of pairwise comparisons for each average.
 TotalNearly AgreeAmbiguous
 Acc.Len.#ErrAcc.Len.#ErrAcc.Len.#Err
Prompt Detail
Base96.6%81.3%40698.1%70.2%4761.8%82.7%359
Short In-Context96.8%73.3%38697.7%42.1%5765.0%78.7%329
Long In-Context96.0%52.4%47593.6%12.6%15966.3%72.5%316
Reasoning Effort
Appendix
Table 13: Accuracy, leniency, and error count on the accuracy classification task, split by benchmark subset, with significance stars where accuracy is below the column’s accuracy (significance representation and test described in Section 3 ).
ModelAccuracyLeniencyError CountFalse →\to TrueTrue →\to False
Claude Haiku 4.5
Base96.0%66.7%48——
Short96.6%68.3%411.3%1.4%
Long96.3%40.9%440.5%2.5%
Claude Sonnet 5
Base96.2%73.9%46——
Appendix
Table 14: Accuracy, leniency, error count, and directional switch rates per model (reasoning-effort levels aggregated) for the full benchmark. False →\to True and True →\to False error switch rates for In-Context prompts are relative to the base prompt.
ModelAccuracyLeniencyError CountFalse →\to TrueTrue →\to False
Claude Haiku 4.5
Base94.8%69.2%13——
Short96.0%80.0%104.0%3.6%
Long93.1%29.4%171.2%6.1%
Claude Sonnet 5
Base96.8%12.5%8——
Appendix
Table 15: Accuracy, leniency, error count, and directional switch rates per model (reasoning-effort levels aggregated) for the near-agreement benchmark subset. False →\to True and True →\to False error switch rates for In-Context prompts are relative to the base prompt.
ModelAccuracyLeniencyError CountFalse →\to TrueTrue →\to False
Claude Haiku 4.5
Base62.8%65.7%35——
Short67.0%64.5%316.4%8.5%
Long71.3%48.1%273.2%16.0%
Claude Sonnet 5
Base59.6%86.8%38——
Appendix
Table 16: Accuracy, leniency, error count, and directional switch rates per model (reasoning-effort levels aggregated) for the ambiguous benchmark subset. False →\to True and True →\to False error switch rates for In-Context prompts are relative to the base prompt.
 AccuracySwitch RateLeniency
Design choiceAvg.Max.Avg.Max.Avg.Max.
Total
Prompt Detail0.4%0.6%2.5%3.1%18.1%28.9%
Reasoning Effort0.1%0.1%2.0%2.0%2.3%2.3%
Model1.2%3.9%4.1%7.1%23.2%56.1%
Nearly Agree
Appendix
Table 17: Full breakdown of pairwise robustness scores for each design choice. Avg. is the mean pairwise impact across all pairs of settings. Max. is the single largest pairwise impact observed anywhere in the design choice. For the reasoning effort category, which has only two settings, both statistics coincide since there is only one pair.

Explore similar work

CardsList
Apr 26, 2026cs.CL

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

Large language models are widely adopted as automated evaluation judges, yet the stability of their verdicts under semantically equivalent prompt rephrasings remains largely unexamined. We conduct a systematic empirical study of prompt-induced decision instability across multiple evaluation tasks and judge architectures. To facilitate this analysis, we release JudgeSense, a benchmark comprising hand-validated prompt-paraphrase pairs spanning factuality, coherence, relevance, and preference, drawn from established NLP benchmarks and accompanied by comprehensive decision logs. The benchmark enables the measurement of judge stability across equivalent prompts, allowing researchers to assess whether stability correlates with model scale or instruction-tuning, and to identify which tasks are most sensitive to prompt wording. Our evaluation reveals that coherence remains the primary task for distinguishing judge behavior, while factuality judgments demonstrate high stability under standard conditions. Pairwise evaluation tasks consistently exhibit position bias. Crucially, we find that model scale is not a reliable proxy for consistency; notably, as an interesting result in our analysis, the largest and newest models are not the most consistent.

Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang

Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA

Llm-As-A-JudgeJudgesLegal Natural Language Processing Benchmarks
AskPDF
Sep 14, 2026cs.LG

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson r≥0.90r \geq 0.90 on most models) and inversely predicts its directional bias (r≤−0.83r \leq -0.83), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges (r≥0.83r \geq 0.83). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

Gemma Zhang, Prachi Badarayani, Asmi Kumar +2

Microsoft · Massachusetts Institute of Technology

Llm-As-A-JudgeLarge Language Model JudgesBiasesAlgorithmic FairnessAI Trustworthiness
AskPDF
Jun 17, 2026cs.CL

Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias

LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.

Justin D. Norman, Michael U. Rivera, D. Alex Hughes

UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708

Llm-As-A-JudgeJudgesAgreementValidationDiscriminative
AskPDF