Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

Ilija Subasic; Brett Levert; Andrew Rabinovich; Sajjad Pakdamansavoji; Zhao Chen

cs.CLOct 27, 2026

Prompt Lottery in LLM-as-a-Judge: Measuring and Mitigating Sensitivity to Syntactic Rephrasing

Authors: Ilija Subasic, Brett Levert, Andrew Rabinovich, Sajjad Pakdamansavoji, Zhao Chen

Organizations: Upwork Inc

Abstract

LLM-as-a-Judge has become a standard for evaluating large language models, yet most approaches implicitly assume that a single judge prompt functions as a stable evaluation instrument. Prior work has primarily explored deliberate prompt modifications such as scoring rubrics, role descriptions, or chain-of-thought reasoning in order to optimize accuracy. In contrast, we investigate whether validated prompt variants intended to preserve evaluation semantics while differing in surface realization (paraphrastic rephrasings) yield consistent judgments under a fixed evaluation task. We introduce Prompt Lottery, a phenomenon in which paraphrastic rephrasings produce different LLMaJ judgments. Across a large-scale study spanning diverse evaluation tasks, judge families, and model backbones, we find that these variations can alter evaluation outcomes and, in some settings, reverse relative model rankings. A user study with 10 human annotators shows that LLMaJ judges exhibit 6-9 $\times$ larger agreement degradation under prompt rephrasings than humans, falling below the generally accepted threshold in usability studies. This indicates that single-prompt evaluation is insufficiently robust for model ranking. Finally, we show that multi-prompt aggregation substantially improves robustness by emphasizing judgments that remain consistent across prompt variants.

Figures & tables

Judge Family	Dimensions	Scoring Task	Datasets	References
Dialogue Quality	Interesting, Specific, Understandable, Quality	Binary (Yes/No): Presence of quality traits in conversation context	PersonaChat, PersonaUSR, DailyDialog, Topical-Chat, ConTurE	( mehri2020usr ; Zhang2018PersonaChat ; mehri-eskenazi-2020-unsupervised ; li2017dailydialog ; topical ; ghazarian2022wrongyouleveraginguser )
QA Correctness	Correctness	Likert 1–5: Completeness and accuracy vs. ground-truth	HotpotQA, SQuAD v2, NQ-Open, TopiOCQA, Natural Questions	( zheng2023judging ; yang2018hotpotqa ; rajpurkar2018know ; lee2019latent ; adlakha2022topiocqa ; kwiatkowski2019natural )
QA Groundedness	Groundedness	Likert 1–5: Faithful inference from evidence	HotpotQA, SQuAD v2, NQ-Open, TopiOCQA, Natural Questions	( es2024ragas ; yang2018hotpotqa ; rajpurkar2018know ; lee2019latent ; adlakha2022topiocqa ; kwiatkowski2019natural )
General Response Quality	Helpfulness, Accuracy, Depth	Likert 1–10: Holistic assessment of open-ended responses	LLM-Arena, PersonaChat, PersonaUSR, DailyDialog, Topical-Chat, ConTurE	( zheng2023judging ; Zhang2018PersonaChat ; mehri-eskenazi-2020-unsupervised ; li2017dailydialog ; topical ; ghazarian2022wrongyouleveraginguser )
Coherence, Consistency, Fluency	Summary Quality	Likert 1–5: Summarization quality assessment	SummEval	( fabbri2021summeval )
Semantic Alignment	Content & Style Equivalence	Likert 1–5: Information overlap between reference and prediction	UPHELD	( subasic2026evaluating )

Table 1: Overview of judge families, evaluation tasks, and datasets.

Figure 1: On the left bar-chart: agreement metrics demonstrate low consistency across syntactic variants both across models and prompts. On the right we show Fleiss’ $\kappa$ values and degradation below baselines across judge families for each model. Table 2 provides details.

Judge Family	GPT-4o		Kimi-K2.5		Llama 3.3		Qwen3-VL		GPT-oss-120B		Model Avg
	$\kappa$	$\Delta$	$\kappa$	$\Delta$	$\kappa$	$\Delta$	$\kappa$	$\Delta$	$\kappa$	$\Delta$	$\bar{\kappa}$	$\overline{\Delta}$
Dialogue Quality (Binary)
Interesting	0.45 $\ast$	0.43	0.55 $\ast$	0.32	0.44 $\ast$	0.45	0.36 $\ast$	0.54	0.15 $\ast$	0.74 $\dagger$	0.39 $\ast$	0.50
Quality	0.31 $\ast$	0.58	0.41 $\ast$	0.48	0.21 $\ast$	0.68	0.19 $\ast$	0.71 $\dagger$	0.16 $\ast$	0.73 $\dagger$	0.26 $\ast$	0.64
Specific	0.55 $\ast$	0.33	0.50 $\ast$	0.38	0.23 $\ast$	0.66	0.33 $\ast$	0.57	0.36 $\ast$	0.53	0.39 $\ast$	0.49
Understandable	0.23 $\ast$	0.65	0.15 $\ast$	0.75 $\dagger$	0.35 $\ast$	0.54	0.22 $\ast$	0.68	0.60	0.30	0.31 $\ast$	0.58

Table 2: LLMaJ stability under paraphrastic variation. Fleiss’

\kappa

values and degradation (

\Delta\kappa=\kappa_{\text{baseline}}-\kappa_{\text{rephrased}}

) across judge families.

\ast

– is statistical sig. at

0.05

and

\dagger

0.01

Table 4

Task	Mean $\tau$	Top-1 Flips	Flip %
MTBench (DailyDial.)	1.00	0/4	0.0%
MTBench (Persona)	0.80	2/4	10.0%
QA Correct. (SQuAD v2)	0.17	3/4	43.3%
Quality (SummEval)	0.64	2/4	40.0%
G-Eval (Topical-Chat)	0.83	1/4	10.0%

Table 5: Ranking stability across paraphrastic prompts. Lower Kendall’s

\tau

and higher absolute reversal rates indicate stronger Prompt Lottery effects.

Table 8

Factor	Key Finding
Backbone model choice	Largest source of pairwise agreement variance
Prompt rephrasing	Significant variance despite semantic equivalence
Binary evaluation tasks	High stability, skip aggregation
Non-binary evaluation tasks	High instability, aggregate
Plurality aggregation	Strong robustness improvement
Selective aggregation	Major cost reduction ( $\sim 60\%$ )

Table 9: Takeaways for robust LLMaJ evaluation.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Judge family (dimension)	Binary?	Configs	$n$
PersonaChat	DQ — Interesting	✓	1	149
	DQ — Quality	✓	1	149
	DQ — Specific	✓	1	149
	DQ — Understandable	✓	1	149
	GQ — MT-Bench		1	149
PersonaUSR	DQ — Interesting	✓	1	60

Appendix

Table 10: Overview of the 33 experimental configurations used in this study. Each configuration pairs a dataset with a judge family and is evaluated with

5\times 5=25

distinct LLMaJ triples (5 back-models

\times

5 prompt variants). Binary? indicates whether the judge produces a binary (Yes/No) rather than a Likert-scale score. Judge-family abbreviations— DQ : Dialogue Quality; GQ : General Quality (MT-Bench style); QC : QA Correctness; QG : QA Groundedness; SA : Semantic Alignment;

Dimension	Description	Examples
1. Role Framing	How the judge’s identity/authority is established.	“You are an expert evaluator” vs. “Act as a judge” vs. “You evaluate”
2. Instruction Syntax	Imperative vs. declarative vs. modal phrasing.	“Compare…” vs. “You should compare…” vs. “Your task is to compare…”
3. Criteria Presentation	How evaluation standards are formatted.	Bullet list vs. numbered list vs. prose vs. scale labels
4. Output Format Specification	Strictness of format requirements.	“strictly following this format” vs. “using exactly:” vs. “in the format:”
5. Task Contextualization	How much context/setup precedes the core task.	Verbose explanation vs. direct instruction
6. Linguistic Register	Formality level and vocabulary choices.	“assess” vs. “evaluate” vs. “rate” vs. “determine” / “brief” vs. “short” vs. “concise”

Appendix

Table 11: Qualitative Analysis of Judge Prompts

Dimension	Estimand	Value	Perm. $p$	Holm $p$	Families
Role framing	Mean within-family $\eta^{2}$	0.34	0.34	1.00	8
Instruction syntax	Mean within-family $\eta^{2}$	0.32	0.78	1.00	8
Criteria presentation	Mean within-family $\eta^{2}$	0.26	0.45	1.00	7
Output-format strictness	Slope ( $\kappa$ per step)	$+0.076$	$<0.001$	0.0007	4
Task contextualization (length)	Slope ( $\kappa$ per 100 words)	$-0.044$	0.63	1.00	10
Linguistic register (verb)	Mean within-family $\eta^{2}$	0.48	0.51	1.00	10

Appendix

Table 12: Within-family permutation tests of paraphrastic prompt dimensions against cross-model Cohen’s

\kappa

. Permutation

p

-values are two-sided for slopes and one-sided for eta-squared; Holm adjustment is over the seven contrasts.

Setting	Value
Input field	llm_judge (conversation + judge instruction)
Max. sequence length	512 tokens (transformers); Llama uses char limits (below)
Train / val / test sizes	10k / 2k / 5k (defaults)
Class weights	Inverse frequency from each model’s training split
DeBERTa train sampling	Class-balanced from holdout pool
Threshold tuning (DeBERTa)	Sweep $t\in\{0.05,0.10,\ldots,0.95\}$ on val; metric MCC

Appendix

Table 13: Shared data and evaluation protocol.

Model	Train data	LR	Batch (eff.)	Epochs
Logistic regression	10k (imbal.)	—	—	—
SVM (RBF)	10k (imbal.)	—	—	—
RoBERTa-base	10k (imbal.)	$2\times 10^{-5}$	16	3
DeBERTa + LoRA	balanced	$5.25\times 10^{-5}$	64 (512)	3
Llama 3.1 8B	none (ICL)	—	64 / 4	—

Appendix

Table 14: Model-specific hyperparameters (defaults).

Hyperparameter	Search	Range / values	Fixed
Compute	–	–	1 $\times$ NVIDIA A100 80GB
Learning rate	Log-uniform	$10^{-5}$ – $2\times 10^{-4}$	–
LoRA rank ( $r$ )	Integer	8, 16, 24, 32, 40	–
LoRA alpha	Integer	$r$ to $4r$ (step $r$ )	–
LoRA dropout	Uniform	0.0–0.15	–
Weight decay	Uniform	0.01–0.1	–

Appendix

Table 15: Optuna hyperparameter search space for DeBERTa LoRA.

Group	Task Type	Prompts
G1	Semantic Alignment (Likert 1–5)	P1–P5
G2	QA Correctness (Likert 1–5)	P1–P5
G3	QA Groundedness (Likert 1–5)	P1–P5
G4	General Quality – User Study (Likert 1–5)	P1–P5
G5	MT-Bench Quality (Likert 1–10)	P1–P5
G6	Dialogue – Relevance (Binary)	P1–P5

Appendix

Table 16: Summary of judge prompt groups. Groups G1–G5 cover semantic alignment, QA, and general response quality. Groups G6–G9 cover dialogue-specific dimensions. Groups G10–G13 cover summarization quality. Groups G14–G15 cover context-aware and domain-specific evaluation.

\dagger

G15 is only used for the gated aggregation and thus not semantic variations.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 559 papers

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Apr 26, 2026Rohith Reddy Bellibatlu, Edward Raff, Wenbin ZhangLLM Evaluation LLM-as-a-Judge
Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA
How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models
Oct 4, 2026Laurène Vaugrante, Thilo HagendorffLLM Evaluation LLM-as-a-Judge
AI Safety Research Group, University of Stuttgart
Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness
Oct 22, 2026Asal Mehrabi, Mohammad Aliannejadi, Clemencia SiroLLM Evaluation LLM-as-a-Judge
University of Amsterdam, Amsterdam, The Netherlands · Centrum Wiskunde & Informatica (CWI), Amsterdam, The Netherlands

Apr 26, 2026J/K move · Enter open · S save

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

Rohith Reddy Bellibatlu, Edward Raff, Wenbin Zhang

Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation

Open paper AskPDF arXiv

Large language models are widely adopted as automated evaluation judges, yet the stability of their verdicts under semantically equivalent prompt rephrasings remains largely unexamined. We conduct a systematic empirical study of prompt-induced decision instability across multiple evaluation tasks and judge architectures. To facilitate this analysis, we release JudgeSense, a benchmark comprising hand-validated prompt-paraphrase pairs spanning factuality, coherence, relevance, and preference, drawn from established NLP benchmarks and accompanied by comprehensive decision logs. The benchmark enables the measurement of judge stability across equivalent prompts, allowing researchers to assess whether stability correlates with model scale or instruction-tuning, and to identify which tasks are most sensitive to prompt wording. Our evaluation reveals that coherence remains the primary task for distinguishing judge behavior, while factuality judgments demonstrate high stability under standard conditions. Pairwise evaluation tasks consistently exhibit position bias. Crucially, we find that model scale is not a reliable proxy for consistency; notably, as an interesting result in our analysis, the largest and newest models are not the most consistent.

Abstract

LLM-as-a-Judge

Explore similar work

JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems

How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and Models

Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness