Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

Hashmath Shaik; Gnaneswar Villuri; Alex Doboli

cs.LGOct 5, 2026

Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations

Authors: Hashmath Shaik, Gnaneswar Villuri, Alex Doboli

Organizations: Department of Electrical and Computer Engineering Stony Brook University Stony Brook, NY 11794, USA

Abstract

The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.

Figures & tables

Judge	Pos. flips/ $N$	PFR	AUROC $\uparrow$	AUPRC $\uparrow$	Brier $\downarrow$
Qwen3-1.7B	269/534	.504	.640 [.595,.686]	.615 [.554,.677]	.233 [.224,.241]
Qwen3-4B	168/534	.315	.762 [.719,.804]	.578 [.507,.660]	.178 [.161,.195]
Qwen3-8B	199/534	.373	.850 [.815,.884]	.750 [.686,.812]	.150 [.132,.168]
Llama-3.1-8B	215/534	.403	.621 [.569,.669]	.507 [.447,.583]	.234 [.220,.249]

Table 1: Linear-probe performance on 534 shared JudgeBench pairs. Brackets denote 95% group-bootstrap confidence intervals. Higher AUROC and AUPRC and lower Brier score indicate better performance.

	JudgeBench (out-of-fold)				MT-Bench (frozen transfer)
Judge	PFR	AUROC $\uparrow$	AUPRC $\uparrow$	Brier $\downarrow$	PFR	AUROC $\uparrow$	AUPRC $\uparrow$	Brier $\downarrow$
Qwen3-1.7B	.504	.640	.615	.233	.379	.685	.538	.234
Qwen3-4B	.315	.762	.578	.178	.172	.735	.349	.135
Qwen3-8B	.373	.850	.750	.150	.173	.853	.529	.110
Llama-3.1-8B	.403	.621	.507	.234	.192	.734	.401	.150

Table 2: JudgeBench results concatenate predictions from five fold-specific linear probes. MT-Bench results use one linear probe refit on all 534 JudgeBench pairs and frozen before transfer.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	JudgeBench usable/ $620$	MT-Bench usable/ $2{,}400$	$BA$ changes/ $100$
Qwen3-1.7B	579	2,380	47
Qwen3-4B	586	2,384	19
Qwen3-8B	590	2,396	30
Llama-3.1-8B	547	1,831	43

Appendix

Table 3: Per-judge usable-pair counts and repeated-presentation results. A pair is usable only when both candidate orders satisfy the fixed output format. Repeating

AB

changes no verdict for any judge on the 100-pair control subset; the final column reports verdict changes after swapping to

BA

Pooled AUROC $\uparrow$
Judge	Linear probe	Combined	Without $AB$ choice	Logit margin	Difference
Qwen3-1.7B	.640	.578	.565	.577	+.062 [.018,.107]
Qwen3-4B	.762	.668	.652	.657	+.094 [.054,.135]
Qwen3-8B	.850	.776	.765	.743	+.074 [.041,.109]
Llama-3.1-8B	.621	.508	.469	.518	+.113 [.055,.171]
Choice-matched AUROC $\uparrow$

Appendix

Table 4: JudgeBench AUROC on identical held-out pairs and folds. The combined baseline uses all six baseline features; the baseline without

AB

choice omits that feature. Choice-matched AUROC compares flipping and non-flipping examples with the same

AB

choice. Brackets denote 95% paired group-bootstrap intervals for the linear probe’s improvement over the combined baseline.

Judge	AUROC diff. $\uparrow$	AUPRC diff. $\uparrow$	Brier diff. $\downarrow$
Qwen3-1.7B	+.062 [.018,.107]	+.079 [.022,.133]	-.008 [-.015,-.001]
Qwen3-4B	+.094 [.054,.135]	+.053 [-.004,.110]	-.018 [-.030,-.007]
Qwen3-8B	+.074 [.041,.109]	+.095 [.030,.155]	-.034 [-.048,-.020]
Llama-3.1-8B	+.113 [.055,.171]	+.097 [.038,.161]	-.012 [-.024,.001]

Appendix

Table 5: Paired differences between the linear probe and combined baseline on JudgeBench. Brackets denote 95% paired group-bootstrap intervals. Positive AUROC/AUPRC differences and negative Brier differences favor the linear probe.

Judge	AUROC $\uparrow$	AUPRC $\uparrow$	Brier $\downarrow$
Qwen3-1.7B	.685 [.653,.714]	.538 [.495,.584]	.234 [.228,.240]
Qwen3-4B	.735 [.691,.778]	.349 [.301,.410]	.135 [.121,.152]
Qwen3-8B	.853 [.823,.883]	.529 [.461,.605]	.110 [.097,.124]
Llama-3.1-8B	.734 [.689,.772]	.401 [.334,.472]	.150 [.139,.162]

Appendix

Table 6: Performance on 1,802 shared MT-Bench comparisons for linear probes refit on JudgeBench and frozen before transfer. Brackets denote 95% question-group-bootstrap intervals.

Explore similar work

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
Apr 23, 2026Abel YagubyanInter-Rater Reliability LLM Evaluation
Independent Researcher
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards
Sep 27, 2026Krishna Chytanya AyyagariLLM Evaluation LLM-as-a-Judge
Google

Apr 23, 2026, cs.CLJ/K move · Enter open · S save

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

Abel Yagubyan

Independent Researcher

Inter-Rater Reliability LLM Evaluation Pairwise Comparison LLM-as-a-Judge Bias in LLM-as-a-Judge Position Bias

Open paper AskPDF arXiv

LLM-as-a-Judge is now widely used to rank model outputs, train reward models, and populate public leaderboards, but its run-to-run reliability remains under-characterized. We study repeated identical evaluations on 29 tasks spanning 10 categories using two OpenAI judge models (GPT-4o-mini and GPT-4.1-mini), with 50 pairwise trials and 50 pointwise trials per question, supplemented by temperature and prompt-sensitivity ablations. Across judges, pairwise preferences flip on average 13.6% of the time, with 28% of questions exceeding a 20% flip rate and one question reaching 56%. GPT-4o-mini also exhibits a significant first-position bias (72% A-majority, p = 0.024). At the same time, mean pointwise score gaps are small (0.19--0.36 on a 10-point scale) and not statistically significant in aggregate, producing a pairwise--pointwise gap: judges frequently choose a winner even when their own scalar scores provide little evidence of a meaningful quality difference. Beyond within-judge instability, cross-judge agreement is only 76% (

κ= 0.51

), semantically equivalent prompt templates change majority outcomes in 25% of tested cases, and deterministic decoding reduces but does not eliminate inconsistency. A reliability curve analysis shows that, in our dataset, 11 repeated trials are needed for a majority vote to recover the 50-trial reference verdict with 95% probability on average, rising to 15 for high-variance questions. These findings suggest that single-trial LLM judging is often too noisy for high-stakes evaluation, and that multi-trial aggregation, position randomization, and explicit uncertainty reporting should be standard practice. Because both judges are from a single provider, cross-provider replication remains an important next step.

Abstract

Explore similar work

The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation

Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards