Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Ante Kapetanovic; Kemal Altwlkany; Andro Mercep; Tomislav Duricic; Emanuel Lacic

cs.CLAug 26, 2026

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Authors: Ante Kapetanovic, Kemal Altwlkany, Andro Mercep, Tomislav Duricic, Emanuel Lacic

Organizations: Infobip Split, Croatia · Infobip Sarajevo, Bosnia and Herzegovina · Infobip Zagreb, Croatia

Abstract

Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each judgment is often assumed to be independent of earlier evaluations. We test this assumption using three prompt conditions: no metadata, revision framing, and anchored metadata containing revision, attempt, and prior-score fields. We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values. Across 192,000 attempted evaluations (185,271 successful), seven out of the eight evaluated models have 95% task-stratified bootstrap intervals below zero for the total anchored-metadata effect on 20 fixed texts. Cohen's $d$ , a standardized measure of the difference between score distributions, reaches an absolute value of 0.71. Token-level analysis of selected model-task probes suggests a threshold-like response pattern: introducing anchored metadata produces a marked redistribution of output-score probabilities, while changing the anchor value within the tested below-threshold range produces comparatively little additional variation. On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label, demonstrating the bias extends beyond numerical scoring to categorical decisions. Neither Chain-of-Thought nor a metadata-disregard warning reduces the total effect, although the warning improves the paired accuracy effect relative to baseline in the industry experiment. Reliable LLM evaluation demands careful context engineering rather than an assumption of impartiality. Effective mitigation must be validated for the intended model and task or domain.

Figures & tables

Panel of Figure 1 — Figure 1. Experimental design with three memory conditions. C0 (baseline) includes no contextual metadata. C1 (framing) indicates the submission is a revision without providing a score. C2 (anchoring) adds a prior score $a$ sampled uniformly from $\left[0,3.99\right)$ , always below the acceptance threshold. The primary comparison $\Delta_{\text{C2}-\text{C0}}$ measures the total anchoring effect. The intermediate condition C1 enables additive decomposition: $\Delta_{\text{C2}-\text{C0}}=\Delta_{\text{C1}-\text{C0}}+\Delta_{\text{C2}-\text{C1}}$ , separating framing effects from the incremental influence of the numerical anchor. Schematic of the three experimental conditions C0, C1, and C2 with comparison arrows. Three side-by-side boxes labeled C0 (Baseline, blue), C1 (Framing, green), and C2 (Anchoring, red). C0 contains task, answer, and rubric with no metadata. C1 adds a metadata block stating that the submission is a revision. C2 adds a metadata block listing the attempt number $k$ and a prior score $a$ drawn uniformly from $[0, 3.99)$ . Below the boxes, arrows mark three comparisons: a primary C2 versus C0 arrow labeled total anchoring effect, a smaller C1 versus C0 arrow labeled framing effect, and a C2 versus C1 arrow labeled incremental anchoring.

Model	Triplets	Incomplete
GPT-4.1	8,000	0.0%
Claude-4.5-Sonnet	8,000	0.0%
DeepSeek-R1	7,936	0.8%
Llama-3.3-70B	7,835	2.1%
Gemma-2-9B	7,596	5.0%
Llama-3.1-8B	6,804	15.0%

Table 1. Complete C0/C1/C2 triplets under the trial-index audit. Exclusions are relative to 8,000 possible triplets per model, of which 59,136 remain.

Model	$\Delta_{\text{C2}-\text{C0}}$	95% CI	$d$
Llama-3.2-3B	$-0.254$	$[-0.298,-0.208]$	$-0.71$
GPT-4.1	$-0.411$	$[-0.457,-0.361]$	$-0.67$
Qwen2.5-7B	$-0.233$	$[-0.324,-0.150]$	$-0.51$
Claude-4.5-Sonnet	$-0.706$	$[-1.139,-0.245]$	$-0.44$
DeepSeek-R1	$-0.269$	$[-0.326,-0.221]$	$-0.25$
Gemma-2-9B	$-0.072$	$[-0.130,-0.001]$	$-0.18$

Table 2. Total anchored-metadata effect

\Delta_{\text{C2}-\text{C0}}

by model. Point estimates first average the four temperature-specific contrasts for each task, then equally average the resulting 20 task effects. Intervals resample tasks within category. Cohen’s

d

is descriptive and uses the complete-triplet sample.

Figure 2. Score difference by model, condition, and temperature. Circles show

\Delta_{\text{C2}-\text{C0}}

, whereas squares show

\Delta_{\text{C1}-\text{C0}}

. Error bars are 95% stratified task-bootstrap intervals. Score difference by model and condition across temperatures. Four panels show mean score changes relative to C0 for eight models. Circles denote C2 minus C0 and squares C1 minus C0. Negative values indicate lower scores than baseline.

Figure 3. Threshold-derived acceptance-rate differences by model, condition, and temperature. Circles show

\Delta_{\text{C2}-\text{C0}}

, whereas squares show

\Delta_{\text{C1}-\text{C0}}

. Error bars are 95% stratified task-bootstrap intervals. Threshold-derived acceptance-rate difference by model and condition across temperatures. Four panels show changes in the probability that score is at least 4.0. Circles denote C2 minus C0 and squares C1 minus C0.

Figure 4. Mean total anchored-metadata effect $\Delta_{\text{C2}-\text{C0}}$ by model and task category. Heatmap of mean total anchored-metadata shift by model and task category. Cells show mean C2 minus C0. Code review is generally closest to zero, while the other categories are more negative, with heterogeneity across models.

Term	Estimate	95% CI
Code review, $T=0$	$-0.067$	$[-0.141,\phantom{-}0.011]$
Creative writing	$-0.236$	$[-0.374,-0.071]$
Factual QA	$-0.291$	$[-0.430,-0.134]$
Summarization	$-0.275$	$[-0.381,-0.165]$

Table 3. Task-aware joint model. Category rows are contrasts against code review, and the intercept is the equally weighted fixed-model-set mean at

T=0

. Intervals resample tasks within category.

Digit	$P_{d}^{(\text{C0})}$	$P_{d}^{(\text{C2})}$	Change
“3”	0.000	0.000	$+0.000$
“4”	0.074	1.000	$+0.926$
“5”	0.926	0.000	$-0.926$

Table 4. Score-digit probabilities under C0 and C2 for the selected GPT-4.1 creative-writing probe. Values are rounded to three decimal places.

Model	Score $\Delta$	$P_{4}^{\text{C0}}$	$P_{4}^{\text{C2}}$	Slope
Llama-3.2-3B	$-0.120$	0.995	0.977	$-0.0085$
Llama-3.1-8B	$-0.018$	0.992	0.992	$+0.0009$
Llama-3.3-70B	$+0.016$	1.000	1.000	$+0.0000$

Table 5. Behavioral and modal-token results for the selected Llama probes. Score

\Delta

denotes

\Delta_{\text{C2}-\text{C0}}

, and slope is the within-C2 regression of

P_{4}

on anchor value. Values are rounded to three decimal places except for the slope.

Analysis	Estimate	95% CI
Signed, unadjusted	$+0.211$	$[\phantom{-}0.013,\phantom{-}0.408]$
Signed, adjusted	$+0.297$	$[-0.054,\phantom{-}0.647]$
Absolute, adjusted	$-0.016$	$[-0.139,\phantom{-}0.107]$
Non-ceiling mean	$-0.227$	$[-0.343,-0.098]$

Table 6. Ceiling robustness. Regression predictors use distance to ceiling,

5-\mu_{\text{C0}}

. Regression intervals use task-clustered uncertainty, whereas the non-ceiling interval uses a task bootstrap over the 19 represented tasks.

Strategy	C0	C2	$\Delta_{\text{total}}$ [95% CI]
Baseline	4.245	3.817	$-0.429\;[-0.462,-0.395]$
CoT	4.458	3.825	$-0.633\;[-0.703,-0.565]$
Warning	4.500	4.043	$-0.457\;[-0.479,-0.435]$

Table 7. Numerical mitigation on one GPT-4.1/task pair. The primary metric is

\Delta_{\text{total}}=\mu_{\text{C2}}-\mu_{\text{C0}}

, while within-C2 slopes are 0.267, 0.506, and

-0.007

, respectively.

Strategy	Prompt	Comp.	Total	Token increase
Baseline	397.0	117.7	514.7	—
CoT	437.0	168.7	605.7	17.7%
Warning	437.0	109.4	546.4	6.2%

Table 8. Mean token cost per call. The increase is based on total tokens relative to baseline.

Group	Metric	C0	C1	C2
A	Correction	21.62%	22.97%	11.26%
A	Mislabel persistence	65.77%	65.77%	81.53%
B	Accuracy	76.26%	—	68.49%
B	Anchor agreement	11.42%	—	18.72%

Table 9. Categorical effects with human ground truth. Group A uses C0/C1/C2 conditions. Group B uses paired C0/C2 outputs and a fixed sample-specific wrong anchor.

Strategy	C0 acc.	C2 acc.	$\Delta$ acc.	Induced ( $n/N$ )
Baseline	76.26%	68.49%	$-7.76$ pp	17/167 (10.18%)
CoT	68.04%	62.56%	$-5.48$ pp	20/149 (13.42%)
Warning	74.43%	74.43%	$\phantom{-}0.00$ pp	7/163 (4.29%)

Table 10. Paired Group B results. Induced is the share of each strategy’s C0-correct cases that become its assigned wrong label in C2.

Bias in LLM-as-a-Judge

Bias in large language model (LLM)-as-a-judge evaluation studies systematic distortions in automated assessments of model outputs.

See all 118 papers

Accounting for Bias Enables Sustainable LLM Evaluation
Sep 25, 2026Harshita Katoch, David Antony Selby, Gerrit Großmann +1LLM Evaluation Bias in LLM-as-a-Judge
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Jun 13, 2026Yitao LiLLM Evaluation LLM-as-a-Judge
AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs
Aug 14, 2026Yiderigun Borjigin, Alexander Hermann, Christian Cyron +1Language Model Bias Evaluation Benchmark Design
Saarland University · Hamburg University of Technology · Helmholtz-Zentrum Hereon +1

Sep 25, 2026J/K move · Enter open · S save

Accounting for Bias Enables Sustainable LLM Evaluation

Harshita Katoch, David Antony Selby, Gerrit Großmann, Sebastian Vollmer

Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany

LLM Evaluation Bias in LLM-as-a-Judge LLM-as-a-Judge Learning to Rank Latent Variable Models

Open paper AskPDF arXiv

LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

Abstract

Bias in LLM-as-a-Judge

Explore similar work

Accounting for Bias Enables Sustainable LLM Evaluation

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs