Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each judgment is often assumed to be independent of earlier evaluations. We test this assumption using three prompt conditions: no metadata, revision framing, and anchored metadata containing revision, attempt, and prior-score fields. We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values. Across 192,000 attempted evaluations (185,271 successful), seven out of the eight evaluated models have 95% task-stratified bootstrap intervals below zero for the total anchored-metadata effect on 20 fixed texts. Cohen's d, a standardized measure of the difference between score distributions, reaches an absolute value of 0.71. Token-level analysis of selected model-task probes suggests a threshold-like response pattern: introducing anchored metadata produces a marked redistribution of output-score probabilities, while changing the anchor value within the tested below-threshold range produces comparatively little additional variation. On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label, demonstrating the bias extends beyond numerical scoring to categorical decisions. Neither Chain-of-Thought nor a metadata-disregard warning reduces the total effect, although the warning improves the paired accuracy effect relative to baseline in the industry experiment. Reliable LLM evaluation demands careful context engineering rather than an assumption of impartiality. Effective mitigation must be validated for the intended model and task or domain.
Figures & tables
Figure 1. Experimental design with three memory conditions. C0 (baseline) includes no contextual metadata. C1 (framing) indicates the submission is a revision without providing a score. C2 (anchoring) adds a prior score a sampled uniformly from [0,3.99) , always below the acceptance threshold. The primary comparison ΔC2−C0 measures the total anchoring effect. The intermediate condition C1 enables additive decomposition: ΔC2−C0=ΔC1−C0+ΔC2−C1 , separating framing effects from the incremental influence of the numerical anchor. Schematic of the three experimental conditions C0, C1, and C2 with comparison arrows. Three side-by-side boxes labeled C0 (Baseline, blue), C1 (Framing, green), and C2 (Anchoring, red). C0 contains task, answer, and rubric with no metadata. C1 adds a metadata block stating that the submission is a revision. C2 adds a metadata block listing the attempt number k and a prior score a drawn uniformly from [0,3.99). Below the boxes, arrows mark three comparisons: a primary C2 versus C0 arrow labeled total anchoring effect, a smaller C1 versus C0 arrow labeled framing effect, and a C2 versus C1 arrow labeled incremental anchoring.
Model
Triplets
Incomplete
GPT-4.1
8,000
0.0%
Claude-4.5-Sonnet
8,000
0.0%
DeepSeek-R1
7,936
0.8%
Llama-3.3-70B
7,835
2.1%
Gemma-2-9B
7,596
5.0%
Llama-3.1-8B
6,804
15.0%
Table 1. Complete C0/C1/C2 triplets under the trial-index audit. Exclusions are relative to 8,000 possible triplets per model, of which 59,136 remain.
Model
ΔC2−C0
95% CI
d
Llama-3.2-3B
−0.254
[−0.298,−0.208]
−0.71
GPT-4.1
−0.411
[−0.457,−0.361]
−0.67
Qwen2.5-7B
−0.233
[−0.324,−0.150]
−0.51
Claude-4.5-Sonnet
−0.706
[−1.139,−0.245]
−0.44
DeepSeek-R1
−0.269
[−0.326,−0.221]
−0.25
Gemma-2-9B
−0.072
[−0.130,−0.001]
−0.18
Table 2. Total anchored-metadata effect ΔC2−C0 by model. Point estimates first average the four temperature-specific contrasts for each task, then equally average the resulting 20 task effects. Intervals resample tasks within category. Cohen’s d is descriptive and uses the complete-triplet sample.
Figure 2. Score difference by model, condition, and temperature. Circles show ΔC2−C0 , whereas squares show ΔC1−C0 . Error bars are 95% stratified task-bootstrap intervals. Score difference by model and condition across temperatures. Four panels show mean score changes relative to C0 for eight models. Circles denote C2 minus C0 and squares C1 minus C0. Negative values indicate lower scores than baseline.
Figure 3. Threshold-derived acceptance-rate differences by model, condition, and temperature. Circles show ΔC2−C0 , whereas squares show ΔC1−C0 . Error bars are 95% stratified task-bootstrap intervals. Threshold-derived acceptance-rate difference by model and condition across temperatures. Four panels show changes in the probability that score is at least 4.0. Circles denote C2 minus C0 and squares C1 minus C0.
Figure 4. Mean total anchored-metadata effect ΔC2−C0 by model and task category. Heatmap of mean total anchored-metadata shift by model and task category. Cells show mean C2 minus C0. Code review is generally closest to zero, while the other categories are more negative, with heterogeneity across models.
Term
Estimate
95% CI
Code review, T=0
−0.067
[−0.141,−0.011]
Creative writing
−0.236
[−0.374,−0.071]
Factual QA
−0.291
[−0.430,−0.134]
Summarization
−0.275
[−0.381,−0.165]
Table 3. Task-aware joint model. Category rows are contrasts against code review, and the intercept is the equally weighted fixed-model-set mean at T=0 . Intervals resample tasks within category.
Digit
Pd(C0)
Pd(C2)
Change
“3”
0.000
0.000
+0.000
“4”
0.074
1.000
+0.926
“5”
0.926
0.000
−0.926
Table 4. Score-digit probabilities under C0 and C2 for the selected GPT-4.1 creative-writing probe. Values are rounded to three decimal places.
Model
Score Δ
P4C0
P4C2
Slope
Llama-3.2-3B
−0.120
0.995
0.977
−0.0085
Llama-3.1-8B
−0.018
0.992
0.992
+0.0009
Llama-3.3-70B
+0.016
1.000
1.000
+0.0000
Table 5. Behavioral and modal-token results for the selected Llama probes. Score Δ denotes ΔC2−C0 , and slope is the within-C2 regression of P4 on anchor value. Values are rounded to three decimal places except for the slope.
Analysis
Estimate
95% CI
Signed, unadjusted
+0.211
[−0.013,−0.408]
Signed, adjusted
+0.297
[−0.054,−0.647]
Absolute, adjusted
−0.016
[−0.139,−0.107]
Non-ceiling mean
−0.227
[−0.343,−0.098]
Table 6. Ceiling robustness. Regression predictors use distance to ceiling, 5−μC0 . Regression intervals use task-clustered uncertainty, whereas the non-ceiling interval uses a task bootstrap over the 19 represented tasks.
Strategy
C0
C2
Δtotal [95% CI]
Baseline
4.245
3.817
−0.429[−0.462,−0.395]
CoT
4.458
3.825
−0.633[−0.703,−0.565]
Warning
4.500
4.043
−0.457[−0.479,−0.435]
Table 7. Numerical mitigation on one GPT-4.1/task pair. The primary metric is Δtotal=μC2−μC0 , while within-C2 slopes are 0.267, 0.506, and −0.007 , respectively.
Strategy
Prompt
Comp.
Total
Token increase
Baseline
397.0
117.7
514.7
—
CoT
437.0
168.7
605.7
17.7%
Warning
437.0
109.4
546.4
6.2%
Table 8. Mean token cost per call. The increase is based on total tokens relative to baseline.
Group
Metric
C0
C1
C2
A
Correction
21.62%
22.97%
11.26%
A
Mislabel persistence
65.77%
65.77%
81.53%
B
Accuracy
76.26%
—
68.49%
B
Anchor agreement
11.42%
—
18.72%
Table 9. Categorical effects with human ground truth. Group A uses C0/C1/C2 conditions. Group B uses paired C0/C2 outputs and a fixed sample-specific wrong anchor.
Strategy
C0 acc.
C2 acc.
Δ acc.
Induced ( n/N )
Baseline
76.26%
68.49%
−7.76 pp
17/167 (10.18%)
CoT
68.04%
62.56%
−5.48 pp
20/149 (13.42%)
Warning
74.43%
74.43%
−0.00 pp
7/163 (4.29%)
Table 10. Paired Group B results. Induced is the share of each strategy’s C0-correct cases that become its assigned wrong label in C2.
LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.
Harshita Katoch, David Antony Selby, Gerrit Großmann +1
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany
Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.
The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in human judgment and decision-making, and recent work suggests that large language models (LLMs) exhibit similar behavior. However, existing work on anchoring in LLMs typically evaluates only a narrow set of anchor pathways and rarely distinguishes irrelevant from plausible anchors. We introduce AnchorBench, a benchmark for the anchoring effect in LLMs that evaluates multiple anchor pathways under an explicit anchor relevance axis. Across fourteen models, including ten open-weight models and four frontier API models, and a large set of controlled prompts, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually induce larger shifts than irrelevant ones when introduced through stronger pathways, (3) anchor influence generally weakens as the anchor moves farther from the evidence-supported answer, most clearly on External and RAG, and (4) high task accuracy on the anchor-free control condition (Acc10: answers within 10 points of gold) does not guarantee robustness: even frontier API models above 95% control accuracy remain susceptible to plausible anchors.
Yiderigun Borjigin, Alexander Hermann, Christian Cyron +1
Saarland University · Hamburg University of Technology · Helmholtz-Zentrum Hereon +1