AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

Yiderigun Borjigin; Alexander Hermann; Christian Cyron; Roland Aydin

cs.AIAug 14, 2026

AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

Authors: Yiderigun Borjigin, Alexander Hermann, Christian Cyron, Roland Aydin

Organizations: Saarland University · Hamburg University of Technology · Helmholtz-Zentrum Hereon · German Research Centre for Artificial Intelligence (DFKI)

Abstract

The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in human judgment and decision-making, and recent work suggests that large language models (LLMs) exhibit similar behavior. However, existing work on anchoring in LLMs typically evaluates only a narrow set of anchor pathways and rarely distinguishes irrelevant from plausible anchors. We introduce AnchorBench, a benchmark for the anchoring effect in LLMs that evaluates multiple anchor pathways under an explicit anchor relevance axis. Across fourteen models, including ten open-weight models and four frontier API models, and a large set of controlled prompts, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually induce larger shifts than irrelevant ones when introduced through stronger pathways, (3) anchor influence generally weakens as the anchor moves farther from the evidence-supported answer, most clearly on External and RAG, and (4) high task accuracy on the anchor-free control condition (Acc $_{10}$ : answers within 10 points of gold) does not guarantee robustness: even frontier API models above 95% control accuracy remain susceptible to plausible anchors.

Figures & tables

Figure 1: AnchorBench is a unified benchmark for studying anchoring in LLMs, (a) inspired by four human anchoring paradigms, (b) AnchorBench varies anchor pathway and anchor relevance, (c) evaluates them in a shared numeric judgment setup with standardized inference and metrics, (d) compares anchoring behavior across fourteen LLMs.

Panel of Figure 2 — Figure 2: Example AnchorBench item (External suite, pricing domain). All conditions share the same scenario, evidence, question, and answer instruction. Only the anchor sentence differs. Additional examples are in Appendix A.2 .

Model	External			History			ICL			RAG			Tool
Model	Acc	U ${}_{\text{irr}}$	U ${}_{\text{pls}}$	Acc	U ${}_{\text{irr}}$	U ${}_{\text{pls}}$	Acc	U ${}_{\text{irr}}$	U ${}_{\text{pls}}$	Acc	U ${}_{\text{irr}}$	U ${}_{\text{pls}}$	Acc	U ${}_{\text{irr}}$	U ${}_{\text{pls}}$
2.5-1.5B	34%	0.02	0.20	32%	0.29	0.98	59%	0.01	0.00	22%	0.02	0.23	46%	0.26	0.82
2.5-3B	64%	0.01	0.09	61%	$-$ 0.08	0.49	78%	$-$ 0.04	$-$ 0.04	74%	$-$ 0.02	0.08	89%	0.08	0.18
2.5-7B	72%	0.01	0.27	79%	$-$ 0.06	0.37	88%	$-$ 0.03	$-$ 0.01	64%	$-$ 0.00	0.20	99%	0.00	0.17
3.2-1B	47%	0.13	0.15	49%	0.48	0.14	55%	0.30	0.22	60%	0.10	0.09	—	—	—
3.2-3B	69%	0.14	0.29	72%	0.15	0.73	81%	0.07	0.14	72%	0.14	0.22	95%	0.03	0.21

Table 1: Task accuracy (Acc,

=

Acc 10 , control-only) and absolute anchor influence on irrelevant (U

{}_{\text{irr}}

) and plausible (U

{}_{\text{pls}}

) anchors for fourteen models across five suites. Any positive U

{}_{\text{irr}}

is an unjustified shift toward a value that carries no task information; U

{}_{\text{pls}}

is sensitivity to a plausibly framed value, and the plausible–irrelevant gap Disc

{}_{\Delta}=\text{U}_{\text{pls}}-\text{U}_{\text{irr}}

is analyzed in § 4.2 . Anchoring varies strongly by pathway: External and RAG show the broadest positive effects; ICL is near zero; History and Tool vary by model. Bold marks the largest U

{}_{\text{pls}}

per suite. Caveats: Llama-1B/Tool is undefined (parse rate 0.9%); History uses a single-stage control (Appendix A.10 ); Tool format varies by model (Appendix A.12 ). For space, instruction-tuned model names are abbreviated by omitting the “-Instruct” suffix; full model identifiers are listed in Table 3 .

Figure 3: Anchor influence attenuates with distance (suites with designer-controlled offsets only). UAI

{}_{\text{pls}}

decreases monotonically with anchor offset

\delta

on External and RAG; ICL (dotted) stays near zero at all distances. Lines show suite means across models; shaded bands are 95% CIs (

\pm

1.96 SE). API models show monotonic attenuation on External; on RAG, they peak at

\delta{=}25

before decreasing. History is omitted because its offsets are not designer-controlled (§ 3.3 ); Tool is omitted due to format confounds (Appendix A.12 ).

Model	UAI ${}_{\text{pls}}$			UAI ${}_{\text{irr}}$
Model	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}1$	$k{=}2$	$k{=}3$
3-4B-it	0.21	0.20	0.25	0.14	$-$ 0.13	0.13
3.1-8B	0.68	0.36	0.50	0.25	$-$ 0.03	0.20
2-13B	0.48	0.34	0.29	0.19	0.11	0.06
2.5-7B	0.53	0.34	0.36	0.22	0.01	0.02
Rational ceiling ( $w{=}1$ )	0.50	0.33	0.25	0	0	0

Table 2: Anchoring under partial evidence on External . Only

k

of the five ratings are shown while gold remains the full-five mean, so the model is genuinely uncertain. An updater treating the plausible anchor as one extra rating of equal credibility would produce UAI

{}_{\text{pls}}=w/(k{+}w)

(last row); for an irrelevant anchor the rational value is

0

at every

k

. Bold marks cells above the ceiling.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Model card	Params	Access	Route
Qwen/Qwen2.5-1.5B-Instruct	1.5B	Open-weight	Local
Qwen/Qwen2.5-3B-Instruct	3B	Open-weight	Local
Qwen/Qwen2.5-7B-Instruct	7B	Open-weight	Local
meta-llama/Llama-3.2-1B-Instruct	1B	Open-weight	Local
meta-llama/Llama-3.2-3B-Instruct	3B	Open-weight	Local
meta-llama/Llama-3.1-8B-Instruct	8B	Open-weight	Local

Appendix

Table 3: Model panel used in AnchorBench . Open-weight models were served locally on H100 GPUs; API models were accessed through OpenRouter.

Panel of Figure 4 — Figure 4: Example AnchorBench item for the History suite. The model first answers from partial evidence; that Stage-1 answer then serves as a self-generated anchor when the full evidence is revealed in Stage 2. In anchored conditions, this two-stage interaction creates the self-generated anchor pathway; the control instead presents the full evidence in a single turn.

Panel of Figure 5 — Figure 5: Example AnchorBench item for the ICL suite. Three demonstrations precede the target item. In the main-benchmark variant ( ICL-metadata ), the anchor appears only in the demonstration header metadata; demo evidence, demo answers, and target evidence are unchanged across conditions.

Panel of Figure 6 — Figure 6: Example AnchorBench item for the RAG suite. A frozen three-document mini-corpus is concatenated into the prompt. One document serves as the anchor slot, while the surrounding documents provide anchor-free context. This suite operationalizes retrieved-document context as an anchor pathway, without a live retriever.

Panel of Figure 7 — Figure 7: Example AnchorBench item for the Tool suite. Two simulated tool outputs are injected before the model answers. The first is anchor-free; the second carries the condition-dependent anchor. This suite operationalizes tool-output context as an anchor pathway rather than full agentic tool use.

Model	MAE c $\,\downarrow$	Acc 10 (%) $\,\uparrow$	UAI irr	UAI pls	TAR irr	TAR pls	Disc Δ	Parse
2.5-1.5B-Instruct	17.93	34.2%	0.02	0.20	0.22	0.48	0.18	1.00
2.5-3B-Instruct	9.62	63.7%	0.01	0.09	0.21	0.50	0.08	0.99
2.5-7B-Instruct	7.74	72.5%	0.01	0.27	0.15	0.67	0.26	1.00
3.2-1B-Instruct	16.79	47.2%	0.13	0.15	0.38	0.49	0.03	0.91
3.2-3B-Instruct	9.21	69.4%	0.14	0.29	0.36	0.59	0.15	1.00
3.1-8B-Instruct	5.61	82.6%	0.07	0.36	0.22	0.69	0.30	0.98

Appendix

Table 4: External suite full results (360 items, 1,800 prompts per model).

Model	MAE c $\,\downarrow$	Acc 10 (%) $\,\uparrow$	UAI irr	UAI pls	TAR irr	TAR pls	Disc Δ	Parse
2.5-1.5B-Instruct	19.02	32.5%	0.29	0.98	0.52	0.71	0.70	1.00
2.5-3B-Instruct	9.56	60.6%	$-$ 0.08	0.49	0.46	0.63	0.57	1.00
2.5-7B-Instruct	7.11	78.6%	$-$ 0.06	0.37	0.37	0.57	0.43	1.00
3.2-1B-Instruct	17.23	48.8%	0.48	0.14	0.65	0.59	$-$ 0.35	0.92
3.2-3B-Instruct	9.20	72.2%	0.15	0.73	0.49	0.70	0.58	0.99
3.1-8B-Instruct	5.74	82.9%	0.13	0.23	0.31	0.42	0.10	0.99

Appendix

Table 5: History suite full results (360 items, two-stage protocol). Anchor values are model-generated (Stage 1 answers). API models show near-zero discrimination, consistent with strong self-correction from full evidence in Stage 2.

Model	MAE c $\,\downarrow$	Acc 10 (%) $\,\uparrow$	UAI irr	UAI pls	TAR irr	TAR pls	Disc Δ	Parse
2.5-1.5B-Instruct	10.27	59.4%	0.01	0.00	0.23	0.25	$-$ 0.01	1.00
2.5-3B-Instruct	7.16	77.5%	$-$ 0.04	$-$ 0.04	0.15	0.21	0.00	1.00
2.5-7B-Instruct	5.50	87.5%	$-$ 0.03	$-$ 0.01	0.17	0.23	0.01	1.00
3.2-1B-Instruct	16.26	54.5%	0.30	0.22	0.42	0.49	$-$ 0.09	0.92
3.2-3B-Instruct	6.98	80.8%	0.07	0.14	0.36	0.44	0.06	1.00
3.1-8B-Instruct	7.31	76.7%	0.03	0.15	0.40	0.51	0.12	0.97

Appendix

Table 6: ICL suite full results (360 items). Anchors appear only in demo-header metadata; demo evidence and answers are identical across conditions.

Model	MAE c $\,\downarrow$	Acc 10 (%) $\,\uparrow$	UAI irr	UAI pls	TAR irr	TAR pls	Disc Δ	Parse
2.5-1.5B-Instruct	22.15	21.9%	0.02	0.23	0.18	0.49	0.21	1.00
2.5-3B-Instruct	8.12	74.1%	$-$ 0.02	0.08	0.15	0.39	0.10	1.00
2.5-7B-Instruct	9.83	63.9%	$-$ 0.00	0.20	0.11	0.48	0.20	1.00
3.2-1B-Instruct	13.73	60.5%	0.10	0.09	0.29	0.35	$-$ 0.01	0.96
3.2-3B-Instruct	10.60	71.5%	0.14	0.22	0.29	0.36	0.08	0.99
3.1-8B-Instruct	4.66	85.1%	$-$ 0.02	0.12	0.22	0.36	0.14	0.99

Appendix

Table 7: RAG suite full results (360 items).

Model	MAE c $\,\downarrow$	Acc 10 (%) $\,\uparrow$	UAI irr	UAI pls	TAR irr	TAR pls	Disc Δ	Parse
2.5-1.5B-Instruct	26.17	45.9%	0.26	0.82	0.28	0.72	0.57	0.97
2.5-3B-Instruct	4.62	89.2%	0.08	0.18	0.24	0.37	0.10	0.98
2.5-7B-Instruct	0.67	98.9%	0.00	0.17	0.08	0.51	0.17	1.00
3.2-1B-Instruct	—	—	—	—	—	—	—	0.01
3.2-3B-Instruct	2.33	94.5%	0.03	0.21	0.17	0.34	0.19	0.80
3.1-8B-Instruct	12.15	73.3%	0.31	0.48	0.28	0.65	0.17	0.96

Appendix

Table 8: Tool suite full results (360 items). Tool outputs are structured ( role: "tool" ) for Qwen and Llama, and plaintext for Gemma, OLMo, and API models. Llama-3.2-1B metrics are undefined (parse rate 0.9%).

Suite	UAI irr	UAI pls	Disc Δ	$\frac{+}{n}$
External	0.05	0.22	0.17	14/14
History	0.05	0.34	0.28	10/14
ICL (metadata)	0.02	0.03	0.00	7/14
ICL (ICL-dist)	0.09	0.16	0.07	8/10
RAG	0.03	0.13	0.10	12/14
Tool	0.07	0.18	0.11	12/13

Appendix

Table 9: Mean anchoring influence by relevance and suite.

+/n

: cells with Disc

{}_{\Delta}>0

. ICL-dist uses distribution-matching demonstrations (open-weight models only).

Contrast	Est.	95% CI	$p_{\mathrm{BH}}$
Plausible $>$ irrelevant (mean $\Delta$ UAI)
External	0.17	[0.13, 0.22]	$<$ 0.01
History	0.28	[0.12, 0.47]	0.02
ICL	0.00	[ $-$ 0.02, 0.03]	0.81
RAG	0.10	[0.05, 0.17]	$<$ 0.01
Tool	0.11	[0.04, 0.20]	$<$ 0.01

Appendix

Table 10: Statistical summary. Top: Wilcoxon signed-rank tests for UAI

{}_{\text{pls}}>\text{UAI}_{\text{irr}}

(

n{=}14

models per suite,

n{=}13

on Tool;

p

-values BH-corrected). Bottom: cross-suite discrimination range and Acc–Disc correlation. All CIs are bootstrap 95%.

Figure 8: Task accuracy (Acc 10 ) vs. discrimination (Disc Δ ) for all 69 computable model–suite cells. Circles: open-weight; diamonds: API; color: suite. The weak Pearson

r

(

-

0.24, 95% CI [

-

0.43,

-

0.00]) shows that accuracy and anchoring robustness are only weakly associated.

	UAI irr	UAI pls
In $[0,1]$	77.1%	73.4%
Overshoot ( $>1$ )	4.2%	7.0%
Reverse ( $<0$ )	18.7%	19.7%
Mean	0.04	0.17
Median	0.00	0.00
Capped $[0,1]$ mean	0.10	0.22

Appendix

Table 11: Distribution of per-item UAI values across all models and suites. The majority of items fall in

[0,1]

, but roughly 4–7% overshoot and

\sim

19% show reverse shifts. Overshoot concentrates in History (23.8% for plausible).

Suite	$\Delta$ MAE irr	$\Delta$ MAE pls
External	$+$ 0.08	$+$ 3.61
History	$-$ 1.60	$+$ 0.12
ICL	$+$ 0.14	$-$ 0.04
RAG	$-$ 0.04	$+$ 0.59
Tool	$-$ 0.67	$+$ 3.39

Appendix

Table 12: Mean change in MAE from control to anchored conditions (averaged across all models per suite). Positive values indicate accuracy degradation. Plausible anchors consistently increase MAE; irrelevant anchors are generally small, though Tool shows a modest negative effect (

-

0.67).

Model	MAE std	MAE ts	Disc std	Disc match
2.5-1.5B-Instruct	5.49	24.87	0.89	0.27
2.5-3B-Instruct	3.65	17.67	0.18	0.51
2.5-7B-Instruct	3.77	10.33	0.18	0.97
3.2-1B-Instruct	15.71	20.87	0.01	0.39
3.2-3B-Instruct	9.46	15.42	0.22	0.20
3.1-8B-Instruct	7.02	8.05	0.08	0.33

Appendix

Table 13: History suite: standard vs. matched-format (two-stage) control analysis. MAE std : MAE on single-stage control; MAE ts : MAE on two-stage control; Disc std : Disc Δ using single-stage control; Disc match : Disc Δ using two-stage control. Both columns are computed from the dedicated matched-control evaluation run and should be compared against each other; Disc std values may differ from Table 1 because the matched-control run used a separate item subset and inference pass.

Model	UAI pls,std	UAI pls,dist	Disc std	Disc dist
2.5-1.5B-Instruct	0.00	0.43	$-$ 0.01	0.09
2.5-3B-Instruct	$-$ 0.04	0.15	0.00	0.07
2.5-7B-Instruct	$-$ 0.01	0.06	0.01	0.07
3.2-1B-Instruct	0.22	0.22	$-$ 0.09	0.04
3.2-3B-Instruct	0.14	0.11	0.06	0.10
3.1-8B-Instruct	0.15	0.26	0.12	$-$ 0.03

Appendix

Table 14: Standard ICL (metadata anchors) vs. ICL-dist (numerically relevant demonstrations).

Model	Disc struct	Disc plain	MAE struct	MAE plain
2.5-1.5B-Instruct	0.58	0.07	11.34	15.01
2.5-3B-Instruct	0.08	0.11	2.19	15.74
2.5-7B-Instruct	0.17	0.02	0.71	1.32
3.2-3B-Instruct	0.23	0.10	2.28	9.49
3.1-8B-Instruct	0.16	0.14	5.47	6.77

Appendix

Table 15: Tool suite: structured ( role: "tool" ) vs. plaintext format for all five models that used structured format in the main benchmark. Disc: Disc Δ ; MAE: control-condition MAE.

Suite	Err $\uparrow$ (%)	Err $\downarrow$ (%)	Irr err $\uparrow$	Pls err $\uparrow$
External	33.9	20.0	19.9	48.0
History	39.2	44.0	29.5	48.7
ICL	22.9	20.0	20.3	25.4
RAG	22.2	20.2	16.4	27.9
Tool	28.6	17.5	17.6	39.9
All	28.7	23.2	20.2	37.3

Appendix

Table 16: Error decomposition relative to the evidence-only target (aggregated over fourteen models). Err

\uparrow

/Err

\downarrow

: share of anchored items where the response shifts away from / toward the gold answer. Irr err

\uparrow

/Pls err

\uparrow

: error-increasing rate by anchor relevance. Overall, error-increasing shifts (29%) outnumber error-reducing ones (23%); the gap is driven by plausible anchors (37% vs. 20% for irrelevant), with the largest differences on External and Tool (

\geq

22 pp).

Figure 9: Harmful shift share by model and relevance condition. Each point shows the percentage of anchored items on which the model’s error increased relative to the unanchored control. Plausible anchors (right) consistently produce a larger harmful share than irrelevant anchors (left), with the widest gaps for External and Tool suites.

Suite	Diff.	Acc 10	MAE c	UAI irr	UAI pls	Disc Δ
External	easy	77.0%	7.2	0.05	0.16	0.11
	hard	65.5%	9.6	0.04	0.28	0.23
History	easy	77.1%	7.5	0.03	0.36	0.33
	hard	65.1%	10.1	0.08	0.31	0.23
ICL	easy	76.7%	6.7	0.03	0.02	$-$ 0.01
	hard	72.4%	7.8	0.02	0.03	0.01

Appendix

Table 17: Anchoring metrics by item difficulty (mean across 14 models per suite, 13 on Tool). Bold highlights the higher Disc Δ within each suite. Hard items show stronger discrimination on External and RAG; History shows the opposite pattern.

Suite	Model	Disc Δ greedy	Disc Δ sampled	Sign
External	Qwen-7B	0.26	0.29 $\pm$ 0.02	✓
	Llama-8B	0.30	0.24 $\pm$ 0.07	✓
RAG	Qwen-7B	0.19	0.18 $\pm$ 0.02	✓
	Llama-8B	0.16	0.06 $\pm$ 0.04	✓
ICL-dist	Qwen-7B	0.07	0.08 $\pm$ 0.02	✓
	Llama-8B	0.03	0.00 $\pm$ 0.02	$\times$

Appendix

Table 18: Sampling robustness. Sampled values are mean

\pm

SD over 3 seeds (temperature 0.7, top-

p

0.9). Sign: whether greedy and sampled mean share the same Disc Δ sign.

Suite	Strategy	Disc Δ		MAE ctrl
		Qwen-7B	Llama-8B	Qwen-7B	Llama-8B
External	baseline	0.26	0.30	7.7	5.5
	ignore	0.16	0.10	5.0	3.4
	self-check	0.45	0.28	8.8	7.5
	CoT	0.14	0.31	1.1	6.1
RAG	baseline	0.20	0.16	9.8	4.8

Appendix

Table 19: Mitigation headroom probe. Disc Δ and control MAE for three prompt-based strategies vs. unmodified baseline. Ignore consistently reduces discrimination across both models and suites. CoT strongly helps Qwen-7B but is inconsistent for Llama-8B. Self-check increases discrimination for Qwen-7B on both suites but slightly reduces it for Llama-8B.

Model	External	History	Rag	Tool	Icl
Claude-H4.5	0.71	0.00	0.29	0.03	0.00
GPT-5.4-mini	0.79	0.23	0.47	0.33	0.00
Gemini-2.5-Flash	0.38	0.00	1.03	0.29	0.00
Gemma-1B	1.10	0.76	0.22	0.00	0.00
Gemma-4B	3.74 ∗	5.00 ∗	0.03	1.37	0.00
Grok-3-mini	0.97	0.53	0.11	0.08	0.02

Appendix

Table 20: Implied anchor weight

w_{\mathrm{imp}}

on plausible anchors, defined as the smallest weight under a Gaussian-conjugate Bayesian model that would make the observed mean UAI rational (with

n=5

evidence ratings). Under the reference assumption that an anonymous anchor is worth one evidence item (

w=1

), the rational UAI ceiling is 0.167;

w_{\mathrm{imp}}>1

means the model treats the anchor as more credible than a single evidence item.

w_{\mathrm{imp}}>n

means the model treats the anchor as more credible than all visible evidence combined. ∗ marks cells where the 95% CI for UAI pls lies entirely above the rational ceiling.

Model	External	History	Rag	Tool	Icl
Claude-H4.5	-0.042	-0.226	-0.112	-0.161	-0.184
GPT-5.4-mini	-0.030	-0.123	-0.080	-0.105	-0.173
Gemini-2.5-Flash	-0.096	-0.171	+0.005	-0.111	-0.166
Gemma-1B	+0.013	-0.034	-0.125	-0.195	-0.193
Gemma-4B	+0.261 ∗	+0.333 ∗	-0.161	+0.048	-0.228
Grok-3-mini	-0.004	-0.071	-0.145	-0.151	-0.163

Appendix

Table 21: Excess UAI on plausible anchors above the rational Bayesian ceiling 0.167 (assuming the anchor is worth one evidence item,

w=1

n=5

evidence ratings). Positive values indicate the model shifted further toward the anchor than a rational Bayesian update could justify. ∗ marks cells where the 95% CI for UAI pls lies entirely above the ceiling.

Model	UAI placebo	UAI irr	UAI pls	UAI auth
GPT-5.4-mini	-0.01	+0.01	+0.13	+0.40
Gemma-4B	+0.38	+0.20	+0.43	+0.58
Llama-8B	+0.08	+0.07	+0.36	+0.42
OLMo-13B	+0.02	-0.01	+0.16	+0.49
Qwen-7B	-0.00	+0.01	+0.27	+0.56
Mean	+0.09	+0.05	+0.27	+0.49

Appendix

Table 22: Plausibility spectrum on External: mean UAI for each of four anchor types — placebo (anchor refers to an extraneous numeric property such as document length), irrelevant (case-number anchor), plausible (“a recent report suggested

\sim

23”), and authority (“a panel of senior analysts estimated

\sim

23”). A monotone increase along the plausibility axis would be consistent with rational use of source credibility; any positive UAI on the placebo column is unambiguously anchoring bias.

Model	Acc 10			UAI pls			UAI irr
Model	k=1	k=2	k=3	k=1	k=2	k=3	k=1	k=2	k=3
Gemma-4B	54%	60%	64%	0.21	0.20	0.25	0.14	$-$ 0.13	0.13
Llama-8B	47%	61%	69%	0.68	0.36	0.50	0.25	$-$ 0.03	0.20
OLMo-13B	54%	76%	81%	0.48	0.34	0.29	0.19	0.11	0.06
Qwen-7B	42%	64%	75%	0.53	0.34	0.36	0.22	0.01	0.02
Rational ceiling ( $w{=}1$ )	—	—	—	0.50	0.33	0.25	—	—	—

Appendix

Table 23: Uncertain-judgment results. Only k of 5 ratings are shown to the model; the gold answer remains the full-5 mean. An ideal Bayesian updater treating the plausible anchor as one additional rating of equal credibility (

w{=}1

) would produce UAI

{}_{\mathrm{pls}}=w/(k+w)

(Rational ceiling row). Measured UAI pls above the ceiling indicates classical (super-rational) anchoring; below indicates the model weights the anchor less than one full additional evidence point. UAI irr should stay near zero for capable models at every

k

Model	External			RAG			History
Model	Mild	Std	Strong	Mild	Std	Strong	Mild	Std	Strong
Gemma-4B	0.45	0.42	0.40	0.00	$-$ 0.06	0.04	0.61	0.58	0.60
Llama-8B	0.26	0.39	0.42	0.10	0.09	0.29	0.44	0.20	1.57
OLMo-13B	0.07	0.16	0.29	0.02	0.09	0.10	0.45	0.24	0.91
Qwen-7B	0.23	0.31	0.49	0.07	0.19	0.29	0.37	0.24	0.52
Mean	0.25	0.32	0.40	0.05	0.08	0.18	0.47	0.31	0.90

Appendix

Table 24: Cross-pathway plausibility-intensity dose-response. The same mild/standard/strong source-credibility manipulation is applied across three pathways. Positive Strong

>

Mild gaps indicate models weight source credibility; near-flat curves at high UAI indicate numeric anchoring dominates regardless of source. UAI values are mean across the panel; details in supplementary CSV.

		UAI irr			UAI pls			Disc Δ
Suite	Model	bus	med	oth	bus	med	oth	bus	med	oth
external	Gemma-4B	0.20	0.28	0.20	0.43	0.21	0.67	0.23	-0.07	0.46
	Llama-8B	0.07	0.03	0.20	0.36	0.25	0.46	0.30	0.21	0.26
	OLMo-13B	-0.01	0.08	-0.03	0.16	0.09	0.22	0.17	0.01	0.24
	Qwen-7B	0.01	0.07	0.01	0.27	0.38	0.45	0.26	0.32	0.44
history	Gemma-4B	0.23	0.17	0.03	0.50	0.41	0.40	0.27	0.24	0.37

Appendix

Table 25: Eleven-domain robustness panel: 6 business domains (published benchmark), 3 medical pilot domains, and 2 further pilot domains (legal contract compliance, consumer purchase decision). The pattern UAI

{}_{\mathrm{pls}}>

UAI

{}_{\mathrm{irr}}>0

replicates across all three domain families on both External and History, demonstrating that anchoring is not a business-domain artifact.

	UAI irr		UAI pls		Disc Δ
Model	mean	wmean	mean	wmean	mean	wmean
Gemma-4B	0.20	0.20	0.43	0.47	0.23	0.28
Llama-8B	0.07	0.06	0.36	0.34	0.30	0.28
OLMo-13B	-0.01	0.03	0.16	0.20	0.17	0.17
Qwen-7B	0.01	0.01	0.27	0.27	0.26	0.26

Appendix

Table 26: Anchor influence on External under the published mean gold standard (“mean”) vs. a weighted-mean gold standard with weights

[1,1,1.5,1.5,2]

(“wmean”). The bias structure (UAI

{}_{\mathrm{pls}}\gg

UAI irr ) is preserved under either scoring choice; the size of the anchoring effect is not an artifact of the simple-mean aggregation.

		UAI irr			UAI pls			Disc Δ
Suite	Model	base	CoT	$\Delta$	base	CoT	$\Delta$	base	CoT	$\Delta$
external	GPT-5.4-mini	0.01	-0.01	-0.01	0.13	0.14	+0.01	0.12	0.14	+0.02
	Gemma-4B	0.09	0.01	-0.08	0.37	0.23	-0.14	0.28	0.22	-0.06
	Llama-8B	0.08	0.06	-0.03	0.39	0.37	-0.03	0.31	0.31	-0.00
	OLMo-13B	0.06	0.05	-0.01	0.15	0.18	+0.03	0.08	0.13	+0.04
	Qwen-7B	0.01	0.00	-0.01	0.27	0.16	-0.12	0.26	0.15	-0.11

Appendix

Table 27: Reasoning-allowed (Chain-of-Thought) prompting vs. “final answer only” baseline across the locked 5-model panel and all three suites (External, RAG, History). CoT lowers plausible-anchor influence in 11 of the 15 model–suite cells, but the effect is model- and pathway-dependent rather than uniform: it lowers both UAI irr and UAI pls in only 6 cells, and raises UAI pls in 4. UAI pls remains positive in 13 of 15 cells, so anchoring is attenuated but not an artifact of the answer-only protocol.

Model	UAI pls			UAI irr
Model	Baseline	+Rule	+Judgment	Baseline	+Rule	+Judgment
Gemma-4B	0.51	0.08	0.38	0.27	0.02	0.25
Llama-8B	0.32	0.01	0.28	0.05	0.00	0.14
OLMo-13B	0.20	0.01	0.12	0.04	0.00	0.08
Qwen-7B	0.27	0.13	0.27	0.01	0.02	$-$ 0.00
Mean	0.33	0.06	0.26	0.09	0.01	0.12

Appendix

Table 28: Task-specification ablation on External. Baseline = published prompt (return integer only); +Rule appends ‘Your estimate should be the unweighted arithmetic mean of the visible ratings’; +Judgment appends ‘Your estimate should be a weighted average … weighting each piece according to which sources you find more or less credible’. A drop from Baseline to +Rule indicates the published anchoring effect is partly driven by task underspecification.

Model	Plausible (UAI)				Irrelevant (UAI)
Model	Base	R1	R5+D	R5+D+Sc	Base	R1	R5+D	R5+D+Sc
Gemma-4B	$-$ 0.06	0.00	0.01	0.01	0.03	$-$ 0.04	0.01	0.02
Llama-8B	0.09	0.12	0.14	$-$ 0.04	$-$ 0.02	0.01	0.01	0.01
OLMo-13B	0.09	0.02	0.21	0.04	0.04	0.09	0.13	0.05
Qwen-7B	0.19	0.23	0.14	0.11	0.01	$-$ 0.02	$-$ 0.02	0.00
Mean	0.08	0.10	0.13	0.03	0.02	0.01	0.03	0.02

Appendix

Table 29: RAG realism ablation. Base = anchored doc at position 2 (published RAG layout); R1 = anchor at top of retrieval (rank 1); R5+D = anchor at bottom of a 5-doc list with 2 plausible distractor documents adjacent to the anchor; R5+D+Sc adds synthetic per-doc relevance scores. A drop from R1 to R5+D indicates models down-weight low-ranked retrieved anchors.

Model	Plausible (UAI)			Irrelevant (UAI)
Model	Base	Elicited	Noisy	Base	Elicited	Noisy
Gemma-4B	0.13	0.28	0.29	0.31	0.35	0.39
Llama-8B	0.62	0.54	0.58	0.45	0.40	0.45
OLMo-13B	0.06	0.06	0.15	0.06	0.02	$-$ 0.01
Qwen-7B	0.17	0.02	$-$ 0.01	$-$ 0.00	0.01	0.00
Mean	0.25	0.23	0.25	0.20	0.19	0.21

Appendix

Table 30: Tool realism ablation. Base reproduces the published Tool suite (externally injected tool output); Elicited prepends a synthetic model-planned tool-call turn (provenance signal); Noisy wraps the anchor value in a realistic JSON envelope containing additional metadata fields. Closing the gap between Base and Elicited indicates the anchor effect is not driven by the externally-injected framing; reductions under Noisy indicate the anchor’s salience competes with surrounding metadata.

Model	External			History			ICL			RAG			Tool
Model	Acc 10	UAI irr	UAI pls	Acc 10	UAI irr	UAI pls	Acc 10	UAI irr	UAI pls	Acc 10	UAI irr	UAI pls	Acc 10	UAI irr	UAI pls
Llama-3.3-70B	95%	$-$ 0.02	0.23	91%	$-$ 0.02	0.01	87%	0.01	0.01	95%	$-$ 0.02	0.04	98%	0.01	0.05
Qwen2.5-72B	91%	0.01	0.16	93%	0.01	0.12	96%	$-$ 0.01	$-$ 0.03	88%	0.00	0.04	99%	0.01	$-$ 0.00
GPT-5.4	100%	0.01	0.19	99%	0.03	0.02	100%	0.00	0.00	100%	0.00	0.09	100%	0.00	0.03
Claude-Sonnet-4.6	99%	$-$ 0.00	0.19	99%	0.02	$-$ 0.02	72%	$-$ 0.02	$-$ 0.01	95%	0.03	0.07	95%	$-$ 0.00	0.06
Grok-4.3	97%	0.05	0.25	99%	0.09	0.03	97%	0.04	0.05	96%	0.02	0.18	91%	0.10	0.25

Appendix

Table 31: Extended large-model panel. Task accuracy (Acc 10 , control-only) and absolute anchor influence UAI irr /UAI pls across all five suites for two 70B-class open-weight models (top) and three frontier API models (bottom). The qualitative pattern of the main panel persists at scale: irrelevant anchors are near-zero, plausible anchors are positive.

Model	Item	$a^{hi}$	$y_{c}$	$y_{p}^{hi}$	$y_{i}^{hi}$	$\Delta_{p}-\Delta_{i}$
Qwen-7B	EXT-legal_policy-e-off40-005	85	43	75	43	+32
Qwen-7B	EXT-resource_consumption-e-off15-002	50	75	44	75	+31
Qwen-7B	EXT-pricing_wtp-e-off40-006	83	44	74	44	+30
Llama-8B	EXT-market_demographics-h-off40-005	70	23	100	26	+74
Llama-8B	EXT-resource_consumption-h-off25-010	65	100	14	56	+42
Llama-8B	EXT-transportation_logistics-h-off40-002	97	38	100	45	+55

Appendix

Table 32: Per-item case studies (External): for each model we surface the top discriminative items where the plausible-anchor shift cleanly exceeds the irrelevant- anchor shift, illustrating that the aggregate UAI pattern is driven by interpretable per-item behaviour rather than averaging artifacts. Full scenarios and per-condition answers are tabulated in the supplementary material.

Suite	$\|d_{\text{pls}}\|$	$\|d_{\text{irr}}\|$	$\|d_{\text{hi-lo}}\|$
External	0.35	0.11	0.59
History	0.24	0.19	0.46
RAG	0.16	0.09	0.28
Tool	0.23	0.12	0.36

Appendix

Table 33: LLM anchoring magnitude in

|

Cohen’s

d|

(open-weight panel mean). Read against the conventional small / medium / large bands of

d=0.2/0.5/0.8

Language Model Bias Evaluation

Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.

See all 493 papers

Jun 11, 2026

Localizing Anchoring Pathways in Language Models

Irrelevant numbers in a prompt can shift language model judgments, producing anchoring effects in numerical reasoning. We study where this anchor-sensitive signal is carried inside language models using a controlled multiple-choice setup with shared answer options. We define a logit-difference metric comparing the correct answer option with the answer option corresponding to the anchor, and validate that it tracks behavioral anchoring. Using attribution-based circuit localization on 7B--8B Qwen and Llama base and instruction-tuned models, we find that edge-level methods recover this signal more faithfully than node-level methods. Low- and high-anchor circuits transfer strongly within a model, suggesting shared pathway structure across anchor direction. However, sparse transfer across base and instruction-tuned variants is less reliable, indicating that post-training changes which pathways matter most. Overall, our results provide a mechanistic account of how anchoring-related decision signals are carried inside language models.

Hillary N. Owusu, Sarah Wiegreffe, Naomi H. Feldman

University of Maryland, College Park

Numerical Reasoning in Language Models LLM Interpretability Circuits in Language Models

Ask PDF

Aug 26, 2026

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each judgment is often assumed to be independent of earlier evaluations. We test this assumption using three prompt conditions: no metadata, revision framing, and anchored metadata containing revision, attempt, and prior-score fields. We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values. Across 192,000 attempted evaluations (185,271 successful), seven out of the eight evaluated models have 95% task-stratified bootstrap intervals below zero for the total anchored-metadata effect on 20 fixed texts. Cohen's

d

, a standardized measure of the difference between score distributions, reaches an absolute value of 0.71. Token-level analysis of selected model-task probes suggests a threshold-like response pattern: introducing anchored metadata produces a marked redistribution of output-score probabilities, while changing the anchor value within the tested below-threshold range produces comparatively little additional variation. On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label, demonstrating the bias extends beyond numerical scoring to categorical decisions. Neither Chain-of-Thought nor a metadata-disregard warning reduces the total effect, although the warning improves the paired accuracy effect relative to baseline in the industry experiment. Reliable LLM evaluation demands careful context engineering rather than an assumption of impartiality. Effective mitigation must be validated for the intended model and task or domain.

Ante Kapetanovic, Kemal Altwlkany, Andro Mercep +2

Infobip Split, Croatia · Infobip Sarajevo, Bosnia and Herzegovina · Infobip Zagreb, Croatia

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Bias Evaluation

Ask PDF

May 13, 2026

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such as position, verbosity, and style preferences, but largely focuses on outcomes, leaving judge explanations underexplored. We instead ask whether LLM judges are cue-invariant, i.e., whether their rankings and explanations remain stable when non-evidential cues are perturbed while holding the underlying texts fixed. We introduce a suite of cue interventions (Blind, Truth, Flip, Placebo, Reveal-After) and tie-aware metrics that quantify outcome anchoring and rationale anchoring, including label-aligned rhetoric and explanation drift, alongside consistency and stereotype-intrusion checks. We design anchoring attacks using verbosity and confidence cues, and compare two mitigations: structured chain-of-thought prompting and PROOF-BEFORE-PREFERENCE (evidence lock, score, rank). Using a new dataset of 1,000 summaries from traditional extractive models and LLMs, we find substantial cue-anchored rationalization under label and placebo perturbations, while PROOF-BEFORE-PREFERENCE markedly improves cue invariance over baselines.

Riya Tapwal, Abhishek Kumar, Carsten Maple

Indian Institute of Technology (IIT) Mandi · The Alan Turing Institute, London, U.K. · Warwick Manufacturing Group University of Warwick, U.K.

LLM-as-a-Judge Faithfulness of Language Model Explanations Language Model Bias Evaluation Bias in LLM-as-a-Judge Causal Interventions in Language Models

Ask PDF

Abstract

Language Model Bias Evaluation

Explore similar work

Localizing Anchoring Pathways in Language Models

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges