cs.AINov 7, 2025

Anchors in the Machine: Behavioral and Attributional Evidence of Anchoring Bias in LLMs

Authors: Felipe Valencia-Clavijo

Organizations: Dataplicada

Abstract

Large language models (LLMs) are increasingly studied as behavioral subjects (e.g., synthetic participants or users) and decision systems (e.g., LLM-as-a-judge), yet it remains unclear how observed cognitive biases relate to changes in model output probabilities. Anchoring bias, a classic human judgment bias, provides a useful test case. Prior research largely relies on generated responses, leaving distributional effects and input-field contributions less explored. This paper advances the study of anchoring in LLMs through three contributions: (1) a log-probability-based behavioral analysis examining how numerical anchors shift distributions over candidate outputs, using alternative questions and anchor pairs alongside a positive control to reduce reliance on potentially memorized stimuli; (2) exact Banzhaf-value attribution over structured prompt fields to quantify anchor contributions to log-probabilities; and (3) an Anchoring Bias Sensitivity Score (ABSS) integrating behavioral and attributional evidence across six open-source models. Results reveal predominantly positive behavioral anchoring effects, varying substantially across models. Gemma-2B, Phi-2, and Llama-2-7B achieve the highest ABSS rankings, whereas other models exhibit weaker behavioral-attributional alignment despite sometimes showing substantial behavioral shifts. Attributional effects also vary across prompts, indicating that behavioral anchoring does not necessarily imply stronger anchor-field contributions. These findings raise questions about LLM reliability as synthetic participants and potential vulnerabilities to numerical cues in automated evaluation and decision-making. More broadly, the framework connects behavioral science with explainable AI, particularly interpretability, and AI safety, providing a reproducible approach to investigating cognitive-bias-like effects in LLMs.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Aug 14, 2026

AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in human judgment and decision-making, and recent work suggests that large language models (LLMs) exhibit similar behavior. However, existing work on anchoring in LLMs typically evaluates only a narrow set of anchor pathways and rarely distinguishes irrelevant from plausible anchors. We introduce AnchorBench, a benchmark for the anchoring effect in LLMs that evaluates multiple anchor pathways under an explicit anchor relevance axis. Across fourteen models, including ten open-weight models and four frontier API models, and a large set of controlled prompts, we find that (1) anchoring is strongly pathway-dependent, (2) plausible anchors usually induce larger shifts than irrelevant ones when introduced through stronger pathways, (3) anchor influence generally weakens as the anchor moves farther from the evidence-supported answer, most clearly on External and RAG, and (4) high task accuracy on the anchor-free control condition (Acc10_{10}: answers within 10 points of gold) does not guarantee robustness: even frontier API models above 95% control accuracy remain susceptible to plausible anchors.
Aug 26, 2026

Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence

Large language models (LLMs) increasingly assess generated content, giving rise to the LLM-as-a-Judge paradigm. These systems now score outputs, filter content, and gate iterative refinement in production pipelines, where each judgment is often assumed to be independent of earlier evaluations. We test this assumption using three prompt conditions: no metadata, revision framing, and anchored metadata containing revision, attempt, and prior-score fields. We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values. Across 192,000 attempted evaluations (185,271 successful), seven out of the eight evaluated models have 95% task-stratified bootstrap intervals below zero for the total anchored-metadata effect on 20 fixed texts. Cohen's dd, a standardized measure of the difference between score distributions, reaches an absolute value of 0.71. Token-level analysis of selected model-task probes suggests a threshold-like response pattern: introducing anchored metadata produces a marked redistribution of output-score probabilities, while changing the anchor value within the tested below-threshold range produces comparatively little additional variation. On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label, demonstrating the bias extends beyond numerical scoring to categorical decisions. Neither Chain-of-Thought nor a metadata-disregard warning reduces the total effect, although the warning improves the paired accuracy effect relative to baseline in the industry experiment. Reliable LLM evaluation demands careful context engineering rather than an assumption of impartiality. Effective mitigation must be validated for the intended model and task or domain.
Apr 1, 2026

CogBias: Measuring and Mitigating Cognitive Bias in Large Language Models

Large Language Models (LLMs) are increasingly deployed in high-stakes decision-making contexts. While prior work has shown that LLMs exhibit cognitive biases behaviorally, whether these biases correspond to identifiable internal representations and can be mitigated through targeted intervention remains an open question. We define LLM cognitive bias as systematic, reproducible deviations from correct answers in tasks with computable ground-truth baselines, and introduce LLM CogBias, a benchmark organized around four families of cognitive biases: Judgment, Information Processing, Social, and Response. We evaluate three LLMs and find that cognitive biases emerge systematically across all four families, with magnitudes and debiasing responses that are strongly family-dependent: prompt-level debiasing substantially reduces Response biases but backfires for Judgment biases. Using linear probes under a contrastive design, we show that these biases are encoded as linearly separable directions in model activation space. Finally, we apply activation steering to modulate biased behavior, achieving 26--32% reduction in bias score (fraction of biased responses) while preserving downstream capability on 25 benchmarks (Llama: negligible degradation; Qwen: up to −-19.0pp for Judgment biases). Despite near-orthogonal bias representations across models (mean cosine similarity 0.01), steering reduces bias at similar rates across architectures (r(246)r(246)=.621, pp<.001), suggesting shared functional organization.