Cognitive Bias Signatures in Search-Equipped LLM Agents: A Stage-Resolved Analysis

Yanru Xiao; Yingwei Yu

cs.CLNov 3, 2026

Cognitive Bias Signatures in Search-Equipped LLM Agents: A Stage-Resolved Analysis

Authors: Yanru Xiao, Yingwei Yu

Organizations: Amazon Web Services

Abstract

Search can help LLM agents correct mistakes, but their starting conditions may shape which evidence they find and use. Drawing on cognitive-bias paradigms, we develop a controlled search harness to examine result order, records of prior work, and initial hypotheses while holding each question and document collection fixed. Matched controls distinguish order effects from differences in document composition and the effects of failed work from context length. Across 10 models and 1,000 synthetic search worlds, we track queries, retrieved documents, document openings, and final answers. Our comparisons suggest that final answers are more sensitive to initial evidence order during active search than when all evidence is supplied in full. Dead-end histories can reduce strategy switching relative to matched neutral histories, indicating an effect beyond context length. Under wrong hypotheses, stage measurements distinguish reduced retrieval or opening from errors that persist after counter-evidence is opened. Exploratory interventions show that opening counter-evidence more often can coincide with lower accuracy, while accuracy can improve with little change in opening.

Figures & tables

Figure 1: Illustrative paired paths for three history manipulations. (a) Anchoring swaps

R/W

order on the same first page. (b) Sunk cost compares dead-end and matched neutral prefills; switching is detected in returned search results. (c) Confirmation compares no seed with a wrong seed, here showing different answers after both agents open counter-evidence. Each paradigm uses its own corpus and endpoint.

Figure 2: History effects across models. (a) Wrong-candidate adoption increase (

W

-first minus

R

-first), with paired 95% CIs. Active arms reverse first-page

W/R

order; passive arms provide full evidence with independently arranged remainders. † : attrition-sensitive. (b) Raw switch rates by prefilled dead-end steps. Open diamonds: neutral at

k=10

(offset; eight at 100%). Rates use condition-specific usable trials, unlike Table 1 ’s common triads; Fable was unrun. (c) No-seed minus wrong-seed accuracy and surfacing drops (pp), with paired 95% CIs; Fable is excluded for attrition. Table 3 adds opening and post-open diagnostics.

	Anchoring			Sunk cost			Confirmation bias
Model	Adoption $\Delta^{\mathrm{anchor}}$	Passive (pp)	no_choice (fraction)	Switch drop $\Delta^{\mathrm{sunk\text{-}ctrl}}$	Load $p_{0}-p_{N}$	Provenance $\Delta^{\mathrm{prov}}$	Accuracy loss $\Delta^{\mathrm{confirm}}$	Surfacing drop $\Delta^{\mathrm{surf}}$
Haiku 4.5	$\boldsymbol{+21.3}$	$+0.5$	0.64	$\boldsymbol{+6.7}$	$+0.0$	$-1.5$	$\boldsymbol{+15.2}$	$+0.0$
Sonnet 4.6	$\boldsymbol{+16.5}$	$+0.0$	0.67	$\boldsymbol{+7.5}$	$+0.0$	$-0.5$	$-1.0$	$+0.0$
Sonnet 5	$\boldsymbol{+15.0}$	$+0.0$	0.14	$\boldsymbol{+8.0}$	$+0.0$	$+0.0$	$-2.0$	$-1.0$
Opus 5	$\boldsymbol{+26.0}$	$+0.0$	0.03	$\boldsymbol{+15.0}$	$+0.0$	$+0.0$	$-2.0$	$-2.0$
Fable 5	$+9.6$ $\dagger$	$+0.0$	0.27	—	—	$+1.0$	—	—

Table 1: Paired history effects (pp;

n=135

–200). The first column per paradigm is primary: positive values mean more wrong adoption or less switching/accuracy. no_choice pools valid outcomes from both active arms (fraction). Bold marks detected effects (§ 3 ). † : attrition-sensitive. Fable’s sunk-cost dashes denote unrun comparisons; confirmation is excluded for attrition. Passive uses a full-evidence task; provenance is other

-

own switching. Appendix C reports CIs and counts.

Model	$R$ opened	$R$ unopened	Pattern
	Wrong-candidate adoption
Haiku 4.5	0.12 (175)	1.00 (24)	no-open
Sonnet 4.6	0.16 (190)	1.00 (10) $\ddagger$	no-open
Sonnet 5	0.34 (165)	0.51 (35)	post-open
Opus 5	0.00 (58)	0.43 (142)	no-open
Fable 5 †	0.34 (106)	0.17 (54)	excluded

Table 2: Wrong-candidate adoption can persist after opening right-supporting evidence. Adoption-proxy fractions (conditional

n

) use all usable

W

-first trials, split by

R

opening. Descriptive pattern rules appear in Appendix B . † : Fable is attrition-sensitive and excluded from patterns. ‡ :

n<20

Model	Accuracy $\Delta^{\mathrm{confirm}}$	Surfaced $\Delta^{\mathrm{surf}}$	Opened $\Delta^{\mathrm{open}}$	Follow seed $\mid$ opened fraction ( $n$ )	no_choice shift (pp)	Observed category
Haiku 4.5	$\boldsymbol{+15}$ $\boldsymbol{.2}$	$+0$ $.0$	$+1$ $.0$	0.18 (148)	$+4$ $.6$	post-open
Sonnet 4.6	$-1$ $.0$	$+0$ $.0$	$-1$ $.5$	0.00 (150)	$-0$ $.5$	no detected effect
Sonnet 5	$-2$ $.0$	$-1$ $.0$	$-1$ $.5$	0.01 (156)	$-0$ $.5$	no detected effect
Opus 5	$-2$ $.0$	$-2$ $.0$	$-0$ $.5$	0.00 (154)	$+0$ $.0$	no detected effect
GPT-5.6 Luna	$\boldsymbol{+43}$ $\boldsymbol{.5}$	$\boldsymbol{+55}$ $\boldsymbol{.5}$	$\boldsymbol{+55}$ $\boldsymbol{.0}$	0.12 (41)	$+0$ $.0$	retrieval selection
GPT-5.6 Sol	$\boldsymbol{+42}$ $\boldsymbol{.0}$	$\boldsymbol{+41}$ $\boldsymbol{.0}$	$\boldsymbol{+42}$ $\boldsymbol{.5}$	0.00 (66)	$-0$ $.5$	retrieval selection

Table 3: Confirmation-bias stage diagnostics. Drops (pp) are no-seed minus wrong-seed; no_choice reverses the difference. Paired

n=200

, except Haiku (197) and GLM-5 (198). Follow seed is the adoption proxy among wrong-seed counter-evidence openers (fraction; conditional

n

). Categories first require detected accuracy loss, then the earliest detected drop

\geq 20

pp in surfacing or opening; post-open is residual and permits smaller upstream changes. Bold marks detections in the three

\Delta

columns. Fable is excluded for attrition.

Figure 3: Opening and accuracy can change in opposite directions under intervention. Points show marginal changes from untreated wrong-seed trials (pp), with paired 95% CIs. Categories follow Table 3 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$\Delta^{\mathrm{anchor}}$	passive
Haiku 4.5	+21.3 ∗ [+15.1, +27.8]	+0.5 [-1.4, +2.8]
Sonnet 4.6	+16.5 ∗ [+11.0, +22.5]	+0.0 [-1.9, +1.9]
Sonnet 5	+15.0 ∗ [+7.2, +22.6]	+0.0 [-1.9, +1.9]
Opus 5	+26.0 ∗ [+19.8, +32.5]	+0.0 [-1.9, +1.9]
Fable 5	+9.6 † [+2.7, +16.7]	+0.0 [-1.9, +1.9]
GPT-5.6 Luna	+33.0 ∗ [+26.2, +39.7]	-1.0 [-3.6, +1.0]

Appendix

Table 4: Full canonical effects (pp; paired 95% CIs). ∗ : detected under the CI and Holm-adjusted McNemar rule in Appendix B.2 . † : Fable complete-case estimates are attrition-sensitive; its confirmation cells are excluded (Appendix D.4 ). Anchoring pairs: 200 per model except Haiku 197, GLM-5 198, and Fable 135.

Figure 4: Anchoring order effects by number of wrong-supporting documents. Left: paired order effects as wrong-supporting documents vary (

n_{W}=1,2,3

; three right-supporting documents throughout). Right: change from

n_{W}=1

3

, with bootstrap 95% CIs from independent resampling of paired world differences within each dose; ∗ marks CIs excluding zero. † : Fable complete-case estimates are attrition-sensitive.

Model	$W$ -first (%)	$R$ -first (%)	Shift (pp) [95% CI]
Haiku 4.5	62.4	65.0	-2.5 [-10.3,+5.3]
Sonnet 4.6	68.5	65.0	+3.5 [-4.8,+11.7]
Sonnet 5	11.0	16.0	-5.0 [-11.1,+1.0]
Opus 5	4.5	0.5	+4.0 [+0.9,+7.9]
Fable 5	26.7	29.6	-3.0 [-12.4,+6.5]
GPT-5.6 Luna	38.5	43.0	-4.5 [-13.3,+4.4]

Appendix

Table 5: Active anchoring: paired no_choice rates by page order and their

W

-first minus

R

-first shift (pp; 95% CI). Both arm rates use the same paired worlds. These outcomes count as non-adoption in the primary endpoint.

Model	$n$	Reached (%)	Switch $\mid$ reached % ( $n$ )	Switch (%)
Haiku 4.5	196	38.8	48.7 (76)	18.9
Sonnet 4.6	200	21.5	79.1 (43)	17.0
Sonnet 5	200	62.5	77.6 (125)	48.5
Opus 5	200	99.5	85.4 (199)	85.0
Fable 5	188	49.5	80.6 (93)	39.9
GPT-5.6 Luna	200	58.5	81.2 (117)	47.5

Appendix

Table 6: Exploratory self-enacted sunk cost: the first five searches favor the dead-end family, within a shared 12-call search/read budget. Reached means reaching an unrigged search; switching counts only searches after the fifth. Switch

\mid

reached uses the displayed conditional

n

, while Switch uses all trials.

Model	Seed	$n$	Surfaced	Opened	no_choice	Opened $\mid$ surfaced ( $n$ )	Adoption $\mid$ opened ( $n$ )
Haiku 4.5	None	198	0.753	0.753	0.005	1.00 (149)	0.05 (149)
	Wrong	199	0.754	0.744	0.050	0.99 (150)	0.18 (148)
Sonnet 4.6	None	200	0.755	0.735	0.005	0.97 (151)	0.00 (147)
	Wrong	200	0.755	0.750	0.000	0.99 (151)	0.00 (150)
Sonnet 5	None	200	0.795	0.765	0.005	0.96 (159)	0.01 (153)
	Wrong	200	0.805	0.780	0.000	0.97 (161)	0.01 (156)

Appendix

Table 7: Underlying confirmation-bias rates by seed condition. The first three rates share the arm’s

n

; conditional rates give their own denominators in parentheses. Surfacing and opening refer to counter-evidence; adoption is the wrong-candidate proxy

F^{X}

. Conditional rates describe post-treatment subsets; Table 3 reports paired marginal effects.

Model	Instruction	$\Gamma^{\mathrm{surfaced}}$	$\Gamma^{\mathrm{opened}}$	$\Gamma^{\mathrm{correct}}$	$n$
GPT-5.6 Luna	read-both-sides	+20.5 [+13.8, +27.0]	+20.0 [+13.3, +26.5]	+16.5 [+9.2, +23.6]	200
	seek-disconfirming	+35.0 [+27.3, +42.0]	+34.0 [+26.4, +41.0]	+30.0 [+22.4, +37.1]	200
GPT-5.6 Sol	read-both-sides	+21.0 [+14.4, +27.3]	+21.0 [+14.4, +27.3]	+21.0 [+14.4, +27.3]	200
	seek-disconfirming	+39.5 [+32.2, +46.0]	+39.5 [+32.3, +45.9]	+40.0 [+32.8, +46.4]	200
Qwen3-Next	read-both-sides	-8.0 [-13.4, -2.5]	+4.0 [-1.4, +9.4]	+1.5 [-3.9, +6.9]	200
	seek-disconfirming	+9.5 [+5.3, +13.8]	+22.5 [+16.2, +28.5]	+23.0 [+16.5, +29.2]	200

Appendix

Table 8: Exploratory instruction effects versus the untreated wrong-seed arm (pp; paired 95% CIs). Each row’s pair count applies to all three outcomes. Figure 3 plots opening and accuracy.

Model	pro- $X$ share	pro- $Y$ share	Seed echo
Haiku 4.5	0.30 $\to$ 0.31	0.34 $\to$ 0.33	0.00 $\to$ 0.02
Sonnet 4.6	0.32 $\to$ 0.36	0.33 $\to$ 0.29	0.00 $\to$ 0.06
Sonnet 5	0.35 $\to$ 0.41	0.43 $\to$ 0.39	0.10 $\to$ 0.14
Opus 5	0.39 $\to$ 0.46	0.36 $\to$ 0.32	0.16 $\to$ 0.29
GPT-5.6 Luna	0.40 $\to$ 0.84	0.41 $\to$ 0.09	0.14 $\to$ 0.79
GPT-5.6 Sol	0.38 $\to$ 0.77	0.44 $\to$ 0.16	0.15 $\to$ 0.68

Appendix

Table 9: Query direction, control

\to

wrong seed. Shares average within trials with at least one search. Pro-

X

/pro-

Y

means more returned documents support that stance than its opponent (

X

: seeded;

Y

: counter-evidence). Seed echo is the fraction of queries containing the seeded candidate’s name.

Anchoring
Model	Missing	$n$	Bounds (pp)
Haiku 4.5	1/2	197	$[+20.0,+21.5]$
GLM-5	0/2	198	$[+6.5,+7.5]$
Fable 5	40/34	135	$[-8.5,+28.5]$
Dead-end-specific
Haiku 4.5	4/2/1	194	$[+6.0,+7.5]$

Appendix

Table 10: Missing primary outcomes and sharp completed-cohort point bounds (200 worlds per arm;

n

: original complete pairs, or common triads for sunk cost). Arm order:

W

R

-first; no-history/neutral/dead-end; control/wrong-seed. Missing outcomes are empty responses for Fable and errors for all other rows. Bounds are not CIs; Fable’s confirmation contrast remains excluded.

Model	v1 per-turn forced	v2 per-call forced	v2 identical-set subset	v3 permutation (canonical)
Haiku 4.5	+31.0 ∗ [+24.7, +37.7]	+32.0 ∗ [+24.9, +39.0]	+36.0 ∗ [+24.5, +47.3]	+21.3 ∗ [+15.1, +27.8]
Sonnet 4.6	+29.5 ∗ [+23.3, +36.2]	+27.5 ∗ [+21.2, +34.1]	+18.7 ∗ [+10.0, +28.9]	+16.5 ∗ [+11.0, +22.5]
Sonnet 5	+27.5 ∗ [+21.0, +34.2]	+29.5 ∗ [+23.2, +36.2]	+13.3 ∗ [+5.4, +22.9]	+15.0 ∗ [+7.2, +22.6]
Opus 5	+7.0 ∗ [+2.2, +12.2]	+36.0 ∗ [+29.1, +42.9]	+16.0 ∗ [+6.5, +26.1]	+26.0 ∗ [+19.8, +32.5]
Fable 5	+16.8 ∗ [+10.6, +23.7]	+20.6 ∗ [+14.1, +27.7]	+8.8 [-1.3, +19.8]	+9.6 † [+2.7, +16.7]
GPT-5.6 Luna	+24.5 ∗ [+18.7, +30.9]	+25.5 ∗ [+19.6, +32.0]	+9.3 ∗ [+2.5, +18.0]	+33.0 ∗ [+26.2, +39.7]

Appendix

Table 11: Anchoring under alternative interfaces (pp; paired 95% CIs). Versions and the identical-set subset are defined in Appendix E . ∗ : detected under the CI and Holm-adjusted McNemar rule. † : Fable’s canonical complete-case estimate is attrition-sensitive.

Anchoring Bias in Language Models

See all 8 papers

SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs
Sep 9, 2026Jiacheng Sang, Mengyuan Li, Sanxing Chen +3Agentic Search LLM Interpretability
Duke University · University of Pennsylvania
When Is Enough Not Enough? Illusory Completion in Search Agents
Feb 7, 2026Dayoon Ko, Jihyuk Kim, Sohyeon Kim +5AI Agent Reliability LLM Answer Verification
Seoul National University · LG AI Research · University of Illinois Chicago +1
Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory
Oct 30, 2026Nipun D. Pathirage, Kwan SrijomkwanAgentic Search LLM Agents
Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA · Department of Cognitive Science, Rensselaer Polytechnic Institute, Troy, NY, USA

Sep 9, 2026J/K move · Enter open · S save

SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs

Jiacheng Sang, Mengyuan Li, Sanxing Chen, Yukun Huang+2

Duke University · University of Pennsylvania

Agentic Search LLM Interpretability LLM Agent Evaluation

Open paper AskPDF arXiv

LLM search agents are often evaluated on final-answer accuracy, overlooking the process. Analyzing a search strategy requires understanding how credible evidence is retrieved to address question constraints. This valuable information is buried in raw search trajectories that are long and difficult to parse. We introduce SearchAtlas, a framework that converts search trajectories into structured graphs whose edges represent how evidence is propagated across the reasoning trace, from the query that retrieves it to the final answer. Our automated parsing pipeline achieves a mean edge F1 of 86.0% against human-annotated graphs and remains consistent across repeated runs. We analyze five search agents on three benchmarks, revealing systematic differences in search scale and evidence aggregation. SearchAtlas exposes fragmented answer support, question constraints that do not reach the answer, and unverified parametric knowledge entering the response. These process failures are strongly associated with incorrect answers, even more so than an LLM judge given either the raw trajectory or the ordered query list, suggesting that the constructed graphs provide useful interpretability. Moreover, an audit of cases in which process-diagnostic scores disagree with final-answer correctness shows that they capture information not reducible to answer accuracy.

Abstract

Anchoring Bias in Language Models

Explore similar work

SearchAtlas: Analyzing Agentic Search Strategies via Evidential Query Graphs

When Is Enough Not Enough? Illusory Completion in Search Agents

Confident Fools or Careful Searchers? A Behavioural Study of When LLMs Search Beyond Their Static Internal Memory