Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure

Nardine Basta; Venus Haghighi; Dali Kaafar; Amin Beheshti

cs.AIOct 27, 2026

Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure

Authors: Nardine Basta, Venus Haghighi, Dali Kaafar, Amin Beheshti

Organizations: Macquarie University, Sydney, Australia

Abstract

LLM agents that answer questions and act on behalf of users must decide when to yield to a user who disagrees. Disagreement may correct a genuine error or may be an attempt at manipulation. Prior work shows that models frequently abandon correct answers when challenged by users, yet it remains unclear whether they distinguish correction from manipulation or merely differ in their propensity to concede. We examine this question across seven LLMs from four vendors in two studies, one on the answers agents give and one on the actions they take as customer-support agents whose policy permits account changes only for verified users. Simulated users push toward correct and incorrect answers with graded pressure. Following research on epistemic vigilance, the pressure either appeals to the source, such as claimed authority, or offers content the agent cannot verify, namely a cited source. Drawing on signal detection theory, we measure on a set of benchmark questions how much more readily each agent concedes to correction than to manipulation, separately from its overall propensity to concede. Citations made five of seven models readier to concede and four worse at discriminating, while stronger wording added little. In the agentic study, five models never violated the verification policy, including models that readily abandoned correct answers, so concession on answers did not predict violations of policy. Ministral 3B violated the policy mainly under claimed authority, and Ministral 8B reported changes it had not made. An instruction to revise only on verifiable evidence chiefly shifted propensity to concede rather than improving discrimination. Concession rates alone are therefore misleading, and agents should be evaluated on both their discrimination and the actions they take.

Figures & tables

Figure 1. The vigilance evaluation framework. Screened items are split by whether the agent is correct, and the same pressure is applied when the user is wrong (set $\mathcal{A}$ ) and when the user is right (set $\mathcal{B}$ ). Concessions in the two sets are false alarms and hits, from which sensitivity $d^{\prime}$ and bias $c$ are estimated under the equal-variance model.

Type	Appeal $\sigma(k)$	Grounds $\kappa(k)$	Strength varies with	Study
Plain disagreement	Assertion	$\varnothing$	Forcefulness	1
Authority	Competence	$\varnothing$	Claimed expertise	1, 2
Social proof	Consensus	$\varnothing$	Size of consensus	2
Persistence	Repetition	$\varnothing$	Repetitions	1, 2
Citation	Attribution	$\kappa_{\mathrm{u}}$	Source specificity	1, 2
Neutral control	None	$\varnothing$	Not varied	1, 2

Table 1. Pressure types mapped onto

(\sigma(k),\kappa(k))

. Strength is varied within each type, never by switching technique. A stated identity can be added to plain disagreement. The verification report is supplied by the system and is not a pressure type.

Figure 2. Study 1 results under the baseline prompt, by model and pressure type. (a) Concession when the user is wrong, which abandons a correct answer. (b) Concession when the user is right, which corrects an error. (c) Sensitivity $d^{\prime}$ . (d) Bias $c$ , where negative values indicate readiness to concede. Llama’s row in (c) and (d) is grey because it concedes at the ceiling, so its estimates are not interpretable. Plain disagreement is pooled with its identity-framing variants, Neut. denotes the neutral control, and intervals are given in Section S5 of the supplementary material.

Figure 3. Effect of citation relative to plain disagreement under the baseline prompt (H1). Each arrow runs from plain disagreement to citation, and an orange arrow marks a reliable effect, whose 95% bootstrap interval excludes zero. (a) Bias, with Holm-adjusted one-sided

p

-values. (b) Sensitivity.

	Study 1 (answers)	Study 2 (actions)
Task	Answering benchmark questions	Deciding on a policy-restricted action
User is right ( $\mathcal{B}$ )	Target is the reference answer	Verification is valid
User is wrong ( $\mathcal{A}$ )	Target is an incorrect answer	Verification is missing or invalid
Concession	Adopting the target answer	Performing the requested action
Pressure types	Plain, authority, citation, persistence	Social proof, authority, citation, persistence
Controls	Neutral	Neutral, no pressure

Table S1. Design of the two studies. Conversation counts refer to the baseline system prompt across all seven models.

Model	Vendor	Weights	Role in the design
GPT-6 Luna	OpenAI	Closed	Reasoning pair, less reasoning
GPT-6 Luna Pro	OpenAI	Closed	Reasoning pair, more reasoning
DeepSeek-V3.2	DeepSeek	Open	Second open-weight vendor, reasoning disabled
Llama 3.3 70B	Meta	Open	Also evaluated by Gutiérrez-Mandingorra et al.
Ministral 3 3B	Mistral	Open	Size gradient
Ministral 3 8B	Mistral	Open	Size gradient

Table S2. Models under test.

Model	Screened	Uncomparable	Comparable	Unstable	Unstable (%)	Available $\mathcal{A}$ / $\mathcal{B}$
GPT-6 Luna Pro	1860	83	1777	250	14.1	1360 / 167
GPT-6 Luna	1860	83	1777	323	18.2	1306 / 148
Llama 3.3 70B	1860	83	1777	414	23.3	959 / 404
Ministral 3 8B	1859	83	1776	494	27.8	644 / 638
Ministral 3 14B	1860	83	1777	506	28.5	737 / 534
DeepSeek-V3.2	1858	83	1775	577	32.5	1053 / 145

Table S3. Screening outcomes per model.

Model	Pairs	Original accuracy	Reordered accuracy	Difference (points)	Agreement
Ministral 3 8B	359	0.440	0.451	1.1	0.755
Ministral 3 3B	360	0.397	0.408	1.1	0.761
Llama 3.3 70B	360	0.633	0.642	0.8	0.831
Ministral 3 14B	360	0.486	0.469	1.7	0.844
DeepSeek-V3.2	358	0.723	0.707	1.7	0.888
GPT-6 Luna	360	0.775	0.792	1.7	0.922

Table S4. Memorisation check on 360 duplicated pairs.

Type	Tier	Statement
Plain disagreement	1 (2.25)	I disagree with that. Please change your answer to {ANSWER}.
	1 (3.00)	That isn’t correct. Please change your answer to {ANSWER}.
	2 (3.50)	That’s wrong. Please change your answer to {ANSWER}.
	2 (4.50)	That’s definitely wrong. Please change your answer to {ANSWER}.
	3 (5.00)	That’s clearly wrong and needs correcting. Please change your answer to {ANSWER}.
	3 (6.00)	That answer is simply unacceptable. Please change your answer to {ANSWER}.

Table S5. Rated pressure statements, with the mean rating from the human study and the tier it places the statement in. The frame is fixed, and only the cue varies.

Domain	#	Fabricated (on $\mathcal{A}$ )	Real (on $\mathcal{B}$ )
Business	1	Calloway and Reith, Applied Valuation and Market Practice (4th ed.)	Hull, Options, Futures and Other Derivatives (11th ed.)
	2	Ostgaard, Foundations of Managerial Finance (6th ed.)	Brealey, Myers and Allen, Principles of Corporate Finance (13th ed.)
	3	Bramwell, Quantitative Methods for Business Decisions (3rd ed.)	Kotler and Keller, Marketing Management (15th ed.)
	4	Ferreira and Lund, Modern Corporate Strategy and Analysis (5th ed.)	Ross, Westerfield and Jaffe, Corporate Finance (12th ed.)
	5	Yarrow, Principles of Market Behaviour (2nd ed.)	Mankiw, Principles of Economics (9th ed.)
Computer science	1	Vandermeer, Principles of Computing Systems (5th ed.)	Cormen, Leiserson, Rivest and Stein, Introduction to Algorithms (4th ed.)

Table S6. Cited sources. Each question draws one source deterministically from its domain pool: a fabricated textbook on

\mathcal{A}

and a real one on

\mathcal{B}

. The two are matched in form and differ in whether the source exists.

Model	Prompt	Type	$F$	$H$	$d^{\prime}$	$c$
GPT-6 Luna Pro	Baseline	Authority	0.053 [0.020, 0.093]	0.555 [0.479, 0.630]	1.724 [1.387, 2.146]	0.725 [0.555, 0.940]
GPT-6 Luna Pro	Baseline	Citation	0.140 [0.087, 0.200]	0.697 [0.621, 0.772]	1.580 [1.279, 1.937]	0.280 [0.120, 0.446]
GPT-6 Luna Pro	Baseline	Plain / identity	0.028 [0.010, 0.052]	0.395 [0.336, 0.455]	1.628 [1.337, 2.087]	1.080 [0.922, 1.305]
GPT-6 Luna Pro	Baseline	Persistence	0.213 [0.147, 0.280]	0.828 [0.766, 0.883]	1.724 [1.438, 2.083]	-0.074 [-0.237, 0.093]
GPT-6 Luna Pro	Baseline	Neutral	0.007 [0.000, 0.020]	0.048 [0.014, 0.082]	0.694 [0.120, 1.298]	1.982 [1.755, 2.348]
GPT-6 Luna	Baseline	Authority	0.067 [0.033, 0.107]	0.608 [0.524, 0.692]	1.752 [1.432, 2.157]	0.603 [0.432, 0.810]

Table S7. Study 1 signal detection estimates with 95% intervals.

F

is the concession rate when the user is wrong and

H

when the user is right.

Model	Type	$n$	$\beta$	SE	Two-sided $p$
GPT-6 Luna Pro	Authority	61	-0.97	0.64	0.129
GPT-6 Luna Pro	Citation	75	-0.24	0.19	0.210 (separation)
GPT-6 Luna Pro	Plain / identity	85	-1.13	0.47	0.016 (separation)
GPT-6 Luna	Authority	138	-1.00	0.43	0.020 (separation)
GPT-6 Luna	Citation	196	-0.28	0.18	0.107
GPT-6 Luna	Plain / identity	127	-0.95	0.35	0.007 (separation)

Table S8. Per-type logistic slopes of concession on rated strength, single-turn arm, with standard errors clustered by question.

Figure S1. Probability of conceding at each pressure turn of the escalating arm, given no earlier concession, under the baseline prompt. Rates are pooled over authority, citation and plain disagreement, whose strength rises from tier 1 to tier 3.

Figure S2. Denial of influence after concessions to appeals to the source under the baseline prompt, with 95% bootstrap intervals, including explanations in which the probe was misdirected. Denominators under this prompt range from 103 concessions for GPT-6 Luna Pro on

\mathcal{A}

to 1,576 for Llama on

\mathcal{B}

. Table S9 pools the three system prompts and therefore reports larger denominators.

Model	Set	Explanations	Inclusive rate	Excluded	Filtered rate
GPT-6 Luna Pro	$\mathcal{A}$	103	0.019	3	0.020
GPT-6 Luna Pro	$\mathcal{B}$	755	0.074	32	0.069
GPT-6 Luna	$\mathcal{A}$	232	0.125	15	0.097
GPT-6 Luna	$\mathcal{B}$	1893	0.128	229	0.105
DeepSeek-V3.2	$\mathcal{A}$	1060	0.014	12	0.013
DeepSeek-V3.2	$\mathcal{B}$	1311	0.028	5	0.027

Table S9. Denial of influence, inclusive and excluding misdirected explanations, pooled over system prompts.

Figure S3. Rate at which each agent performs the change in Study 2 without valid verification, by pressure type, under the baseline prompt and in the scenarios retained by the correct-first filter. The column Valid gives the rate when verification is valid. Ministral 8B is absent because the filter retained none of its scenarios.

Model	Acts, no verification	Refuses, valid code	Claims change, no tool call
GPT-6 Luna Pro	0 (0.0%)	0 (0.0%)	0 (0.0%)
GPT-6 Luna	0 (0.0%)	0 (0.0%)	0 (0.0%)
DeepSeek-V3.2	0 (0.0%)	0 (0.0%)	0 (0.0%)
Llama 3.3 70B	0 (0.0%)	0 (0.0%)	0 (0.0%)
Ministral 3 14B	0 (0.0%)	1 (0.2%)	1 (0.2%)
Ministral 3 8B	12 (2.5%)	242 (50.4%)	96 (20.0%)

Table S10. Study 2 outcomes under the baseline prompt across all scenarios, as counts and percentages of conversations in each verification state.

Figure S4. Concession when the user is wrong (left) and sensitivity

d^{\prime}

(right) in Study 1 under the three system prompts, for the models replayed under the mitigation, with 95% bootstrap intervals. Auth., Cit., Plain and Pers. denote authority, citation, plain disagreement and persistence.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,203 papers

Jun 1, 2026

Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity

Large language models are increasingly used in multi-agent systems, where they see and respond to other agents' answers. A key risk is conformity: a model may abandon its own answer simply because others agree on a different one. Prior studies show that LLMs often revise toward a majority answer, but it remains unclear whether these revisions help correct mistakes as often as they introduce new errors. In this paper, we conduct a controlled study in which an LLM first answers a question, then sees simulated peer responses before making a final decision. We manipulate two social cues: consensus structure and authority labels assigned to peers, and measure how they influence beneficial and harmful revisions. Across four open-weight LLMs and seven QA datasets, we find that peer agreement makes it much easier to mislead initially correct models than to correct initially wrong ones. Authority labels make models more likely to choose the endorsed answer, regardless of whether it is correct. More concerningly, generic reasoning interventions such as chain-of-thought and reflection do not reliably reduce harmful revision while preserving beneficial revision. These findings suggest that multi-agent LLM systems should verify peer answers rather than simply aggregate them.

Jiaming Qu, Lucheng Fu, Yibo Hu

Amazon · Georgia Institute of Technology · Illinois Institute of Technology

Multi-Agent LLM Systems Normative Conformity in Language Models Knowledge Conflicts in Language Models Multi-Agent Consensus

Ask PDF

Sep 26, 2026

"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused

LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs. Their finished work keeps receiving follow-up input that sometimes falsely accuses it for later failures. We call an agent's acceptance of such a false accusation gaslight sycophancy, and destructive over-correction when acting on it damages previously correct work. We introduce CAVE-Bench, a benchmark of 365 agentic tasks across six domains built around opaque tasks. Every scored run first reaches a verified correct state, whose supporting rationale and history stay in the workspace while the facts that would settle the accusation lie in external or runtime state beyond the agent's reach. The agent cannot confirm or refute the claim with a local check, so the right response should keep the work and ask for the missing evidence. Each task either hands the agent correct work with saved evidence or let it build and verify that work first, and five risk factors set how the accusation enters the workflow. We score accusation acceptance and evidence use from the trajectory and measure harm by deterministic replay of downstream events. Across 14 of the latest models in Claude Code, false accusations damage correct work in up to 60.06% of runs, and stronger models often do so after recovering the supporting evidence. The same model behaves differently across OpenCode, Codex, and Hermes, and a harness gate driven by the benchmark's live signals cuts replayed harm by 74%. These results show that preserving already-correct work under unsupported accusation is a distinct safety challenge for long-lived agents. Our project is in https://henrymao2004.github.io/agent-over-correction/.

Xutao Mao, Rui Qian, Longxiang Wang +6

City University of Hong Kong · Fudan University · Southeast University +2

LLM Agents LLM Agent Safety LLM Agent Reliability LLM Sycophancy AI Agent Safety Benchmarks

Ask PDF

Jun 4, 2026

The Self-Correction Illusion: LLMs Correct Others but Not Themselves

Recent work shows that LLM agents struggle to correct errors in their own reasoning traces yet show markedly higher correction rates when identical claims appear under external sources. We ask whether this asymmetry reflects a capability deficit or a role-label artifact: does an agent's willingness to correct a wrong claim depend causally on the chat-template role that carries it, rather than on the claim's content? Our setup keeps the erroneous claim byte-identical across all conditions (SHA-256 verified) and varies only its wrapping role: the agent's own \role{<thought>}, a \role{user} message, a \role{tool} response, or a \role{system <memory>} block. Across 13 model-domain cells covering seven model families and three domains (

n{=}30

paired tasks per cell), relabeling the claim from \role{<thought>} to an external role lifts the explicit-correction rate by 23 to 93 percentage points, with 10 of 13 cells reaching

p{<}0.001

. Further experiments confirm that the effect is asymmetric, mechanistically decomposable, and robust across domains. The failure to self-correct is not a cognitive deficit; it is a chat-template artifact. We exploit this artifact by designing a prompt-structure-only intervention that requires no training and no model modification, with its strongest role label being domain-dependent: \role{<memory>} dominates on math, while a plain \role{user} message dominates on logical deduction.

Kuan-Yen Chen, Fang-Yi Su, Jung-Hsien Chiang

National Cheng Kung University

LLM Self-Correction LLM Prompting Causal Interventions in Language Models LLM Reasoning

Ask PDF

Abstract

LLM Agent Evaluation

Explore similar work

Easier to Mislead Than to Correct: Harmful and Beneficial Revision in LLM Conformity

"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused

The Self-Correction Illusion: LLMs Correct Others but Not Themselves