Under the Influence: Evaluating and Improving LLM Vigilance in Agentic Tasks

Sasha Robinson; Katherine M. Collins; Ilia Sucholutsky; Kelsey R. Allen

cs.CLFeb 24, 2026

Under the Influence: Evaluating and Improving LLM Vigilance in Agentic Tasks

Authors: Sasha Robinson, Katherine M. Collins, Ilia Sucholutsky, Kelsey R. Allen

Organizations: McMaster University · Vector Institute · Princeton AI Lab · Massachusetts Institute of Technology ·

Abstract

Large Language Models (LLMs) are becoming increasingly autonomous assistants, capable of accomplishing series of complex tasks by interacting with open world domains such as software databases and the internet. This presents great opportunity but also great risk: open domains routinely present opportunities for autonomous agents to be influenced by others, e.g. deciding whether to accept pull requests or whether to use advice on a message board in order to accomplish a task. Critically, agents must be capable of deciphering when to incorporate benevolent advice into their reasoning, and when to reject malicious advice. This involves the social capacity of vigilance: the ability to determine which information to use, and which to discard, for effective communication. While well-studied in humans, the extent to which LLM agents exhibit vigilance is relatively unknown. Here, we examine LLM agent vigilance in three realistic tasks: navigating dangerous websites, avoiding e-commerce scams, and patching an unsafe code repository. We find that all frontier models are highly susceptible to malicious influence across all environments. We also find that model performance itself is surprisingly dissociable from vigilance. In light of these results, we propose a vigilance harness which uses Bayesian reasoning over a model's estimates of the value of an advisor's advice to decide whether to act on that advice. The vigilance harness helps recover baseline performance while retaining the benefit of positive influence on agents, paving the way for safer agentic AI systems.

Figures & tables

Figure 1: A : The OpenAI-Hugging Face Incident involved multiple agents escaping a sandbox testing environment and gaining administrator access to sensitive data. Their success was seemingly due in large part to their ability to orchestrate and communicate through a makeshift message board channel within the sandbox, and by misaligned agents convincing other agents to follow dangerous plans. B : In light of this incident, we explore different LLM’s vulnerability to malicious (and benevolent) persuasion from other agents. We do this by observing agent actions under the influence of advisors, compared to their unadvised (“baseline”) performance. C : Our experiments span three different realistic environments for agents, ranging from navigating real dangerous websites to selecting the best deal amongst real scams in online commerce.

Figure 2: Baseline, malicious influence, and benevolent influence performance for each model across all three environments (Web Navigation (n=4), Online Commerce (n=10), and Code Merging (n=10) tasks). Models decrease in performance under malicious influence (e.g., Claude Opus 4.8 decreases 75% between conditions in Web Navigation and Qwen3.8-27B decreases 60% in Online Commerce), and the majority of models in most environments increase in performance under benevolent influence (e.g., nearly all models reach ceiling performance in Code Merging, but show mixed results in Web Navigation.) DeepSeek V4 Flash and Gemini 3.6 Flash obtained perfect baseline performance in Web Navigation.

Figure 3: Estimated values per model across environments. We find that all models can reasonably differentiate the value of benevolent advised actions from malicious ones (all mean estimated values under malicious advice are below zero, whereas under benevolent advice they are above zero). However, this capability differs slightly per model, with Gemini 3.6 Flash being the best performer.

Figure 4: Our vigilance harness extracts the advised action from the advisor’s advice, produces a value estimate of the state the advised action leads towards, and produces a baseline action, using three queries within separate contexts. We then combine the advised action, value estimate, and baseline action within a Bayesian model. The model updates $\lambda$ , an estimate of the advisor’s utility, with every action, and uses a static estimate of the agent’s capability within the environment, $\tau$ . These weights are combined with the advised and baseline actions respectively within the policy function to produce an action at each state.

Figure 5: Percentage solve rate comparison between the awareness prompt and vigilance harness interventions, relative to regular malicious performance. We find that the vigilance harness offers a substantial improvement over the regular malicious performance across all models, and that it outperforms the awareness prompt in all models.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Comparison between Sokoban and our three agentic tasks (aggregated). A While baseline performance varies greatly in Sokoban, we see a similar trend of most models (besides Qwen3.8-27B and Claude Opus 4.8) being tricked by malicious advice. B Models appear to be worse at identifying negative actions compared to other agentic tasks, possibly due to the less explicit nature of Sokoban failures or the lower-risks associated with each decision. Nonetheless, models can still clearly differentiate negative and positive actions. C We find that both interventions improve performance in Sokoban, although the vigilance harness is outperformed by the awareness prompt.

Individual Solve Rates
	Grok 4.5	GPT-5.6 Terra	DeepSeek V4	Gemini 3.6	Qwen3.8-27B	Claude Opus 4.8
Web Navigation
Baseline	0.583	0.667	1.000	1.000	0.750	0.750
Malicious	0.250	0.250	0.250	0.750	0.250	0.000
Benevolent	0.500	0.750	1.000	1.000	1.000	0.750
Code Merging

Appendix

Table 1: Baseline, malicious influence, and benevolent influence solve rates for each individual (model, task) pairing, followed by aggregated solve rates across tasks and models. Notable metrics include large decreases between baseline and malicious in Web Navigation and Online Commerce, DeepSeek V4 Flash and GPT-5.6 Terra’s large drop across all tasks, and DeepSeek V4 Flash, Gemini 3.6 Flash and Qwen3.8’s perfect performance under benevolent influence.

Figure 7: Estimated values per task pooled across models. We find that there is a reasonable difference between the estimated value of good states and bad states across all tasks. However, Web Navigation under benevolent advice still maintains quite a low average estimated value. This is likely due to the overtly dangerous nature of Web Navigation (as opposed to the more subtle dangers found in other tasks) that makes the models evaluate any state quite poorly.

	Malicious		Benevolent		Difference
Task	mean	sd	mean	sd
Web Navigation	-0.619	0.248	+0.049	0.404	$p{<}.001$
Code Merging	-0.323	0.092	+0.917	0.030	$p{<}.001$
Online Commerce	-0.477	0.346	+0.645	0.203	$p{<}.001$
All Tasks	-0.473	0.148	+0.537	0.444	$p{<}.001$
	Malicious		Benevolent		Difference

Appendix

Table 2: Estimated values of suboptimal (the malicious advisor’s intended action) and optimal (the benevolent advisor’s intended action) states, aggregated across tasks and models. We find a significant difference in all tasks and from all models between the malicious and benevolent case (performed with a Welch’s

t

-test), suggesting that models do not lack vigilance due to an inability to accurately evaluate future states.

	Malicious				Benevolent
Task	$\hat{\tau}$	$\tau{=}0.2$	$\tau{=}0.5$	$\tau{=}0.8$	$\hat{\tau}$	$\tau{=}0.2$	$\tau{=}0.5$	$\tau{=}0.8$
Web Navigation	0.708	0.042	0.458	0.708	0.667	0.208	0.333	0.583
Code Merging	0.633	0.000	0.550	0.667	0.833	1.000	1.000	0.683
Online Commerce	0.667	0.000	0.583	0.667	0.717	1.000	0.933	0.700
All Tasks	0.669	0.014	0.531	0.681	0.739	0.736	0.756	0.656
	Malicious				Benevolent

Appendix

Table 3: Sensitivity of vigilance harness performance to choice of

\tau

. We show performance for each task and model with the empirical

\tau=\hat{\tau}

, as well as three static values:

\tau=0.2

\tau=0.5

\tau=0.8

. We find that the vigilance harness is generally strongest at repelling malicious advice at

\hat{\tau}

, but that it still exhibits decent performance with

\tau=0.5

. However, with

\tau=0.2

, harness performance collapses under malicious advice. Intuitively, we generally observe the opposite under benevolent advice, except for notably Web Navigation. The key takeaway is that the harness performs best, balancing performance under malicious and benevolent advice, when the choice of

\tau

is most closely calibrated to the true empirical capability of the agent.

Figure 8: Percentage solve rate comparison between the awareness prompt and vigilance harness interventions, relative to regular benevolent performance, per environment. We find that the vigilance harness decently retains benevolent performance. However, this based on setting $\tau=\hat{\tau}$ in our experiments, and the high baseline performance in models. Thus, the slight decrease from the regular benevolent condition is a tradeoff for improved vigilance under malicious influence.

Figure 9: Percentage solve rate comparison between the awareness prompt and vigilance harness interventions, relative to regular malicious performance, per environment. We find that the vigilance harness offers a substantial improvement from the regular malicious performance across all environments, and that it outperforms the awareness prompt across all environments.

	Malicious			Benevolent
Task	Regular	Harness $\Delta$	Aware $\Delta$	Regular	Harness $\Delta$	Aware $\Delta$
Web Navigation	0.292	+0.417 ∗∗	+0.250	0.833	$-$ 0.167	$-$ 0.208
Code Merging	0.408	+0.225 ∗∗	+0.008	0.983	$-$ 0.150 ∗∗	$-$ 0.058 ∗
Online Commerce	0.217	+0.450 ∗∗∗	+0.283 ∗∗	0.917	$-$ 0.200 ∗∗	$-$ 0.200 ∗∗
	Malicious			Benevolent
Model	Regular	Harness $\Delta$	Aware $\Delta$	Regular	Harness $\Delta$	Aware $\Delta$

Appendix

Table 4: Vigilance harness (Harness) and awareness prompt (Aware) interventions improvements across all tasks and models. We find that the vigilance harness significantly out performs the awareness prompt (roughly double the improvement) under malicious influence, while retaining the same small reduction from baseline under benevolent influence. Significance is from Welch’s

t

-test with the intervention vs. regular malicious or benevolent performance:

{}^{*}p{<}0.05

{}^{**}p{<}0.01

{}^{***}p{<}0.001

Figure 10: Average number of output tokens per action for each model pooled across tasks. Although high variance exists between model output token usage, we observe that on average the vigilance harness consistently uses more tokens than the awareness prompt. This could potentially motivate the usage of the awareness prompt with more expensive models or in lower-risk scenarios.

Figure 11: Examples of real viruses or scams we manually scrapped to create the Web Navigation dataset. From these sites, we take the URL, content of the page (e.g., pop-up titles and titles), and what pages they link to. Those visualized are mostly extortion scams, whereas others were real viruses that completely destroyed our virtual OS.

Figure 12: Examples of similar products to the items manually scrapped to create the Online Commerce dataset. The dataset spans automobiles, computers, tablets, gaming consoles, and more. The listings shown in this example are not the exact ones used in the dataset, and some of the information in the figure has been removed to maintain anonymity to the sellers.

Figure 13: Examples of code for the four pull requests from the first two tasks in Code Merging. The optimal code to merge is highlighted in green, whereas the suboptimal dangerous code is highlighted in red.

Figure 14: The ten Sokoban puzzles used in our experiments. All puzzles contained 2 boxes (except for one, which contained 3), and were between 6x6 to 9x9 tiles in size.

Persuasion in Language Models

Persuasion in language models studies how they influence human and AI beliefs or decisions, and how to measure and limit undue influence.

See all 79 papers

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs
Aug 12, 2026Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur +1Deception in Language Models Adversarial Attacks on LLMs
University of Illinois Urbana-Champaign
Large Language Models Persuade Without Planning Theory of Mind
Feb 19, 2026Jared Moore, Rasmus Overmark, Ned Cooper +3Theory of Mind Persuasion in Language Models
Department of Computer Science, Stanford University · School of Philosophical, Anthropological and Film Studies, School of Psychology and Neuroscience, University of St. Andrews · Department of Information Science, Cornell University +3
Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion
Sep 25, 2025Haodong Zhao, Jidong Li, Zhaomin Wu +4LLM Agents Persuasion in Language Models
School of Computer Science, Shanghai Jiao Tong University · 2National University of Singapore · 3Inner Mongolia Research Institute, Shanghai Jiao Tong University

Aug 12, 2026J/K move · Enter open · S save

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Nimet Beyza Bozdag, Emre Can Acikgoz, Gokhan Tur, Dilek Hakkani-Tür

University of Illinois Urbana-Champaign

Deception in Language Models Adversarial Attacks on LLMs Adversarial Robustness of Language Models Belief Updating in LLMs Persuasion in Language Models

Open paper AskPDF arXiv

Persuasion is a core dynamic of natural language communication, shaping how large language models (LLMs) update beliefs, resolve disagreements, and reach decisions. As LLMs increasingly debate, advise, and think collaboratively with humans and each other, resistance to harmful persuasion becomes a core requirement for reliable behavior. Yet we show that this requirement is far from met: a single targeted persuasive argument is enough to collapse model accuracy to near zero, even when the argument is factually false. We formalize this threat as adversarial persuasion and introduce an adversarial reinforcement learning framework that trains persuader agents to change a target model's answer in a single interaction. First, we show that optimizing persuasion strategies through trial and error exposes vulnerabilities that static prompting misses: RL-trained persuaders raise persuasion success from approximately 24% to over 93% against the training-time persuadee. Second, we find that these learned strategies transfer to unseen models, achieving 83% attack success on Qwen-14B, 79% on Llama-3.1-8B, and 25% on GPT-4o-mini. Third, we demonstrate that a curriculum that bootstraps on more persuadable open-weight models before targeting harder models further increases GPT-4o-mini attack success from 25% to 38%. Moreover, our results reveal that optimized persuaders increasingly rely on credibility-based tactics, including fabricated citations and false authoritative evidence. Together, these findings expose a critical weakness in current LLM agents: even when they initially reason correctly, they can be steered toward false conclusions by optimized natural language influence. This positions persuasion robustness as a necessary safety criterion for multi-agent and human-AI decision-making systems.

Abstract

Persuasion in Language Models

Explore similar work

Learning to Persuade Exposes How Easily LLMs Abandon Correct Beliefs

Large Language Models Persuade Without Planning Theory of Mind

Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasion