"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems

Xinfeng Li; Shenyu Dai; Kelong Zheng; Yue Xiao; Gelei Deng; Wei Dong; Xiaofeng Wang

cs.HCFeb 24, 2026

"Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems

Authors: Xinfeng Li, Shenyu Dai, Kelong Zheng, Yue Xiao, Gelei Deng, Wei Dong, Xiaofeng Wang

Organizations: Nanyang Technological University · KTH · William & Mary

Abstract

Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare. However, this deepening trust introduces a novel attack surface: Agent-Mediated Deception (AMD), where compromised agents are weaponized against their human users. While extensive research focuses on agent-centric threats, human susceptibility to deception by a compromised agent remains unexplored. We present the first large-scale empirical study with 303 participants to measure human susceptibility to AMD. This is based on HAT-Lab (Human-Agent Trust Laboratory), a high-fidelity research platform we develop, featuring nine carefully crafted scenarios spanning everyday and professional domains (e.g., healthcare, software development, human resources). Our 10 key findings reveal significant vulnerabilities and provide future defense perspectives. Specifically, only 8.6% of participants perceive AMD attacks, while domain experts show increased susceptibility in certain scenarios. We identify six cognitive failure modes in users and find that their risk awareness often fails to translate to protective behavior. The defense analysis reveals that effective warnings should interrupt workflows with low verification costs. With experiential learning based on HAT-Lab, over 90% of users who perceive risks report increased caution against AMD. This work provides empirical evidence and a platform for human-centric agent security research.

Figures & tables

Fig. 1: Examples of stealthy LLM agent-mediated deception. (1) An agent manipulated by a hidden note in a resume biases the hiring decision. (2) An agent hijacked by a malicious email exfiltrates sensitive data and deletes the evidence.

Scenario	Boundary	Attack Type	Assigned Task	Attack Consequence
CoverLetterAgent	Perception	Indirect Prompt Injection	Generate professional cover letters	Low-quality letter due to injected instructions
HR-Agent	Perception	Indirect Prompt Injection	Select the best candidate from CVs	Candidate ranking manipulated via hidden prompt
DoctorAgent	Perception	Indirect Prompt Injection	Generate a medical report	Skewed diagnosis due to tampered questionnaire
CodeEvalAgent	Perception	Indirect Prompt Injection	Assess candidate code quality	Flawed reasoning caused by poisoned comment
WebSearchAgent	Perception	Indirect Prompt Injection	Answer complex query with cited info	Tool disabled via fake citation with prompt trap
CodeRepoAgent	Memory	RAG Poisoning	Get library suggestion for a new function	Agent suggests malicious Python package

TABLE I: Overview of nine task scenarios, detailing the trust boundary, attack type, task, and consequence for each.

Fig. 3: Illustrations of three guardrail conditions.

(1) Age	N	%	(2) LLM Use Freq.	N	%
18–24	99	32.7	(5) Everyday	155	51.2
25–34	113	37.3	(4) Frequently	109	36.0
35–44	51	16.8	(3) Occasionally	29	9.6
45–54	20	6.6	(2) Rarely	6	2.0
55+	20	6.6	(1) Very Rarely	4	1.3
(3) Gender	N	%	(4) Task-Oriented?	N	%

TABLE II: Demographics of study participants (N=303) .

Fig. 4: The three-phase procedure of our experiment. Participants are randomly assigned to a guardrail in Phase 2.

Fig. 5: Pre-survey attitudes towards LLM agents (N=303). Users have a mixed, contradictory view. (a) 97.0% have a moderate to high trust in AI to do tasks correctly. (b) 75.9% say they check AI outputs often. (c) 96.4% have a moderate to very high confidence in identifying risks in AI systems. (d) Only 28.4% report they never experience AI issues.

Metric	Experienced	Inexperienced	Change	p -value $\star$
Metric	(n=193)	(n=110) $\natural$	(Abs. (Rel. %))	p -value $\star$
Trust (1–5) $\downarrow$	3.65 $\pm$ 0.70	4.30 $\pm$ 0.72	-0.65(-15.1%)	$<$ .001
Vigilance (1–5) $\uparrow$	4.31 $\pm$ 0.92	3.82 $\pm$ 1.21	+0.49 (+12.8%)	.0003
Confidence (1–5) $\downarrow$	3.96 $\pm$ 0.77	4.11 $\pm$ 0.99	-0.15 (-3.6%)	0.024

TABLE III: Effect of negative AI experience on user attitudes.

Group / Perception Rate (%)	Guard 1	Guard 2	Guard 3
Overall	8.6% [183]	20.0% [60]	25.0% [60]
Domain Professionals	6.8% [69]	18.4% [29]	22.2% [21]
General Users	9.6% [114]	21.5% [31]	26.5% [39]

TABLE IV: Risk perception rates for users with different technical expertise. Domain experts show lower perception rates.

Fig. 6: User-rated severity and attack perception rate across nine agent scenarios. A positive correlation is observed, where higher severity is associated with higher perception rate.

Metric	Perceived	Unperceived	Change	p -value $\star$
Metric	(n=85)	(n=218)	(Abs. (Rel. %))	p -value $\star$
Trust (1–5) $\downarrow$	3.68 $\pm$ 0.76	3.96 $\pm$ 0.77	-0.28 (-7.1%)	0.007
Vigilance (1–5) $\uparrow$	4.31 $\pm$ 0.98	4.06 $\pm$ 1.09	+0.24 (+5.9%)	0.054
Confidence (1–5) $\downarrow$	3.98 $\pm$ 0.80	4.03 $\pm$ 0.88	-0.056 (-1.4%)	0.392
Experience $\natural$ $\uparrow$	0.81 $\pm$ 0.39	0.65 $\pm$ 0.48	+0.16 (+25.6%)	0.007

TABLE V: Attitudes of users who perceive an attack vs. those who do not. Perceived users start with lower trust, higher caution, and are highly likely to have experienced AI errors.

Guardrail	Num	Prop.	Risk Perc. (%)		Perc.	Prop. in
Guardrail	Users	Proact.	All	Proact.	Lift	Perc. Users
G1 (Static)	183	$40.8$	$8.6$	12.9	+51.2	$61.7$
G2 (Persistent)	60	$26.1$	$20.0$	34.0	+70.2	$44.4$
G3 (Interactive)	60	$36.1$	$25.0$	32.3	+29.2	$46.7$
Overall	303	37.0	14.1	19.6	+39.5	51.6

TABLE VI: How proactive user judgment affects risk perception. Users exhibiting proactive scrutiny (relying on personal judgment) show higher risk perception rates.

Factor	Group	Perception Rate
		(%) $\uparrow$	p -value $\star$
Consistency	Low ( $\leq$ 2)	24.29	$<$ .0001
	High ( $\geq$ 3)	6.91
In-Situ Trust	Low ( $\leq$ 2)	23.19	$<$ .0001
	High ( $\geq$ 3)	7.02
In-Situ Stress	Low ( $\leq$ 2)	8.35	.1645

TABLE VII: Multi-factor correlation analysis of risk perception. Results reveal that cognitive processes during interaction are more predictive than pre-existing user characteristics.

Fig. 7: Changes in user trust after interacting with HAT-Lab , broken down by guardrail (G1, G2, G3) and whether the user perceives a risk. Users who missed an attack often reported increased trust, which makes them more susceptible.

Fig. 8: Effectiveness of the three guardrails. A clear order of effectiveness shows: G3

>

>

G1, from (i) guardrail visibility, to (ii) risk perception, to (iii) accurate identification.

Fig. 9: Per-scenario breakdown of guardrail effectiveness. Guard 2 and Guard 3 consistently yield the highest rates of both risk perception (top) and accurate identification (bottom).

Condition	Pleasantness $\uparrow$			Recommendation $\uparrow$
Condition	Perceived	Unperceived	p -value	Trust+ $\natural$	Trust–	p -value
Guard 1	3.86 $\pm$ 0.83	4.12 $\pm$ 0.97	0.117	4.43 $\pm$ 0.65	4.03 $\pm$ 1.10	0.002
Guard 2	3.87 $\pm$ 1.10	4.30 $\pm$ 0.81	0.115	4.50 $\pm$ 0.74	4.16 $\pm$ 0.86	0.110
Guard 3	3.96 $\pm$ 0.87	4.59 $\pm$ 0.86	0.007	4.66 $\pm$ 0.55	4.45 $\pm$ 0.68	0.205

TABLE VIII: Pleasantness ratings and trust effects on recommendation likelihood. Left: How HAT-Lab ’s pleasantness ratings differ between users who perceived an attack and those who did not. Right: How trust affects recommendation likelihood.

Fig. 10: User responses to “ Will you be more cautious in the future? ” The experience with HAT-Lab is a powerful lesson, especially for users who perceived AMD attacks.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Scenario	COMET	Scenario	COMET	LLM Backend	ASR
CoverLetterAgent	0.82	CodeEvalAgent	0.81	GPT-4o	98.9
HR-Agent	0.75	SecretaryAgent	0.76	Gemini 2.0 Flash	94.4
TripAgent	0.91	MailAgent	0.84	Gemini 2.5 Flash	93.3
DoctorAgent	0.73	WebSearchAgent	0.79	Gemini 2.5 Pro	96.3
CodeRepoAgent	0.78	Overall	0.80	GPT-4.1	96.7
Baseline COMET Lower=0.54; Upper=0.83 $\natural$				DeepSeek-V3	93.8

Appendix

TABLE IX: COMET scores for scenarios and ASR for LLMs.

Fig. 11: User feedback on HAT-Lab ’s usability and experience. (a) 81.6% of users rated the experience as “pleasant” or better. (b) A vast majority of users (78.6%) think the system is not difficult, even very easy to use. (c) 100% of users feel the tasks are realistic. (d) 84.5% are highly “likely” to recommend HAT-Lab for AI (agent) safety education.

Protection Measure (N=303)	Count	(%)
Real-time warnings for risky outputs/suspicious links	185	61.1
Periodic reminders for users to review system outputs	169	55.8
Limit autonomous permissions for high-risk actions	164	54.1
Human-in-the-loop confirmation for sensitive actions	157	51.8
Isolate permissions and data between different apps	96	31.7
Others $\natural$	23	7.6

Appendix

TABLE X: User-preferred protection for agentic systems.

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers
Jun 19, 2026Kerri Prinos, Lilianne Brush, Cameron DentonCybersecurity AI Agent Security
Horizon3, San Francisco, CA, 94111, USA
DecepEval: A Benchmark for Evaluating Deception in LLM Agents
Oct 6, 2026Yiming Xu, Hongyue Yu, Beihua Yang +8Deception in Language Models LLM Agent Evaluation
Xi’an Jiaotong University · University of Virginia · Griffith University +2
Intentional Deception as Controllable Capability in LLM Agents
Mar 8, 2026Jason Starace, Terence SouleDeception in Language Models LLM Agent Safety
Department of Computer Science, University of Idaho, Moscow, ID 83844

Jun 19, 2026J/K move · Enter open · S save

Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers

Kerri Prinos, Lilianne Brush, Cameron Denton

Horizon3, San Francisco, CA, 94111, USA

Cybersecurity AI Agent Security AI Agent Security Benchmarks LLMs for Cybersecurity

Open paper AskPDF arXiv

The empirical foundation of cyber deception relies on human-centered hypotheses, but the rapid emergence of autonomous, AI-enabled attackers challenges whether this foundation transfers to AI agents. To address this, we introduce an automated evaluation framework adapted from the Honeyquest instrument to assess LLM attacker judgment at scale. Our 21-LLM cohort spanned 10 providers, diverse architectures and specializations, open- and closed-weight models, and parameter scales from 8B to over 1T. We evaluated the performance of this LLM cohort (yielding 10,962 responses) against the 47-participant human baseline across an identical set of 174 reconnaissance queries. Our empirical evaluation reveals three key findings that establish LLMs as a distinct attacker class: (1) every model in our cohort falls for deceptive traps at a significantly higher rate than human attackers; (2) the defensive attention-diversion effect observed in humans is statistically absent in our LLM cohort; and (3) a critical recognition-action gap, where LLMs successfully articulate trap recognition in their reasoning but exploit the deceptive elements anyway 73.4% of the time. Across the 21 models, trap recognition in reasoning text did not predict fell-for-trap behavior (Spearman r = +0.08, p = 0.73). Ultimately, these findings demonstrate that human-centered deception hypotheses do not reliably transfer to AI attackers, highlighting the critical need for new research into AI-native active defense frameworks.

Abstract

AI Agent Security

Explore similar work

Honeyquest for LLMs: Rethinking Cyber Deception for AI Attackers

DecepEval: A Benchmark for Evaluating Deception in LLM Agents

Intentional Deception as Controllable Capability in LLM Agents