AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

Tanqiu Jiang; Yuhui Wang; Jiacheng Liang; Ting Wang

cs.AIFeb 18, 2026

AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks

Authors: Tanqiu Jiang, Yuhui Wang, Jiacheng Liang, Ting Wang

Organizations: Department of Computer Science, Stony Brook University, Stony Brook, USA.

Abstract

LLM agents are increasingly deployed in long-horizon, complex environments to solve challenging problems, but this expansion exposes them to long-horizon attacks that exploit multi-turn user-agent-environment interactions to achieve objectives infeasible in single-turn settings. To measure agent vulnerabilities to such risks, we present AgentLAB, the first benchmark dedicated to evaluating LLM agent susceptibility to adaptive, long-horizon attacks. Currently, AgentLAB supports five novel attack types including intent hijacking, tool chaining, task injection, objective drifting, and memory poisoning, spanning 28 realistic agentic environments, and 644 security test cases. Leveraging AgentLAB, we evaluate representative LLM agents and find that they remain highly susceptible to long-horizon attacks; moreover, defenses designed for single-turn interactions fail to reliably mitigate long-horizon threats. We anticipate that AgentLAB will serve as a valuable benchmark for tracking progress on securing LLM agents in practical settings. The benchmark is publicly available at https://tanqiujiang.github.io/AgentLAB_main.

Figures & tables

Figure 1: Overall framework of AgentLAB.

Attack type	Adversary	Attack strategy	Attack vector
Intent hijacking	User	Deceive agent into executing malicious task	Adversarial prompting
Tool chaining	User	Chain benign tool calls to achieve malicious task	Adversarial prompting
Objective drifting	Environment	Shift agent’s objective from benign to malicious task	Indirect prompt injection
Task injection		Inject malicious task alongside benign task
Memory poisoning		Persist malicious task in agent’s memory

Table 1: Overview of long-horizon attack types supported by AgentLAB. Each type exploits the distinct aspects of extended user-agent-environment interactions.

Figure 2: A multi-agent framework for long-horizon attacks.

Panel of Figure 3 — Figure 3: Task Injection. Coordinated injections hijack a benign task into unauthorized Slack commands.

Panel of Figure 4 — Figure 4: Memory Poisoning Attack. Hidden injections in routine content (emails, code, products) are stored as “user preferences.” When a harmful request arrives, retrieved memories provide false context that disables safety filtering, causing sensitive data leakage.

Figure 5: Distribution of tasks cross different risk categories.

Agent	Intent	Tool	Objective	Task	Memory	Overall
Agent	Hijacking	Chaining	Drifting	Injection	Poisoning	Overall
Qwen-3	78.1	96.3	92.2	93.1	48.0	81.5
Llama-3.1	53.3	90.4	67.4	86.6	34.6	66.5
GPT-4o	74.0	94.1	79.2	79.9	63.3	78.1
GPT-5.1	59.8	94.6	73.7	21.5	51.3	69.9
Gemini-3	46.2	95.9	15.8	43.1	67.3	53.7

Table 2: Average effectiveness (ASR) of long-horizon attacks on various LLM agents.

Agent	One-Shot Attack	Long-Horizon Attack
Qwen-3	81.9%	93.1%
Llama-3	50.7%	86.8%
GPT-4o	62.5%	79.9%
GPT-5.1	2.08%	21.5%
Gemini-3	41.0%	43.1%
Claude-4.5	0.0 %	0.0 %

Table 3: Comparison of the attack effectiveness of one-shot and long-horizon injection attacks.

Agent	Intent Hijacking			Tool Chaining			Objective Drifting			Task Injection			Memory Poisoning
Agent	/	SR	LG	/	SR	LG	/	RP	DD	/	RP	DD	/	RP	DD
Qwen-3	78.1	41.4	73.3	96.3	81.5	48.2	92.2	69.0	25.3	93.1	95.1	6.9	48.0	32.4	41.2
Llama-3.1	53.3	24.1	50.0	90.4	76.0	66.5	67.4	37.9	19.8	86.8	68.1	35.4	34.6	24.3	30.9
GPT-4o	74.0	39.3	71.0	94.1	82.4	64.7	79.2	51.7	13.3	79.9	66.7	12.7	63.3	56.8	31.6
GPT-5.1	59.8	20.7	55.4	94.6	75.7	48.6	73.7	31.0	7.8	21.5	20.8	2.8	51.3	27.0	38.5
Gemini-3	46.2	10.3	46.4	95.9	61.2	49.0	15.8	10.3	15.8	43.1	31.9	13.8	67.3	43.2	55.1

Table 4: Evaluation of baseline defenses against long-horizon attacks.

Figure 6: ASR (%) vs. T2S across five attack categories for six LLM-based agents. Upper-left indicates higher vulnerability; lower-right indicates greater robustness.

Figure 7: Attack success rate (ASR) as a function of maximum attack turns (

n_{\text{turn}}

) for intent hijacking and task injection on Qwen-3 and GPT-4o.

Figure 8: Attack success rate (ASR) as a function of allowed optimization steps (

n_{\text{opt}}

) for intent hijacking and task injection on Qwen-3 and GPT-4o.

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 327 papers

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
Jul 20, 2026Devina Jain, David Hartmann, Chuan LiAdversarial Attacks on LLMs LLM Agent Evaluation
Lambda
DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security
Sep 21, 2026Ivan Aleksandrov, German Kochnev, Sabrina Sadiekh +1LLM Agent Security AI Agent Security Benchmarks
AI Security Lab, ITMO University · Hive Trace Lab · Independent Researcher
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2

Jul 20, 2026J/K move · Enter open · S save

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

Devina Jain, David Hartmann, Chuan Li

Lambda

Adversarial Attacks on LLMs LLM Agent Evaluation AI Agent Security Benchmarks Prompt Injection Attacks Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. Most safety benchmarks evaluate defenders against fixed attack pools collected before evaluation, single-turn or multi-turn. We present a 21-scenario benchmark for adaptive multi-round attacks against memoryless LLM defenders: an autonomous LLM attacker observes prior defender responses and pivots across rounds, while each defender response is evaluated as a fresh interaction. Holding the 21 scenarios, attackers, defenders, and structured-output scoring fixed, restricting scoring to the first attacker turn yields

0

1\%

attack success rate (ASR); allowing 15 rounds of adaptive attack yields

5.4

14.0\%

. Pooling three frontier attacker LLMs uncovers

1.4

2.2\times

as many unique successful attacks as the best single attacker, and the generated attacks have low cosine similarity (

0.02

0.14

) to attacks in existing benchmarks. Claude Opus 4.6 and GPT-5.4 are tied in aggregate (

5.4\%

each; overlapping

95\%

CIs), but their weaknesses differ sharply: on one scenario Opus reaches

60\%

ASR (

95\%

36

80\%

) while GPT-5.4 and Gemini each stay at

7\%

(CI

1

30\%

; the gap is preserved in a higher-

N

replication).

13

21

scenarios distinguish at least one defender pair, yet rankings disagree across scenarios (Kendall's

W = 0.19

). We release the benchmark -- 21 evaluation scenarios, 10 public development scenarios, the orchestrator, baseline harnesses, and a multi-attacker CLI -- plus 945 transcripts from the 3

\times

3 frontier matrix, an attack-replay dataset, and 18{,}422 gpt-oss-20b battles from an open competition's final scoring rounds.

Abstract

LLM Agent Security

Explore similar work

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis