cs.CRMar 19, 2026

A Framework for Formalizing LLM Agent Security

Authors: Vincent Siu, Jingxuan He, Kyle Montgomery, Zhun Wang, Neil Gong, Chenguang Wang, Dawn Song

Organizations: UC Santa Cruz · UC Berkeley · Duke University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Security in LLM agents is inherently contextual. For example, the same action taken by an agent may represent legitimate behavior or a security violation depending on whose instruction led to the action, what objective is being pursued, and whether the action serves that objective. However, existing definitions of security attacks against LLM agents often fail to capture this contextual nature. As a result, defenses face a fundamental utility-security tradeoff: applying defenses uniformly across all contexts can lead to significant utility loss, while applying defenses in insufficient or inappropriate contexts can result in security vulnerabilities. In this work, we present a framework that systematizes existing attacks and defenses from the perspective of contextual security. To this end, we propose four security properties that capture contextual security for LLM agents: task alignment (pursuing authorized objectives), action alignment (individual actions serving those objectives), source authorization (executing commands from authenticated sources), and data isolation (ensuring information flows respect privilege boundaries). We further introduce a set of oracle functions that enable verification of whether these security properties are violated as an agent executes a user task. Using this framework, we reformalize existing attacks, such as indirect prompt injection, direct prompt injection, jailbreak, task drift, and memory poisoning, as violations of one or more security properties, thereby providing precise and contextual definitions of these attacks. Similarly, we reformalize defenses as mechanisms that strengthen oracle functions or perform security property checks. Finally, we discuss several important future research directions enabled by our framework.

Figures & tables

Figure 1: Contextual security resolves the utility–security trade-off in LLM-agent systems. These examples illustrate that the same prompt may correspond to either an attack or a legitimate task depending on the execution context. In the absence of contextual considerations in attack definitions, prior detection methods primarily classify prompts as malicious or benign based on surface patterns, without accounting for how the agent is executing the task. This inevitably forces a trade-off: blocking patterns such as “delete file” improves security but prevents legitimate cleanup operations, while allowing them preserves utility but enables data-destruction attacks. Our framework formalizes context by checking four contextual security properties: Source Authorization , Task Alignment , Action Alignment , and Data Isolation . This enables the agent to distinguish legitimate file cleanup from data destruction even when the underlying prompts are identical, thereby achieving both security and utility.
Figure 2: Graphical representation of an agent solving a task with example, showing how the memory MtM_{t} , Environment E , and Trajectory Tr evolve over time.
Figure 3: Data isolation violations occur when information inappropriately crosses session boundaries. (A) An agent observes admin credentials during debugging on Day 1, then reuses those credentials for an unrelated access check on Day 5, violating memory constraints. (B) Current benchmarks reset context between tasks, making cross-session credential reuse invisible to evaluation. Data isolation checks require tracking information flow across the trajectory.
More in this topic

LLM Agent Security

Large language model agent security studies attacks and safeguards for systems that plan, use tools, retain memory, and act across applications.

See all 320 papers

Explore similar work

CardsList
Jun 9, 2026

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, and act on external environments. This transition changes the nature of security risk. In agentic settings, failures are no longer limited to unsafe text generation. Untrusted content may redirect control flow, misuse tool privileges, corrupt persistent state, leak sensitive information, or trigger harmful external actions. At the same time, research on LLM agent security is expanding quickly but remains fragmented across attack families, defense layers, application domains, and evaluation settings. This paper synthesizes 247 papers through a lifecycle-based, systems-oriented framework that models agent security around the interaction of information flow, delegated authority, and persistent state. We organize the literature around four questions: how LLM agent security should be modeled, which threat surfaces and attack families dominate, what defenses have been proposed and with what tradeoffs, and how security claims are evaluated. We find that prompt injection and tool-mediated control-flow hijacking still dominate the field, while persistent state corruption and multi-agent propagation are becoming central emerging concerns. We further find that current defenses provide useful building blocks but remain weakly compositional, and that existing benchmarks still underrepresent long-horizon, stateful, and deployment-sensitive risks. We argue that secure LLM agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and evaluation practices aligned with realistic operational settings.

Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Technical University of Munich, Heilbronn, Germany

Prompt InjectionLLM Agent SecurityAI Agent Security Benchmarks
AskPDF
Jul 24, 2026

Agent Security Needs Redefinition through a Holistic Framework

Agent security is widely treated as a question about action content. Defenses ask whether an instruction looks malicious. Benchmarks ask whether an agent performs a harmful sounding action. We argue that agent security is fundamentally a contextual problem, and that the current content based framing systematically misdefines it. A command to ``delete user data'' might be a routine administrative request or a prompt injection attacking production systems, and the content alone cannot distinguish the two. Authorization context can. Across every injection task in AgentDojo and WASP, the same action is one an authenticated user would plausibly request in a routine workflow, which makes the conflation a structural property of evaluating security through content. We operationalize contextual security through four properties that must hold jointly and be evaluated continuously across the agent's trajectory. Source Authorization asks who issued the command. Task Alignment specifies the agent's authorized objective. Action Alignment evaluates whether each action serves that objective. Data Isolation governs information flows across privilege boundaries. Under this reframing, indirect prompt injection becomes a Source Authorization violation. Snapshot benchmarks are structurally incapable of evaluating Data Isolation. Existing defenses are reorganized around the property they actually approximate. The contextual reframing changes which defenses are coherent, which evaluations measure something useful, and which attack patterns evaluation can see at all.

Vincent Siu, Jingxuan He, Kyle Montgomery +3

1UC Santa Cruz · 2UC Berkeley

AI Agent SecurityAI Agent Security BenchmarksPrompt Injection Attacks on AI Agents
AskPDF
Jun 26, 2026

LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity

Large language model (LLM) agents are rapidly being integrated into real-world systems. Their autonomy and tool-use capabilities generate substantial value while simultaneously expanding the security attack surface. This survey provides a comprehensive overview of the opportunities and challenges of LLM agents in security, focusing on two core areas: (1) threats to LLM agents themselves and corresponding mitigation strategies (LLM agents self-security), and (2) the role of LLM agents in empowering the cybersecurity lifecycle across offense and defense (LLM agents empowered cybersecurity). We first examine the internal and external attack surfaces of agents, propose a taxonomy organized by threat sources, and analyze associated mitigations and evaluation frameworks. We then investigate how agent capabilities are applied in cybersecurity practice and present, to our knowledge, the first agent-empowerment framework aligned with the full cyber offense-defense lifecycle. By systematically surveying these two areas, we are the first to highlight a positive feedback synergy between LLM agents self-security and empowered cybersecurity, offering new insights for the advancement of both. We further identify current limitations and outline promising directions for future research. The insights provided aim to catalyze the coordinated development of LLM agents self-security and agent empowered cybersecurity, paving the way for more capable and robust agent applications.

Yiwei Xu, Yong Zhuang, Xuanming Liu +6

School of Cyber Science and Engineering, Wuhan University, Wuhan, China. · Department of Computer Science and Engineering, University at Buffalo, Buffalo, USA.

Autonomous Cyber DefenseLLM Agent SecurityLLMs for Cybersecurity
AskPDF