Security Considerations for Multi-agent Systems

Tam Nguyen; Moses Ndebugre; Dheeraj Arremsetty

cs.CRMar 9, 2026

Security Considerations for Multi-agent Systems

Authors: Tam Nguyen, Moses Ndebugre, Dheeraj Arremsetty

Organizations: Crew Scaler · North Carolina A&T State University

Abstract

Multi-agent artificial intelligence systems or MAS are systems of autonomous agents that exercise delegated tool authority, share persistent memory, and coordinate via inter-agent communication. MAS introduces qualitatively distinct security vulnerabilities from those documented for singular AI models. Existing security and governance frameworks were not designed for these emerging attack surfaces. This study systematically characterizes the threat landscape of MAS and quantitatively evaluates 16 security frameworks for AI against it. A four-phase methodology is proposed: constructing a deep technical knowledge base of production multi-agent architectures; conducting generative AI-assisted threat modeling scoped to MAS cybersecurity risks and validated by domain experts; structuring survey plans at individual-threat granularity; and scoring each framework on a three-point scale against the cybersecurity risks. The risks were organized into 193 distinct main threat items across nine risk categories. The expected minimal average score is 2. No reviewed framework achieves majority coverage of any single category. Non-Determinism (mean score 1.231 across all 16 frameworks) and Data Leakage (1.340) are the most under-addressed domains. The OWASP Agentic Security Initiative leads overall at 65.3% coverage and in the design phase; the CDAO Generative AI Responsible AI Toolkit leads in development and operational coverage. These results provide the first empirical cross-framework comparison for MAS security and offer evidence-based guidance for framework selection. Please check back for information on the published journal version.

Figures & tables

Vulnerability	Description
Prompt injection	In May 2022 researchers at Preamble responsibly disclosed to OpenAI that GPT-3 could be coerced to ignore safety instructions by embedding malicious commands in the user prompt or in external content. Preamble initially called it a “command injection” because it resembled SQL injection; the term “prompt injection” was adopted later. They warned that AI agents increase the likelihood of prompt injection because agents integrate more APIs and have a larger attack surface [ 2383 ]
Adversarial examples & jailbreaks	[ 2383 ] demonstrated that handcrafted adversarial examples could cause GPT-3 and BERT models to output erroneous or harmful text. The authors highlighted a “major security vulnerability” in GPT-3, showing that minimal token-level perturbations significantly degrade performance and bypass quality checks.
Training-data leakage & membership inference	LLMs memorize parts of their training data. Research found that two prominent privacy risks, i.e., training-data extraction and membership inference attacks are interconnected. Attackers can prompt an LLM to generate large amounts of text and then apply membership inference to determine whether specific data were in the training set. Follow-on work showed that despite claims of strong generalization, training-data extraction is feasible, and membership-inference techniques can differentiate training samples from non-training samples [ 2384 ]
Model inversion & data exfiltration	Early experiments showed adversaries could reconstruct sensitive data (e.g., medical images) from model outputs [ 2384 ] . These attacks illustrated that even monolithic LLMs can leak confidential training data when given cleverly crafted prompts.

TABLE I: Key security vulnerabilities identified in early monolithic LLMs (c. 2022–2023).

Vulnerability	Description
Memory/context poisoning	Microsoft’s red-team taxonomy describes memory poisoning: attackers inject malicious instructions into an agent’s long-term memory, causing future actions to be manipulated (e.g., adding a hidden CC address to every email) [ 9 ] . An attack called MINJA (Memory Injection) shows that query-only interactions can achieve over 95% injection success and 70% attack success [ 2389 ] . The attack embeds hidden instructions in seemingly benign queries and uses bridging steps to ensure they are stored and later retrieved. Memory poisoning is serious because agents rely on past context when deciding to execute tools.
Targeted knowledge-base poisoning (RAG poisoning)	Attackers can poison a retrieval knowledge base by inserting malicious documents or instructions. Microsoft’s taxonomy notes that targeted knowledge-base poisoning becomes more impactful as RAG systems allow agents to ingest large volumes of untrusted data [ 9 ] . BadRAG and AgentPoison attacks demonstrate that embedding carefully crafted triggers into a small fraction of documents (0.1 %) can cause the agent to retrieve malicious examples whenever a trigger word appears. AgentPoison achieves $\geq$ 80 % attack success with negligible impact on benign performance.
Cross-domain prompt injection (XPIA)	Agents with tool-calling capabilities risk executing arbitrary commands. The National Vulnerability Database reported CVE-2023-37274 in Auto-GPT: the execute_python_code command did not sanitize file names, allowing path-traversal to overwrite any .py file outside the workspace. Attackers could overwrite autogpt/main.py and achieve arbitrary code execution on the host. Running Auto-GPT inside a VM was recommended as a workaround [ 9 ] . This incident shows how unsanitized tool arguments can turn LLM autonomy into remote-code-execution vulnerabilities.
Denial of service via recursive invocation	Trend Micro notes that poorly configured agents can recursively invoke themselves or other agents, leading to infinite loops and service exhaustion [ 1 ] .

TABLE II: Emerging vulnerabilities in tool-augmented and RAG-enabled single-agent systems.

Vulnerability	Description
Inter-agent trust exploitation / peer-trust blind spot	The Dark Side of LLMs study demonstrates that multi-agent systems introduce three main attack vectors: direct prompt injection (success rate 41.2 %), RAG backdoor attacks (52.9 %), and inter-agent trust exploitation (82.4 %). Even when a model resists malicious commands from a human, it may execute the same command if another agent requests it [ 2393 ] . EmergentMind explains that attackers can craft malicious metadata or error messages so that an orchestration agent trusts a malicious agent; the victim agent executes commands because it assumes peer messages are trustworthy. This peer-trust blind spot underscores that the trust boundary has shifted from human vs. model to agent vs. agent.
Communication-layer attacks (Agent-in-the-Middle)	He et. al. [ 2395 ] introduces the Agent-in-the-Middle (AiTM) attack. Unlike attacks that directly compromise an agent, AiTM intercepts and manipulates messages between agents. The attacker eavesdrops on inter-agent communication and injects malicious instructions, thereby altering the system’s output. The researchers show that communication frameworks are a critical yet unexplored vulnerability, and by intercepting messages the adversary can compromise entire multi-agent systems. They highlight that existing multi-agent research mainly secured individual agents, leaving communication channels unprotected.
Cascading failures	OWASP explains that multi-agent systems are prone to cascading failures: one compromised agent can poison downstream agents via shared memory or message passing [ 6 ] .
Supply-chain and external-dependency poisoning	Multi-agent systems often depend on microservices, plug-ins or other agents hosted by third parties. OWASP’s Agentic Top 10 lists Agentic Supply Chain Vulnerabilities (ASI04): dynamic multi-component pipelines (MCPs) can be poisoned via malicious updates or compromised dependencies; natural-language execution paths can lead to remote-code execution [ 6 ] .
Memory and context poisoning at scale	Multi-agent systems share memory modules or vector stores. A memory injection attack (MINJA) can inject malicious instructions into shared memory, causing multiple agents to retrieve and act on poisoned data [ 2389 ] . OWASP documents that malicious calendar invites in Gemini could implant persistent instructions that re-emerge across sessions and trigger actions like opening smart-home devices [ 2394 ] .
Insecure external communication and zero-click prompt injection	The EchoLeak case (CVE-2025-32711) showed that a single crafted email could exploit multiple weaknesses in Microsoft 365 Copilot. The attack chain evaded Microsoft’s Cross-Prompt-Injection-Attempt classifier, bypassed link redaction via reference-style Markdown, used auto-fetched images, and abused a Microsoft Teams proxy domain to exfiltrate data. The result was remote, unauthenticated data exfiltration through zero user interaction. NIST and OWASP subsequently called indirect prompt injection “generative AI’s greatest security flaw” [ 2394 ] .

TABLE III: Novel attack vectors in compound and decentralized multi-agent architectures.

ASI code	Risk and example (from OWASP Top 10)
ASI01: Agent Goal Hijack	Hidden prompts within external content can hijack an agent’s objectives. EchoLeak is a real example.
ASI02: Tool Misuse	Agents misuse privileged tools when manipulated. Incidents include Amazon Q Code Assistant executing destructive shell commands because a malicious extension enabled a “YOLO” mode (auto-approve all tools).
ASI03: Identity and Privilege Abuse	Compromised agents inherit sensitive privileges (database access, cloud APIs). Attackers have used VS Code’s AGENTS.MD file to convince chat agents to email internal data.
ASI04: Agentic Supply-Chain Vulnerabilities	Dynamic multi-component pipelines can be poisoned through compromised plug-ins (e.g., Langflow AI RCE vulnerability).
ASI05: Unexpected Code Execution	Natural-language execution paths can lead to remote code execution (Auto-GPT RCE is an example).
ASI06: Memory and Context Poisoning	Poisoned data reshapes agent behavior long after the initial interaction—Gemini memory attack demonstrated how hidden prompts changed saved information.

TABLE IV: The OWASP Agentic AI Top 10 (2025): Risks and observed industry incidents.

Panel of Figure 5 — Fig. 1: Coverage of 193 agentic AI threat items per framework, stacked by coverage tier. Frameworks sorted by total coverage (score $\geq$ 2) in descending order. OWASP ASI leads at 65.3%; DIU RAI provides the narrowest coverage at 6.2%.

Panel of Figure 6 — Fig. 2: Mean coverage score (averaged across all 16 frameworks) per threat category, sorted weakest to strongest. No category reaches the 1.6 threshold where a majority of frameworks provide meaningful coverage.

Category	Best Framework	Avg
Agent-Tool Coupling	OWASP ASI	2.00
Data Leakage	CDAO GenAI	1.71
Injection	OWASP ASI / CDAO / MITRE	2.14
Identity/Provenance	OWASP ASI	2.16
Memory Poisoning	OWASP ASI	2.12
Non-Determinism	CDAO GenAI	1.68

TABLE V: Highest-scoring framework per threat category.

Panel of Figure 8 — Fig. 3: Mean coverage of the top four frameworks across three lifecycle phases. Design: RATC, RIP, RWA (56 items); Development: RMP, RTM (28 items); Operation: RIDC, RDL, RTE, RND (109 items).

AI Agent Security

Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.

See all 508 papers

Apr 25, 2026

Architecture Matters for Multi-Agent Security

Multi-agent systems (MAS), composed of networks of two or more autonomous AI agents, have become increasingly popular in production deployments, yet introduce security risks that do not arise in single-agent settings. Even if individual agents exhibit robust security, architectural decisions governing their coordination can create attack surfaces that have not been systematically characterized. In this work, we present an empirical study of how MAS design decisions shape the tradeoff between task performance and attack resistance. Across three agentic environments (browser, desktop, and code) and 13 architectural configurations, we use stagewise evaluations that distinguish planning refusal, execution-stage interception, partial harmful execution, and successful attack completion to study three key design choices: (i) agent roles, which determine how authority and responsibility are allocated; (ii) communication topology, which shapes how and when agents interact; and (iii) memory, which determines the context and state visibility accessible to each agent. We find that multi-agent architectures are more vulnerable than standalone agents in the majority of configurations, with attack success rates varying by up to 3.8x at comparable or higher benign accuracy, and that no single design is universally safer. These results motivate the development of further evaluations that move beyond the security properties of a single agent.

Ben Hagag, William L. Anderson, Christian Schroeder de Witt +1

Carnegie Mellon University · MATS · Oxford Witt Lab +1

Multi-Agent Systems Multi-Agent System Security Multi-Agent Communication

Ask PDF

Sep 1, 2026

SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

Safe agents can fail together. Multi-agent LLM systems (MAS) move information, state, decisions, and authority across principal boundaries, creating failures that local checks may miss. Without an execution-level view, a multi-agent setting can easily be mistaken for evidence of a genuinely multi-agent security effect. We thus systematize MAS security through an execution-centered analysis of 197 works, covering six interaction interfaces, four adversary positions, seven system-level risks, and eight recurring attack paths. We introduce an A-I-R framework that organizes attacks by adversary position, interaction interface, and resulting system-level risk, unifying otherwise fragmented attack mechanisms across MAS. We organize defenses through a five-part contract covering path target, observation, intervention, trust boundary, and recovery, and identify path closure and recovery as key challenges. We audit 44 evaluation and benchmark works and identify open challenges in isolating interaction effects, designing comparable and diagnostic metrics, supporting reuse across MAS designs, and evaluating open-system operation. Together, these findings motivate an interaction-aware view of MAS security: trace attacks end to end, test whether defenses close those paths, and evaluate system-level effects with appropriate counterfactuals.

Rui Yang, Junjie Xu, Zhengyu Liu +4

Johns Hopkins University · Nanyang Technological University

AI Agent Security Benchmarks Multi-Agent System Security Multi-Agent LLMs

Ask PDF

Mar 16, 2026

TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems

With the rapid development of LLM-based multi-agent systems (MAS), their significant safety and security concerns have emerged, which introduce novel risks going beyond single agents or LLMs. Despite attempts to address these issues, the existing literature lacks a cohesive safeguarding system specialized for MAS risks. In this work, we introduce TrinityGuard, a comprehensive safety evaluation and monitoring framework for LLM-based MAS, grounded in the OWASP standards. Specifically, TrinityGuard encompasses a three-tier fine-grained risk taxonomy that identifies 20 risk types, covering single-agent vulnerabilities, inter-agent communication threats, and system-level emergent hazards. Designed for scalability across various MAS structures and platforms, TrinityGuard is organized in a trinity manner, involving an MAS abstraction layer that can be adapted to any MAS structures, an evaluation layer containing risk-specific test modules, alongside runtime monitor agents coordinated by a unified LLM Judge Factory. During Evaluation, TrinityGuard executes curated attack probes to generate detailed vulnerability reports for each risk type, where monitor agents analyze structured execution traces and issue real-time alerts, enabling both pre-development evaluation and runtime monitoring. We further formalize these safety metrics and present detailed case studies across various representative MAS examples, showcasing the versatility and reliability of TrinityGuard. Overall, TrinityGuard acts as a comprehensive framework for evaluating and monitoring various risks in MAS, paving the way for further research into their safety and security.

Kai Wang, Biaojie Zeng, Zeming Wei +7

Shanghai AI Laboratory

LLM Agent Safety Multi-Agent System Security Multi-Agent LLM Systems AI Safety Evaluation AI Agent Monitoring

Ask PDF

Abstract

AI Agent Security

Explore similar work

Architecture Matters for Multi-Agent Security

SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems

TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems