Prompt Injection Attacks on AI Agents

Follow this topic Ask about this topic

Momentum

26 papers in the last four weeks, up 271% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

Constrained-Action AI Remediation for SIEM/XDR via a NeMo-Guardrails Proxy
Oct 7, 2026Georgios Koutidis, Nikolaos Kekatos, Tom Nianios +1LLMs for Cybersecurity Tool Access Control for LLM Agents
Clone Systems, Larnaca, Cyprus · University of Thessaly, Department of Intelligent Energy Systems, Larissa, Greece
Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents
Oct 7, 2026Sarthak Choudhary, Mihai Christodorescu, Ashish Hooda +3Computer-Use Agents AI Agent Security
University of Wisconsin–Madison · Google · Google DeepMind
Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
Oct 7, 2026Yupu Wang, Zhengyuan Jiang, Reachal Wang +1AI Agent Security Adversarial Attacks
Duke University
AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model
Oct 6, 2026Sarim Hashmi, Mukul Ranjan, Kshitij Mishra +3Adversarial Training Sim-to-Real Transfer
Mohamed bin Zayed University of Artificial Intelligence · Amazon · Massachusetts Institute of Technology
Surviving the Router: Optimizing Skill Injections for Retrieval and Execution
Oct 6, 2026Haneen Najjar, Luca Scionis, Haritz Puerto +1AI Agent Security Benchmarks LLM Agent Skill Retrieval
Max Planck Institute for Intelligent Systems; Tel Aviv University · Sapienza University of Rome; University of Cagliari · ELLIS Institute Tübingen
Towards a Unified Misuse Monitoring Benchmark
Oct 5, 2026Aniruddh Pramod, James Oldfield, Adel BibiAI Agent Security AI Agent Monitoring
University of Oxford
Can CaMeLs Talk? Securing Multi-Agent Systems Against Indirect Prompt Injection Attacks
Oct 5, 2026James Peters-Gill, Avi Semler, Henning Bartsch +2AI Agent Security Benchmarks Information Flow Control
Independent · University of Oxford · MATS Research +1
Readable Before Actionable: Causal Tracing of Indirect Prompt Injection
Oct 4, 2026Zhe Yu, Wenpeng Xing, Xingxing Yang +1Causal Reasoning in Language Models Indirect Prompt Injection
Zhejiang University · Binjiang Institute of Zhejiang University · Hong Kong Baptist University
Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection
Oct 4, 2026Jingkai Liu, Yufei Han, Xiaoting Lyu +2Prompt Injection AI Agent Auditing
Beijing Jiaotong University · Mohamed bin Zayed University of Artificial Intelligence · INRIA Rennes-Bretagne-Atlantique +1
APEX: Active Protection at Execution Boundaries for LLM Agents
Oct 3, 2026Xinran Zheng, Xin Fan Guo, Zhiqiang Hao +7AI Agent Security Prompt Injection Defense
Tsinghua University · Imperial College London · Nanjing University +6
Chaining Skills to Hijack LLM Agents
Oct 1, 2026Tian Dong, Zixuan Ma, Haodong Zhao +3LLM Agent Security AI Agent Safety
The University of Hong Kong · Shandong University · Shanghai Jiao Tong University +1
ActionGuard: Tool Call Authorization under Poisoned Skills
Sep 30, 2026Jihun Han, Yejin Jang, Byung Il Kwak +1Runtime Enforcement for AI Agents Prompt Injection Defense
Korea University Republic of Korea
Can Agents Trust Their Skills? Uncovering Unsafe Chains of Trust in Skill-Based LLM Agents
Sep 30, 2026Yan Wang, Zhihao Zhang, Ke Chen +5LLM Agent Security Prompt Injection Attacks on AI Agents
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyberspace Security, University of Chinese Academy of Sciences · Worcester Polytechnic Institute
The Backdrop Exposes What the World Around an Agent Costs It
Sep 29, 2026Nusrat Jahan Lia, Shubhashis Roy DiptaAI Agent Reliability Prompt Injection Attacks on AI Agents
University of Dhaka · University of Maryland, Baltimore County
Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?
Sep 29, 2026Michael Lee, Zhipeng Wei, Yue Dong +1Indirect Prompt Injection Prompt Injection Attacks
International Computer Science Institute · DSO National Laboratories · UC Riverside +1
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
Share-Borne AI Virus: Memory-Hopping Attacks Across LLM Agents
Sep 28, 2026Sidharth Pulipaka, Ansh Sharma, Stanislau Hlebik +4AI Agent Security Agent Memory Poisoning
SPAR · University of Cambridge · APTA AI +1
Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection
Sep 28, 2026Yan Zhan, Yunze Song, Mengkai Hou +3Prompt Injection Prompt Injection Defense
Peking University · National University of Singapore · BYD Company Limited +1
DGF-Bench: A Benchmark for Simulating and Auditing Deception Against Multi-Agent Governance Boards
Sep 28, 2026Jeremy CanaleAI Agent Auditing AI Agent Security Benchmarks
MMSkillRisk: Can Agents Stay Safe When Multimodal Skills Become Traps?
Sep 28, 2026Lingqi Jiang, Jialuo Chen, Jianan Ma +8Prompt Injection Attacks on AI Agents Agent Skill Security Auditing
Zhejiang University · Ant Group · Hangzhou Dianzi University +1
Certified Multi-Source Integrity for Structured Agent Actions
Sep 28, 2026Anmol Pandey, Aditya Jain, Liang Chen +2Data Provenance LLM Agent Security
University of Warwick · University of Hertfordshire
When Consent Outlives Context: Residual Authority Replay in Long-Lived Agents
Sep 27, 2026Zhihao Zhang, Chao Wang, Rujia Li +3LLM Agent Security Prompt Injection Attacks on AI Agents
Worcester Polytechnic Institute · Tsinghua University · Independent Researcher
Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning
Sep 27, 2026Chenlong Yin, Xiaolong Jin, Wei Zou +2Adversarial Attacks on LLMs LLM Red Teaming
The Pennsylvania State University · Purdue University
Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions
Sep 23, 2026Tiantong Wu, Wei Yang Bryan LimAdversarial Prompt Generation Prompt Injection Attacks
Nanyang Technological University Singapore
ActGov: Governing LLM Agent Actions via Policy-Constrained Validation
Sep 21, 2026Kaiyuan Zhang, Yuke Peng, Ke Jiang +1LLM Agent Security Runtime Enforcement for AI Agents
Southern University of Science and Technology
Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
Sep 17, 2026Alex Remedios, Simon Storf, Fabien Roger +1AI Agent Security AI Agent Monitoring
Anthropic Fellows Program · Anthropic
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems
Sep 15, 2026Deepak Akkil, Tamer Abuelsaad, Karthik Vikram +5Multi-Agent LLM Systems Long-Horizon Agent Evaluation
Emergence AI
Corrupt Plans, Clean Traces: Evading Chain-of-Thought Monitoring with Plan Injection
Sep 14, 2026Keertana Chidambaram, Andrew Ilyas, Vasilis SyrgkanisLLM Security Prompt Injection Attacks on AI Agents
Stanford University · CMU
Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
Sep 14, 2026Xiaoyan Li, Yunli WangAdversarial Attacks on LLMs Backdoor Defense in LLMs
Department of Computer Science University of Toronto Toronto, Ontario, Canada · Digital Technologies Research Centre National Research Council Canada Ottawa, Ontario, Canada
ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents
Sep 14, 2026Bingzheng Wang, Xiaoyan Gu, Wentao Wang +3Runtime Enforcement for AI Agents Prompt Injection Defense
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Key Laboratory of Cyberspace Security Defense, Beijing, China +3

Oct 7, 2026, cs.CRJ/K move · Enter open · S save

Constrained-Action AI Remediation for SIEM/XDR via a NeMo-Guardrails Proxy

Georgios Koutidis, Nikolaos Kekatos, Tom Nianios, Alexios Lekidis

Clone Systems, Larnaca, Cyprus · University of Thessaly, Department of Intelligent Energy Systems, Larissa, Greece

LLMs for Cybersecurity Tool Access Control for LLM Agents Prompt Injection Attacks on AI Agents LLM Guardrails AI Agent Security Benchmarks

Open paper AskPDF arXiv

Security Operations Centers (SOCs) for information technology and operational technology share one incident-response problem: a flood of correlated alerts and too few analysts. Large Language Models (LLMs) are increasingly proposed as reasoning engines that triage alerts and, in autonomous deployments, issue commands that block IPs, kill processes, or quarantine files on production hosts. This coupling introduces a new risk: a single adversarial alert can become a remote code path through the LLM's reasoning, leading it to recommend an action the SOC then executes. We present a constrained-action architecture with two coordinated layers: (i) a SIEM/XDR control plane that grounds remediation in correlated host events and confines the LLM's output to a closed intent vocabulary whose templated commands are executed by thin endpoint agents, backstopped by an argument validator; and (ii) a NeMo-Guardrails proxy that wraps the SOC-analyst LLM with input- and output-rail policies, evaluated out-of-the-box against a SOC-specific adversarial corpus we release. The stock proxy lifts injection recall from 25.0% to 94.5% at a 0.1% false-positive rate, and a live red-team exercise confirms that the closed intent vocabulary and argument validator contain the observed LLM failure modes before any command crosses the trust boundary. As an architectural fit (not yet a measured operational-technology deployment), the constrained-action property suits critical-infrastructure settings where a wrong remediation has physical, not merely operational, consequences. The loop is best run human-in-the-loop or delayed: the measured rail latency keeps inline control out of scope.

Momentum

Related topics

Latest papers 108