Prompt Injection Attacks on LLMs

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 367% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
Jun 1, 2026Hiskias Dingeto, William LeeneyLLM Red Teaming LLM Agent Security
StackOne Technologies
Send a SCOUT First: Pre-hoc Reasoning for Adaptive Detector Allocation in Prompt-Injection Defense
May 29, 2026Shuhao Zhang, Jiarui Li, Qi Cao +2AI Agent Security Benchmarks LLM Security
UC San Diego · University of Illinois Urbana-Champaign
Persona Attack: Incremental Memory Injection Jailbreak Attack against Large Language Models
May 29, 2026Junyoung Park, Seongyong Ju, Sunghwan Park +1Adversarial Attacks on LLMs LLM Safety
Chung-Ang University
Depth-Dependent Indirect Prompt Injection in Tool-Calling ReAct Agents: Injection Depth, Payload Framing, and Turn-Budget Sensitivity
May 29, 2026Mohammadreza RashidiLLM Agent Evaluation AI Agent Security
Department of Computer Science AI and Media Analysis Lab Berlin, Germany
The Surface You Test Is Not the Surface That Breaks
May 28, 2026Syed Nazmus Sakib, Nafiul Haque, Shahrear Bin Amin +1LLM Agent Security AI Agent Security Benchmarks
Department of Robotics and Mechatronics Engineering, University of Dhaka · Department of Computer Science and Engineering, University of Dhaka
Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screening
May 27, 2026Mohan Zhang, Yuqi Jia, Zhen Tan +4LLM Security Indirect Prompt Injection
University of North Carolina at Chapel Hill · Duke University · Arizona State University +2
Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals
May 26, 2026Akindoyin Akinrele, Shreyank N GowdaLanguage Model Safety Evaluation Prompt Injection Attacks on LLMs
School of Computer Science, University of Nottingham, Jubilee Campus, Wollaton Road, Nottingham, NG8 1BB, United Kingdom
IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization
May 23, 2026Zixuan Chen, Jiaxiang Chen, Li Luo +4LLM Agent Security Indirect Prompt Injection
1Shanghai Jiao Tong University · 2The University of Hong Kong
Poisoning the Watchtower: Prompt Injection Attacks Against LLM-Augmented Security Operations Through Adversarial Log Content
May 23, 2026Rohan Pandey, Archit BhujangIndirect Prompt Injection LLMs for Cybersecurity
DigitalOcean · Arizona State University
Blind Spots in the Guard: How Domain-Camouflaged Injection Attacks Evade Detection in Multi-Agent LLM Systems
May 21, 2026Aaditya PaiMulti-Agent LLM Systems AI Agent Security
Data Science Institute Columbia University
LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent Security AI Agent Safety
University of Pennsylvania
ESLD (External Surrogate Latent Defense): A Latent-Space Architecture for Faster, Stronger Prompt-Injection Defense
May 18, 2026Yash NarendraAI Agent Security Safety Filtering
Microsoft
AI Agents May Always Fall for Prompt Injections
May 17, 2026Sahar Abdelnabi, Eugene BagdasarianContextual Integrity AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center · University of Massachusetts Amherst
ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
May 17, 2026Udari Madhushani Sehwag, Zhengyang Shan, Heming Liu +3LLM Agent Security AI Agent Security Benchmarks
Scale AI · 2Boston University · 4Human Frontier Collective +3
CALYREX: Cross-Attention LaYeR EXtended Transformers for System Prompt Anchoring
May 10, 2026Li LixingSelf-Attention Cross-Attention
Cornell University Ithaca, NY 14853
AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents
May 10, 2026Yassin H. Rassul, Tarik A. RashidLLM Agent Security AI Agent Monitoring
Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewlêr, Erbil, Iraq · Artificial Intelligence and Innovation Centre, University of Kurdistan Hewlêr, Erbil, Iraq
Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents
May 8, 2026Jun Wen LeongLanguage Model Safety Evaluation AI Agent Security
MIPIAD: Multilingual Indirect Prompt Injection Attack Defense with Qwen -- TF-IDF Hybrid and Meta-Ensemble Learning
May 8, 2026Al Muhit Muhtadi, Mostafa Rifat TazwarPrompt Injection Defense Prompt Injection Attacks on LLMs
Bangladesh University of Engineering and Technology
LoopTrap: Termination Poisoning Attacks on LLM Agents
May 7, 2026Huiyu Xu, Zhibo Wang, Wenhui Zhang +4Adversarial Attacks on LLMs LLM Red Teaming
The State Key Laboratory of Blockchain and Data Security Zhejiang University Hangzhou, Zhejiang, P. R. China · School of Cyber Science and Engineering Southeast University Nanjing, Jiangsu, P. R. China
VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models
May 2, 2026Pang Liu, Yingjie LaoVLM Evaluation VLM Robustness
Department of Electrical and Computer Engineering, Tufts University
A Sentence Relation-Based Approach to Sanitizing Malicious Instructions
May 1, 2026Soumil Datta, Melissa Umble, Daniel S. Brown +1Prompt Injection Defense RAG Security
University of Utah
CleanBase: Detecting Malicious Documents in RAG Knowledge Databases
May 1, 2026Weifei Jin, Xilong Wang, Wei Zou +2RAG Poisoning Attacks RAG Security
Duke University · The Pennsylvania State University
Latent Adversarial Detection: Adaptive Probing of LLM Activations for Multi-Turn Attack Detection
Apr 30, 2026Prashant KulkarniPrompt Injection Attacks on LLMs Adversarial Example Detection
Mountain View, CA
Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models
Apr 27, 2026Nay Myat Min, Long H. Pham, Jun SunLLM Backdoor Attacks Adversarial Attacks on LLMs
Singapore Management University
Evaluation of Prompt Injection Defenses in Large Language Models
Apr 26, 2026Priyal Deep, Shane Emmons, Amy Fox +4Data Leakage Privacy Leakage in Language Models
Swept AI[1] · University of Michigan
RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agents
Apr 24, 2026Wenjie Xiao, Xuehai Tang, Biyu Zhou +2LLM Agent Security Prompt Injection Attacks on LLMs
University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences
Beyond Pattern Matching: Seven Cross-Domain Techniques for Prompt Injection Detection
Apr 20, 2026Thamilvendhan MunirathinamPrompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents
Independent Researcher · prompt-shield project
SafeAgent: A Runtime Protection Architecture for Agentic Systems
Apr 19, 2026Hailin Liu, Eugene Ilyushin, Jie Ni +1Adversarial Attacks on LLMs AI Agent Security
Lomonosov Moscow State University · Central University
Kill-Chain Canaries: Stage-Level Tracking of Prompt Injection Across Attack Surfaces and Five Production LLMs
Mar 30, 2026Haochuan Kevin Wang, Zechen ZhangMulti-Agent LLM Systems Language Model Safety Evaluation
Massachusetts Institute of Technology · University of Chicago
Prompt Injection as Role Confusion
Feb 22, 2026Charles Ye, Jasmine Cui, Dylan Hadfield-MenellAdversarial Attacks on LLMs Prompt Injection Attacks
*Equal contribution 1Independent · 2Massachusetts Institute of Technology, Cambridge, MA, United States

Jun 1, 2026, cs.CRJ/K move · Enter open · S save

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

Hiskias Dingeto, William Leeney

StackOne Technologies

LLM Red Teaming LLM Agent Security AI Agent Security Benchmarks Indirect Prompt Injection Prompt Injection Defense Prompt Injection Attacks on LLMs+4

Open paper AskPDF arXiv

Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls. Existing benchmarks under-measure the threat: most cover only a handful of integrations with the same attack payload replayed across runs, and open-source guards are trained on chat-style data rather than tool-response content. We introduce AGENTREDBENCH, a dynamic LLM-driven redteaming benchmark of 215 subtle underspecified authorization (attacks at the boundary of what the user's request authorises) scenarios across 24 enterprise integrations in nine functional families and five attack types. Across an eight-model panel (Anthropic, OpenAI, Google), no-guard ASR (attack success rate) ranges from 32% (Claude Sonnet 4.6) to 81% (Gemini 3 Flash). To keep the scenario set out of training corpora and preserve headline ASR meaning over time, we release the codebase, integration schemas, and AGENTREDGUARD model openly; the canonical scenarios are evaluated through a maintainer-mediated channel with immutable versioning. We release AGENTREDGUARD alongside the benchmark: a guard trained on an integration-diverse corpus of adversarial tool-response content. AGENTREDGUARD cuts panel ASR from 69.9% to 2.4% at 0.37% false-positive rate, outperforming every open-source baseline with non-trivial detection (Llama Guard, PromptGuard 2, ProtectAI) on both axes. Cross-integration and cross-attack type holdouts both confirm the gain transfers beyond the training subset.

Momentum

Related topics

Latest papers 95