Prompt Injection Attacks on LLMs

LLM: Large Language Model

Overview

Prompt injection studies how content supplied as data—such as retrieved documents, webpages, tool outputs, memories, images, and audio—can be treated as instructions and divert a model or agent from its intended task. Attack methods range from role and delimiter tricks to hidden or fragmented payloads, adaptive black-box search, gradient-based optimization, and reinforcement-learning red teaming; evaluations cover retrieval-augmented generation, web and computer-use agents, coding, and multi-turn tasks. Defenses include detection and sanitization, provenance-aware separation of trusted instructions from untrusted data, runtime action auditing, and model-level training or activation interventions. Results depend strongly on the model, input channel, and attacker adaptivity, leaving a central tension between blocking harmful actions and preserving useful content and authorized tasks; success on static benchmarks may not predict deployment robustness.

Latest papers

CardsList

131 papers9 papers in the last four weeks, down 55% on the four weeks before. 0.1% of all new papers.

  1. I Think, Therefore I Attack: Evaluating Thought Injection Attacks on Reasoning Agents

    Nov 6, 2026Parth Illendula, Garik Livingston, Zain Shamsi +2Prompt Injection Attacks on LLMsPrompt Injection Attacks on AI Agents

  2. Understanding and Mitigating Role Spoofing in LLMs

    Oct 29, 2026Ezra Edelman, Benjamin Edelman, Surbhi Goel +1Adversarial Attacks on LLMsPrompt Injection Attacks on LLMs

  3. CLIC: Closed-Loop Intent Consistency for Securing LLM-Driven Control in Open RAN

    Oct 29, 2026Sunder Ali Khowaja, Kapal Dev, Joongheon Kim +2Language Model-Based ControlPrompt Injection Attacks on LLMs

  4. REALM: Identifying Minimal Task Assets for LLM Agents

    Oct 29, 2026Zihou Ren, Byungchul Tak, Long WangAI Agent SecurityTool Access Control for LLM Agents

  5. Post-Training Large Language Models with LLMON

    Oct 28, 2026Basel Shbita, Michael Hind, Chad DeLuca +8LLM Post-TrainingPrompt Injection Defense

  6. Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

    Oct 27, 2026Igor Shilov, Xiaoxue Yang, Alexander Panfilov +6LLM SecurityPrompt Injection Defense

  7. SoK: PIAtlas: Decomposing the Unbounded Prompt Injection Attack Space

    Oct 18, 2026Stavros Damianakis, Konstantinos PsounisPrompt Injection AttacksPrompt Injection Attacks on LLMs

  8. StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

    Oct 12, 2026Anning Hu, Yangyu Li, Yuqi Huang +2LLM-as-a-JudgePrompt Injection Defense

  9. LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense

    Oct 8, 2026Luman Zhao, Minghui Xu, Yue Zhang +1LLM SecurityPrompt Injection Defense

  10. Secure Speculative Decoding for Large Language Models

    Oct 6, 2026Yichi Zhang, Zhiqi Wang, Neil Gong +1Adversarial Attacks on LLMsSpeculative Decoding

  11. RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems

    Oct 6, 2026Niveen O. Jaffal, Ahmet Yuksel, David MohaisenRAG SecurityPrompt Injection Attacks on LLMs

  12. RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

    Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection DefenseLLM Agents

  13. Can CaMeLs Talk? Securing Multi-Agent Systems Against Indirect Prompt Injection Attacks

    Oct 5, 2026James Peters-Gill, Avi Semler, Henning Bartsch +2AI Agent Security BenchmarksInformation Flow Control

  14. Readable Before Actionable: Causal Tracing of Indirect Prompt Injection

    Oct 4, 2026Zhe Yu, Wenpeng Xing, Xingxing Yang +1Causal Reasoning in Language ModelsIndirect Prompt Injection

  15. Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection

    Oct 4, 2026Jingkai Liu, Yufei Han, Xiaoting Lyu +2Prompt InjectionAI Agent Auditing

  16. Large Language Models and Augmented Democracy

    Oct 3, 2026Jairo Gudiño-RoseroDigital TwinsLarge Language Model-Based Social Simulation

  17. Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?

    Sep 29, 2026Michael Lee, Zhipeng Wei, Yue Dong +1Indirect Prompt InjectionPrompt Injection Attacks

  18. Render Before Reading: Visual Rendering as a Prompt Injection Defense

    Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language ModelsModality Gap in VLMs

  19. Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection

    Sep 28, 2026Yan Zhan, Yunze Song, Mengkai Hou +3Prompt InjectionPrompt Injection Defense

  20. CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents

    Sep 28, 2026Xiao Yang, Yangchen Ou, Yuhan Gao +3Adversarial TrainingLLM Agent Security

  21. Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning

    Sep 27, 2026Chenlong Yin, Xiaolong Jin, Wei Zou +2Adversarial Attacks on LLMsLLM Red Teaming

  22. ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation

    Sep 24, 2026Qingyu Wu, Zeyu Feng, Yongda Yu +2Adversarial Prompt GenerationPrompt Injection Attacks

  23. Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs

    Sep 24, 2026Lukáš Brůna, Robert Bridges, Adam EkAdversarial Attacks on LLMsPrompt Injection Attacks

  24. ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents

    Sep 14, 2026Bingzheng Wang, Xiaoyan Gu, Wentao Wang +3Runtime Enforcement for AI AgentsPrompt Injection Defense