Prompt Injection Attacks on LLMs

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 367% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 95

All topics

Secure Speculative Decoding for Large Language Models
Oct 6, 2026Yichi Zhang, Zhiqi Wang, Neil Gong +1Adversarial Attacks on LLMs Speculative Decoding
The Pennsylvania State University · Duke University
RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems
Oct 6, 2026Niveen O. Jaffal, Ahmet Yuksel, David MohaisenRAG Security Prompt Injection Attacks on LLMs
Birzeit University, Birzeit, Palestine · Independent Researcher · University of Central Florida, Orlando, FL, USA
RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection Defense LLM Agents
LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1
Can CaMeLs Talk? Securing Multi-Agent Systems Against Indirect Prompt Injection Attacks
Oct 5, 2026James Peters-Gill, Avi Semler, Henning Bartsch +2AI Agent Security Benchmarks Information Flow Control
Independent · University of Oxford · MATS Research +1
Readable Before Actionable: Causal Tracing of Indirect Prompt Injection
Oct 4, 2026Zhe Yu, Wenpeng Xing, Xingxing Yang +1Causal Reasoning in Language Models Indirect Prompt Injection
Zhejiang University · Binjiang Institute of Zhejiang University · Hong Kong Baptist University
Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection
Oct 4, 2026Jingkai Liu, Yufei Han, Xiaoting Lyu +2Prompt Injection AI Agent Auditing
Beijing Jiaotong University · Mohamed bin Zayed University of Artificial Intelligence · INRIA Rennes-Bretagne-Atlantique +1
Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?
Sep 29, 2026Michael Lee, Zhipeng Wei, Yue Dong +1Indirect Prompt Injection Prompt Injection Attacks
International Computer Science Institute · DSO National Laboratories · UC Riverside +1
CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering
Sep 29, 2026Mark RussinovichLLM Agent Security Indirect Prompt Injection
Microsoft Azure
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection
Sep 28, 2026Yan Zhan, Yunze Song, Mengkai Hou +3Prompt Injection Prompt Injection Defense
Peking University · National University of Singapore · BYD Company Limited +1
CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents
Sep 28, 2026Xiao Yang, Yangchen Ou, Yuhan Gao +3Adversarial Training LLM Agent Security
School of Computer Science and Engineering, Beihang University · State Key Laboratory of Complex & Critical Software Environment, Beihang University
Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning
Sep 27, 2026Chenlong Yin, Xiaolong Jin, Wei Zou +2Adversarial Attacks on LLMs LLM Red Teaming
The Pennsylvania State University · Purdue University
ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation
Sep 24, 2026Qingyu Wu, Zeyu Feng, Yongda Yu +2Adversarial Prompt Generation Prompt Injection Attacks
Defense Innovation Institute, Academy of Military Science, Beijing, China · Nanjing University, Nanjing, China
Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs
Sep 24, 2026Lukáš Brůna, Robert Bridges, Adam EkAdversarial Attacks on LLMs Prompt Injection Attacks
Uppsala University · AI Sweden
ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents
Sep 14, 2026Bingzheng Wang, Xiaoyan Gu, Wentao Wang +3Runtime Enforcement for AI Agents Prompt Injection Defense
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Key Laboratory of Cyberspace Security Defense, Beijing, China +3
DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents
Sep 12, 2026Asif Pinjari, Mithun Paul Saint-GermainAI Agent Security Benchmarks AI Agent Monitoring
School of Informatics, Computing, and Cyber Systems Northern Arizona University Flagstaff, AZ, USA
An Experimental Evaluation of Multimodal Prompt Injection Attacks on Agentic AI Frameworks
Sep 10, 2026Viet K. Nguyen, Mohammad I. HusainAI Agent Security AI Agent Security Benchmarks
California State Polytechnic University, Pomona Pomona, CA 91768, USA
CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement
Sep 7, 2026Boyang Zhang, Qingxin Xiao, Lingwei Dang +1Reinforcement Learning AI Agent Security
School of Software Engineering, South China University of Technology
HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation
Sep 1, 2026Nikita Oblakov, Sabrina Sadiekh, Evgeniy KokuykinLanguage Model Safety Evaluation LLM Guardrails
HiveTraceLab
Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents
Aug 31, 2026Yunseok Lee, Yunji Kim, Woojin LeeLLM Agent Security AI Agent Security Benchmarks
Dongguk University-Seoul
Stealing Reasoning Traces from Proprietary LLM APIs
Aug 10, 2026Alexander Panfilov, David Schmotz, Ilia Shumailov +5Model Extraction Attacks Adversarial Attacks
MATS Research · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +4
Toward Metacognitive One-Shot Indirect Prompt Injection: Strategy Abstraction Via Outcome-Conditioned Reflection
Aug 9, 2026Sihan Hou, Xinmeng Hou, Zhijun Zhang +5AI Agent Security Benchmarks Indirect Prompt Injection
1Nankai University · 2Nanyang Technological University · 3Technical Lead, R&D, Tianjin 712 Mobile Communication Co., Ltd. +1
BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes
Aug 8, 2026Laiqiao Qin, Tianqing Zhu, Longxiang Gao +1Prompt Injection Defense Prompt Injection Attacks on LLMs
City University of Macau, Macau · Qilu University of Technology
The Anatomy of a Prompt Injection: A Component Model for Structured Analysis
Aug 7, 2026Jeremy McHughLLM Security Prompt Injection Attacks
Preamble, Inc.
NiyamAI - An Intent-Bound AI Agent with Cryptographically Verifiable Guardrails using Zero-Knowledge Proofs
Aug 7, 2026Aditya Katkar, Om Karkele, Kartik Mandhane +2Tool-Augmented Language Model Agents LLM Guardrails
Department of Computer Engineering, Vishwakarma Institute of Technology, Pune, Maharashtra, India
StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection
Aug 6, 2026Zhuoxin Zhan, Akbar Rafiey, Avery Ma +2Computer-Use Agents AI Agent Safety
Simon Fraser University · RBC Borealis · New York University
Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots
Aug 6, 2026S. M . Bhagya P. Samarakoon, M. A. Viraj J. Muthugala, W. K. R. Sachinthana +1VLM Robustness Adversarial Attacks on VLMs
Engineering Product Development Pillar · Singapore University of Technology and Design · Somapah Rd, Singapore 487372
Robust Context-Aware Detection of Malicious Instructions in Text
Aug 5, 2026Buzhao Liu, Xinhang Ma, Yevgeniy VorobeychikAdversarial Training Prompt Injection Defense
Washington University in St. Louis
AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injection
Aug 4, 2026Shihao Weng, Yang Feng, Xiaofei Xie +1Continual Learning for LLM Agents LLM Agent Security
Nanjing University · Singapore Management University · Nanyang Technological University
Your Agentic LLMs Secretly Encode Latent Signals of Indirect Prompt-Injection Exposure
Aug 1, 2026Jianshuo Dong, Yiming Liu, Maosen Zhang +6Indirect Prompt Injection Prompt Injection Defense
Tsinghua University · MatrixOrigin · Nanyang Technological University +1

Oct 6, 2026, cs.CRJ/K move · Enter open · S save

Secure Speculative Decoding for Large Language Models

Yichi Zhang, Zhiqi Wang, Neil Gong, Yuchen Yang

The Pennsylvania State University · Duke University

Adversarial Attacks on LLMs Speculative Decoding LLM Security LLM Jailbreak Attacks Prompt Injection Attacks on LLMs

Open paper AskPDF arXiv

Speculative decoding accelerates inference for a large language model (LLM), referred to as the \emph{target model}, by first using a smaller model, referred to as the \emph{draft model}, to generate candidate tokens and then verifying them with the target model for acceptance or rejection. Prior studies primarily focused on the efficiency-utility trade-off of speculative decoding, e.g., lossy speculative decoding, leaving its security implications largely unexplored. In this work, we bridge this gap by providing the \emph{first} systematic study of the security implications of speculative decoding. Through a large-scale measurement study, we reveal a pronounced security-utility asymmetry: across a wide range of lossy speculative decoding methods, improvements in inference efficiency come at a disproportionately high cost to security, with attack success rates for jailbreak and prompt injection attacks increasing much faster than utility degrades. We then propose SecureSD, a new theory-guided speculative decoding method that enhances security while maintaining efficiency and utility. Specifically, our theoretical analysis reveals that security degradation primarily originates from the early tokens generated by the draft model. Motivated by this insight, SecureSD applies a stricter verification criterion to draft-model tokens at early decoding positions. Extensive experiments on both security and utility benchmarks demonstrate that SecureSD significantly improves security while preserving efficiency and utility compared to existing speculative decoding methods.

Momentum

Related topics

Latest papers 95