Prompt Injection Defense

Follow this topic Ask about this topic

Latest papers 101

17 papers in the last four weeks, up 113% on the four weeks before. 0.1% of all new papers.

All topics

StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge
Oct 12, 2026Anning Hu, Yangyu Li, Yuqi Huang +2LLM-as-a-Judge Prompt Injection Defense
Shanghai Jiao Tong University
Only What You've Signed up For: Secure Workflows with Constrained Diffusion Agents
Oct 10, 2026Michael Cardei, Anjun Hu, Chenrui Ma +4AI Agent Security Prompt Injection Defense
University of Virginia · University of Oxford · Google DeepMind +1
LTBD: Learnable Trust-Boundary Delimiters for Prompt Injection Defense
Oct 8, 2026Luman Zhao, Minghui Xu, Yue Zhang +1LLM Security Prompt Injection Defense
Shandong University, Shandong Academy of Artificial Intelligence
RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
Oct 5, 2026Mohamed Dhouib, Clement Elliker, Alexi Canesse +5Prompt Injection Defense LLM Agents
LIX, École polytechnique, Institut Polytechnique de Paris, CNRS · AMIAD (Agence Ministérielle pour l’IA de Défense) · IRT SystemX +1
Readable Before Actionable: Causal Tracing of Indirect Prompt Injection
Oct 4, 2026Zhe Yu, Wenpeng Xing, Xingxing Yang +1Causal Reasoning in Language Models Indirect Prompt Injection
Zhejiang University · Binjiang Institute of Zhejiang University · Hong Kong Baptist University
APEX: Active Protection at Execution Boundaries for LLM Agents
Oct 3, 2026Xinran Zheng, Xin Fan Guo, Zhiqiang Hao +7AI Agent Security Prompt Injection Defense
Tsinghua University · Imperial College London · Nanjing University +6
Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
Oct 3, 2026Zixuan Wang, Hao Li, Fengyu Gao +5LLM Agent Safety Prompt Injection Defense
Johns Hopkins University · Washington University in St. Louis · University of Virginia +2
ActionGuard: Tool Call Authorization under Poisoned Skills
Sep 30, 2026Jihun Han, Yejin Jang, Byung Il Kwak +1Runtime Enforcement for AI Agents Prompt Injection Defense
Korea University Republic of Korea
ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents
Sep 29, 2026Yanjie Li, Xiangyu He, Xuelong Dai +1Runtime Enforcement for AI Agents Prompt Injection Defense
Hong Kong Polytechnic University · Shandong University
CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering
Sep 29, 2026Mark RussinovichLLM Agent Security Indirect Prompt Injection
Microsoft Azure
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection
Sep 28, 2026Yan Zhan, Yunze Song, Mengkai Hou +3Prompt Injection Prompt Injection Defense
Peking University · National University of Singapore · BYD Company Limited +1
CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents
Sep 28, 2026Xiao Yang, Yangchen Ou, Yuhan Gao +3Adversarial Training LLM Agent Security
School of Computer Science and Engineering, Beihang University · State Key Laboratory of Complex & Critical Software Environment, Beihang University
Ajar: Measuring Open Privilege in Agent Defenses
Sep 22, 2026Reshabh K Sharma, Linxi Jiang, Shuo Chen +1Tool Access Control for LLM Agents AI Agent Auditing
University of Washington Seattle, Washington, USA · The Ohio State University Columbus, Ohio, USA · Microsoft Research Redmond, Washington, USA
Decoding Guardrails: XAI-Guided Perturbation Analysis of Prompt Injection Detection
Sep 21, 2026Fernando Outeda, Gustavo Betarte, Juan Diego Campo +1LLM Guardrails Prompt Injection Defense
Pedeciba Informática, Uruguay · InCo, Facultad de Ingeniería, Universidad de la República · Departamento de Informática e Inteligencia Artificial, Universidad Católica del Uruguay
Beyond Single-Model Injection: A Threat Model and Defense Architecture for Prompt Injection in Multi-Agent Systems
Sep 19, 2026Rudrendu Kumar Paul, Sourav NandyPrompt Injection Defense Prompt Injection Attacks on AI Agents
Boston University, Boston, MA, USA · University of Texas at Austin, Austin, TX, USA
Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents
Sep 18, 2026Justin Szczepaniak, Elad Feldman, Naum Viner +1Prompt Injection Attacks Prompt Injection Attacks on AI Agents
Tel Aviv University
ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents
Sep 14, 2026Bingzheng Wang, Xiaoyan Gu, Wentao Wang +3Runtime Enforcement for AI Agents Prompt Injection Defense
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Key Laboratory of Cyberspace Security Defense, Beijing, China +3
Architecting the Secure AI-SOC: A Neurosymbolic Framework for Pipeline Integrity and Threat Mitigation
Sep 9, 2026Anna Gazani, Spyridon Kounoupidis, Panagiotis Katsaros +3Cybersecurity Indirect Prompt Injection
Aristotle University of Thessaloniki, Greece · Clone Systems, Cyprus
CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement
Sep 7, 2026Boyang Zhang, Qingxin Xiao, Lingwei Dang +1Reinforcement Learning AI Agent Security
School of Software Engineering, South China University of Technology
LongPIBench: A Long-Context Benchmark for Prompt Injection
Aug 28, 2026Yupei Liu, Yuqi Jia, Neil Zhenqiang Gong +1Prompt Injection Prompt Injection Attacks
The Pennsylvania State University · Duke University
CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?
Aug 28, 2026Zi Liang, Xiaoyu Xu, Yanyun Wang +3Prompt Injection Defense AI Agent Security
The Hong Kong Polytechnic University · The Chinese University of Hong Kong
The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detection
Aug 26, 2026Jaturong Kongmanee, Smile ThanapattheerakulInterpretable ML Prompt Injection Defense
Research Incubation, TrendAI™Research
Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors
Aug 24, 2026Joshua PenmanLLM Security Prompt Injection Defense
Mitigating Database Leakage in RAG Systems with Keyword-Grounded Fact Substitution
Aug 21, 2026Ziliang Zhang, Yubo Zhu, Wei Tong +4RAG Security Prompt Injection Defense
Nanjing University
SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation
Aug 21, 2026Yibo Peng, Long Lian, David Wagner +1Prompt Injection Defense AI Agent Security
University of California, Berkeley
Defending Retrieval-Augmented Intrusion Detection Against Knowledge Poisoning and Prompt Injection
Aug 8, 2026Kaysarul Anas Apurba, Md. Hasibul Hasan, Mahedee Zaman Moon +2Retrieval-Augmented Classification RAG Poisoning Attacks
Laurentian University Ontario, Canada · Centennial College Ontario, Canada · The University of Osaka Osaka, Japan
BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes
Aug 8, 2026Laiqiao Qin, Tianqing Zhu, Longxiang Gao +1Prompt Injection Defense Prompt Injection Attacks on LLMs
City University of Macau, Macau · Qilu University of Technology
NiyamAI - An Intent-Bound AI Agent with Cryptographically Verifiable Guardrails using Zero-Knowledge Proofs
Aug 7, 2026Aditya Katkar, Om Karkele, Kartik Mandhane +2Tool-Augmented Language Model Agents LLM Guardrails
Department of Computer Engineering, Vishwakarma Institute of Technology, Pune, Maharashtra, India
Robust Context-Aware Detection of Malicious Instructions in Text
Aug 5, 2026Buzhao Liu, Xinhang Ma, Yevgeniy VorobeychikAdversarial Training Prompt Injection Defense
Washington University in St. Louis

Oct 12, 2026J/K move · Enter open · S save

StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

Anning Hu, Yangyu Li, Yuqi Huang, Shengchao Hu+1

Shanghai Jiao Tong University

LLM-as-a-Judge Prompt Injection Defense Adversarial Robustness of Language Models Prompt Injection Attacks on LLMs

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as paper review and recommendation. However, LLM-as-a-Judge pipelines are vulnerable to score-hijacking prompt injection embedded in evaluated content, which corrupts scoring signals and distorts induced rankings. Existing defenses often enforce robustness by modifying the judge model itself, making safety dependent on a specific model's scale and training and requiring repeated effort when the judge is updated or replaced. We propose StableJudge, a lightweight defense framework that externalizes prompt-injection mitigation as a reusable pre-scoring capability. StableJudge centers on a compact 0.13B Sentry that localizes injected spans and performs targeted sanitization prior to evaluation, and is trained using adversarially evolving attacks to improve robustness beyond a fixed prompt set. To stabilize measurement, StableJudge derives deterministic, fine-grained scalar scores from the judge's output distribution. Across held-out scientific abstracts and out-of-distribution attack variants, StableJudge improves score recovery under attack while largely preserving general-purpose performance, demonstrating that robust LLM-based judging can be achieved by decoupling safety from judge scale via reusable defense capabilities. Code and data will be released.