Adversarial Attacks on LLMs

LLM: Large Language Model

This field studies how attackers can make large language models produce unsafe or unreliable outputs, leak information, manipulate judgments, or misuse tools. Common approaches include black-box prompt search and multi-turn red teaming, gradient-based suffix optimization, reinforcement-learning attackers, and interventions on model activations or weights. Researchers evaluate these attacks on benchmarks such as HarmBench, AdvBench, and JailbreakBench, as well as in multilingual, multimodal, and tool-using agent settings. A central challenge is that adaptive attacks often expose weaknesses missed by static tests, while defenses must improve robustness without excessive false refusals, lost capability, or new system-level vulnerabilities.

Latest papers

358 papers51 papers in the last four weeks, up 89% on the four weeks before. 0.3% of all new papers.

Understanding and Mitigating Role Spoofing in LLMs
Oct 29, 2026Ezra Edelman, Benjamin Edelman, Surbhi Goel +1Adversarial Attacks on LLMs Prompt Injection Attacks on LLMs
University of Pennsylvania · Princeton University · Apple
Adversarial Self-Distillation
Oct 28, 2026Mohammad Sadegh Akhondzadeh, Soroush H. Zargarbashi, Aleksandar BojchevskiAdversarial Robustness of Language Models Safety-Preserving Fine-Tuning of Language Models
University of Cologne · CISPA Helmholtz Center for Information Security
SEMA-Auto: Sighted and Efficient Multi-Turn Attack with Autonomy
Oct 26, 2026Mingqian Feng, Xiaodong Liu, Weiwei Yang +8Adversarial Attacks on LLMs Jailbreak Attacks
University of Rochester · Microsoft Research · Microsoft Azure
Do AI-for-Science Agents Preserve Model Safety?
Oct 23, 2026Michael Yew Chuan Lee, Manya Sriram, N. Benjamin ErichsonLLM Agent Safety LLM Safety Evaluation
DSO National Laboratories · UC Berkeley · ICSI and Berkeley Lab
$M^3$ : A Multi-LLM System for Multi-Turn Jailbreaking as a Markov Decision Process
Oct 22, 2026Yuqing Huang, Jack Ji, Jithin Pradeep +2Adversarial Attacks on LLMs Multi-Turn Jailbreak Attacks
The Vanguard Group∗
Diffusion Language Models for Perplexity-Aware Black-Box Attacks
Oct 22, 2026Ali Al-Lawati, Elsayed Eshra, Dongwon Lee +1Adversarial Attacks on LLMs LLM Safety Evaluation
The Pennsylvania State University, USA
AgentPhishing: A Benchmark of LLM Email Agents Under Adaptive Attack
Oct 22, 2026Justin Tan, Miao Li, Ding ZhaoLLM Agent Security AI Agent Security Benchmarks
Carnegie Mellon University
Fooling LLM Safety Guards into Judging the Wrong Text
Oct 20, 2026Utkarsh KanwatLLM Guardrails Adversarial Attacks on LLMs
Independent Researcher
After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents
Oct 20, 2026Zhiyue Xu, Xukai Wang, Zhiyuan Huang +3Adversarial Attacks on LLMs Prompt Injection Attacks on AI Agents
MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artiﬁcial Intelligence, UCAS · Zhongguancun Academy +2
Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing
Oct 20, 2026Diego Miguel Lozano, Erik Derner, Juan Antonio Pérez-Ortiz +1Language Model Safety Evaluation Adversarial Robustness of Language Models
ELLIS Alicante, Spain · Czech Technical University in Prague, Czech Republic · Universidad de Alicante, Spain
Global Security Blind Spots: Translation as a Safety Bypass in Low-Resource Languages
Oct 19, 2026Jalil Alva, David Harper, Abigail E. Thornton +8LLM Safety Language Model Safety Evaluation
Welo Data
Fake News, Real Crashes: Information Injection Attacks on LLM Financial Trading Agents
Oct 19, 2026Mengxiao Wang, Nitesh SaxenaAdversarial Attacks on LLMs AI Agent Security
Texas A&M University
The One-Pixel of Reasoning: One Truthful Relation Is Enough to Mislead LLM Agents
Oct 17, 2026Jinwei Hu, Youcheng Sun, Yi Dong +1Adversarial Attacks on LLMs LLM Agent Reliability
School of Computer Science and Informatics, University of Liverpool · Department of Computer Science, MBZUAI
Pair-Aware Optimization: Coordinating Exploration and Refinement for Jailbreaking LLM Agents
Oct 17, 2026Juanyang Xu, Zheng Wang, Xingyu Zhao +2Adversarial Attacks on LLMs Jailbreak Attacks
University of Macau · WMG, University of Warwick · Wuhan University
LLM Trojans that Increase Response Lengths and Cost
Oct 17, 2026Zhengxing Li, Guangmingmei Yang, Nader Alfares +2LLM Backdoor Attacks Adversarial Attacks on LLMs
Departments of Electrical Engineering and Computer Science & Engineering Penn State University University Park, PA 16802, USA
S-GAP: Stealthy Graph Attack Primitives for Graph Large Language Models
Oct 16, 2026Wenda Li, Tongya Zheng, Shunyu Liu +8Adversarial Attacks on LLMs LLMs for Graph Learning
1Zhejiang University · 2Hangzhou City University · 3Nanyang Technological University +1
Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs
Oct 14, 2026Adel Khorramrouz, Yi-Chia Wang, Sharon LevyAdversarial Attacks on LLMs LLM Safety Evaluation
Rutgers University · Carnegie Mellon University
Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers
Oct 13, 2026Xiang Li, Aichun Huang, Mingming Zhang +1Phishing Detection Language Model Safety Evaluation
Nankai University · Zhongguancun Laboratory
Narrative Paradoxes in LLM Inference: Shaping Reasoning Trajectories toward Misalignment
Oct 12, 2026Haoming Yang, Ke Ma, Yangbangyan Jiang +6Adversarial Attacks on LLMs LLM Safety
School of Electronic, Electrical and Communication Engineering, UCAS · School of Computer Science and Technology, UCAS · Nanyang Technological University, Singapore +3
REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming
Oct 11, 2026Abrar Alotaibi, Moataz AhmedAdversarial Prompt Generation Adversarial Attacks on LLMs
King Fahd University of Petroleum & Minerals, Information and Computer Science Department, Dhahran, Saudi Arabia · Imam Abdulrahman Bin Faisal University, College of Computer Science and Information Technology, Dammam, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia
Security in LLM-Assisted Coding: A Systematic Literature Review
Oct 11, 2026Liang Yuan, Maryam Khanian Najafabadi, Mashud Rana +1LLM Security Software Security
Australian Catholic University, North Sydney, New South Wales, Australia
Attention Sees, MLP Refuses: Tracing the Attention-to-MLP Safety Pathway in LLMs
Oct 9, 2026Zixuan Weng, Haoqi He, Weifei Jin +2Refusal Behavior in Language Models Language Model Safety Evaluation
Georgia Institute of Technology · Independent Researcher · Duke University +1
One Word Opens the Gate: The Option-Channel Attack on Typed Decision Models as Agent Guardrails
Oct 8, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLLM Guardrails LLM Agent Security
University of Southern California (USC), USA
BRANCH: Bypassing Multi-Scanner AI Guardrails
Oct 7, 2026William Hackett, Peter GarraghanLLM Guardrails Adversarial Attacks on LLMs
Mindgard
Safe at One Loop, Risky at Another: Aligning Safety Across Recurrent Depths in Looped Language Models
Oct 7, 2026Yi Wang, Xiuyuan Qi, Dongqi Han +2LLM Safety Alignment Language Model Safety Evaluation
ShanghaiTech University · Microsoft Research Asia
Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
Oct 6, 2026Pavan MaddulaLanguage Model Safety Evaluation Adversarial Robustness of Language Models
Independent Researcher
How Fragile Is On-Device Language Model Safety? Localizing Safety-Critical Parameters for Sparse Fault Analysis
Oct 6, 2026Muhammad Zeeshan Karamat, Christiana Chamon GarciaLLM Safety Language Model Safety Evaluation
Virginia Tech
Secure Speculative Decoding for Large Language Models
Oct 6, 2026Yichi Zhang, Zhiqi Wang, Neil Gong +1Adversarial Attacks on LLMs Speculative Decoding
The Pennsylvania State University · Duke University
Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge
Oct 5, 2026Wonjun Lee, Kyungsik Yang, Gaeun Ji +5Adversarial Attacks on LLMs LLM Security
UNC Chapel Hill · Kyung Hee University · AIM Intelligence +1
Jailbreaking Open-Weight LLMs via Random Embedding Perturbations
Oct 5, 2026Abhinav Sudhakar Dubey, Scott Sirri, Vaggos Chatziafratis +1Adversarial Attacks Adversarial Attacks on LLMs
University of California Santa Cruz

Oct 29, 2026J/K move · Enter open · S save

Understanding and Mitigating Role Spoofing in LLMs

Ezra Edelman, Benjamin Edelman, Surbhi Goel, Eran Malach

University of Pennsylvania · Princeton University · Apple

Adversarial Attacks on LLMs Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.