Adversarial Attacks on LLMs

LLM: Large Language Model

Ask about this topic

Overview

This field studies how attackers can make large language models produce unsafe or unreliable outputs, leak information, manipulate judgments, or misuse tools. Common approaches include black-box prompt search and multi-turn red teaming, gradient-based suffix optimization, reinforcement-learning attackers, and interventions on model activations or weights. Researchers evaluate these attacks on benchmarks such as HarmBench, AdvBench, and JailbreakBench, as well as in multilingual, multimodal, and tool-using agent settings. A central challenge is that adaptive attacks often expose weaknesses missed by static tests, while defenses must improve robustness without excessive false refusals, lost capability, or new system-level vulnerabilities.

Latest papers

CardsList

358 papers51 papers in the last four weeks, up 89% on the four weeks before. 0.3% of all new papers.

  1. ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Covering

    Mar 14, 2026Hussein Jawad, Nicolas J-B BrunelLLM Agent SecurityAdversarial Attacks on LLMs

    Capgemini Invent, Paris, France · LaMME, University Paris-Saclay, Evry, France · ENSIIE, Evry, France

  2. Colluding LoRA: A Compositional Vulnerability in LLM Safety Alignment

    Mar 13, 2026Sihao DingLLM SecurityLLM Safety Alignment

    Mercedes-Benz Research & Development North America, USA

  3. Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover

    Mar 11, 2026Indranil Halder, Annesya Banerjee, Cengiz PehlevanAdversarial Attacks on LLMsPrompt Injection Attacks on LLMs

    John A. Paulson School of Engineering And Applied Sciences, Harvard University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology · Speech and Hearing Bioscience and Technology, Harvard Medical School +2

  4. Multi-Stream Perturbation Attack: Breaking Safety Alignment of Thinking LLMs Through Concurrent Task Interference

    Mar 10, 2026Fan YangAdversarial Attacks on LLMsLLM Jailbreak Attacks

    Jinan University

  5. Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models

    Mar 10, 2026Ali Raza, Gurang Gupta, Nikolay Matyunin +1Adversarial Attacks on LLMsLLM Jailbreak Attacks

    Honda Research Institute Europe Offenbach, Hesse, Germany · IIT KharagPur Kharagpur, India

  6. ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models

    Mar 10, 2026Harry Owiredu-AshleyLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    Independent Researcher, New Jersey, United States

  7. Invisible Safety Threat: Malicious Finetuning for LLM via Steganography

    Mar 9, 2026Guangnian Wan, Xinyin Ma, Gongfan Fang +1Language Model SteganographyLLM Security

    National University of Singapore

  8. Intentional Deception as Controllable Capability in LLM Agents

    Mar 8, 2026Jason Starace, Terence SouleDeception in Language ModelsLLM Agent Safety

    Department of Computer Science, University of Idaho, Moscow, ID 83844

  9. Targeted Bit-Flip Attacks on LLM-Based Agents

    Mar 7, 2026Jialai Wang, Ya Wen, Zhongmou Liu +4Adversarial Attacks on LLMsAI Agent Security

    National University of Singapore, Singapore · Tsinghua University, China · Huazhong University of Science and Technology, China +1

  10. Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

    Mar 6, 2026Jinman Wu, Yi Xie, Shiqian Zhao +1Adversarial Attacks on LLMsLLM Jailbreak Attacks

    Xidian University, Xi’an, China · Tsinghua University, Shenzhen, China · Nanyang Technological University, Singapore

  11. Efficient Refusal Ablation in LLM through Optimal Transport

    Mar 4, 2026Geraldin Nanfack, Eugene Belilovsky, Elvis DohmatobAdversarial Attacks on LLMsJailbreak Attacks

    Concordia University, Montreal, Canada · Mila – Quebec AI Institute, Montreal, Canada

  12. In-Context Environments Induce Evaluation-Awareness in Language Models

    Mar 4, 2026Maheep ChaudharyEvaluation Awareness in Language ModelsLLM Evaluation

    Independent

  13. TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models

    Mar 3, 2026Zhi Xu, Jiaqi Li, Xiaotong Zhang +2Adversarial Attacks on LLMsJailbreak Attacks

    Dalian University of Technology, Dalian, China

  14. Co-Evolutionary Multi-Modal Alignment via Structured Adversarial Evolution

    Mar 2, 2026Guoxin Shi, Haoyu Wang, Zaihui Yang +2Multimodal Multi-Turn Safety AlignmentAdversarial Training

    Nanyang Technological University · Tsinghua University

  15. Jailbreaking Embodied LLMs via Action-level Manipulation

    Mar 2, 2026Xinyu Huang, Qiang Yang, Leming Shen +2AI Agent SecurityLLM Security

    The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Cambridge, Cambridge, United Kingdom

  16. Silent Sabotage During Fine-Tuning: Few-Shot Rationale Poisoning of Compact Medical LLMs

    Feb 28, 2026Jingyuan Xie, Wenjie Wang, Ji Wu +1Adversarial Attacks on LLMsLLM Security

    Department of Electronics Engineering, Tsinghua University, Beijing, China · College of AI, Tsinghua University, Beijing, China · Beijing National Research Center for Information Science and Technology, Beijing, China

  17. Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence

    Feb 28, 2026Quoc Minh Nguyen, Trung Le, Jing Wu +2Safety-Preserving Fine-Tuning of Language ModelsLLM Fine-Tuning

    Department of Electrical and Computer Systems Engineering, Monash University, Australia · Department of Data Science and AI, Monash University, Australia

  18. Alignment-Weighted DPO: A principled reasoning approach to improve safety alignment

    Feb 24, 2026Mengxuan Hu, Vivek V. Datla, Anoop Kumar +4LLM AlignmentLLM Safety Alignment

    University of Virginia · Capital One

  19. OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services

    Feb 24, 2026Longxiang Wang, Xiang Zheng, Xuhao Zhang +3Privacy Leakage in Language ModelsLLM Security

    City University of Hong Kong · ByteDance Inc.

  20. Prompt Injection as Role Confusion

    Feb 22, 2026Charles Ye, Jasmine Cui, Dylan Hadfield-MenellAdversarial Attacks on LLMsPrompt Injection Attacks

    *Equal contribution 1Independent · 2Massachusetts Institute of Technology, Cambridge, MA, United States

  21. When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models

    Feb 21, 2026Zafir Shamsi, Nikhil Chekuru, Zachary Guzman +1Adversarial Attacks on LLMsLanguage Model Safety Evaluation

    Algoverse AI Research

  22. TFL: Targeted Bit-Flip Attack on Large Language Model

    Feb 19, 2026Jingkai Guo, Chaitali Chakrabarti, Deliang FanAdversarial AttacksAdversarial Attacks on LLMs

    School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA

  23. Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMs

    Feb 19, 2026Zachary Coalson, Bo Fang, Sanghyun HongAdversarial Attacks on LLMsLLM Interpretability

    Oregon State University, Corvallis, OR, USA · University of Texas at Arlington, Arlington, TX, USA

  24. Fail-Closed Alignment for Large Language Models

    Feb 19, 2026Zachary Coalson, Beth Sohler, Aiden Gabriel +1LLM AlignmentRefusal Behavior in Language Models

    Oregon State University, Corvallis, OR USA.

  25. DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMs

    Feb 18, 2026Justin Albrethsen, Yash Datta, Kunal Kumar +1LLM SafetyJailbreak Defense

  26. IndicJR: A Judge-Free Benchmark of Jailbreak Robustness in South Asian Languages

    Feb 18, 2026Priyaranjan Pattnayak, Sanchari ChowdhuriAdversarial Robustness of Language ModelsLLM Safety Benchmarks

    Oracle America Inc.

  27. Can Adversarial Code Comments Fool AI Security Reviewers -- Large-Scale Empirical Study of Comment-Based Attacks and Defenses Against LLM Code Analysis

    Feb 18, 2026Scott ThorntonAdversarial AttacksAdversarial Attacks on LLMs

  28. ER-MIA: Black-Box Adversarial Memory Injection Attacks on Long-Term Memory-Augmented Large Language Models

    Feb 17, 2026Mitchell Piehl, Zhaohan Xi, Zuobin Xiong +2Adversarial Attacks on LLMsLLM Security

    Department of Computer Science, The University of Iowa · School of Computing, State University of New York at Binghamton · Department of Computer Science, University of Nevada, Las Vegas +1

  29. Boundary Point Jailbreaking of Black-Box LLMs

    Feb 16, 2026Xander Davies, Giorgi Giglemiani, Edmund Lau +3Adversarial Attacks on LLMsJailbreak Attacks

    UK AI Security Institute · OATML, University of Oxford

  30. Exposing the Systematic Vulnerability of Open-Weight Models to Prefill Attacks

    Feb 16, 2026Lukas Struppek, Adam Gleave, Kellin PelrineAdversarial Attacks on LLMsLLM Safety