Adversarial Attacks on LLMs

LLM: Large Language Model

Ask about this topic

Overview

This field studies how attackers can make large language models produce unsafe or unreliable outputs, leak information, manipulate judgments, or misuse tools. Common approaches include black-box prompt search and multi-turn red teaming, gradient-based suffix optimization, reinforcement-learning attackers, and interventions on model activations or weights. Researchers evaluate these attacks on benchmarks such as HarmBench, AdvBench, and JailbreakBench, as well as in multilingual, multimodal, and tool-using agent settings. A central challenge is that adaptive attacks often expose weaknesses missed by static tests, while defenses must improve robustness without excessive false refusals, lost capability, or new system-level vulnerabilities.

Latest papers

CardsList

358 papers51 papers in the last four weeks, up 89% on the four weeks before. 0.3% of all new papers.

  1. Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

    Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM AlignmentLLM-as-a-Judge

    University of North Carolina at Chapel Hill · 2Carnegie Mellon University · 3Yale University +1

  2. Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model

    Feb 8, 2026Tianyi Wang, Huawei Fan, Yuanchao Shu +2LLM SecurityAdversarial Attacks on LLMs

    Zhejiang University

  3. Causal Front-Door Adjustment for Robust Jailbreak Attacks on LLMs

    Feb 5, 2026Yao Zhou, Zeen Song, Wenwen Qiang +2Adversarial Attacks on LLMsLLM Jailbreak Attacks

    Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

  4. Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

    Jan 29, 2026Xiang Zheng, Yutao Wu, Hanxun Huang +5Prompt InjectionAI Coding Agents

    City University of Hong Kong · Deakin University · The University of Melbourne +3

  5. In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement

    Jan 19, 2026Anudeex Shetty, Aditya Joshi, Salil S. KanhereLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    School of Computer Science and Engineering, UNSW Sydney, Australia · School of Computing and Information System, the University of Melbourne, Australia

  6. Safety Is Not Universal: The Selective Safety Trap in LLM Alignment

    Jan 7, 2026Iago Alves Brito, Walcy Santos Rezende Rios, Julia Soares Dollis +2LLM Safety BenchmarksLLM Alignment

    Advanced Knowledge Center for Immersive Technologies (AKCIT) Federal University of Goiás (UFG)

  7. Hidden State Poisoning Attacks against Mamba-based Language Models

    Jan 5, 2026Alexandre Le Mercier, Chris Develder, Thomas DemeesterState Space Language ModelsAdversarial Attacks on LLMs

    IDLab–T2K, Ghent University–imec

  8. AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens

    Dec 19, 2025Tung-Ling Li, Yuhao Wu, Hongliang LiuLLM-as-a-JudgeAdversarial Attacks on LLMs

    Palo Alto Networks

  9. One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

    Dec 14, 2025Yixin Tan, Zhe Yu, Rui Wen +1LLM SecurityAdversarial Attacks on LLMs

    Institute of Science Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan

  10. Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety

    Dec 8, 2025Max Zhang, Derek Liu, Kai Zhang +2Adversarial Attacks on LLMsLLM Safety Alignment

  11. Epistemic Bias Injection: Manipulating LLM Opinion via Selective Context Retrieval

    Nov 30, 2025Hao Wu, Prateek SaxenaAdversarial Attacks on LLMsLLM Security

    National University of Singapore

  12. Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks

    Oct 16, 2025Trilok Padhi, Pinxian Lu, Abdulkadir Erol +5Adversarial Attacks on LLMsAI Agent Security Benchmarks

    1Georgia State University · 2Georgia Institute of Technology · University of California, Berkeley

  13. MetaBreak: Jailbreaking Online LLM Services via Special Token Manipulation

    Oct 11, 2025Wentian Zhu, Zhen Xiang, Wei Niu +1Adversarial Attacks on LLMsLLM Security

    University of Georgia

  14. Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimization

    Oct 8, 2025Tiancheng Xing, Jerry Li, Yixuan Du +1Adversarial Attacks on LLMsLLM Reranking

    National University of Singapore · University of Southern California · Georgetown University +1

  15. NonTextual Target Attack

    Oct 3, 2025Xinzhe Huang, Wenjing Hu, Tianhang Zheng +6Adversarial Prompt GenerationAdversarial Attacks on LLMs

    State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou 310007, China · Hangzhou High-Tech Zone, Binjiang Institute of Blockchain and Data Security, Hangzhou 310053, China · School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing 210094, China +3

  16. There are Levels to It: Red Teaming Dialogue With Hierarchical Reinforcement Learning

    Aug 6, 2025Roman Belaire, Arunesh Sinha, Pradeep VarakanthamLLM Red TeamingAdversarial Attacks on LLMs

    Singapore Management University · Rutgers University

  17. Data Security in Large Language Models: Risks, Defense, and Directions

    Aug 4, 2025Kang Chen, Xiuze Zhou, Yuanhui Yu +4Adversarial Attacks on LLMsLLM Security

    School of Computer Engineering, Jimei University, Xiamen, 361021, China · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China · School of Informatics, Xiamen University, Xiamen, 361102, China +1

  18. Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness

    Jun 30, 2025Tung-Ling Li, Hongliang LiuLLM AlignmentLanguage Model Safety Evaluation

    Palo Alto Networks

  19. In-Context Bias Propagation in LLM-Based Tabular Data Generation

    Jun 11, 2025Pol G. Recasens, Alberto Gutierrez, Jordi Torres +4Synthetic Data GenerationSynthetic Tabular Data Generation

    Barcelona Supercomputing Center (BSC), Spain · Universitat Politècnica de Catalunya (UPC), Spain · IBM Research Ireland

  20. Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models

    Jun 9, 2025Mickel Liu, Liwei Jiang, Yancheng Liang +4Self-Play RLAdversarial Attacks on LLMs

    Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA · Department of Computer Science, Stanford University, Stanford, CA, USA

  21. Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models

    Jun 8, 2025Ren-Jian Wang, Ke Xue, Zeyu Qin +7LLM Red TeamingAdversarial Prompt Generation

    National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology, Hong Kong, SAR, China +3

  22. HauntAttack: When Attack Follows Reasoning as a Shadow

    Jun 8, 2025Jingyuan Ma, Rui Li, Zheng Li +4Adversarial Attacks on LLMsJailbreak Attacks

    State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Department of Computing, The Hong Kong Polytechnic University +1

  23. Auditing Prompt Injection Defenses Before They Reach Users: What Strong Adversaries Reveal

    May 21, 2025Xiaoxue Yang, Bozhidar Stevanoski, Matthieu Meeus +1Prompt Injection DefensePrompt Injection Attacks on LLMs

    Imperial College London

  24. Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models

    Mar 8, 2025Thomas Winninger, Boussad Addad, Katarzyna KapustaAdversarial Prompt GenerationAdversarial Attacks on LLMs

    *Work started during internship at Thales.

  25. UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Models

    Feb 18, 2025Huawei Lin, Yingjie Lao, Tony Geng +2LLM Backdoor AttacksAdversarial Attacks on LLMs

    Rochester Institute of Technology · Tufts University · University of Rochester +1

  26. Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robots

    Dec 21, 2024Xuancun Lu, Zhengxian Huang, Xinfeng Li +3Large Language Model-Based Robot PlanningAdversarial Attacks on LLMs

    Zhejiang University

  27. Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

    Dec 21, 2024Yanxu Mao, Peipei Liu, Tiehan Cui +3Adversarial Attacks on LLMsMultimodal Large Language Models

    School of Computer Science and Technology, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Laboratory, Beijing, China · School of Software, Henan University, Kaifeng, China +1

  28. Learning diverse attacks on large language models for robust red-teaming and safety tuning

    May 28, 2024Seanie Lee, Minsu Kim, Lynn Cherif +8Adversarial Attacks on LLMsLLM Red Teaming

    KAIST · Mila – Québec AI Institute · Université de Montréal +3