Adversarial Attacks on LLMs

LLM: Large Language Model

Ask about this topic

Overview

This field studies how attackers can make large language models produce unsafe or unreliable outputs, leak information, manipulate judgments, or misuse tools. Common approaches include black-box prompt search and multi-turn red teaming, gradient-based suffix optimization, reinforcement-learning attackers, and interventions on model activations or weights. Researchers evaluate these attacks on benchmarks such as HarmBench, AdvBench, and JailbreakBench, as well as in multilingual, multimodal, and tool-using agent settings. A central challenge is that adaptive attacks often expose weaknesses missed by static tests, while defenses must improve robustness without excessive false refusals, lost capability, or new system-level vulnerabilities.

Latest papers

CardsList

358 papers51 papers in the last four weeks, up 89% on the four weeks before. 0.3% of all new papers.

  1. Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment

    May 1, 2026Tung-Ling Li, Hongliang Liu, Yuhao WuLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    Palo Alto Networks

  2. FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

    May 1, 2026Yutao Hou, Yihan Jiang, Yuhan Xie +5Multilingual Language Model EvaluationLanguage Model Safety Evaluation

    Shanghai University of Finance and Economics · Beihang University · Southern University of Science and Technology +1

  3. RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs

    May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language ModelsAdversarial Attacks on LLMs

    University of Bristol Bristol, United Kingdom

  4. Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance

    May 1, 2026Minchan Kwon, Sunghyun Baek, Minseo Kim +3Language Model Safety EvaluationAdversarial Attacks on LLMs

    1KAIST · 2Naver AI Lab.

  5. The Power of Order: Fooling LLMs with Adversarial Table Permutations

    May 1, 2026Xinshuai Dong, Haifeng Chen, Xuyuan Liu +5Table QAAdversarial Attacks on LLMs

    CMU & NEC Labs · NEC Labs · Dartmouth College & NEC Labs +1

  6. Attention Is Where You Attack

    Apr 30, 2026Aviral Srivastava, Sourav PandaSelf-AttentionAdversarial Attacks

    Amazon · The Pennsylvania State University

  7. Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation

    Apr 29, 2026Jon-Paul CacioliLLM EvaluationLanguage Model Generation Evaluation

    Independent Researcher, Melbourne, Australia

  8. Adaptive Prompt Embedding Optimization for LLM Jailbreaking

    Apr 27, 2026Miles Q. Li, Benjamin C. M. Fung, Boyang Li +2Adversarial Prompt GenerationAdversarial Attacks on LLMs

    School of Information Studies, McGill University, Montreal, QC, Canada · Department of Computer Science, Kean University, Union, NJ, United States · Mila - Quebec AI Institute, Montreal, QC, Canada +1

  9. Layerwise Convergence Fingerprints for Runtime Misbehavior Detection in Large Language Models

    Apr 27, 2026Nay Myat Min, Long H. Pham, Jun SunLLM Backdoor AttacksAdversarial Attacks on LLMs

    Singapore Management University

  10. When AI reviews science: Can we trust the referee?

    Apr 26, 2026Jialiang Wang, Yuchen Liu, Hang Xu +7LLM AuditingAdversarial Attacks on LLMs

    Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR 999077, China · Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, School of Computer Science and Engeering, Southeast University, Nanjing 210000, China · These authors contributed equally +2

  11. Semantic Denial of Service in LLM-controlled robots

    Apr 25, 2026Jonathan Steinberg, Oren GalLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    Swarms & AI Lab (SAIL) · University of Haifa

  12. Training a General Purpose Automated Red Teaming Model

    Apr 24, 2026Aishwarya Padmakumar, Leon Derczynski, Traian Rebedea +1Adversarial Attacks on LLMsLLM Red Teaming

    NVIDIA

  13. Transient Turn Injection: Exposing Stateless Multi-Turn Vulnerabilities in Large Language Models

    Apr 23, 2026Naheed Rayhan, Sohely JahanLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    Jagannath University · University of Barishal

  14. Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models

    Apr 22, 2026Yannis Belkhiter, Giulio Zizzo, Sergio Maffeis +2Adversarial Attacks on LLMsAI Agent Security

    IBM Research Europe · Trinity College Dublin · Imperial College London +1

  15. AVISE: Framework for Evaluating the Security of AI Systems

    Apr 22, 2026Mikko Lempinen, Joni Kemppainen, Niklas RaesalmiLanguage Model Safety EvaluationAdversarial Attacks on LLMs

    University of Oulu

  16. Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs

    Apr 22, 2026Krishiv Agarwal, Ramneet Kaur, Colin Samplawski +6Language Model Safety EvaluationLLM Auditing

    University of Florida · SRI

  17. ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System

    Apr 20, 2026Jiacheng Liang, Yao Ma, Tharindu Kumarage +5Reward ModelingAdversarial Attacks on LLMs

    Stony Brook University · Amazon Nova Responsible AI

  18. Towards Understanding the Robustness of Sparse Autoencoders

    Apr 20, 2026Ahson Saiyed, Sabrina Sadiekh, Chirag AgarwalAdversarial Attacks on LLMsJailbreak Robustness

    University of Virginia · Independent Researcher

  19. Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety

    Apr 20, 2026Marcello Galisai, Susanna Cifani, Francesco Giarrusso +5LLM Safety BenchmarksAdversarial Attacks on LLMs

    DEXAI – Icaro Lab · Sapienza University of Rome · Sant’Anna School of Advanced Studies

  20. Evaluating Answer Leakage Robustness of LLM Tutors against Adversarial Student Attacks

    Apr 20, 2026Jin Zhao, Marta Knežević, Tanja KäserLLM EvaluationIntelligent Tutoring Systems

    †Work done during exchange at EPFL.

  21. SafeAgent: A Runtime Protection Architecture for Agentic Systems

    Apr 19, 2026Hailin Liu, Eugene Ilyushin, Jie Ni +1Adversarial Attacks on LLMsAI Agent Security

    Lomonosov Moscow State University · Central University

  22. Bit-Flip Vulnerability of Shared KV-Cache Blocks in LLM Serving Systems

    Apr 19, 2026Yuji Yamamoto, Satoshi MatsuuraLLM ServingAdversarial Attacks on LLMs

    Institute of Science Tokyo, Tokyo, Japan

  23. Jailbreaking Large Language Models with Morality Attacks

    Apr 18, 2026Ying Su, Mingen Zheng, Weili Diao +1Moral Reasoning in Language ModelsLanguage Model Safety Evaluation

    South China University of Technology1 · HKUST2 · Beihang University3

  24. When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints

    Apr 18, 2026Yuheng Chen, Zhiyu Wu, Bowen Cheng +2Adversarial Attacks on LLMsLLM Safety Alignment

    Kagoshima University · Fudan University · China University of Petroleum-Beijing

  25. Stochasticity in Tokenisation Improves Robustness

    Apr 17, 2026Sophie Steger, Rui Li, Sofiane Ennadir +4Adversarial Attacks on LLMsNeural Network Robustness

    Institute of Signal Processing and Speech Communication, Graz University of Technology, Graz, Austria · ELLIS Institute Finland & Aalto University, Espoo, Finland · King AI Labs, Microsoft Gaming +2

  26. Conjunctive Prompt Attacks in Multi-Agent LLM Systems

    Apr 17, 2026Nokimul Hasan Arif, Qian Lou, Mengxin ZhengMulti-Agent LLM SystemsAdversarial Attacks on LLMs

    University of Central Florida, USA

  27. Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theory

    Apr 14, 2026Shaopeng Fu, Di WangAdversarial TrainingAdversarial Robustness of Language Models

    Provable Responsible AI and Data Analytics (PRADA) Lab, King Abdullah University of Science and Technology, Saudi Arabia

  28. TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

    Apr 14, 2026Qingchao Shen, Zibo Xiao, Lili Huang +3Adversarial Attacks on LLMsLLM Security

    School of Computer Software, Tianjin University, Tianjin, China · Monash University, Monash, Australia

  29. Homoglyph-based Adversarial Perturbation of Introductory Computer Science Theory Problems

    Apr 13, 2026Aidan Alexander, Chitrangada Juneja, Napaluck Tontrasathien +4Adversarial Text AttacksAdversarial Attacks on LLMs

  30. Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Models

    Apr 13, 2026Shuhao Zhang, Yuli Chen, Jiale Han +2Adversarial Attacks on LLMsLLM Watermarking

    State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications · Hong Kong University of Science and Technology