Adversarial Attacks on LLMs

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

37 papers in the last four weeks, up 311% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 238

All topics

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
Jun 3, 2026Kyungmin Park, Taesup KimAdversarial Attacks on LLMs LLM Safety Alignment
Hankuk University of Foreign Studies · Seoul National University
When Autoregressive Consistency Hurts Safety Alignment
Jun 2, 2026Bochen Lyu, Yiyang Jia, Xiaohao Cai +1Adversarial Attacks on LLMs LLM Safety Alignment
University of Southampton · Independent Researcher
Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks
Jun 2, 2026Malia Barker, Bishal Lakha, Edoardo Serra +1Numerical Reasoning in Language Models Mathematical Reasoning Benchmarks
Department of Computer Science, Boise State University, Boise, ID, USA · University of L'Aquila, L'Aquila, Italy
PsychoPass: Geometric Profiling of Multi-Turn Adversarial LLM Conversations
Jun 2, 2026Muberra Ozmen, Subhabrata MajumdarAdversarial Attacks on LLMs LLM Safety
Coveo · Indian Institute of Management Bangalore
MaskForge: Structure-Aware Adaptive Attacks for Jailbreaking Diffusion Large Language Models
Jun 1, 2026Yingzi Ma, Zhengyue Zhao, Xiaogeng Liu +3Masked Diffusion Models Adversarial Attacks on LLMs
University of Wisconsin-Madison · Johns Hopkins University · Responsible AI Research (RAIR) Centre, The University of Adelaide +1
Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents
Jun 1, 2026Yoshinari Fujinuma, Varun Gangal, Traian Rebedea +4Adversarial Attacks on LLMs LLM Agent Security
Patronus AI · NVIDIA
Dive into Ambiguity: A*-Inspired Multi-Agents Commonsense Obfuscation Attack on LLM Prompts
May 31, 2026Boxuan Wang, Zhuoyun Li, Xiaowei Huang +1Adversarial Prompt Generation Adversarial Attacks
University of Liverpool Merseyside, United Kingdom
Inference Cost Attacks for Retrieval-Augmented Large Language Models
May 31, 2026Chengliang Liu, Liangbo Ning, Yujuan Ding +1Retrieval-Augmented Generation Adversarial Attacks
The Hong Kong Polytechnic University Hong Kong, Hong Kong
Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
May 30, 2026Rana Muhammad UsmanAdversarial Attacks on LLMs LLM Agent Evaluation
Independent Researcher
Cross-Generational Transfer of Adversarial Attacks Reveals Non-Monotonic Safety Alignment in LLMs
May 30, 2026Subhadip MitraAdversarial Attacks on LLMs LLM Red Teaming
Research Lead, Rota Labs
Quality-Diversity Evolution for Discovering Diverse Vulnerabilities in LLM Safety
May 30, 2026Subhadip MitraAdversarial Prompt Generation Adversarial Attacks on LLMs
Research Lead, Rota Labs
MultiTurnPSB: Evaluating Multi-Turn Jailbreak Attacks an dClassifier-Based Defenses for Medical AI Safety
May 30, 2026Anushka Sheoran, Yiduo HaoLanguage Model Safety Evaluation Adversarial Attacks on LLMs
University of Pennsylvania
Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models
May 30, 2026Mohammed Sameer Syed, Rozhin YasaeiAdversarial Attacks on LLMs Prompt Injection Attacks
University of Arizona
EvoDefense: Co-Evolving Black-Box Defense with Large Language Models
May 29, 2026Yu Li, Yuenan Hou, Yingmei Wei +2Evolutionary Optimization Adversarial Attacks on LLMs
National University of Defense Technology · Shanghai AI Laboratory
Persona Attack: Incremental Memory Injection Jailbreak Attack against Large Language Models
May 29, 2026Junyoung Park, Seongyong Ju, Sunghwan Park +1Adversarial Attacks on LLMs LLM Safety
Chung-Ang University
Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs
May 28, 2026David Gros, Adam GleaveLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
May 28, 2026Alexander Sternfeld, Andrei Kucharavy, Ljiljana DolamicPrompt Sensitivity Adversarial Attacks on LLMs
IEM, HES-SO, Le Foyer, Techno-Pôle 1, Sierre, Switzerland · Cyber-Defence Campus, armasuisse Science and Technology, Thun, Switzerland
Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
May 28, 2026Wajdi Zaghouani, Kholoud K. Aldous, Yicheng GaoMultilingual Language Model Evaluation LLM Evaluation
Northwestern University in Qatar
SciIntBench: Measuring LLM Compliance with Research Integrity Norms Under Adversarial Framing
May 28, 2026Almene De Meran Meguimtsop, Maria Leonor Pacheco, Daniel E. AcunaAI for Science Adversarial Attacks on LLMs
Department of Computer Science, University of Colorado Boulder, Boulder, CO 80309
Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills
May 28, 2026Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang +1Adversarial Prompt Generation AI Coding Agents
Department of Computer Science National Yang Ming Chiao Tung University · Department of Electronics and Electrical Engineering National Yang Ming Chiao Tung University · School of Computing Institute of Science Tokyo
Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activations
May 27, 2026Matteo Gioele Collu, Riccardo Conte, Alberto Giaretta +4LLM Interpretability Adversarial Attacks on LLMs
University of Padua, Italy · 2Örebro University, Sweden · 3Fondazione Bruno Kessler, Italy
Out of Sight, Not Out of Mind: Unveiling Latent Attack in Latent-based Multi-Agent Systems
May 27, 2026Chenxi Wang, Ruiyang Huang, Jiayan Sun +2Multi-Agent LLM Systems Adversarial Attacks
Southeast University, Nanjing, China · Peking University, Beijing, China
HARP: Measuring Harm Amplification in Multi-Agent LLM Systems
May 26, 2026Md Hafizur Rahman, Zafaryab Haider, Tanzim Mahfuz +1Multi-Agent LLM Systems Adversarial Attacks on LLMs
Electrical and Computer Engineering University of Maine, ME, USA
Cordyceps: Covert Control Attacks on LLMs via Data Poisoning
May 26, 2026Zedian Shao, Charles Fleming, Teodora BalutaLLM Backdoor Attacks Adversarial Attacks on LLMs
Georgia Institute of Technology · Cisco Systems
Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks
May 26, 2026Kevin Kuo, Virginia Smith, Chhavi YadavLLM Fine-Tuning Adversarial Attacks on LLMs
Carnegie Mellon University · Simons Institute, UC Berkeley
LLM-as-a-Reviewer: Benchmarking Their Ability, Divergence, and Prompt Injection Resistance as Paper Reviewers
May 25, 2026Lingyao Li, Junjie Xiong, Changjia Zhu +5LLM Evaluation LLM-as-a-Judge
University of South Florida · Missouri University of Science and Technology · University of Alabama +3
Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS
May 25, 2026Bingyu Yan, Xiaoming Zhang, Jinyu Hou +4Adversarial Attacks on LLMs LLM Agent Security
1Beihang University · 2Beijing University of Posts and Telecommunications · 3China Academy of Information and Communications Technology
Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications
May 24, 2026Xiaoyue Lu, Xianglin Yang, Haijun Liu +4Automated Software Testing Language Model Safety Evaluation
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China · National University of Singapore, Singapore · Independent Researcher
Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges
May 24, 2026Xianglin Yang, Bryan Hooi, Gelei Deng +2LLM Evaluation LLM-as-a-Judge
School of Computing, National University of Singapore, Singapore · Nanyang Technological University, Singapore
Reasoning as an Attack Surface: Adaptive Evolutionary CoT Jailbreaks for LLMs
May 23, 2026Jianan Li, Simeng Qin, Xiaojun Jia +5Adversarial Attacks on LLMs LLM Jailbreak Attacks
Northeastern University · The University of Hong Kong, Hong Kong, China · BraneMatrix AI, China +7

Jun 3, 2026, cs.AIJ/K move · Enter open · S save

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

Kyungmin Park, Taesup Kim

Hankuk University of Foreign Studies · Seoul National University

Adversarial Attacks on LLMs LLM Safety Alignment Adversarial Robustness of Language Models Inference-Time Language Model Alignment

Open paper AskPDF arXiv

Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.

Momentum

Related topics

Latest papers 238