LLM Security

LLM: Large Language Model

LLM security examines how models and the systems around them can be manipulated, made to leak information, or induced to take unsafe actions. Attacks include prompt injection and jailbreaks, poisoned training data and backdoors, model extraction, side channels in shared serving systems, and vulnerabilities in generated code or tool-using agents. Defenses range from safer training and fine-tuning to runtime detectors, access controls, secure serving, and application-level isolation; evaluations increasingly use adaptive attacks and realistic agent, coding, and deployment benchmarks. A central tension is that model-level safeguards can fail under new inputs or system interactions, while stronger controls may reduce utility, privacy, or efficiency, making end-to-end testing and clear threat models essential.

Latest papers

327 papers45 papers in the last four weeks, up 45% on the four weeks before. 0.3% of all new papers.

LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters
May 13, 2026Beomjin Ahn, Jungmin Kwon, Chanyong Jung +1Low-Rank Adaptation LLM Security
Samsung Research · Amazon Web Services · University of Michigan +1
CoT-Guard: Small Models for Strong Monitoring
May 12, 2026Nirav Diwan, Han Wang, Berkcan Kapusuzoglu +6LLM Auditing Adversarial Attacks on LLMs
†University of Illinois Urbana-Champaign · ♢Capital One
Robust LLM Unlearning Against Relearning Attacks: The Minor Components in Representations Matter
May 12, 2026Zeguan Xiao, Xuanzhe Xu, Yun Chen +4Neural Representation Geometry LLM Security
Shanghai University of Finance and Economics · Southern University of Science and Technology · Alibaba Group +1
Natural Language based Specification and Verification
May 11, 2026Zhaorui Li, Chengyu SongFormal Verification LLM Security
University of California Riverside
Continuous Discovery of Vulnerabilities in LLM Serving Systems with Fuzzing
May 11, 2026Yunze Zhao, Yibo Zhao, Yuchen Zhang +2LLM Serving LLM Security
University of Maryland · New York University
LLMs for Secure Hardware Design and Related Problems: Opportunities and Challenges
May 11, 2026Johann Knechtel, Ozgur Sinanoglu, Ramesh KarriElectronic Design Automation Cybersecurity
New York University Abu Dhabi · NYU Tandon School of Engineering
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
May 11, 2026Keitaro Sakamoto, Pierre Ablin, Federico Danieli +1Open-Weight Language Models LLM Security
Apple
Identified-Set Geometry of Distributional Model Extraction under Top- $K$ Censored API Access
May 11, 2026Wenhua Nie, ZiCheng Zhu, Jianan Wu +3Parameter Identifiability Partial Identification
National Taiwan University
When Prompts Become Payloads: A Framework for Mitigating SQL Injection Attacks in Large Language Model-Driven Applications
May 11, 2026Farzad Nourmohammadzadeh Motlagh, Mehrdad Hajizadeh, Mehryar Majd +3LLM Security Text-to-SQL
Hasso Plattner Institute, University of Potsdam, Germany · Chemnitz University of Technology, Chemnitz, Germany
Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing
May 11, 2026Qinghua Mao, Xi Lin, Jinze Gu +3Knowledge Editing Knowledge Conflicts in Language Models
School of Computer Science · Shanghai Jiao Tong University
OTora: A Unified Red Teaming Framework for Reasoning-Level Denial-of-Service in LLM Agents
May 9, 2026Xinyu Li, Ronghui Mu, Lin Li +2Adversarial Attacks on LLMs LLM Red Teaming
Department of Computer Science, University of Exeter · Department of Computer Science, University of Oxford · Department of Mathematics and Computer Science, Eindhoven University of Technology
Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs
May 8, 2026Hanlin Cai, Kai Li, Houtianfu Wang +4LLM Fine-Tuning LLM Security
Centre for neXt Communications (CXC), Department of Engineering, University of Cambridge, CB3 0FA Cambridge, U.K. · Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, L-1855, Luxembourg · Telecommunication Networks group (TKN) at the School of Electrical Engineering and Computer Science, TU Berlin, Germany +1
Seed Hijacking of LLM Sampling and Quantum Random Number Defense
May 8, 2026Ziyang You, Xiaoke Yang, Zhanling Fan +3LLM Backdoor Attacks LLM Security
School of Electronic, Electrical and Physics, Fujian University of Technology, Fuzhou 350118, China · School of Humanities, Fujian University of Technology, Fuzhou 350118, China · Department of Investigation, Fujian Police College, Fuzhou 350007, China +1
Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions
May 7, 2026Chengjie Wang, Jingzheng Wu, Xiang Ling +2Software Engineering LLM Security
Intelligent Software Research Center, Institute of Software, Chinese Academy of Sciences, Beijing, China · Key Laboratory of System Software (Chinese Academy of Sciences), Beijing, China
SoK: Robustness in Large Language Models against Jailbreak Attacks
May 6, 2026Feiyue Xu, Hongsheng Hu, Chaoxiang He +9Language Model Safety Evaluation LLM Security
Shanghai Jiao Tong University, China · Microsoft Corporation
On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
May 6, 2026Zhengyi Li, Yakai Wang, Kang Yang +6Transformer Inference Model Extraction Attacks
Shanghai Jiao Tong University · State Key Laboratory of Cryptology · Shanghai Qizhi Institute
Shattering the Echo Chamber: Hidden Safeguards in Manuscripts Against the AI Takeover of Peer Review
May 6, 2026Oubo Ma, Ruixiao Lin, Jiahao Chen +3LLM Security Indirect Prompt Injection
Zhejiang University
Breaking Safety at the Token Boundary: How BPE Tokenization Creates Exploitable Gaps in LLM Alignment
May 1, 2026Tung-Ling Li, Hongliang Liu, Yuhao WuLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Palo Alto Networks
TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning
Apr 30, 2026Bowen Sun, Chaozhuo Li, Yaodong Yang +2Jailbreak Attacks LLM Security
Johns Hopkins University · Microsoft Research Asia · Peking University +1
Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoors
Apr 30, 2026Zi Li, Tian Zhou, Wenze Li +3Data Leakage LLM Backdoor Attacks
Nanjing University
From Prompt to Physical Actuation: Holistic Threat Modeling of LLM-Enabled Robotic Systems
Apr 29, 2026Neha Nagaraja, Hayretdin Bahsi, Carlo R. da CunhaLarge Language Model-Based Robot Planning Cybersecurity
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia
DSIPA: Detecting LLM-Generated Texts via Sentiment-Invariant Patterns Divergence Analysis
Apr 29, 2026Siyuan Li, Aodu Wulianghai, Guangyan Li +5AI-Generated Text Detection LLM Security
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, Shanghai, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Cross-Lingual Jailbreak Detection via Semantic Codebooks
Apr 28, 2026Shirin Alanova, Bogdan Minko, Sabrina Sadiekh +1Multilingual Language Model Evaluation LLM Security
AI Talent Hub, ITMO University · HiveTraceLab
Defusing the Trigger: Plug-and-Play Defense for Backdoored LLMs via Tail-Risk Intrinsic Geometric Smoothing
Apr 27, 2026Kaisheng Fan, Weizhe Zhang, Yishu Gao +2LLM Backdoor Attacks Backdoor Defense in LLMs
School of Cyber Science and Technology Harbin Institute of Technology Harbin, China · Department of New Networks Peng Cheng Laboratory Shenzhen, China · SnT University of Luxembourg Luxembourg City, Luxembourg
Evaluation of Prompt Injection Defenses in Large Language Models
Apr 26, 2026Priyal Deep, Shane Emmons, Amy Fox +4Data Leakage Privacy Leakage in Language Models
Swept AI[1] · University of Michigan
Semantic Denial of Service in LLM-controlled robots
Apr 25, 2026Jonathan Steinberg, Oren GalLanguage Model Safety Evaluation Adversarial Attacks on LLMs
Swarms & AI Lab (SAIL) · University of Haifa
Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggers
Apr 23, 2026Jiali Wei, Ming Fan, Guoheng Sun +3LLM Backdoor Attacks Backdoor Attacks
School of Cyber Science and Engineering, Xi’an Jiaotong University, Xi’an 710049, China · Ministry of Education Key Lab for Intelligent Networks and Network Security, Xi’an Jiaotong University, Xi’an 710049, China
AVISE: Framework for Evaluating the Security of AI Systems
Apr 22, 2026Mikko Lempinen, Joni Kemppainen, Niklas RaesalmiLanguage Model Safety Evaluation Adversarial Attacks on LLMs
University of Oulu
Mythos and the Unverified Cage: Z3-Based Pre-Deployment Verification for Frontier-Model Sandbox Infrastructure
Apr 22, 2026Dominik BlainSoftware Vulnerability Detection Formal Verification
COBALT Formal Verification, QreativeLab Inc. Montr´eal, Qu´ebec, Canada
Text Steganography with Dynamic Codebook and Multimodal Large Language Model
Apr 22, 2026Jianxin Gao, Ruohan Lei, Wanli PengLLM Security Language Model Steganography
China Agricultural University Beijing, China

May 13, 2026J/K move · Enter open · S save

LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters

Beomjin Ahn, Jungmin Kwon, Chanyong Jung, Jaewook Chung

Samsung Research · Amazon Web Services · University of Michigan+1

Low-Rank Adaptation LLM Security

Open paper AskPDF arXiv

Foundation models and low-rank adapters enable efficient on-device generative AI but raise risks such as intellectual property leakage and model recovery attacks. Existing defenses are often impractical because they require retraining or access to the original dataset. We propose LoREnc, a training-free framework that secures both FMs and adapters via spectral truncation and compensation. LoREnc suppresses dominant low-rank components of FM weights, compensates for the missing information in authorized adapters, and further applies orthogonal reparameterization to obscure structural fingerprints of the protected adapter. Unauthorized users produce structurally collapsed outputs, while authorized users recover exact performance. Experiments demonstrate that LoREnc provides strong protection against model recovery with under 1% computational overhead.