LLM Safety

Large language model (LLM) safety research focuses on mitigating the risks associated with these powerful models, primarily by preventing the generation of harmful or biased content and enhancing their robustness against adversarial attacks. Current research emphasizes developing and evaluating various defense mechanisms, including alignment techniques (like preference optimization and constrained direct preference optimization), and analyzing attack methods such as jailbreaks and prompt injection, often using reinforcement learning and adversarial training. This field is crucial for responsible LLM deployment, impacting both the development of safer models and the creation of effective content moderation and security protocols for a wide range of applications.

Papers

March 20, 2024

Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal
Rahul Pankajakshan, Sumitra Biswal, Yuvaraj Govindarajulu, Gilad Gressel
LLM Safety Threat Model LLM Integrated Application

March 18, 2024

Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models
Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong
Language Model Structured Output LLM Safety Enhancing Safety

March 4, 2024

Enhancing LLM Safety via Constrained Direct Preference Optimization
Zixuan Liu, Xiaolin Sun, Zizhan Zheng
Large Language Model Direct Preference Optimization Constrained Reinforcement Learning LLM Safety

March 3, 2024

Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
Arijit Ghosh Chowdhury, Md Mofijul Islam, Vaibhav Kumar, Faysal Hossain Shezan, Vaibhav Kumar, Vinija Jain, Aman Chadha
Natural Language Processing Adversarial Attack New Attack LLM Safety LLM Attack

February 29, 2024

LoRA-as-an-Attack! Piercing LLM Safety Under The Share-and-Play Scenario
Hongyi Liu, Zirui Liu, Ruixiang Tang, Jiayi Yuan, Shaochen Zhong, Yu-Neng Chuang, Li Li, Rui Chen, Xia Hu
LLM Safety LoRA Fine Tuning LoRA Based Transferable Backdoor Attack Collaborative Game

February 19, 2024

Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic
Rishabh Bhardwaj, Do Duc Anh, Soujanya Poria
Language Model LLM Safety Fine Tuned Language Model Safety Alignment Concept Activation Vector Arithmetic Task Safety Evaluation Benchmark

February 14, 2024

Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey
Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, Yu Qiao
Timely Survey Global Evaluation New Attack LLM Safety Harmful Content

February 8, 2024

In-Context Learning Can Re-learn Forbidden Tasks
Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, Dhanya Sridhar
Large Language Model Context Learning LLM Generated LLM Safety Knowledge Unlearning Safety Critical Task

January 30, 2024

January 18, 2024

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu
Large Language Model LLM Agent High Quality Risk Description LLM Safety Risk Awareness

December 30, 2023

The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness
Neeraj Varshney, Pavel Dolin, Agastya Seth, Chitta Baral
General Analysis NLP Research LLM Safety Natural Language Processing Application Unsafe Input

November 19, 2023

A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Models
Erik Derner, Kristina Batistič, Jan Zahálka, Robert Babuška
Large Language Model LLM Safety Prompt Attack Risk Taxonomy Security Risk

November 13, 2023

MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, Yuning Mao
Adversarial Prompt Red Teaming LLM Safety LLM Attack

November 7, 2023

Identifying and Mitigating Vulnerabilities in LLM-Integrated Applications
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Boxin Wang, Jinyuan Jia, Bo Li, Radha Poovendran
LLM Safety LLM Integrated Application

October 16, 2023

NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, Jonathan Cohen
Easy to Use Toolkit LLM Safety Conversational System Guardrail Model Dialogue Management Programmable Rail

September 6, 2023

Certifying LLM Safety against Adversarial Prompting
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, Himabindu Lakkaraju
Adversarial Prompt LLM Safety Token Embeddings Adversarial Pattern Adversarial Suffix Adversarial Prompting

August 14, 2023

LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked
Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, Duen Horng Chau
Large Language Model Text Generation Adversarial Prompt LLM Safety Self Feedback

July 5, 2023

Jailbroken: How Does LLM Safety Training Fail?
Alexander Wei, Nika Haghtalab, Jacob Steinhardt
Strong Generalization Jailbreak Attack Security Vulnerability LLM Safety Demographic Parity Safety Mechanism Adversarial Misuse