LLM Safety Benchmarks

LLM: Large Language Model

This field measures how reliably language models recognize and handle risks such as harmful requests, jailbreaks, bias, privacy violations, and unsafe tool use. Benchmarks use curated and expert-annotated scenarios, adversarial red-teaming, multi-turn and agent simulations, and increasingly multilingual, multimodal, and domain-specific tests. Researchers also compare refusal and moderation methods, use repeated sampling and statistical techniques to estimate reliability, and assess both harmful compliance and over-refusal. A central challenge is making results comparable and trustworthy while ensuring tests reflect changing, context-dependent behavior in real deployments.

Latest papers

284 papers44 papers in the last four weeks, up 7% on the four weeks before. 0.3% of all new papers.

When Personas Become Harmful: Guiding LLMs with Global Citizenship Values Alignment
Oct 24, 2026Siqi Shen, Lajanugen Logeswaran, Moontae Lee +2LLM Safety LLM Safety Alignment
University of Michigan · LG AI Research · University of Illinois at Chicago
BoardSafety: AI-Ready Data from Manufacturer Documentation for Development-Board Safety
Oct 24, 2026Linjian Yang, Xinyan Wang, Kunpeng LiuAI Safety Evaluation LLM Safety Benchmarks
Clemson University Clemson, SC, USA · Portland State University Portland, OR, USA
Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability
Oct 22, 2026Junhao Wang, Yidou Chen, Li Lu +6LLM Safety Benchmarks MLLM Safety
State Key Laboratory of Blockchain and Data Security, Zhejiang University · Alibaba Group
Scaling Laws for Safety Probes
Oct 22, 2026Yuchen Wu, Jerry WeiLLM Safety Monitoring Language Model Probing
University of Washington · Anthropic
TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics
Oct 21, 2026Yegor Anichkov, Valery Vinogradov, Andrey Furov +2Language Model Safety Evaluation LLM Safety Benchmarks
Trusted AI Research Center, RAS, Moscow, Russia. · Center for Applied Artificial Intelligence, Institute of Social Sciences, Moscow, Russia.
Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics
Oct 21, 2026Hongyu Li, Xinlong Huang, Zhan Yu +13LLM Safety Evaluation LLM Safety Benchmarks
Hunyuan Team, Tencent · Beijing Normal University
Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents
Oct 20, 2026Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang +1LLM Agent Safety Refusal Behavior in Language Models
Harvard University · Clarkson University · Carleton University
Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing
Oct 20, 2026Diego Miguel Lozano, Erik Derner, Juan Antonio Pérez-Ortiz +1Language Model Safety Evaluation Adversarial Robustness of Language Models
ELLIS Alicante, Spain · Czech Technical University in Prague, Czech Republic · Universidad de Alicante, Spain
ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values
Oct 20, 2026Liping Yang, Yongqi Leng, Jingting Zheng +3LLM Safety Benchmarks Language Model Safety Evaluation
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, Tianjin, China · Yuanhui AI
When Health Assistants Agree Too Readily: A Benchmark of Time-Critical Self-Diagnosis
Oct 20, 2026Marilyn Zhang, Tianfeng Chen, Patricia M Pechter +1Clinical Language Model Evaluation LLM Safety Benchmarks
Pareto Inc · NYU Langone
African Languages AI Safety Readiness Report
Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Tajuddeen Gwadabe +1Language Model Safety Evaluation African Language Processing
Masakhane · Saarland University · Mila - Quebec AI Institute, McGill University, and Canada CIFAR AI Chair
Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates
Oct 19, 2026Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro +2Jailbreak Attack Evaluation LLM Safety Benchmarks
Carl von Ossietzky University of Oldenburg, Oldenburg, Germany · Chalmers University of Technology, Gothenburg, Sweden
AfriGuard: Safety and Robust Refusal For African Languages
Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Iffat Maab +20LLM Safety Language Model Safety Evaluation
Saarland University, Saarland Informatic Campus, Germany · Masakhane NLP · National Institute of Informatics, Japan +13
VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)
Oct 19, 2026Leanne Tan, Keith Chua, Shaun Khoo +7LLM Safety Benchmarks Language Model Safety Evaluation
GovTech Singapore · MOH Office for Healthcare Transformation · Singapore University of Technology and Design +2
MoveBench: A Rubric-Based Benchmark for Evaluating Large Language Models in Sports and Health Coaching
Oct 19, 2026Yitong Li, Jiang Bian, Jingling Wang +2LLM Evaluation Benchmark Design
Keep, Beijing, China
BrandEval: A Two-Track Multi-Agent Benchmark for Risk-Sensitive LLM Crisis Communication
Oct 18, 2026Haotian Gan, Weidong Tang, Hanbin Sun +5LLM Agent Evaluation LLM Safety Benchmarks
Shaanxi University of Science and Technology · China Agricultural University · Tianjin University +3
Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On
Oct 18, 2026Vahid Jebraeeli, Jonathon Romero, Yijin Wang +2Language Model Safety Evaluation LLM Safety Benchmarks
Lenovo
SoK: PIAtlas: Decomposing the Unbounded Prompt Injection Attack Space
Oct 18, 2026Stavros Damianakis, Konstantinos PsounisPrompt Injection Attacks Prompt Injection Attacks on LLMs
University of Southern California
ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs
Oct 17, 2026Juhyeon Park, Sumin Yu, Taesup MoonContextual Integrity LLM Safety Benchmarks
IPAI, Seoul National University · ECE, Seoul National University · ASRI / INMC / AIIS, Seoul National University
BanglaShield: A Script-Robust Safety Classifier and Benchmark for Bangla Native, Romanized, and Code-Mixed Prompts
Oct 16, 2026Mohsinul Kabir, Md. Tariquzzaman, Mohammad Rashidujjaman Rifat +2LLM Safety Benchmarks Language Model Safety Evaluation
University of Manchester, UK · Islamic University of Technology, Bangladesh · University of Notre Dame, USA +1
Refusal Is Not Robustness: Measuring Harmful Reachability in Autoregressive Language Models
Oct 15, 2026Parv Kapoor, Teresa Fortes, Claudia Mamede +3Language Model Safety Evaluation LLM Safety
Carnegie Mellon University, Pittsburgh, PA, USA. · General Robotics · University of Porto, Porto, Portugal.
In the Guise of Correction: How Framing and Cultural Adaptation Erode LLM Safety on Real-World Criminal Cases
Oct 15, 2026Mohsinul Kabir, Md. Tariquzzaman, Karam S. R. Elradie +4LLM Safety Evaluation LLM Safety Benchmarks
The University of Manchester · ELLIS Manchester · Islamic University of Technology +2
Recoverability-Controlled Safety Miscalibration in Large Language Models
Oct 15, 2026Yuwei Wang, Xingwang Liang, Yu Xie +2Language Model Safety Evaluation LLM Safety
School of Automation and Software Engineering, Shanxi University · School of Computer and Information Technology, Shanxi University
Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
Oct 14, 2026Zahraa Al Sahili, Ioannis Patras, Matthew PurverMultilingual Language Model Evaluation Language Model Safety Evaluation
School of Electronic Engineering and Computer Science, Queen Mary University of London, London, UK · Department of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia
When Protection Overrides Consequences: Understanding Protection Mode in Large Language Model Decision-Making
Oct 14, 2026Yuanfan Li, Junyi Chen, Qi Zhou +3LLM Decision-Making Language Model Safety Evaluation
Xiaohongshu · Shanghai Jiao Tong University · Xi’an Jiaotong University
Moral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMs
Oct 14, 2026Adel Khorramrouz, Yi-Chia Wang, Sharon LevyAdversarial Attacks on LLMs LLM Safety Evaluation
Rutgers University · Carnegie Mellon University
Safety for Whom? Benchmarking Understanding Beyond Refusal in Egocentric Multimodal Assistants
Oct 13, 2026Yewon Han, Jisu Jeon, Suhyun Choi +5AI Safety Evaluation LLM Safety Benchmarks
Graduate School of Data Science, Seoul National University
REPRO: Reflective Evolutionary Prompt Optimizer for Automated LLM Red-Teaming
Oct 11, 2026Abrar Alotaibi, Moataz AhmedAdversarial Prompt Generation Adversarial Attacks on LLMs
King Fahd University of Petroleum & Minerals, Information and Computer Science Department, Dhahran, Saudi Arabia · Imam Abdulrahman Bin Faisal University, College of Computer Science and Information Technology, Dammam, Saudi Arabia · SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia
UDHRBench: Benchmarking Human Rights in Large Language Models
Oct 10, 2026Alak Raad, Noreldin Morssy, Zeerak Suhail +2LLM Alignment LLM Safety Benchmarks
Hamad Bin Khalifa University, Qatar · Princess Sumaya University for Technology, Jordan
Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models
Oct 10, 2026Giovana Kerche Bonás, Thiago Laitz, Thales Sales Almeida +1LLM Safety Benchmarks LLM Safety
Tropic AI · Maritaca AI

Oct 24, 2026J/K move · Enter open · S save

When Personas Become Harmful: Guiding LLMs with Global Citizenship Values Alignment

Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee+1

University of Michigan · LG AI Research · University of Illinois at Chicago

LLM Safety LLM Safety Alignment Personalized Language Model Alignment LLM Safety Benchmarks

Open paper AskPDF arXiv

LLMs are increasingly being used by users from diverse demographics with pluralistic values and distinct backgrounds. Maintaining safety while respecting value-laden personas is uniquely challenging because persona-conditioned requests are often not overtly malicious, unlike those in existing safety benchmarks. Alignment methods aimed at adversarial requests therefore may miss the harm that emerges from persona use, leaving models to either refuse legitimate value expression or blindly follow unsafe personas. We construct a dataset of 11k controversial topics paired with value-based personas mapping to value dimensions from the Schwartz Value Survey. Through an agentic pipeline that iteratively refines the topic and persona, we surface and collect harmful responses from well-aligned commercial LLMs with a more than 47% success rate. We use UNESCO's global citizenship principles as a minimal common ground for distinguishing valid value expressions from harmful behaviors, and introduce a controllable alignment model that lets users balance persona adherence and safety at inference time. The resulting model is safer and more persona-adherent than baselines on most samples.