LLM Safety Benchmarks

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense
Oct 7, 2026Zhankai Ye, Yanning Wang, Yukai Jin +5Refusal Behavior in Language Models LLM Safety Alignment
Florida State University · Texas Christian University · University of Pennsylvania +1
Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
Oct 6, 2026Pavan MaddulaLanguage Model Safety Evaluation Adversarial Robustness of Language Models
Independent Researcher
The Pushback Paradox: A Two-Probe Diagnostic for Language Model Compliance
Oct 5, 2026Stefan Bühler, David Exler, Markus Reischl +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent Researcher · Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, Karlsruhe · Duale Hochschule Baden-Württemberg, Ravensburg
False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Oct 1, 2026Amit Singh Bhatti, Vishal VaddinaLLM Safety Benchmarks Distribution Shift Robustness
Quantiphi Analytics
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1LLM Safety Benchmarks LLM Auditing
Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA
RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
Sep 28, 2026Jianpeng Zhao, Haihua Xu, Haoyang Zhang +7LLM Safety Benchmarks LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China
Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails
Sep 27, 2026Gert Lek, Abele Malan, Chaoyi Zhu +3LLM Safety Benchmarks LLM Guardrails
University of Neuchâtel · Delft University of Technology · IBM Research +1
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Sep 24, 2026Ruoqi Guo, Yi Liu, Gelei Deng +6LLM Safety Benchmarks LLM Alignment
Griffith University · Nanyang Technological University · UNSW +4
ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts
Sep 24, 2026Firoj Alam, Md. Rafiul Biswas, Mohamed Bayan Kmainasi +5LLM Safety Benchmarks Arabic NLP
Qatar Computing Research Institute, Qatar · Hamad Bin Khalifa University, Qatar · APAVI.AI, France +1
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Sep 24, 2026Suvradip Paul, Chandra Bhushan, Harsh Sharma +4LLM Safety Benchmarks Financial Services
National Payments Corporation of India Mumbai, India
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
Sep 21, 2026Fatih Deniz, Yazan Boshmaf, Issa KhalilLLM Safety Benchmarks LLM Evaluation
Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar
SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Sep 17, 2026Chenxi Wu, Zimu Wang, Haiyang Zhang +2LLM Safety Benchmarks AI Risk Management
School of Advanced Technology, Xi’an Jiaotong-Liverpool University
Benchmarking LLM Compliance with China AI Generated Content Regulations
Sep 17, 2026Chenrui Cui, Hongye Fang, Lisha Song +3LLM Safety Benchmarks LLM Alignment
School of Computer Science and Technology, Tongji University, Shanghai, China · Law School of Tongji University, Shanghai, China
From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization
Sep 17, 2026Diba Afroze, Xingli Zhang, Yazhou Tu +1LLM Safety Benchmarks LLM Safety Evaluation
University of Louisiana at Lafayette, Lafayette, LA 70503, USA · Auburn University, Auburn, AL 36849, USA
Do Frontier Models Seek Safety Evidence Before Acting?
Sep 15, 2026Omer TafveezLLM Safety Benchmarks LLM Safety Evaluation
University of Michigan
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2LLM Safety Benchmarks Healthcare
Department of Neurosurgery, National Hospital Organization Kagoshima Medical Center, Kagoshima, Japan · Department of Neurology, Imamura General Hospital, Kagoshima, Japan · Iwamoto Neurosurgery, Kanoya, Kagoshima, Japan +1
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9LLM Safety Benchmarks Mental Health Counseling
School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4
RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety
Sep 11, 2026Adithiyan Rajan Indira Saravanan, Kathleen C. FraserLLM Safety Benchmarks LLM Safety
Faculty of Engineering University of Ottawa Ottawa, Canada
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
Sep 11, 2026Abhinav Rajeev Kumar, Harshit Arora, Varun Singh +1LLM Safety Benchmarks LLM Guardrails
SRM Institute of Science and Technology
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
Sep 8, 2026Leyuan Tang, Kangda Wei, Tianyu Jiang +1LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Engineering, Texas A&M University · Department of Computer Science, University of Cincinnati
Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
Sep 8, 2026Tejasvi C. AddagadaLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts
Sep 8, 2026Yongxi Zhou, Wenbo Ye, Yuanzhe Liu +2LLM Safety Benchmarks LLM-as-a-Judge
Northeastern University, Massachusetts, USA · University of Southern California, California, USA · Georgia Institute of Technology, Georgia, USA
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Sep 7, 2026Pengfei Li, Naufal Suryanto, Sicheng Zhang +2LLM Safety Benchmarks LLM Safety Evaluation
Khalifa University · University of Western Australia
The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability
Sep 7, 2026Xin XuLLM Safety Benchmarks LLM Safety Evaluation
Carnegie Mellon University
FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models
Sep 3, 2026Yalun Wu, Junfeng Fang, Jiawei Wang +6LLM Safety Benchmarks LLM Safety Evaluation
NExT++ Lab, School of Computing, National University of Singapore · 3Xiamen University · University of Manchester +4
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India
FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs
Sep 2, 2026Zhengyi Jin, Ru Zhang, Xiao Chen +5LLM Safety Benchmarks Jailbreak Robustness
School of Cyberspace Security, Beijing University of Posts and Telecommunications, China
SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue
Sep 1, 2026Stephanie Fong, Yiwen Jiang, Zimu Wang +12LLM Safety Benchmarks Social Bias in Language Models
Monash University · University of Liverpool · University of Edinburgh +2
CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs
Sep 1, 2026Maryam Alshehyari, Dushyant Singh Chauhan, Samuele Poppi +3LLM Safety Benchmarks Direct Preference Optimization
Computing and Mathematical Science Division Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts
Sep 1, 2026Rui Yang, Yang Hong, Yichao Xu +3LLM Safety Benchmarks LLM Refusal Behavior
Johns Hopkins University
WildSEEK: Evaluating Language Models for Information-Seeking
Aug 31, 2026Tanise Ceron, Joachim Baumann, Elisa Bassignana +3LLM Safety Benchmarks LLM Sycophancy
1Bocconi University · 2Stanford University · 3IT University of Copenhagen +1
Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
Aug 25, 2026Yujing Chang, Thinh Pham, Van-Phat Thai +4LLM Safety Benchmarks LLM Evaluation
1ATMRI, Nanyang Technological University (NTU), Singapore · Centre of AI Research, VinUniversity, Vietnam · School of Management, Indian Institute of Technology Mandi, India
Stress Testing Unlearning Algorithms
Aug 23, 2026Noam Diamant, Neta Glazer, Ethan FetayaLLM Safety Benchmarks Machine Unlearning
Bar-Ilan University
Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance
Aug 21, 2026Alexander Thomas, Hubert P. H. Shum, Darren Nellis +4LLM Safety Benchmarks Language Model Safety Evaluation
NCB Hazcheck Limited, UK · Durham University, UK
AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit
Aug 17, 2026Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2LLM Safety Benchmarks AI Agent Safety Benchmarks
School of Aeronautic Science and Engineering, Beihang University
Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluation
Aug 12, 2026Haoyu Zhang, Haowen Xu, Xiao Luo +7LLM Safety Benchmarks Language Model Safety Evaluation
Northeastern University
TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent
Aug 10, 2026Waleed Jamil, Raphael SchmittLLM Safety Benchmarks LLM Safety Evaluation
1Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany
Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction
Aug 10, 2026Yuqiao Xu, Osama Zafar, Alexander Nemecek +1LLM Safety Benchmarks LLM Safety
Case Western Reserve University
ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models
Aug 10, 2026Yilin Jiang, Xiaorong Zhu, Fei Tan +9LLM Safety Benchmarks LLM Evaluation
1East China Normal University · 2The Hong Kong University of Science and Technology (Guangzhou) · 3Shanghai Jiao Tong University +1
When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs
Aug 9, 2026Yu Ma, Hongli Shi, Jing Li +2LLM Safety Benchmarks Language Model Safety Evaluation
Google · University of New South Wales · University of Technology Sydney +2
Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
Aug 8, 2026Alizishaan Khatri, Dun Li ChanLLM Safety Benchmarks LLM Inference
Wrynx Inc · INTI International College Penang
MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
Aug 8, 2026Yilian Shi, Yueming Lyu, Haoxiang Tan +5LLM Safety Benchmarks Multimodal Robustness
Nanjing University
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Aug 6, 2026Ro Encarnación, Tina Behzad, Emma Lurie +1LLM Safety Benchmarks LLM Evaluation
University of Pennsylvania · Stony Brook University
From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs
Aug 6, 2026Jiawei Qiu, Yichen Xu, Jianzhe Ma +5LLM Safety Benchmarks Language Model Safety Evaluation
School of Information, Renmin University of China Beijing, China
Item Response Theory for AI Safety
Aug 5, 2026Joshua Fonseca Rivera, Neil Shah, David Demitri Africa +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent · 1Independent · UK AI Security Institute +1
DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
Aug 5, 2026Jared Moore, Andrea Mock, Yifan Mai +9LLM Safety Benchmarks LLM Safety
Stanford University Stanford, California, USA · University of Chicago Chicago, Illinois, USA · Carnegie Mellon University Pittsburgh, Pennsylvania, USA +2
Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings
Aug 5, 2026Agatha Duzan, Asa Cooper SticklandLLM Safety Benchmarks CoT Monitoring
EPFL · UK AI Security Institute
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Aug 5, 2026Mouxiao Bian, Zhi Chen, Ruiyao Chen +8LLM Safety Benchmarks Healthcare
Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1
ADMITBench: A Safety-Governed Reference Framework for Evaluating the Admissibility of Industrial LLM Advisories
Aug 4, 2026Yash Misra, Javal Vyas, Siddharth Gutta +1LLM Safety Benchmarks AI-Assisted Decision Making
Refiant, Inc. · Imperial College London
CARE-Bench: Benchmarking Patient-Facing LLM Triage
Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaLLM Safety Benchmarks Clinical Triage
1Harvard University · University of Technology Sydney
AI Security Leaderboard: Methodology, Results and Minimal Standard
Aug 4, 2026Jasper Timm, Lukas Struppek, Ziwei Xu +12LLM Safety Benchmarks Cybersecurity
onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
Aug 3, 2026Saman Sarker Joy, Niloy FarhanLLM Safety Benchmarks LLM Sycophancy
Universiti Malaya, Malaysia · BRAC University, Bangladesh
EduZone: A Framework for Evaluating LLM Safety for K-12 Students and Teachers
Aug 3, 2026Junyeong Park, Jieun Han, Haneul Yoo +3LLM Safety Benchmarks Generative AI in Education
KAIST · NYU · Google Cloud AI Research
A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models
Aug 3, 2026Shu Quan, Tianfang Hao, Sitong Fang +8LLM Safety Benchmarks LLM Safety
Institute for Artificial Intelligence, Peking University · The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification
Aug 2, 2026Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous +1LLM Safety Benchmarks Arabic NLP
Northwestern University in Qatar, Qatar · Hamad Bin Khalifa University, Qatar

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense

Zhankai Ye, Yanning Wang, Yukai Jin, Bo Mei+4

Florida State University · Texas Christian University · University of Pennsylvania+1

Refusal Behavior in Language Models LLM Safety Alignment LLM Jailbreak Attacks Language Model Safety Evaluation LLM Safety Benchmarks

Open paper AskPDF arXiv

Safety-aligned language models often refuse a harmful request stated directly but answer the same request inside a role-play or narrative wrapper. We measure this vulnerability across languages and registers: attack success on Qwen3-1.7B is already 89.4% in English and 93.0% in modern Chinese, and reaches 95.7% in Classical Chinese. We build GUISE, a benchmark for systematically studying this vulnerability. It includes parallel requests in English, modern Chinese, and Classical Chinese, matched harmful and benign pairs, wrapper types held out for evaluation, and a stricter criterion that counts warn-then-answer responses as attack successes. Representation analysis shows that language and register move harmful-request representations only slightly away from the model's refusal direction, whereas narrative wrappers move them much farther away. We propose AXIS, which combines preference optimisation with a rotation objective that aligns harmful-request representations with the refusal direction and a commitment objective that trains the model to refuse completely rather than produce a warn-then-answer response. Across Qwen3-1.7B, Qwen3-4B and GLM-4-9B, AXIS achieves the highest combined safety and usability score among the compared methods.

Momentum

Related topics

Latest papers 141