LLM Safety Benchmarks

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense
Oct 7, 2026Zhankai Ye, Yanning Wang, Yukai Jin +5Refusal Behavior in Language Models LLM Safety Alignment
Florida State University · Texas Christian University · University of Pennsylvania +1
Quad-State Safety Evaluation of Open-Weight Large Language Models on Non-Canonical Inputs
Oct 6, 2026Pavan MaddulaLanguage Model Safety Evaluation Adversarial Robustness of Language Models
Independent Researcher
The Pushback Paradox: A Two-Probe Diagnostic for Language Model Compliance
Oct 5, 2026Stefan Bühler, David Exler, Markus Reischl +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent Researcher · Institute for Automation and Applied Informatics, Karlsruhe Institute of Technology, Karlsruhe · Duale Hochschule Baden-Württemberg, Ravensburg
False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift
Oct 1, 2026Amit Singh Bhatti, Vishal VaddinaLLM Safety Benchmarks Distribution Shift Robustness
Quantiphi Analytics
OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation
Oct 1, 2026Negin Ashrafi, Jia Luo, Stacey M. Frumm +1LLM Safety Benchmarks LLM Auditing
Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · Lowe Center for Thoracic Oncology, Dana-Farber Cancer Institute, and Harvard Medical School, Boston, MA, USA
RGDT-Bench: Benchmarking LLM Reasoning for Rule-Governed Decisions and Their Justifications
Sep 28, 2026Jianpeng Zhao, Haihua Xu, Haoyang Zhang +7LLM Safety Benchmarks LLM Decision-Making
State Key Laboratory of Internet of Things for Smart City and Institute of Smart City Technologies, University of Macau, Macau SAR, China · ByteDance, Beijing, China
Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails
Sep 27, 2026Gert Lek, Abele Malan, Chaoyi Zhu +3LLM Safety Benchmarks LLM Guardrails
University of Neuchâtel · Delft University of Technology · IBM Research +1
Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures
Sep 24, 2026Ruoqi Guo, Yi Liu, Gelei Deng +6LLM Safety Benchmarks LLM Alignment
Griffith University · Nanyang Technological University · UNSW +4
ArGuard Shared Task: Harmful Content Detection in Arabic Memes and LLM Prompts
Sep 24, 2026Firoj Alam, Md. Rafiul Biswas, Mohamed Bayan Kmainasi +5LLM Safety Benchmarks Arabic NLP
Qatar Computing Research Institute, Qatar · Hamad Bin Khalifa University, Qatar · APAVI.AI, France +1
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Sep 24, 2026Suvradip Paul, Chandra Bhushan, Harsh Sharma +4LLM Safety Benchmarks Financial Services
National Payments Corporation of India Mumbai, India
EADC: Evaluation of Advanced and Deep-level Compliance in Large Language Models
Sep 22, 2026Yan Zhang, Ruien Li, Yaoyao Peng +4LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison · Law School, University of Chinese Academy of Social Sciences
SSP-Bench: A Hybrid Data Generation Framework for Safety, Security, and Privacy Evaluation
Sep 21, 2026Fatih Deniz, Yazan Boshmaf, Issa KhalilLLM Safety Benchmarks LLM Evaluation
Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar
SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment
Sep 17, 2026Chenxi Wu, Zimu Wang, Haiyang Zhang +2LLM Safety Benchmarks AI Risk Management
School of Advanced Technology, Xi’an Jiaotong-Liverpool University
Benchmarking LLM Compliance with China AI Generated Content Regulations
Sep 17, 2026Chenrui Cui, Hongye Fang, Lisha Song +3LLM Safety Benchmarks LLM Alignment
School of Computer Science and Technology, Tongji University, Shanghai, China · Law School of Tongji University, Shanghai, China
From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization
Sep 17, 2026Diba Afroze, Xingli Zhang, Yazhou Tu +1LLM Safety Benchmarks LLM Safety Evaluation
University of Louisiana at Lafayette, Lafayette, LA 70503, USA · Auburn University, Auburn, AL 36849, USA
Do Frontier Models Seek Safety Evidence Before Acting?
Sep 15, 2026Omer TafveezLLM Safety Benchmarks LLM Safety Evaluation
University of Michigan
Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models
Sep 15, 2026Keisuke Masuda, Kazutaka Yatsushiro, Hirohumi Iwamoto +2LLM Safety Benchmarks Healthcare
Department of Neurosurgery, National Hospital Organization Kagoshima Medical Center, Kagoshima, Japan · Department of Neurology, Imamura General Hospital, Kagoshima, Japan · Iwamoto Neurosurgery, Kanoya, Kagoshima, Japan +1
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9LLM Safety Benchmarks Mental Health Counseling
School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4
DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis
Sep 11, 2026Abhinav Rajeev Kumar, Harshit Arora, Varun Singh +1LLM Safety Benchmarks LLM Guardrails
SRM Institute of Science and Technology
RAG-Safety-Bench: Reliable Evaluation of Retrieval-Augmented LLM Safety
Sep 11, 2026Adithiyan Rajan Indira Saravanan, Kathleen C. FraserLLM Safety Benchmarks LLM Safety
Faculty of Engineering University of Ottawa Ottawa, Canada
Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
Sep 8, 2026Leyuan Tang, Kangda Wei, Tianyu Jiang +1LLM Safety Benchmarks LLM Evaluation
Department of Computer Science and Engineering, Texas A&M University · Department of Computer Science, University of Cincinnati
Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
Sep 8, 2026Tejasvi C. AddagadaLLM Safety Benchmarks Multilingual Language Model Evaluation
Independent researcher
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts
Sep 8, 2026Yongxi Zhou, Wenbo Ye, Yuanzhe Liu +2LLM Safety Benchmarks LLM-as-a-Judge
Northeastern University, Massachusetts, USA · University of Southern California, California, USA · Georgia Institute of Technology, Georgia, USA
SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs
Sep 7, 2026Pengfei Li, Naufal Suryanto, Sicheng Zhang +2LLM Safety Benchmarks LLM Safety Evaluation
Khalifa University · University of Western Australia
The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability
Sep 7, 2026Xin XuLLM Safety Benchmarks LLM Safety Evaluation
Carnegie Mellon University
FLY-EVAL++: An Evidence-Driven Evaluation Protocol for Safety-Constrained Flight Prediction with Large Language Models
Sep 3, 2026Yalun Wu, Junfeng Fang, Jiawei Wang +6LLM Safety Benchmarks LLM Safety Evaluation
NExT++ Lab, School of Computing, National University of Singapore · 3Xiamen University · University of Manchester +4
Evaluating Criterion-Conditioned Behaviour of Large Language Models in Content Moderation
Sep 3, 2026Danting Zhang, Bei Peng, Robert LoftinLLM Safety Benchmarks LLM Evaluation
1Independent Researcher · University of Sheffield
IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks
Sep 3, 2026Saikat Mondal, Mamta, Deeksha Varshney +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Indian Institute of Technology Jodhpur, Jodhpur, India · King’s College London, London, UK · Indian Institute of Technology Patna, Patna, India
FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs
Sep 2, 2026Zhengyi Jin, Ru Zhang, Xiao Chen +5LLM Safety Benchmarks Jailbreak Robustness
School of Cyberspace Security, Beijing University of Posts and Telecommunications, China

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense

Zhankai Ye, Yanning Wang, Yukai Jin, Bo Mei+4

Florida State University · Texas Christian University · University of Pennsylvania+1

Refusal Behavior in Language Models LLM Safety Alignment LLM Jailbreak Attacks Language Model Safety Evaluation LLM Safety Benchmarks

Open paper AskPDF arXiv

Safety-aligned language models often refuse a harmful request stated directly but answer the same request inside a role-play or narrative wrapper. We measure this vulnerability across languages and registers: attack success on Qwen3-1.7B is already 89.4% in English and 93.0% in modern Chinese, and reaches 95.7% in Classical Chinese. We build GUISE, a benchmark for systematically studying this vulnerability. It includes parallel requests in English, modern Chinese, and Classical Chinese, matched harmful and benign pairs, wrapper types held out for evaluation, and a stricter criterion that counts warn-then-answer responses as attack successes. Representation analysis shows that language and register move harmful-request representations only slightly away from the model's refusal direction, whereas narrative wrappers move them much farther away. We propose AXIS, which combines preference optimisation with a rotation objective that aligns harmful-request representations with the refusal direction and a commitment objective that trains the model to refuse completely rather than produce a warn-then-answer response. Across Qwen3-1.7B, Qwen3-4B and GLM-4-9B, AXIS achieves the highest combined safety and usability score among the compared methods.

Momentum

Related topics

Latest papers 141