LLM Safety Benchmarks

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
May 3, 2026Jialin Song, Xiaodong Liu, Weiwei Yang +4LLM Safety Benchmarks Adversarial Prompt Generation
Simon Fraser University · Microsoft Research
BabelSafe: A Policy-Grounded Multilingual Safety Benchmark for LLMs
May 1, 2026Yunhan Zhao, Zhaorun Chen, Xingjun Ma +1LLM Safety Benchmarks Multilingual Language Models
University of Illinois Urbana-Champaign · Fudan University · University of Chicago
ARMOR 2025: A Military-Aligned Benchmark for Evaluating Large Language Model Safety Beyond Civilian Contexts
Apr 30, 2026Sydney Johns, Heng Jin, Chaoyu Zhang +2LLM Safety Benchmarks LLM Safety Alignment
Virginia Polytechnic Institute and State University, Virginia, United States
Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations
Apr 29, 2026Mingqian Zheng, Malia Morgan, Liwei Jiang +2LLM Safety Benchmarks Language Model Safety Evaluation
Carnegie Mellon University · Allen Institute for AI · University of Washington
How Sensitive Are Safety Benchmarks to Judge Configuration Choices?
Apr 27, 2026Xinran ZhangLLM Safety Benchmarks LLM-as-a-Judge
University of California, Berkeley, Berkeley CA 94720, USA
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models
Apr 21, 2026Josue Torres-Fonseca, Naihao Deng, Yinpei Dai +5LLM Safety Benchmarks Large Language Model-Based Robot Planning
University of Michigan · Boise State University
HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human-LLM Collaborative Writing
Apr 21, 2026Euntae Kim, Soomin Han, Buru ChangLLM Safety Benchmarks Language Model Safety Evaluation
Korea University · Sogang University
Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
Apr 20, 2026Marcello Galisai, Susanna Cifani, Francesco Giarrusso +5LLM Safety Benchmarks Adversarial Attacks on LLMs
DEXAI – Icaro Lab · Sapienza University of Rome · Sant’Anna School of Advanced Studies
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
Apr 17, 2026Manh Luong, Tamas Abraham, Junae Kim +6LLM Safety Benchmarks Language Model Safety Evaluation
Monash University, Australia · Defence Science and Technology Group, Australia
IatroBench: A Pre-Registered Benchmark of Clinical Omission in Language Models
Apr 9, 2026David GringrasLLM Safety Benchmarks Healthcare
Harvard T.H. Chan School of Public Health
AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
Apr 3, 2026Yunhao Feng, Yifan Ding, Yingshui Tan +6LLM Safety Benchmarks Computer-Use Agent Benchmarks
Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Mar 8, 2026David GringrasLLM Safety Benchmarks LLM Agent Evaluation
Harvard University
MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models
Mar 3, 2026Zhongxi Wang, Yueqian Lin, Jingyang Zhang +3LLM Safety Benchmarks Language Model Safety Evaluation
1Duke University · 2Virtue AI
Quantifying Frontier LLM Capabilities for Container Sandbox Escape
Mar 1, 2026Rahul Marchand, Art O Cathain, Jerome Wynne +5LLM Safety Benchmarks LLM Agent Security
University of Oxford · UK AI Security Institute
SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models
Feb 3, 2026Omar Abdelnasser, Fatemah Alharbi, Khaled Khasawneh +2LLM Safety Benchmarks Multilingual Language Model Evaluation
Compumacy for Artificial Intelligence Solutions, Cairo, Egypt · Computer Science Department, College of Computer Science and Engineering, Taibah University, Yanbu 46522, KSA · Electrical and Computer Engineering Department, George Mason University, VA, USA +2
Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context
Jan 25, 2026Zhihao Zhang, Liting Huang, Guanghao Wu +3LLM Safety Benchmarks Language Model Safety Evaluation
Macquarie University1 · UTS2 · MBZUAI3 +1
Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
Jan 7, 2026Iago Alves Brito, Walcy Santos Rezende Rios, Julia Soares Dollis +2LLM Safety Benchmarks LLM Alignment
Advanced Knowledge Center for Immersive Technologies (AKCIT) Federal University of Goiás (UFG)
SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
Aug 21, 2025Xiangyang Zhu, Yuan Tian, Chunyi Li +3LLM Safety Benchmarks Synthetic Benchmark Generation
Shanghai AI Lab · East China Normal University
Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20LLM Safety Benchmarks Language Model Safety Evaluation
Cherise · School of Computation, Information and Technology, Technical University of Munich (TUM), Munich, Germany · Department of Engineering Science, University of Oxford, Oxford, United Kingdom +9
CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
Jan 24, 2025Guangzhi Sun, Xiao Zhan, Shutong Feng +2LLM Safety Benchmarks Language Model Safety Evaluation
Trinity College, University of Cambridge, Cambridge, United Kingdom · Department of Informatics, King’s College London, United Kingdom · Institut für Informatik, Heinrich-Heine-Universität Düsseldorf, Germany +1
Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges
Date pendingRui Yang, Shuang Huang, Junhua Liu +7LLM Safety Benchmarks LLM-as-a-Judge
Anhui SparkShield Intelligent Technology · iFLYTEK · Peking University +1

May 3, 2026, cs.CLJ/K move · Enter open · S save

MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety

Jialin Song, Xiaodong Liu, Weiwei Yang, Wuyang Chen+3

Simon Fraser University · Microsoft Research

LLM Safety Benchmarks Adversarial Prompt Generation Language Model Safety Evaluation Adversarial Scenario Generation Jailbreak Attacks LLM Jailbreak Attacks+1

Open paper AskPDF arXiv

We present MultiBreak, a scalable and diverse multi-turn jailbreak benchmark to evaluate large language model (LLM) safety. Multi-turn jailbreaks mimic natural conversational settings, making them easier to bypass safety-aligned LLM than single-turn jailbreaks. Existing multi-turn benchmarks are limited in size or rely heavily on templates, which restrict their diversity. To address this gap, we unify a wide range of harmful jailbreak intents, and introduce an active learning pipeline for expanding high-quality multi-turn adversarial prompts, where a generator is iteratively fine-tuned to produce stronger attack candidates, guided by uncertainty-based refinement. Our MultiBreak includes 10,389 multi-turn adversarial prompts, spans 2,665 distinct harmful intents, and covers the most diverse set of topics to date. Empirical evaluation shows that our benchmark achieves up to a 54.0 and 34.6 higher attack success rate (ASR)} than the second-best dataset on DeepSeek-R1-7B and GPT-4.1-mini, respectively. More importantly, safety evaluations suggest that diverse attack categories uncover fine-grained LLM vulnerabilities}, and categories that appear benign under single-turn can exhibit substantially higher adversarial effectiveness in multi-turn scenarios. These findings highlight persistent vulnerabilities of LLMs under realistic adversarial settings and establish MultiBreak as a scalable resource for advancing LLM safety.

Momentum

Related topics

Latest papers 141