LLM Safety Benchmarks

LLM: Large Language Model

Overview

This field measures how reliably language models recognize and handle risks such as harmful requests, jailbreaks, bias, privacy violations, and unsafe tool use. Benchmarks use curated and expert-annotated scenarios, adversarial red-teaming, multi-turn and agent simulations, and increasingly multilingual, multimodal, and domain-specific tests. Researchers also compare refusal and moderation methods, use repeated sampling and statistical techniques to estimate reliability, and assess both harmful compliance and over-refusal. A central challenge is making results comparable and trustworthy while ensuring tests reflect changing, context-dependent behavior in real deployments.

Latest papers

CardsList

299 papers27 papers in the last four weeks, down 32% on the four weeks before. 0.2% of all new papers.

  1. MedGuardBench: A Patient-Oriented Benchmark for Personalized Guardrail Development

    Nov 17, 2026Sara Khosravi, Xiaofei Wen, Majid Sarrafzadeh +1LLM GuardrailsLLM Safety Benchmarks

  2. Double Blind Evals: Resolving the Dual Confidentiality Dilemma in AI Safety Auditing

    Nov 9, 2026Andrew Trask, Sol Messing, Vai Pahwa +27AI Safety EvaluationLLM Auditing

  3. FraudBench: A Legal Evaluation of AI Deception

    Nov 9, 2026Kevin Wei, Sumaya Nur Adan, Stephan Llerena +17Deception in Language ModelsLanguage Model Safety Evaluation

  4. System Prompts Affect Response Bias More Than Discriminability in LLM Safety

    Nov 8, 2026Mohan W. Gupta, Sandy TanwisuthLLM Safety EvaluationRefusal Behavior in Language Models

  5. TrustMod-SM: A Multi-Axis Benchmark for Evaluating Trustworthiness of LLMs in Social Media Content Moderation

    Nov 7, 2026Aizan Zafar, Sreekantam Sai Venkata, Gathik Jindal +4LLM Safety BenchmarksLLM Safety Evaluation

  6. Authorization Drift in LLM Agents: Evaluating Boundary Compliance in Multi-Step Tool Use

    Nov 6, 2026Chuxue Cao, Jingru Zeng, Boqin Yin +7LLM Agent SafetyTool Access Control for LLM Agents

  7. CAREFUL-Bench: Effect-Aware, Judge-Free Evaluation of Multi-Turn LLM Function Calling

    Nov 2, 2026Alexander Kovalenko, Artem RedchychTool-Use EvaluationTool Use

  8. What Does a Safety Label Measure? Auditing Situational-Safety Benchmarks for Multimodal LLMs

    Oct 31, 2026Wayner Barrios, Kenneth BarriosLLM Safety BenchmarksMLLM Safety

  9. HEARSAYBENCH: Can AI Navigate from Abstract Human Rights to Lived Lives?

    Oct 28, 2026Ava Iranmanesh, Sobhan Lotfi, Ali Iranmanesh +1LLM Safety BenchmarksLanguage Model Safety Evaluation

  10. When Support Crosses the Line: Evaluating Ethical Boundaries in LLM-Based Mental Health Support in the Chinese Context

    Oct 28, 2026Bichen Wang, Yufeng Jin, Yixin Sun +2Multi-Turn Language Model Safety EvaluationMental Health Conversational Agents

  11. Preserving Safety During Sequential Fine-Tuning: A Factorial and Geometric Analysis of Replay

    Oct 27, 2026George Ruellan, Diana Benavides-PradoSafety-Preserving Fine-Tuning of Language ModelsSafety-Utility Trade-Off in Language Models

  12. When Personas Become Harmful: Guiding LLMs with Global Citizenship Values Alignment

    Oct 24, 2026Siqi Shen, Lajanugen Logeswaran, Moontae Lee +2LLM SafetyLLM Safety Alignment

  13. BoardSafety: AI-Ready Data from Manufacturer Documentation for Development-Board Safety

    Oct 24, 2026Linjian Yang, Xinyan Wang, Kunpeng LiuAI Safety EvaluationLLM Safety Benchmarks

  14. Omni-UBench: Benchmarking Omni-Modal LLM-based Guardrails Across Safety, Utility, and Reliability

    Oct 22, 2026Junhao Wang, Yidou Chen, Li Lu +6LLM Safety BenchmarksMLLM Safety

  15. Scaling Laws for Safety Probes

    Oct 22, 2026Yuchen Wu, Jerry WeiLLM Safety MonitoringLanguage Model Probing

  16. TenetBench: A Clause-Anchored Russian Corpus for Evaluating Language Models against Professional Codes of Ethics

    Oct 21, 2026Yegor Anichkov, Valery Vinogradov, Andrey Furov +2Language Model Safety EvaluationLLM Safety Benchmarks

  17. Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

    Oct 21, 2026Hongyu Li, Xinlong Huang, Zhan Yu +13LLM Safety EvaluationLLM Safety Benchmarks

  18. Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents

    Oct 20, 2026Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang +1LLM Agent SafetyRefusal Behavior in Language Models

  19. Polarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial Framing

    Oct 20, 2026Diego Miguel Lozano, Erik Derner, Juan Antonio Pérez-Ortiz +1Language Model Safety EvaluationAdversarial Robustness of Language Models

  20. ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

    Oct 20, 2026Liping Yang, Yongqi Leng, Jingting Zheng +3LLM Safety BenchmarksLanguage Model Safety Evaluation

  21. African Languages AI Safety Readiness Report

    Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Tajuddeen Gwadabe +1Language Model Safety EvaluationAfrican Language Processing

  22. Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

    Oct 19, 2026Leon Tu Nguyen, Rohan Menon, Rafael Tappe Maestro +2Jailbreak Attack EvaluationLLM Safety Benchmarks

  23. AfriGuard: Safety and Robust Refusal For African Languages

    Oct 19, 2026Israel Abebe Azime, Jesujoba O. Alabi, Iffat Maab +20LLM SafetyLanguage Model Safety Evaluation

  24. VEILED-SH: Vulnerability Expression through Indirect Language Evaluation Dataset (Self-Harm)

    Oct 19, 2026Leanne Tan, Keith Chua, Shaun Khoo +7LLM Safety BenchmarksLanguage Model Safety Evaluation

  25. BrandEval: A Two-Track Multi-Agent Benchmark for Risk-Sensitive LLM Crisis Communication

    Oct 18, 2026Haotian Gan, Weidong Tang, Hanbin Sun +5LLM Agent EvaluationLLM Safety Benchmarks