LLM Safety Benchmarks

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

23 papers in the last four weeks, up 44% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue
Sep 1, 2026Stephanie Fong, Yiwen Jiang, Zimu Wang +12LLM Safety Benchmarks Social Bias in Language Models
Monash University · University of Liverpool · University of Edinburgh +2
CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs
Sep 1, 2026Maryam Alshehyari, Dushyant Singh Chauhan, Samuele Poppi +3LLM Safety Benchmarks Direct Preference Optimization
Computing and Mathematical Science Division Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Same Request, Different Boundary: Evaluating Cybersecurity Assistance across Conversational Contexts
Sep 1, 2026Rui Yang, Yang Hong, Yichao Xu +3LLM Safety Benchmarks LLM Refusal Behavior
Johns Hopkins University
WildSEEK: Evaluating Language Models for Information-Seeking
Aug 31, 2026Tanise Ceron, Joachim Baumann, Elisa Bassignana +3LLM Safety Benchmarks LLM Sycophancy
1Bocconi University · 2Stanford University · 3IT University of Copenhagen +1
Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
Aug 25, 2026Yujing Chang, Thinh Pham, Van-Phat Thai +4LLM Safety Benchmarks LLM Evaluation
1ATMRI, Nanyang Technological University (NTU), Singapore · Centre of AI Research, VinUniversity, Vietnam · School of Management, Indian Institute of Technology Mandi, India
Stress Testing Unlearning Algorithms
Aug 23, 2026Noam Diamant, Neta Glazer, Ethan FetayaLLM Safety Benchmarks Machine Unlearning
Bar-Ilan University
Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance
Aug 21, 2026Alexander Thomas, Hubert P. H. Shum, Darren Nellis +4LLM Safety Benchmarks Language Model Safety Evaluation
NCB Hazcheck Limited, UK · Durham University, UK
AeroCopilotBench: Safety-Gated Evaluation of LLM Agents on Aircraft Emergency Procedures in an Executable Cockpit
Aug 17, 2026Yuchen Yuan, Zhenghuang Wu, Yuangan Li +2LLM Safety Benchmarks AI Agent Safety Benchmarks
School of Aeronautic Science and Engineering, Beihang University
Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluation
Aug 12, 2026Haoyu Zhang, Haowen Xu, Xiao Luo +7LLM Safety Benchmarks Language Model Safety Evaluation
Northeastern University
TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent
Aug 10, 2026Waleed Jamil, Raphael SchmittLLM Safety Benchmarks LLM Safety Evaluation
1Independent Researcher, Edinburgh, United Kingdom · School of Computation, Information and Technology, Technical University of Munich, Germany · Institute of General Practice, Faculty of Medicine and Medical Center, University of Freiburg, Germany
Beyond Detection: Evaluating Defensive LLMs Against AI-Generated Social Engineering in Live Turn-by-Turn Interaction
Aug 10, 2026Yuqiao Xu, Osama Zafar, Alexander Nemecek +1LLM Safety Benchmarks LLM Safety
Case Western Reserve University
ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models
Aug 10, 2026Yilin Jiang, Xiaorong Zhu, Fei Tan +9LLM Safety Benchmarks LLM Evaluation
1East China Normal University · 2The Hong Kong University of Science and Technology (Guangzhou) · 3Shanghai Jiao Tong University +1
When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs
Aug 9, 2026Yu Ma, Hongli Shi, Jing Li +2LLM Safety Benchmarks Language Model Safety Evaluation
Google · University of New South Wales · University of Technology Sydney +2
Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
Aug 8, 2026Alizishaan Khatri, Dun Li ChanLLM Safety Benchmarks LLM Inference
Wrynx Inc · INTI International College Penang
MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs
Aug 8, 2026Yilian Shi, Yueming Lyu, Haoxiang Tan +5LLM Safety Benchmarks Multimodal Robustness
Nanjing University
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Aug 6, 2026Ro Encarnación, Tina Behzad, Emma Lurie +1LLM Safety Benchmarks LLM Evaluation
University of Pennsylvania · Stony Brook University
From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs
Aug 6, 2026Jiawei Qiu, Yichen Xu, Jianzhe Ma +5LLM Safety Benchmarks Language Model Safety Evaluation
School of Information, Renmin University of China Beijing, China
Item Response Theory for AI Safety
Aug 5, 2026Joshua Fonseca Rivera, Neil Shah, David Demitri Africa +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent · 1Independent · UK AI Security Institute +1
DelusionEval: Measuring Delusion-Linked Behaviors in AI Chatbots
Aug 5, 2026Jared Moore, Andrea Mock, Yifan Mai +9LLM Safety Benchmarks LLM Safety
Stanford University Stanford, California, USA · University of Chicago Chicago, Illinois, USA · Carnegie Mellon University Pittsburgh, Pennsylvania, USA +2
Chain-of-Thought Monitoring Can Be Unreliable in Implicit-Influence Settings
Aug 5, 2026Agatha Duzan, Asa Cooper SticklandLLM Safety Benchmarks CoT Monitoring
EPFL · UK AI Security Institute
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Aug 5, 2026Mouxiao Bian, Zhi Chen, Ruiyao Chen +8LLM Safety Benchmarks Healthcare
Shanghai Artificial Intelligence Laboratory, Shanhai, China · Macau University of Science and Technology, Macau, China · First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China +1
ADMITBench: A Safety-Governed Reference Framework for Evaluating the Admissibility of Industrial LLM Advisories
Aug 4, 2026Yash Misra, Javal Vyas, Siddharth Gutta +1LLM Safety Benchmarks AI-Assisted Decision Making
Refiant, Inc. · Imperial College London
CARE-Bench: Benchmarking Patient-Facing LLM Triage
Aug 4, 2026Yining Hua, Hongbin Na, Cyrus AyubchaLLM Safety Benchmarks Clinical Triage
1Harvard University · University of Technology Sydney
AI Security Leaderboard: Methodology, Results and Minimal Standard
Aug 4, 2026Jasper Timm, Lukas Struppek, Ziwei Xu +12LLM Safety Benchmarks Cybersecurity
onepot-Bench 0: towards lab-aware in silico chemistry benchmarks
Aug 3, 2026Brandon Wang, Andrei S. Tyrin, Daniil A. BoikoLLM Safety Benchmarks LLM Evaluation
Onepot AI, Inc.
MedPRESS: A Multi-turn Benchmark for Patient-Pressure-Induced Medical Sycophancy in LLMs
Aug 3, 2026Saman Sarker Joy, Niloy FarhanLLM Safety Benchmarks LLM Sycophancy
Universiti Malaya, Malaysia · BRAC University, Bangladesh
EduZone: A Framework for Evaluating LLM Safety for K-12 Students and Teachers
Aug 3, 2026Junyeong Park, Jieun Han, Haneul Yoo +3LLM Safety Benchmarks Generative AI in Education
KAIST · NYU · Google Cloud AI Research
A Blind Spot in Alignment: Quantifying Biosecurity Risks in Large Language Models
Aug 3, 2026Shu Quan, Tianfang Hao, Sitong Fang +8LLM Safety Benchmarks LLM Safety
Institute for Artificial Intelligence, Peking University · The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification
Aug 2, 2026Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous +1LLM Safety Benchmarks Arabic NLP
Northwestern University in Qatar, Qatar · Hamad Bin Khalifa University, Qatar

Sep 1, 2026, cs.CLJ/K move · Enter open · S save

SDARE-Bench: Evaluating Large Language Models on Conversational Stigma Detection and Response in Dyadic and Group Dialogue

Stephanie Fong, Yiwen Jiang, Zimu Wang, Hongxi Yang+11

Monash University · University of Liverpool · University of Edinburgh+2

LLM Safety Benchmarks Social Bias in Language Models LLM Safety Evaluation

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used in advice seeking and decision making that may affect social judgements. Despite stigma's profound effects on people and communities, benchmarks remain scarce. Existing general-domain evaluations typically rely on static prompts and fixed-format tasks, overlooking conversational contexts and audience effects in everyday communication. To address these gaps, we introduce SDARE-Bench, the first scenario-based benchmark evaluating both stigma detection and open-ended response generation in LLMs, comprising 1,138 dyadic queries and 1,388 group dialogue. Empirical results across 8 LLMs consistently demonstrate poor identification of stigma components, especially in group dialogues. In open-ended response generation, stigma expression was substantially higher in group settings than in dyadic, with weaker resistance to stigma and more unrealistic advice. Responses were evaluated using a classifier trained on 1,392 human annotated responses. In constructed group pressure settings, stigma expression rates further increased to a striking average of 97.5%. Our findings identify stigma response as a recurring LLM safety vulnerability, especially in socially complex conversational contexts.

Momentum

Related topics

Latest papers 141