LLM Safety Evaluation

LLM: Large Language Model

The field tests whether language models refuse harmful requests, handle benign sensitive tasks helpfully, and remain safe when prompts, context, or tools change. Common approaches include fixed safety benchmarks, adversarial red-teaming and jailbreak testing, multi-turn agent simulations, domain-specific scenarios, and automated or human judging. Researchers increasingly measure more than refusal rates or attack success, including harm severity, omissions, false refusals, reasoning and tool-use failures, and effects on users. A central challenge is that results can shift with benchmark design, evaluation conditions, and judge choice, while stronger safeguards may reduce helpfulness and static tests may not predict behavior in deployment.

Latest papers

261 papers29 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing
May 11, 2026Qinghua Mao, Xi Lin, Jinze Gu +3Knowledge Editing Knowledge Conflicts in Language Models
School of Computer Science · Shanghai Jiao Tong University
FragBench: Cross-Session Attacks Hidden in Benign-Looking Fragments
May 10, 2026Astha Mehta, Niruthiha Selvanayagam, Cedric Lam +10LLM Safety Benchmarks Language Model Safety Evaluation
Isabella · ÉTS Montreal / SPAR · Mila / Lida Safety
GLiGuard: Schema-Conditioned Classification for LLM Safeguard
May 8, 2026Urchade Zaratiana, Mary Newhauser, George Hurn-Maloney +1Jailbreak Defense Text Classification
Fastino Labs
Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonLLM Safety Benchmarks LLM Evaluation
University of Bath
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
May 7, 2026Shihao Weng, Yang Feng, Xiaofei XieLLM-as-a-Judge AI Agent Safety
Nanjing University Nanjing, China · Singapore Management University Singapore
Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering
May 7, 2026Xiaomin Li, Jianheng Hou, Zheyuan Deng +6LLM Safety LLM Safety Evaluation
Harvard University · University of Southern California · Brown University +6
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
May 7, 2026Dasol Choi, Eugenia Kim, Jaewon Noh +14LLM Safety Benchmarks Multilingual Language Model Evaluation
AIM Intelligence · Microsoft · Korea AISI +7
Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry
May 6, 2026Oriana Presacan, Andreea Grama, Larisa Irimină +6Selective Prediction Healthcare
National University of Science and Technology Politehnica Bucharest · Psychiatric Hospital Doctor Gheorghe Preda · Oslo Metropolitan University +3
Safety and accuracy follow different scaling laws in clinical large language models
May 5, 2026Sebastian Wind, Tri-Thien Nguyen, Jeta Sopa +9LLM Safety Benchmarks Retrieval-Augmented Generation
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Radiology, University Hospital Erlangen, Erlangen, Germany · Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany +3
Code World Model Preparedness Report
May 1, 2026Daniel Song, Peter Ney, Cristina Menghini +21AI Safety Evaluation LLM Safety Evaluation
MSL Preparedness Team, AI Security Team
ARMOR 2025: A Military-Aligned Benchmark for Evaluating Large Language Model Safety Beyond Civilian Contexts
Apr 30, 2026Sydney Johns, Heng Jin, Chaoyu Zhang +2LLM Safety Benchmarks LLM Safety Alignment
Virginia Polytechnic Institute and State University, Virginia, United States
Useless but Safe? Benchmarking Utility Recovery with User Intent Clarification in Multi-Turn Conversations
Apr 29, 2026Mingqian Zheng, Malia Morgan, Liwei Jiang +2LLM Safety Benchmarks Language Model Safety Evaluation
Carnegie Mellon University · Allen Institute for AI · University of Washington
Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control
Apr 29, 2026Mahiro Nakao, Kazuhiro TakemotoHealthcare Robotics Language Model-Based Control
Kyushu Institute of Technology Iizuka, Fukuoka, Japan
Tatemae: Detecting Alignment Faking via Tool Selection in LLMs
Apr 29, 2026Matteo Leonesi, Francesco Belardinelli, Flavio Corradini +1Deception in Language Models LLM Safety Alignment
Department of Computing Imperial College London
From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language Models
Apr 28, 2026Mengya Hu, Qiong Wei, Sandeep AtluriLanguage Model Safety Evaluation LLM Safety
Responsible AI, Microsoft
Grammar-Constrained Refinement of Safety Operational Rules Using Language in the Loop: What Could Go Wrong
Apr 26, 2026Khouloud Gaaloul, Zaid Ghazal, Madhu Latha Pulimi +1Cyber-Physical Systems Safety-Critical Control
University of Michigan–Dearborn Dearborn, MI, USA
AI Safety Training Can be Clinically Harmful
Apr 25, 2026Suhas BN, Andrew M. Sherrill, Rosa I. Arriaga +2Mental Health Counseling LLM Alignment
College of Information Sciences and Technology, Penn State University, USA · Department of Psychiatry and Behavioral Sciences, Emory University, USA · School of Interactive Computing, Georgia Tech, USA +1
Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards
Apr 25, 2026Taha Hammadia, Lucas Rea, Ahmad Mohammad Saber +2LLM Jailbreak Attacks LLM Safety Evaluation
ECE Department, University of Toronto, Toronto, ON Canada · CIISE, Concordia University, Montréal, QC, Canada
Discovering Agentic Safety Specifications from 1-Bit Danger Signals
Apr 25, 2026Víctor GallegoAgentic Reasoning LLM Safety Evaluation
Komorebi AI
SHAPE: Unifying Safety, Helpfulness and Pedagogy for Educational LLMs
Apr 24, 2026Sihang Zhao, Kangrui Yu, Youliang Yuan +2Intelligent Tutoring Systems Generative AI in Education
Center for Data Science, New York University Shanghai · Courant Institute of Mathematical Sciences, New York University · School of Data Science, The Chinese University of Hong Kong, Shenzhen
Large Language Models Outperform Humans in Fraud Detection and Resistance to Motivated Investor Pressure
Apr 22, 2026Nattavudh PowdthaveeFinancial Fraud Detection LLM Decision-Making
Nanyang Technological University
Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents
Apr 22, 2026Yeran GamageAI Agent Monitoring LLM Security
University of South Florida · Independent AI Security Researcher
When Safety Fails Before the Answer: Benchmarking Harmful Behavior Detection in Reasoning Chains
Apr 21, 2026Ishita Kakkar, Enze Zhang, Rheeya Uppaal +1AI Safety Evaluation LLM Safety Evaluation
Department of Computer Sciences, University of Wisconsin-Madison
Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety
Apr 20, 2026Marcello Galisai, Susanna Cifani, Francesco Giarrusso +5LLM Safety Benchmarks Adversarial Attacks on LLMs
DEXAI – Icaro Lab · Sapienza University of Rome · Sant’Anna School of Advanced Studies
Using large language models for embodied planning introduces systematic safety risks
Apr 20, 2026Tao Zhang, Kaixian Qu, Zhibin Li +4Large Language Model-Based Robot Planning Robot Safety
ETH Zurich, Zurich, Switzerland. · University College London, London, United Kingdom. · Stanford University, Stanford, California, United States. +2
TPS-CalcBench: A Benchmark and Diagnostic Evaluation Framework for LLM Analytical Calculation Competence in Hypersonic Thermal Protection System Engineering
Apr 20, 2026Jinglai Zheng, Chuhan Qiao, Haiming HuangSynthetic Benchmark Generation LLM Safety Evaluation
School of Civil Engineering, Beijing Jiaotong University, Beijing 100044, China
MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
Apr 20, 2026Suhyun Lee, Palakorn Achananuparp, Neemesh Yadav +2Mental Health Counseling LLM Evaluation
Department of Artificial Intelligence, Hanyang University, Seoul, Republic of Korea · School of Computing and Information Systems, Singapore Management University, Singapore
On Safety Risks in Experience-Driven Self-Evolving Agents
Apr 18, 2026Weixiang Zhao, Yichen Zhang, Yingshuo Wang +8LLM Safety Evaluation Over-Refusal
Harbin Institute of Technology · Singapore Management University · SERES
When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
Apr 18, 2026Yuheng Chen, Zhiyu Wu, Bowen Cheng +2Adversarial Attacks on LLMs LLM Safety Alignment
Kagoshima University · Fudan University · China University of Petroleum-Beijing
HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine
Apr 18, 2026Brandon DentHealthcare AI Agent Benchmarks
GOATnote Inc.

May 11, 2026J/K move · Enter open · S save

Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing

Qinghua Mao, Xi Lin, Jinze Gu, Jun Wu+2

School of Computer Science · Shanghai Jiao Tong University

Knowledge Editing Knowledge Conflicts in Language Models LLM Security LLM Safety Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) increasingly rely on knowledge editing to support knowledge-intensive reasoning, but this flexibility also introduces critical safety risks: adversaries can inject malicious or misleading knowledge that corrupts downstream reasoning and leads to harmful outcomes. Existing knowledge editing benchmarks primarily focus on editing efficacy and lack a unified framework for systematically evaluating the safety implications of edited knowledge on reasoning behavior. To address this gap, we present EditRisk-Bench, a benchmark for systematically evaluating safety risks of knowledge-intensive reasoning under malicious knowledge editing. Unlike prior benchmarks that mainly emphasize edit success, generalization, and locality, EditRisk-Bench focuses on how injected knowledge affects downstream reasoning behavior and reliability. It integrates diverse malicious scenarios, including misinformation, bias, and safety violations, together with multi-level knowledge-intensive reasoning tasks and representative editing strategies within a unified evaluation framework measuring attack effectiveness, reasoning correctness, and side effects. Extensive experiments on both open-source and closed-source LLMs show that malicious knowledge editing can reliably induce incorrect or unsafe reasoning while largely preserving general capabilities, making such risks difficult to detect. We further identify several key factors influencing these risks, including edit scale, knowledge characteristics, and reasoning complexity. EditRisk-Bench provides an extensible testbed for understanding and mitigating safety risks in knowledge editing for LLMs.