Language Model Safety Evaluation

Follow this topic Ask about this topic

Language model safety evaluation asks whether models handle harmful, ambiguous, and high-stakes requests safely while still helping with legitimate ones. Researchers use benchmarks such as HarmBench, JailbreakBench, and XSTest, alongside static and adaptive red-teaming, expert review, and automated judges; newer tests also examine multi-turn, tool-using, speech, and multimodal systems. Psychometric methods, repeated sampling, and internal activation probes can reveal risks that single-run refusal scores miss, while localized and domain-specific evaluations test language, culture, and settings such as healthcare or finance. A central challenge is making results reliable and predictive of deployment: robustness can shift with wording, language, interaction length, or evaluator, and reducing harmful compliance can increase over-refusal or undermine useful performance.

Latest papers

550 papers82 papers in the last four weeks, up 64% on the four weeks before. 0.5% of all new papers.

Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
Dec 3, 2025Oren Rachmil, Avishag Shapira, Roy Betser +5Language Model Safety Evaluation LLM Auditing
Fujitsu Research of Europe · Ben-Gurion University of the Negev
D-STEER: Two-Way Control of DPO-Induced Safety Behavior in Activation Space
Dec 3, 2025Samarth Raina, Saksham Aggarwal, Amit Dhanda +3Causal Interventions in Language Models Activation Steering
IIIT Delhi · Microsoft · Apple (USA) +2
Addressing Benchmarking Gaps in Large Language Models for Health and Medicine with Dynamic Red-Teaming
Jul 30, 2025Jiazhen Pan, Bailiang Jian, Paul Hager +20LLM Safety Benchmarks Language Model Safety Evaluation
Cherise · School of Computation, Information and Technology, Technical University of Munich (TUM), Munich, Germany · Department of Engineering Science, University of Oxford, Oxford, United Kingdom +9
Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness
Jun 30, 2025Tung-Ling Li, Hongliang LiuLLM Alignment Language Model Safety Evaluation
Palo Alto Networks
PRISON: Unmasking the Criminal Potential of Large Language Models
Jun 19, 2025Xinyi Wu, Geng Hong, Pei Chen +3Language Model Safety Evaluation LLM Auditing
Fudan University · Shanghai Innovation Institute
Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Models
Jun 8, 2025Ren-Jian Wang, Ke Xue, Zeyu Qin +7LLM Red Teaming Adversarial Prompt Generation
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · The Hong Kong University of Science and Technology, Hong Kong, SAR, China +3
ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
May 22, 2025Changyi Li, Jiayi Wang, Xudong Pan +2Language Model Safety Evaluation LLM Safety Benchmarks
Fudan University · Shanghai Innovation Institute
When Do Large Language Models Exhibit Unsolicited Deception?
Mar 31, 2025Samuel M. Taylor, Benjamin K. BergenLanguage Model Safety Evaluation Deception in Language Models
Department of Cognitive Science University of California, San Diego
GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methods
Feb 24, 2025Ruixuan Huang, Xunguang Wang, Zongjie Li +2Language Model Safety Evaluation LLM Jailbreak Attacks
The Hong Kong University of Science and Technology, Hong Kong SAR, China
CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
Jan 24, 2025Guangzhi Sun, Xiao Zhan, Shutong Feng +2LLM Safety Benchmarks Language Model Safety Evaluation
Trinity College, University of Cambridge, Cambridge, United Kingdom · Department of Informatics, King’s College London, United Kingdom · Institut für Informatik, Heinrich-Heine-Universität Düsseldorf, Germany +1

Dec 3, 2025J/K move · Enter open · S save

Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs

Oren Rachmil, Avishag Shapira, Roy Betser, Omer Hofman+4

Fujitsu Research of Europe · Ben-Gurion University of the Negev

Language Model Safety Evaluation LLM Auditing LLM Safety

Open paper AskPDF arXiv

As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we derive policy-violation scores directly from normalized representations of LLM hidden activations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks. It achieves up to 86.0% F1, outperforming both fine-tuned and LLM-as-a-judge baselines while requiring substantially less computation. Our code is publicly available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection