Language Model Safety Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 58% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 347

All topics

Harm Laundering in GPT Models: Evidence That Gender Discrimination Is Transformed Rather Than Reduced Across Safety-Trained Generations
Sep 17, 2026Sarah Wyer, Sue Black, Noura Al MoubayedGender Bias in Language Models Language Model Safety Evaluation
Durham University
Benchmarking LLM Compliance with China AI Generated Content Regulations
Sep 17, 2026Chenrui Cui, Hongye Fang, Lisha Song +3LLM Safety Benchmarks LLM Alignment
School of Computer Science and Technology, Tongji University, Shanghai, China · Law School of Tongji University, Shanghai, China
Full-Duplex Speech Models Take the Floor When Asked, Not When Needed
Sep 17, 2026Linkai Peng, Baorian Nuchged, Kaiqi Fu +1Language Model Safety Evaluation Spoken Dialogue Systems
University of Connecticut · The University of Texas at Austin · X Square Robot +1
The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention
Sep 15, 2026Srijith RavikumarLanguage Model Safety Evaluation Language Model Calibration
Amazon.com LLC
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
Sep 14, 2026Laura M. Vowels, Matthew J. Vowels, Shivali Sharma +9LLM Safety Benchmarks Mental Health Counseling
School of Psychology, University of Roehampton, London, United Kingdom · Kivira Health, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom +4
HazardAuditor: From Executable Threats to Safer Computer-Use Agents
Sep 14, 2026Yunhao Feng, Ruixiao Lin, Ming Wen +5Language Model Safety Evaluation AI Agent Auditing
Ant Group · Hunan Institute of Advanced Technology · Zhejiang University +3
SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
Sep 14, 2026Md Jueal Mia, Yanzhao Wu, Selcuk Uluagac +1Language Model Safety Evaluation Jailbreak Robustness
Knight Foundation School of Computing and Information Sciences, Florida International University, USA · Security, Optimization, and Learning for InterDependent Networks (solid) Laboratory, Miami, Florida, USA
Semantic Fibers and Cross-Gram Interference: A Calculus of Safety Drift in Overcomplete Representations
Sep 14, 2026Mohammed Ahnouch, Lotfi ElaachackLanguage Model Safety Evaluation LLM Safety
Université Paris 1 Paris, France · Faculty of Science and Technology of Tangier Abdelmalek Essaadi University Tangier, Morocco
Decision Shifts, Lost Label Functionality, and an Inconclusive Grounding Audit in Correctness-Gated Multi-Teacher Distillation
Sep 9, 2026Xiaofei FengLanguage Model Safety Evaluation Multi-Teacher Knowledge Distillation
University of Illinois Urbana-Champaign
DuplexJail: Spoken Interruption Attacks on Full-Duplex Speech Models
Sep 8, 2026Jaechul Roh, Deepak Chandran, Amir Houmansadr +1Language Model Safety Evaluation Jailbreak Attacks
University of Massachusetts Amherst · Dolby Laboratories
ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdicts
Sep 8, 2026Yongxi Zhou, Wenbo Ye, Yuanzhe Liu +2LLM Safety Benchmarks LLM-as-a-Judge
Northeastern University, Massachusetts, USA · University of Southern California, California, USA · Georgia Institute of Technology, Georgia, USA
The Profit Alignment Problem: How Profit Mandates Induce Alignment Failures in LLMs
Sep 7, 2026Eric SoLLM Alignment Language Model Safety Evaluation
MIT Sloan School of Management Massachusetts Institute of Technology
Recall Is Not Protection: Evaluating Safety Monitors Against Model Compliance
Sep 5, 2026Sripad KarneLanguage Model Safety Evaluation LLM Guardrails
Columbia University
HiveTraceGuard-Pro: A Compact Generative Guardrail for Prompt Injection, Jailbreaks, and Adversarial Obfuscation
Sep 1, 2026Nikita Oblakov, Sabrina Sadiekh, Evgeniy KokuykinLanguage Model Safety Evaluation LLM Guardrails
HiveTraceLab
Validity-Aware Jailbreak Evaluation for Large Language Models
Aug 31, 2026Qilong Wu, Sahil Wadhwa, Pranab Mohanty +2Language Model Safety Evaluation LLM Auditing
University of Illinois Urbana-Champaign · Capital One
Beyond Token Positions: Safety Alignment Across Denoising Steps in Diffusion Language Models
Aug 31, 2026Guoli Wang, Haonan Shi, Tu Ouyang +1Language Model Safety Evaluation Diffusion Language Model Decoding
Case Western Reserve University 10900 Euclid Avenue, Cleveland, OH 44106, USA
EvoFlint: An Evolutionary Atlas of Multi-Turn LLM Vulnerabilities
Aug 31, 2026Feitong Qiao, Liren Peng, Shiming Ren +7Language Model Safety Evaluation LLM Red Teaming
Reinforce Labs USA
Capability-Gated Language Models: Security Composes, Utility Does Not
Aug 31, 2026Patrikas Vanagas, Augustas Mačijauskas, Laurynas LopataLanguage Model Safety Evaluation LLM Security
BPTI Vilnius, Lithuania · Independent Researcher Vilnius, Lithuania · askEarth AG Zürich, Switzerland
BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
Aug 31, 2026Adrians Skapars, Edoardo ManinoLanguage Model Steering Language Model Safety Evaluation
University of Manchester, United Kingdom
You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals
Aug 31, 2026Ruoxuan Li, Pinqiao Wang, Sheng Li +1Language Model Safety Evaluation Pragmatic Reasoning in Language Models
Stony Brook University · University of Virginia
The Fragility of Jailbreak Robustness Across Operational States
Aug 31, 2026Yuna Park, Hwang Youn Kim, Yujin Kim +3Prompt Sensitivity Language Model Safety Evaluation
Yonsei University · Korea Institute of Science and Technology (KIST) · Korea University +1
SingProbe Technical Report
Aug 31, 2026Sing TeamLanguage Model Safety Evaluation LLM Guardrails
AI Security Lab, Ant Group
WildSEEK: Evaluating Language Models for Information-Seeking
Aug 31, 2026Tanise Ceron, Joachim Baumann, Elisa Bassignana +3LLM Safety Benchmarks LLM Sycophancy
1Bocconi University · 2Stanford University · 3IT University of Copenhagen +1
LongGuard: Mechanistic Analysis and Training-Free Mitigation of Long-Context Failure in Safety Guardrails
Aug 27, 2026Ziyang Chen, Xing Wu, Songlin HuLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
Beyond Semantic Accuracy: Consequence-Aware Evaluation for Safety-Critical Language Understanding
Aug 25, 2026Yujing Chang, Thinh Pham, Van-Phat Thai +4LLM Safety Benchmarks LLM Evaluation
1ATMRI, Nanyang Technological University (NTU), Singapore · Centre of AI Research, VinUniversity, Vietnam · School of Management, Indian Institute of Technology Mandi, India
Hidden in the Request: Explaining Unethical LLM Compliance through Token Relevance
Aug 24, 2026Or Biton, Tomer Krichli, Itai Allouche +1Language Model Decoding Language Model Safety Evaluation
Faculty of Electrical and Computer Engineering, Technion, Haifa, Israel
Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance
Aug 21, 2026Alexander Thomas, Hubert P. H. Shum, Darren Nellis +4LLM Safety Benchmarks Language Model Safety Evaluation
NCB Hazcheck Limited, UK · Durham University, UK
Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers
Aug 14, 2026Thiago Sandoval, Ufuk TopcuDistribution Shift Selective Prediction
The University of Texas at Austin
Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluation
Aug 12, 2026Haoyu Zhang, Haowen Xu, Xiao Luo +7LLM Safety Benchmarks Language Model Safety Evaluation
Northeastern University

Sep 17, 2026, cs.CLJ/K move · Enter open · S save

Harm Laundering in GPT Models: Evidence That Gender Discrimination Is Transformed Rather Than Reduced Across Safety-Trained Generations

Sarah Wyer, Sue Black, Noura Al Moubayed

Durham University

Gender Bias in Language Models Language Model Safety Evaluation LLM Safety Language Model Bias Evaluation

Open paper AskPDF arXiv

Safety evaluations for large language models rely on surface-form classifiers that report declining harm scores across model generations. We provide evidence that this methodology is systematically incomplete: explicit discriminatory content is transformed rather than removed. We call this \emph{harm laundering}. Analysing 450,000 gender-directed completions across 15 models spanning GPT-2 through to GPT-5 (OpenAI GPT lineage; three demographic conditions), we show that sexual violence clusters prevalent in GPT-2 women-directed output disappear by GPT-4, while men-directed completions gain positive representational territory (caregiving, emotional range, ally identity) that women-directed completions do not. The pattern is most visible at GPT-5: Topic~~5 (1,997~~documents) frames breast cancer as a men's rights debate, while zero equivalent clusters appear in women-directed output. Three independent classifiers score this content as non-toxic. Sentiment scores invert at GPT-4: early models demean women; later models over-correct. Topic diversity in women-directed completions falls 36% relative to men at the GPT-4 alignment boundary (W/M~

= 0.58

, from

0.91

at GPT-2). REGARD representational harm disparity correlates with release date (

ρ= +0.55

p = .034

) while Detoxify does not (

ρ= -0.23

p = .42

): toxicity scores fall as representational harm grows. We formalise harm laundering as a three-criteria test and provide a three-stage detection protocol applicable to any generative model. Within the OpenAI GPT lineage, toxicity score reduction is not a sufficient proxy for harm reduction.

Momentum

Related topics

Latest papers 347