Language Model Safety Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 58% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 347

All topics

Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis
May 13, 2026Zvi TopolLanguage Model Safety Evaluation LLM Jailbreak Attacks
MuyVentive, LLC
Before the Last Token: Diagnosing Final-Token Safety Probe Failures
May 12, 2026Shravan DodaLanguage Model Safety Evaluation Safety Filtering
Kipo AI
DisaBench: A Participatory Evaluation Framework for Disability Harms in Language Models
May 12, 2026Eugenia Kim, Ioana Tanase, Christina MallonLLM Safety Benchmarks LLM Evaluation
Microsoft
Classifier Context Rot: Monitor Performance Degrades with Context Length
May 12, 2026Sam Martin, Fabien RogerLanguage Model Safety Evaluation Long-Context Language Model Evaluation
Anthropic Fellows Program · Anthropic
Confidently Deceptive: On the Relationship Between Confidence and Deception in LLMs
May 12, 2026Ali Asad, Stephen Obadinma, Anshul Pattoo +2Language Model Safety Evaluation Confidence Estimation in Language Models
Department of Electrical and Computer Engineering & Ingenuity Labs Research Institute Queen’s University, Kingston, Canada
To Whom Do Language Models Align? Measuring Principal Hierarchies Under High-Stakes Competing Demands
May 12, 2026Fangyi Yu, Nabeel Seedat, Jonathan Richard Schwarz +1LLM Alignment Language Model Safety Evaluation
Thomson Reuters Foundational Research, 5 Canada Square, London · Imperial College London · University of Oxford
Safety-Oriented Evaluation of Language Understanding Systems for Air Traffic Control
May 12, 2026Yujing Chang, Yash Guleria, Duc-Thinh Pham +4LLM Grounding Air Traffic Management
1ATMRI, Nanyang Technological University (NTU), Singapore · School of Management, Indian Institute of Technology Mandi, India · Centre of AI Research, VinUniversity, Vietnam
Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers
May 11, 2026Nikita Kezins, Urbas Ekka, Pascal Berrang +1Neural Network Verification Language Model Safety Evaluation
Delft University of Technology · University of Birmingham & Zeroth Research
RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systems
May 11, 2026Joel Rorseth, Parke Godfrey, Lukasz Golab +2Retrieval-Augmented Generation Language Model Safety Evaluation
University of Waterloo · York University · AT&T Chief Data Office
Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks
May 11, 2026Regina Gugg, Selina Niederländer, Andreas Stöckl +1Language Model Safety Evaluation Language Model Bias Evaluation
Dynatrace Research, Linz, Austria · University of Applied Sciences Upper Austria, Hagenberg, Austria
Acceptance Cards:A Four-Diagnostic Standard for Safe Fine-Tuning Defense Claims
May 11, 2026Phongsakon Mark Konrad, Toygar Tanyel, Serkan AyvazFine-Tuning Language Model Safety Evaluation
Centre for Industrial Software, University of Southern Denmark, Alsion 2, Sønderborg, 6400, Denmark · ProMake, Newark, DE, USA
FragBench: Cross-Session Attacks Hidden in Benign-Looking Fragments
May 10, 2026Astha Mehta, Niruthiha Selvanayagam, Cedric Lam +10LLM Safety Benchmarks Language Model Safety Evaluation
Isabella · ÉTS Montreal / SPAR · Mila / Lida Safety
Do Linear Probes Generalize Better in Persona Coordinates?
May 10, 2026Prasad Mahadik, Adrians SkaparsLanguage Model Safety Evaluation Deception in Language Models
Independent Researcher · University of Manchester
MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks
May 10, 2026Xinkai Zhang, Zhipeng Wei, Huanli Gong +4Language Model Safety Evaluation Jailbreak Attacks
1UC Berkeley · 3International Computer Science Institute · 2UC Riverside +1
The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
May 9, 2026Ismail Hossain, Tanzim Ahad, Md Jahangir Alam +3Adversarial Prompt Generation Language Model Safety Evaluation
Department of Computer Science, University of Texas at El Paso, TX, USA 79902 · School of Computing, Southern Illinois University Carbondale, IL, USA 62901 · University of Illinois Urbana-Champaign, IL, USA
Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success
May 9, 2026Carsten Maple, Abhishek Kumar, Riya TapwalLanguage Model Safety Evaluation Jailbreak Attacks
University of Warwick · The Alan Turing Institute · IIT Mandi
Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents
May 8, 2026Jun Wen LeongLanguage Model Safety Evaluation AI Agent Security
The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
May 8, 2026Gabriele La Malfa, Emanuele La Malfa, Saar Cohen +4Self-Play RL Language Model Safety Evaluation
Department of Informatics, King’s College London · Department of Computer Science, University of Oxford · Institute for Decentralized AI +1
A Systematic Investigation of RL-Jailbreaking in LLMs
May 7, 2026Montaser Mohammedalamen, Kevin Roice, Reginald McLean +1Language Model Safety Evaluation Adversarial Attacks on LLMs
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada.
When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels
May 7, 2026Sushant Gautam, Finn Schwall, Annika Willoch Olstad +6LLM Safety Benchmarks LLM Evaluation
1Simula Metropolitan Center for Digital Engineering · 2Oslo Metropolitan University · 4Simula Research Laboratory +2
Algospeak, Hiding in the Open: The Trade-off Between Legible Meaning and Detection Avoidance
May 7, 2026Jan Fillies, Ronald E. Robertson, Jeffrey HancockLanguage Model Safety Evaluation Adversarial Text Attacks
Stanford University Stanford, California, USA
How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation
May 7, 2026Shai Feldman, Yaniv RomanoLLM Evaluation Language Model Safety Evaluation
Department of Computer Science Technion, Israel · Departments of Electrical & Computer Engineering. and Computer Science Technion, Israel
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLLM Evaluation Language Model Generation Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence
Evaluation Awareness in Language Models Has Limited Effect on Behaviour
May 7, 2026Amelie Knecht, Lucas Florin, Thilo HagendorffLLM Alignment Language Model Safety Evaluation
University of Stuttgart
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
May 7, 2026Dasol Choi, Eugenia Kim, Jaewon Noh +14LLM Safety Benchmarks Multilingual Language Model Evaluation
AIM Intelligence · Microsoft · Korea AISI +7
One Turn Too Late: Learning When to Intervene Against Multi-Turn Malicious Intent
May 7, 2026Xinjie Shen, Rongzhe Wei, Peizhi Niu +6Language Model Safety Evaluation LLM Guardrails
Georgia Institute of Technology · University of Illinois Urbana-Champaign · UCSD +3
The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
May 6, 2026Alif Al Hasan, Sumon BiswasLLM Safety Benchmarks Language Model Safety Evaluation
Department of Computer and Data Sciences Case Western Reserve University Cleveland, OH, USA
SoK: Robustness in Large Language Models against Jailbreak Attacks
May 6, 2026Feiyue Xu, Hongsheng Hu, Chaoxiang He +9Language Model Safety Evaluation LLM Security
Shanghai Jiao Tong University, China · Microsoft Corporation
Misaligned by Reward: Socially Undesirable Preferences in LLMs
May 6, 2026Gayane Ghazaryan, Esra DönmezReward Modeling Language Model Safety Evaluation
Institute for Natural Language Processing, University of Stuttgart · 2Interchange Forum for Reflecting on Intelligent Systems, University of Stuttgart
MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
May 5, 2026Jonathan Steinberg, Oren GalLanguage Model Safety Evaluation AI Coding Agents
Swarms & AI Lab (SAIL), University of Haifa

May 13, 2026, cs.CRJ/K move · Enter open · S save

Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis

Zvi Topol

MuyVentive, LLC

Language Model Safety Evaluation LLM Jailbreak Attacks

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed in a wide range of applications, yet remain vulnerable to adversarial jailbreak attacks that circumvent their safety guardrails. Existing evaluation frameworks typically report binary success/failure metrics, failing to capture the temporal dynamics of how attacks succeed under persistent adversarial pressure. This preliminary work proposes a novel evaluation framework that applies survival analysis techniques to characterize LLM jailbreak vuln`erability. Our approach models the time-to-jailbreak as a survival outcome, enabling estimation of hazard functions, survival curves, and risk factors associated with successful attacks. We evaluate three LLMs against a subset of prompts from the HarmBench dataset spanning three attack categories. Our analysis reveals that models exhibit distinct vulnerability profiles: while one model demonstrates rapid degradation under iterative attacks, the two other models show consistent moderate vulnerability. Our framework provides actionable insights for model and LLM application developers and establishes survival analysis as a rigorous methodology for LLM safety evaluation.

Momentum

Related topics

Latest papers 347