Language Model Safety Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 58% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 347

All topics

Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning
Aug 12, 2026Lang CaoLLM Alignment Language Model Safety Evaluation
University of Illinois Urbana-Champaign
Withholding the Completing Chunk: Deterministic Pair-Completion Guardrails for Streaming LLM Output
Aug 10, 2026Christopher M. FrostLanguage Model Safety Evaluation LLM Guardrails
HEOSSI (Pte.) Ltd., Singapore
When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs
Aug 9, 2026Yu Ma, Hongli Shi, Jing Li +2LLM Safety Benchmarks Language Model Safety Evaluation
Google · University of New South Wales · University of Technology Sydney +2
HoloAegis: Frozen Representation, Topological Inference --- Minimally Parametric Safety Manifolds and Their Capability Boundaries for LLM Guardrails
Aug 9, 2026Tak Ho Alex Li, Kaijie Liu, Lik-Hang Lee +3Language Model Safety Evaluation LLM Guardrails
Department of Mathematics, Hong Kong Baptist University, Hong Kong 999077 · Guangdong-Hong Kong-Macao Institute of ESG and New Quality Productive Forces, School of Optoelectronic Engineering, Guangdong Polytechnic Normal University, Guangzhou 510665 · Guangdong Institute of Digital Industry, Guangzhou 510000 +3
Calling the Bluff: Detecting Ever-Shifting Harmful Chat Dialogue via Ordered Reasoning Chain Regularization
Aug 9, 2026Haojie Yu, Ziyou Jiang, Junjie Wang +4Language Model Safety Evaluation Text Classification
State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China · Science and Technology on Integrated Information System Laboratory Institute of Software Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences
Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Families
Aug 8, 2026Alizishaan Khatri, Dun Li ChanLLM Safety Benchmarks LLM Inference
Wrynx Inc · INTI International College Penang
SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs
Aug 8, 2026Debopriyo Banerjee, Kapil Rajesh Kavitha, Angana Borah +11LLM Safety Benchmarks Multilingual Language Model Evaluation
Inception · Mohamed bin Zayed University of Artificial Intelligence · University of Michigan +4
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Aug 6, 2026Ro Encarnación, Tina Behzad, Emma Lurie +1LLM Safety Benchmarks LLM Evaluation
University of Pennsylvania · Stony Brook University
Measuring and Detecting Harmful AI Sycophancy
Aug 6, 2026Bohan Jiang, Dawei Li, Yasin Silva +1LLM Sycophancy Language Model Safety Evaluation
1Arizona State University · 2Loyola University Chicago
From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs
Aug 6, 2026Jiawei Qiu, Yichen Xu, Jianzhe Ma +5LLM Safety Benchmarks Language Model Safety Evaluation
School of Information, Renmin University of China Beijing, China
Item Response Theory for AI Safety
Aug 5, 2026Joshua Fonseca Rivera, Neil Shah, David Demitri Africa +1LLM Safety Benchmarks Language Model Safety Evaluation
Independent · 1Independent · UK AI Security Institute +1
DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuning
Aug 5, 2026Junbo Zhang, Qianli Zhou, Xinyang Deng +1Language Model Safety Evaluation LLM Fine-Tuning
Northwestern Polytechnical University
Risky Business: Measuring The Faithfulness-Safety Tension
Aug 4, 2026Dominik Meier, Luca Joshua Francis, Marco Bernhard Kaiser +3CoT Faithfulness Language Model Safety Evaluation
University of Göttingen · 2Landeskriminalamt NRW
Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili
Aug 4, 2026Ruolei Zhang, Teddy Njuguna, Yue FengMultilingual Language Model Evaluation Language Model Safety Evaluation
University of Birmingham
When Refusal Looks Safe: The Refusal-Cue Shortcut in Safety Guard Models
Aug 4, 2026Yu Feng, Chunting Zang, Chen Shen +4Language Model Safety Evaluation LLM Guardrails
The University of Sydney · Alibaba Group · Jilin University +1
Aligned in Form, Not in Meaning: The Comprehension - Containment Decoupling of LLM Safety in Low-Resource Bangla Derogatory Speech
Aug 3, 2026Shadab Bin Habib, A K M Ferdous Reza Habib, Subarno Neel +1Language Model Safety Evaluation LLM Safety Alignment
Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh
Long-term Measurements: Towards a Longitudinal Understanding of Human-AI Interactions
Aug 3, 2026Nicole Mitchell, Dhruv Agarwal, Maty Bohacek +2Language Model Safety Evaluation Human-AI Interaction
♢Google Research · ♡Google DeepMind · ♠Cornell University +1
Same violence, different answer: how AI responds to coercive control against women across languages
Aug 2, 2026Lyu Chang, Sònia Estradé Albiol, Núria Vergés BoschMultilingual Language Model Evaluation Language Model Safety Evaluation
Department of Electronic and Biomedical Engineering, Faculty of Physics, Universitat de Barcelona, Martí i Franquès 1, 08028 Barcelona, Spain · Department of Sociology, Universitat de Barcelona, Av. Diagonal 690–696, 08034 Barcelona, Spain
ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification
Aug 2, 2026Wajdi Zaghouani, Md. Rafiul Biswas, Kholoud Khalil Aldous +1LLM Safety Benchmarks Arabic NLP
Northwestern University in Qatar, Qatar · Hamad Bin Khalifa University, Qatar
Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity
Aug 1, 2026Yongxi Zhou, Junwei Yao, Yuanzhe Liu +4LLM Safety Benchmarks Prompt Sensitivity
1Northeastern University, Massachusetts, USA · 2Georgia Institute of Technology, Georgia, USA · University of Southern California, California, USA
InfoOps Bench: A live information operations safety benchmark
Jul 30, 2026Dorian Quelle, Lisa-Maria Neudert, Jonathan Bright +1LLM Safety Benchmarks Language Model Safety Evaluation
Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution
Jul 30, 2026I. Kennedy, T. KennedyModel Compression AI Agent Reliability
Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions
Jul 29, 2026Shi Lin, Peng Qian, Dinghao Liu +5Language Model Safety Evaluation LLM Agent Safety
1Zhejiang Gongshang University · 2Shandong University · 3Hainan University +1
When Does Span-Guided Detoxification Help? Human Preferences and Evaluator Diagnostics in a Controlled Comparison
Jul 29, 2026Kyungwon ParkLanguage Model Safety Evaluation Generative AI Evaluation
Department of Artificial Intelligence, Yonsei University
Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting
Jul 28, 2026Mengya Hu, Susie Park, Suzana Ilic +5Language Model Safety Evaluation Content Moderation
Microsoft Responsible AI · Goodfire
Minimizing Targeted Activations: Input-Only Suppression of Evaluation-Awareness Latents in Large Language Models
Jul 28, 2026Deepanshu Mody, Samarth Agarwal, Utkarsh Mittal +1Language Model Safety Evaluation LLM Interpretability
Center for Data Science, New York University
Shieldstral
Jul 28, 2026Antonia Calvi, Avinash Sooriyarachchi, Giada Pistilli +273Language Model Safety Evaluation Content Moderation
When LLM Defenses Backfire: Characterizing Safety, Performance, and Cost Trade-offs
Jul 27, 2026Tong Zhang, Zexin Li, Simin Chen +1LLM Inference Efficiency Language Model Safety Evaluation
Fudan University · University of California, Riverside · Columbia University
Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition
Jul 26, 2026Aarnav Choudhary, Matheus Fonseca Rocha, Jiwon Seo +2Language Model Safety Evaluation Task Vector Merging
UCLA · Algoverse · Purdue University +1
Trustworthiness Costs of Domain Adaptation in Small Language Models:A Cross-Architecture Empirical Study
Jul 23, 2026Ramesh B. ParamkushamLanguage Model Safety Evaluation Domain Adaptation
Independent Researcher; Adjunct Faculty, University of the Cumberlands, Williamsburg, KY, USA

Aug 12, 2026, cs.AIJ/K move · Enter open · S save

Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning

Lang Cao

University of Illinois Urbana-Champaign

LLM Alignment Language Model Safety Evaluation LLM Safety Alignment Language Model Robustness

Open paper AskPDF arXiv

Aligned large language models (LLMs) are expected to exhibit safety behavior based on the content of the user request: they should refuse unsafe requests and comply with safe ones. However, we show that the same request can elicit substantially different safety decisions under different traits assigned in the system prompt, a failure mode we call trait-induced safety variation. To measure this failure, we introduce refusal-based metrics: Trait-Induced Deviation measures dataset-level deviation from the no-trait baseline, while Trait-Induced Flip Rate measures whether the same request receives different safety decisions across traits. We then provide a representation-level analysis of the mechanism behind trait-induced safety shifts and find that traits perturb the model's safety representations within a low-dimensional subspace. To achieve trait-invariant safety, where safety behavior remains stable across traits, we introduce Trait-Invariant Safety Tuning (TIST), a simple yet effective self-distillation framework that aligns an LLM's trait-conditioned behavior with its no-trait behavior. Guided by our analysis, we further propose Trait-Subspace Neutralization (TraSN), an instantiation of TIST, which enforces invariance only within the identified trait subspace. Experiments show that TraSN improves trait-invariant safety and strengthens harmful-request safety while preserving general capability. Our results highlight traits as an important factor in LLM safety and robust model behavior.

Momentum

Related topics

Latest papers 347