Language Model Safety Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 58% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 347

All topics

Safeguards for Speech2Speech LLM-Assistants: A Case Study in Automotive Applications
Jul 23, 2026Gregor Endler, Sebastian Kraus, Lukas StappenLanguage Model Safety Evaluation LLM Guardrails
codemanufaktur GmbH, Germany · BMW Group, Germany
Sound Probabilistic Safety Bounds for Large Language Models
Jul 22, 2026Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani +1Language Model Safety Evaluation Confidence Estimation in Language Models
Department of Computer Science, University of Oxford, UK · Max Planck Institute for Software Systems, Germany · School of Computer Science, University of Birmingham, UK
Adaptive Capitulation: A Structural Failure Mode of LLM Responses in Vulnerability Contexts
Jul 21, 2026Eunna LeeLanguage Model Safety Evaluation LLM Refusal Behavior
Independent Researcher
Don't Want Your LLM to Recommend Nuclear Strike? Try Asking It in Japanese
Jul 21, 2026Rian TouchentMoral Reasoning in Language Models Language Model Safety Evaluation
ALMAnaCH · Sorbonne Université / INRIA Paris ALMAnaCH Team
Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety
Jul 21, 2026Koyar AfrasyabLLM-as-a-Judge Language Model Safety Evaluation
Kinvectum AB, Sweden
Estimating Rare Events in Language Models with Proper Evaluation
Jul 20, 2026Nikita Y. Parulekar, Anqi LiuLanguage Model Safety Evaluation Proper Scoring Rules
Johns Hopkins University
An Early Warning of Emerging Biosecurity Risks in Frontier LLMs
Jul 20, 2026Zhida He, Xia Hu, Baichen Le +20Language Model Safety Evaluation LLM Security
Shanghai Artificial Intelligence Laboratory
AEGIS: Awareness-Enhanced Guidance for Iterative Safeguard
Jul 20, 2026Kyungwon Park, Sangmin Lee, Heejae Chon +1Language Model Safety Evaluation Text Generation
Dept. of Artificial Intelligence, Yonsei University · Dept. of Electrical & Electronic Engineering, Yonsei University · Dept. of Computer Science & Engineering, Yonsei University +1
A Dual-Hypothesis Reasoning Framework for LLM Guardrails
Jul 20, 2026Md Asiful Islam, Fahmida Alam, Mihai SurdeanuSupervised Fine-Tuning Language Model Safety Evaluation
Department of Computer Science University of Arizona, USA
Safety That Does Not Transfer: Cross-Lingual Clinical Correctness Drift in Deployable Medical Language Models
Jul 19, 2026Anthonio Oladimeji Gabriel, Dimeji Olawuyi, Toba Ajayi +1Multilingual Language Model Evaluation Language Model Safety Evaluation
AI SafetyX
LLM Evaluators are Biased across Languages
Jul 16, 2026Ej Zhou, Lucas Resck, Zheng Hui +1Multilingual Language Model Evaluation Language Model Safety Evaluation
Language Technology Lab, University of Cambridge
Protective Capacity Hallucination: When Large Language Models Claim Nonexistent Capabilities
Jul 15, 2026Eunna Lee, Jungpyo Nam, Sunjun HwangLanguage Model Safety Evaluation Hallucination in Language Models
Independent Researcher · Korea Cyber University · Yonsei University
SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing
Jul 15, 2026Tianyu Chen, Chujia Hu, Wenjie WangLanguage Model Safety Evaluation LLM Guardrails
ShanghaiTech University, Shanghai, China
Silent Alarm: A J-Space Protocol for Comparing Danger Recognition Across Models and Quantization Levels
Jul 14, 2026Roman Prosvirnin, Victor Minchenkov, Alexey Soldatov +1LLM Safety Benchmarks Language Model Safety Evaluation
HSE University
A Threshold Exceedance Framework for CBRN Uplift Evaluation in Frontier Language Models
Jul 13, 2026Rahul Gupta, Abhinav Mohanty, Payal Motwani +8Language Model Safety Evaluation
Amazon Nova Responsible AI · Work done while at Amazon. · Nemesys Insights
Relational Positioning as a Measurable Risk Object: History-Carried Lock-in and Self-Confabulation in Multi-Turn Human-AI Dialogue
Jul 13, 2026Jihong ChenLanguage Model Safety Evaluation Human-AI Interaction
Beijing Etown Academy
The Paternalistic Filter: Epistemic Injustice and Differential Refusal in LLM-Mediated History Education for Marginalized Romanian Students
Jul 13, 2026Alexis Popovici, Andrei Ionascu, Adrian-Marius DumitranLanguage Model Safety Evaluation AI in Education
Universitatea din Bucuresti, Bucharest, Romania
The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicators
Jul 12, 2026Dominik SchwarzLanguage Model Safety Evaluation Hidden-State Probing
Independent Researcher
Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
Jul 11, 2026Chigozirim Ifebi, Brent Kong, Ayushi MehrotraLLM Safety Benchmarks Multilingual Language Model Evaluation
California Institute of Technology.
Trivial Prompt Reframing Bypasses Safety Guardrails in Googleś MedGemma-4B
Jul 9, 2026Avi-ad Avraam BuskilaLLM Safety Benchmarks Language Model Safety Evaluation
Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat Gan, Israel
Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring
Jul 9, 2026Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2Language Model Safety Evaluation Adversarial Attacks on LLMs
LASR Labs · University College London · Google DeepMind
Predicting LLM Safety Before Release by Simulating Deployment
Jul 8, 2026Marcus Williams, Hannah Sheahan, Cameron Raymond +8LLM Evaluation AI Risk Management
OpenAI
Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents
Jul 4, 2026Abhishek Kumar, Carsten MapleLanguage Model Safety Evaluation AI Coding Agents
The Alan Turing Institute, London, United Kingdom
Online Safety Monitoring for LLMs
Jul 2, 2026Mona Schirmer, Metod Jazbec, Alexander Timans +3Language Model Safety Evaluation LLM Safety
UvA Bosch-Delta Lab, University of Amsterdam · University of Wisconsin Madison · Johns Hopkins University
HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety
Jul 2, 2026Navaneeth Sangameswaran, Preetham S, Ashmiya LeninLLM Safety Benchmarks Language Model Safety Evaluation
Astroware AI, Delaware, USA · Karunya Institute of Technology and Sciences, Coimbatore, India
OpenSafeIntent: Evaluating Intent-Calibrated Safe Completion Across Dual-Use Prompt Sets
Jul 2, 2026Rheeya Uppaal, Seungwoo Lyu, Selina Sung +1LLM Safety Benchmarks Language Model Safety Evaluation
Department of Computer Sciences University of Wisconsin-Madison · Department of CSE Korea University
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
Jul 1, 2026Brett ReynoldsLLM Safety Benchmarks Language Model Safety Evaluation
Humber Polytechnic & University of Toronto
Harnessing Textual Refusal Directions for Multimodal Safety
Jun 30, 2026Moreno D'Incà, Nicu Sebe, Massimiliano ManciniLanguage Model Safety Evaluation Multimodal Large Language Models
University of Trento
Moral Safety in LLMs: Exposing Performative Compliance with Puzzled Cues
Jun 30, 2026Mohammadamin Shafiei, Shuyue Stella Li, Yulia TsvetkovLanguage Model Safety Evaluation Language Model Bias Evaluation
University of Milan · University of Washington
Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoring
Jun 29, 2026Max Fomin, Elad David, Amit LeViLanguage Model Safety Evaluation AI Agent Monitoring
Zenity

Jul 23, 2026, cs.AIJ/K move · Enter open · S save

Safeguards for Speech2Speech LLM-Assistants: A Case Study in Automotive Applications

Gregor Endler, Sebastian Kraus, Lukas Stappen

codemanufaktur GmbH, Germany · BMW Group, Germany

Language Model Safety Evaluation LLM Guardrails Spoken Dialogue Systems Speech Language Models Conversational Agents

Open paper AskPDF arXiv

Recent advances have introduced speech-to-speech (S2S) conversational assistants capable of producing natural-sounding interactions, including non-verbal cues like tonality and mood. In the automotive domain, this enables intuitive and humanlike in-car dialogue experiences. However, integrating these end-to-end assistants limits architectural options for programmable domain-specific safeguards. This paper discusses two implementation approaches for S2S guardrails: transcript-based and tool-based. Through an empirical evaluation, we demonstrate that both strategies are insufficient for industrial deployment in most cases due to prohibitive latency (delaying each answer by 0 to 1.4 seconds even for computationally cheap checks) and technical impediments (like potentially non-deterministic tool call behavior). Finally, we outline open challenges for S2S guardrails in the automotive context.

Momentum

Related topics

Latest papers 347