AI Safety Evaluation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 91

All topics

Distillation for Incrimination and Distillation for Capabilities
Oct 7, 2026Sebastian Prasanna, Jacqueline Tay, Alek WestoverLLM Safety Alignment Language Model Distillation
Redwood Research
Systemization of Knowledge (SoK): Human-Centered AI Safety for Youth
Oct 6, 2026Pratyasha Saha, Yaman Yu, Yang WangAI Risk Management AI Safety Evaluation
University of Illinois Urbana-Champaign Champaign, IL, United States
Not What a Child Expressed: Auditing the Sign-to-Text Safety Interface in Child-Facing AI
Oct 5, 2026Muhammad Rafiullah Memon, Viet Vo, Wanlun Ma +1Sign Language Translation AI Safety Evaluation
School of Science, Computing and Emerging Technologies Swinburne University of Technology, Hawthorn, VIC, Australia
Inspect Robots: Evaluating the Capabilities and Safety of Embodied AI
Oct 5, 2026Christopher Leet, Achu Menon, Sravanthi Machcha +12AI Agent Evaluation Robotic Policy Evaluation
Robocurve · University of Southern California · San Jose State University +7
Reliability Engineering for AI Systems: Challenges, Methods, and Directions
Sep 28, 2026Rong Pan, Yili Hong, Min XieAI Agent Reliability AI Safety Evaluation
Arizona State University, Tempe, AZ, USA · Virginia Tech, Blacksburg, VA, USA · City University of Hong Kong, Hong Kong
Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance
Sep 28, 2026Dipankar SarkarLLM Guardrails AI Agent Evaluation
Skelf Research
From Protocols to Evidence: Bounded Claims for AI in Service of the Common Good
Sep 10, 2026Nitesh V. Chawla, Paulo BenantiAI Accountability AI Safety Evaluation
University of Notre Dame Notre Dame, USA · LUISS Guido Carli University Rome, Italy
Reducing Catastrophic Risk from AI with Systematic Monitoring and Evaluation of Rogue AI Progression
Sep 2, 2026T. Bauer, W. P. Kegelmeyer, E. Begoli +15AI Risk Management AI Safety Evaluation
Sandia National Laboratories (SNL), Albuquerque, NM, 87185, USA. · Sandia National Laboratories, Livermore, CA 94451, USA. · Oakridge National Laboratories (ORNL), Oak Ridge, TN, 37830, USA. +9
ObserverBench: Testing Mechanistic Estimates for Intervention and Control
Sep 2, 2026Vijay ErramilliMechanistic Interpretability AI Safety Evaluation
Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety
Aug 8, 2026Hannah Cha, Neha Shukla, Solon Barocas +3Human-in-the-Loop Evaluation AI Safety Evaluation
Microsoft Research · Duke University · Abridge AI Inc. +1
Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics
Aug 6, 2026Jessica Y. Bo, Paula Akemi Aoyagui, Shalaleh Rismani +3AI Safety Evaluation Human-Centered AI
University of Toronto · McGill University & Mila Quebec AI Institute
Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
Jul 28, 2026Xinyu Yang, Tianxing Chen, Honghao Su +38AI Safety Evaluation Trust in AI
Why Public Service AI Governance Frameworks Risk Failing in the Age of General-Purpose AI: Lessons from Policing
Jul 28, 2026Sam Relins, Daniel BirksAI Safety Evaluation AI Governance
School of Law, University of Leeds, Leeds, UK · ESRC Vulnerability and Policing Futures Research Centre
SAGE: Safety-First Defense-in-Depth Guardrails for Verified Lifecycle Control of High-Impact Generative AI
Jul 24, 2026Mahdi EslamimehrLLM Guardrails AI Safety Evaluation
Quandary Peak Research
What AI Red-Team Evaluations Can and Cannot Prove
Jul 23, 2026Bandana KaurLLM Safety Benchmarks LLM Red Teaming
APIsec Research Labs
MedFailBench: A Clinician-Built Open-Source Benchmark for Medical AI Safety Boundary Inspection
Jul 16, 2026Goktug OzkanSynthetic Benchmark Generation AI Safety Evaluation
Department of Internal Medicine, Kutahya Emet Dr. Fazil Dogan State Hospital, Kutahya, Turkiye
Adversarial Prompting Framework for AI Safety Assessment
Jul 15, 2026Yash Bhatnagar, Kunal Banerjee, Anirban ChatterjeeAdversarial Prompt Generation Adversarial Attacks on LLMs
Microsoft · Bengaluru, Karnataka, India · Walmart Global Tech
Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric
Jul 14, 2026Oleg SolozobovAI Agent Safety AI Safety Evaluation
Independent Researcher (Global)
Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety
Jul 8, 2026Yujiao ChenAlgorithmic Fairness AI Agent Safety
Massachusetts Institute of Technology · Cambridge, MA 02139
Reasoning Consistency Scanning: A Framework for Auditing Chain-of-Thought Validity in AI Safety Evaluations
Jul 8, 2026Silvia SantanoLLM Auditing Reasoning Consistency in Language Models
When Certificates Fail: A Unified Safety Framework for Embedded Neural Interface Models
Jul 7, 2026Jasmeet Singh BindraBrain-Computer Interfaces Neural Network Robustness
IKSMHA Center, IIT Mandi
Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability
Jul 7, 2026Alicia Parrish, Rajat Shinde, Sanket Badhe +57VLM Evaluation Cross-Cultural Language Model Evaluation
The Foreign Policy AI Evaluation Gap
Jul 3, 2026Charles Pozniak, Jeba SaniaAI Safety Evaluation AI Governance
Belfer Center, Harvard Kennedy School, Cambridge, MA, USA.
Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
Jul 1, 2026Yanhang Li, Zhichao Fan, Zexin ZhuangAI Assurance Benchmark Auditing
Northeastern University, USA · University of Illinois Urbana-Champaign, USA · Southern Methodist University, USA
Evaluating Intellectual Property Guardrails of Generative Image Models: A Technical Report
Jul 1, 2026Austin T. Hoag, Apostolos Modas, Yunhao Ba +9Image Generation AI Safety Evaluation
Sony AI · New York, NY, USA · Zurich, Switzerland +2
Defeat Devices in AI Systems
Jun 27, 2026Emilio FerraraDeception in Language Models AI Safety Evaluation
Thomas Lord Department of Computer Science University of Southern California
RedVox: Safety and Fairness Gaps in Speech Models Across Languages
Jun 25, 2026Beatrice Savoldi, Sara Papi, Wafa Aissa +2Audio-Language Model Evaluation AI Safety Evaluation
Fondazione Bruno Kessler, Italy
The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report
Jun 25, 2026Kwan Soo Shin, In Seok Kang, Yunkyung Min +2Language Model Safety Evaluation AI Safety Evaluation
PolymathMinds Lab, Seoul, Republic of Korea · Department of Chemical Engineering, Pohang University of Science and Technology (POSTECH), Pohang, Republic of Korea · aSSIST University (Seoul School of Integrated Sciences & Technologies), Seoul, Republic of Korea +3
PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation
Jun 23, 2026Leyi Sheng, Han Sun, Zhen Sun +4AI Safety Evaluation Multimodal Jailbreak Attacks
1The Hong Kong University of Technology and Science (Guangzhou) · 2East China Normal University · 3Shanghai Qi Zhi Institute +4
Safety-Aware Evaluation of LLM-Generated Driver Intervention Messages through Multi-Task Risk Fusion
Jun 21, 2026Keito InoshitaLLM Evaluation Autonomous Driving
Faculty of Business and Commerce, Kansai University, 3-3-35, Yamatecho, Suita, 564-8680, Osaka, Japan · Data Science and AI Innovation Research Promotion Center, Shiga University, 1-1-1 Baba, Hikone, 522-8522, Shiga, Japan

Oct 7, 2026, cs.AIJ/K move · Enter open · S save

Distillation for Incrimination and Distillation for Capabilities

Sebastian Prasanna, Jacqueline Tay, Alek Westover

Redwood Research

LLM Safety Alignment Language Model Distillation LLM Auditing AI Safety Evaluation

Open paper AskPDF arXiv

Powerful misaligned AI models might recognize alignment evaluations and strategically behave well on them, rendering direct audits uninformative. However, distilling such a model into a weaker benign student places the teacher in a Distillation Double Bind: if misalignment transfers, the student may conceal it less effectively, revealing evidence about the teacher; if it does not, the student may learn useful capabilities while remaining benign. We introduce two distinct distillation approaches, one targeting each outcome. Distillation for Incrimination (DFI) aims to transfer misalignment but not the ability to conceal it. Distilling AuditBench's secret-keeping models into their underlying instruction-tuned model produces students that are significantly more likely than their teachers to admit their hidden behavior when asked, suggesting that knowledge of the behavior transferred more readily than the propensity to conceal it. Confession gains largely disappear when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment. Among several techniques we evaluate, two are effective: inoculation prompting and training for more epochs on fewer unique examples. Both preserve the capability gains of standard distillation while substantially reducing the subliminal transfer of an animal preference, our proxy for misalignment. Together, these findings demonstrate two ways distillation can be used for AI safety: incriminating misaligned models, and extracting their capabilities without their misalignment.

Momentum

Related topics

Latest papers 91