AI Safety Evaluation

Ask about this topic

Overview

AI safety evaluation asks whether models and AI systems behave safely—not only on isolated prompts, but across conversations, tool use, human-AI teams, and physical or high-stakes settings. Common approaches include red-teaming and adversarial testing, curated benchmarks, simulated agent tasks, human studies, runtime monitoring, and formal verification. Evaluations measure outcomes such as harmful assistance, missed hazards, refusal quality, robustness to shifts, and effects on users or interacting agents. A central challenge is that strong benchmark results provide limited assurance about rare failures or deployment behavior, so evaluations must make their assumptions and evidence limits explicit.

Latest papers

CardsList

143 papers21 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

  1. Breakdowns in Conversational AI: Interactional Failures in Emotionally and Ethically Sensitive Contexts

    Apr 3, 2026Jiawen Deng, Wentao Zhang, Ziyun Jiao +1Human-AI InteractionAI Safety Evaluation

    University of Electronic Science and Technology of China

  2. I must delete the evidence: AI Agents Explicitly Cover up Fraud and Violent Crime

    Apr 2, 2026Thomas RivasseauAI Agent SafetyAI Ethics

    Data Mining and Security Lab, School of Information Studies, McGill University.

  3. UK AISI Alignment Evaluation Case-Study

    Apr 1, 2026Alexandra Souly, Robert Kirk, Jacob Merizian +2AI Agent SafetyAI Safety Evaluation

  4. ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models

    Mar 31, 2026Zhenglin Lai, Sirui Huang, Yuteng Li +3AI Safety EvaluationWorld Model Evaluation

    Shenzhen University, Shenzhen, China.

  5. Empirical Validation of the Classification-Verification Dichotomy for AI Safety Gates

    Mar 31, 2026Arsenios ScrivensAI Agent SafetyAI Safety Evaluation

  6. M-CARE: Standardized Clinical Case Reporting for AI Model Behavioral Disorders, with a 20-Case Atlas and Experimental Validation

    Mar 27, 2026Jihoon JeongAI Agent EvaluationAI Safety Evaluation

    Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST) · ModuLabs

  7. Signals Are Not States: Neuro-Symbolic Safeguards for Culturally Aware Classroom AI

    Mar 24, 2026Sina Bagheri NezhadAI in EducationAI Safety Evaluation

    Independent Researcher, Seattle, WA, USA

  8. AEGIS: An Operational Infrastructure for Post-Market Governance of Adaptive Medical AI Under US and EU Regulations

    Mar 20, 2026Fardin Afdideh, Mehdi Astaraki, Fernando Seoane +1AI GovernanceResponsible AI

    Department of Clinical Science, Intervention and Technology, Karolinska Institutet, 17177 Stockholm, Sweden · Department of Medical Radiation Physics, Stockholm University, Solna, Sweden · Department of Oncology-Pathology, Karolinska Institutet, Solna, Sweden +4

  9. Soft-Label Governance for Distributional Safety in Multi-Agent Systems

    Mar 19, 2026Aizierjiang Aiersilan, Raeli SavittAI Agent GovernanceAI Risk Management

    SWARM AI Safety, New York, USA

  10. SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems

    Mar 18, 2026Rima Hazra, Bikram Ghuku, Ilona Marchenko +5Intelligent Tutoring SystemsAI Safety Evaluation

    Eindhoven University of Technology, Netherlands (TU/e) · Indian Institute of Technology Kharagpur · Taras Shevchenko National University of Kyiv +3

  11. Invisible Orchestrators Suppress Protective Behavior and Dissociate Power-Holders: Safety Risks in Multi-Agent LLM Systems

    Mar 17, 2026Hiroki FukuiMulti-Agent LLM SystemsMulti-Agent Orchestration

    Criminal Psychiatry Research Institute / Sexual Offender Medical Center Department of Neuropsychiatry, Kyoto University

  12. TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems

    Mar 16, 2026Kai Wang, Biaojie Zeng, Zeming Wei +7LLM Agent SafetyMulti-Agent System Security

    Shanghai AI Laboratory

  13. Questionnaire Responses Do not Capture the Safety of AI Agents

    Mar 15, 2026Max Hellrigel-Holderbaum, Edward James YoungAI Safety EvaluationLLM Safety Alignment

    Centre for Philosophy and AI Research Friedrich-Alexander-Universität Erlangen-Nürnberg · Department of Engineering University of Cambridge

  14. Literary Narrative as Moral Probe : A Cross-System Framework for Evaluating AI Ethical Reasoning and Refusal Behavior

    Mar 13, 2026David C. FlynnMoral Reasoning in Language ModelsAI Safety Evaluation

  15. Detecting Intrinsic and Instrumental Self-Preservation in Autonomous Agents: The Unified Continuation-Interest Protocol

    Mar 11, 2026Christopher AltmanAI Agent EvaluationAI Safety Evaluation

    Chief Scientist, Quantum Technology & Artificial Intelligence, Astradyne.

  16. Evaluating Human-AI Safety: A Framework for Measuring Harmful Capability Uplift

    Mar 6, 2026Michelle Vaccaro, Jaeyoon Song, Abdullah Almaatouq +1AI Safety EvaluationLLM Safety Evaluation

    Massachusetts Institute of Technology, Cambridge, USA

  17. Measuring AI R&D Automation

    Mar 4, 2026Alan Chan, Ranay Padarath, Joe Kwon +2AI Risk ManagementHuman Oversight of AI

    GovAI · University of Oxford

  18. SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testing

    Mar 2, 2026Anjali Parashar, Yingke Li, Eric Yang Yu +4AI Safety EvaluationBenchmark Design

    Laboratory for Information & Decision Systems (LIDS), MIT · Saab Inc.

  19. When can we trust untrusted monitoring? A safety case sketch across collusion strategies

    Feb 24, 2026Nelson Gardner-Challis, Jonathan Bostock, Georgiy Kozhevnikov +4AI Agent MonitoringAI Safety

    LASR Labs · Imperial College London · University of Wyoming +2

  20. AI Alignment Breaks at the Edge

    Feb 23, 2026Han Bao, Yue Huang, Xiaoda Wang +5LLM AlignmentAI Safety Evaluation

    University of Notre Dame · Emory University

  21. Fundamental Limits of Black-Box Safety Evaluation: Information-Theoretic and Computational Barriers from Latent Context Conditioning

    Feb 19, 2026Vishal SrivastavaAI Safety EvaluationBlack-Box Model Auditing

    Whiting School of Engineering, John Hopkins University, Baltimore, MD 21218

  22. First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations

    Dec 1, 2025David Wu, Fateme Nateghi Haredasht, Saloni Kumar Maharaj +54HealthcareClinical Decision Support

  23. PB-OEL: A Performance-Bounded Online Ensemble Learning Framework With Mixed Feedback for Real-Time Safety Assessment

    Mar 19, 2025Songqiao Hu, Zeyi Liu, Lufeng Hao +2Ensemble LearningAI Safety Evaluation

    Department of Automation, and the Institute for Embodied Intelligence and Robotics, Tsinghua University, Beijing 100084, China · China Ship Research and Development Academy, Beijing 100192, China