AI Safety Evaluation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 91

All topics

AI Agents May Always Fall for Prompt Injections
May 17, 2026Sahar Abdelnabi, Eugene BagdasarianContextual Integrity AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center · University of Massachusetts Amherst
Is VLA Reasoning Faithful? Probing Safety of Chain-of-Causation in Autonomous Driving Models
May 17, 2026Nicanor Mayumu, Xiaoheng Deng, Patrick MukalaCoT Faithfulness VLMs for Autonomous Driving
School of Computer Science and Engineering Central South University 932 Lushan Road, Changsha, Hunan P. R. China · School of Computer Science University of Wollongong in Dubai UOWD Building, Dubai Knowledge Park Dubai, United Arab Emirates
SLIP & ETHICS: Graduated Intervention for AI Emotional Companions
May 15, 2026Minseo KimAI Safety Evaluation AI Safety
HUA Labs, Seoul, Republic of Korea
Muse Spark Safety & Preparedness Report
May 14, 2026Cristina Menghini, Peter Ney, Hamza Kwisaba +117Cybersecurity AI Safety Evaluation
Sail
Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute
May 14, 2026Eugene Koran, Yejun Yun, Samantha Tetef +2AI Agent Monitoring AI Control
SPAR · Yale University
Training ML Models with Predictable Failures
May 14, 2026Will Schwarzer, Scott NiekumLanguage Model Safety Evaluation LLM Fine-Tuning
University of Massachusetts Amherst
Fusion-fission forecasts when AI will shift to undesirable behavior
May 14, 2026Neil F. Johnson, Frank Yingjie HuoAI Safety Evaluation LLM Safety
Physics Department, The George Washington University, Washington DC 20052, USA
VERA-MH: Validation of Ethical and Responsible AI in Mental Health
May 13, 2026Luca Belli, Kate H. Bentley, Josh Gieringer +6Healthcare Mental Health
Spring Health · Yale University
RISED: A Pre-Deployment Evaluation Framework for High-Stakes AI Decision-Support Systems, with Application to Healthcare
May 13, 2026Rohith Reddy Bellibatlu, Manpreet Singh, Yash Jajoo +2Distribution Shift Robustness Healthcare
Florida International University, Miami, FL, USA. · Boston University, Boston, MA, USA. · New York University, New York, NY, USA. +2
The Evaluation Differential: When Frontier AI Models Recognise They Are Being Tested
May 12, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Auditing AI Safety Evaluation
Department of Computer Science, University of Oxford · Institute for Ethics in AI, University of Oxford · 3Responsible Technology Institute, University of Oxford
Conformity Generates Collective Misalignment in AI Agents Societies
May 11, 2026Giordano De Marzo, Alessandro Bellina, Claudio Castellano +2Opinion Dynamics AI Agent Safety
University of Konstanz, Konstanz, Germany · Centro Ricerche Enrico Fermi, Rome, Italy · Complexity Science Hub, Vienna, Austria +5
Restricting the Model, Missing the System: Measurement and Accountability in Offensive AI Governance
May 10, 2026Michael Alexander Riegler, Finn Schwall, Annika Willoch Olstad +4AI Agent Security AI Safety Evaluation
AI Safety Department SimulaMet and OsloMet Oslo, Norway · Department of Computer Science NTNU Trondheim, Norway
Mental Health AI Safety Claims Must Preserve Temporal Evidence
May 9, 2026Srimonti Dutta, Ratna KandalaHealthcare Mental Health
WAI USA Research Labs · Texas, USA · University of Kansas +1
Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents
May 8, 2026Zhengyang Tang, Yi Zhang, Chenxin Li +18Computer-Use Agents Agent Evaluation
1Tencent Hunyuan · 2The Chinese University of Hong Kong, Shenzhen · 3Tsinghua University
Automated alignment is harder than you think
May 7, 2026Aleksandr Bowkis, Marie Davidsen Buhl, Jacob Pfau +1Scalable Oversight AI Alignment
AI Security Institute
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
May 6, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Evaluation LLM Alignment
Department of Computer Science, University of Oxford
NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims
May 5, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1AI Accountability AI Safety Evaluation
Department of Computer Science, University of Oxford
Position: Safety and Fairness in Agentic AI Depend on Interaction Topology, Not on Model Scale or Alignment
May 1, 2026Tanav Singh Bajaj, Nikhil Singh, Karan Anand +1Opinion Dynamics AI Agent Safety
Department of Computer Science, University of British Columbia, Vancouver, Canada · Department of Artificial Intelligence, IIT Hyderabad, Hyderabad, India · Amazon, Delhi, India.
Code World Model Preparedness Report
May 1, 2026Daniel Song, Peter Ney, Cristina Menghini +21AI Safety Evaluation LLM Safety Evaluation
MSL Preparedness Team, AI Security Team
Persona-Grounded Safety Evaluation of AI Companions in Multi-Turn Conversations
Apr 30, 2026Prerna Juneja, Lika LomidzeAI Companions AI Safety Evaluation
Seattle University
Real-Time GPU-Accelerated Monte Carlo Evaluation of Safety-Critical AEB Systems Under Uncertainty
Apr 29, 2026Akshay Karjol, Shadi AlawnehCollision Avoidance Uncertainty Quantification
Department of Electrical and Computer Engineering, Oakland University, Rochester, MI 48309 USA
Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control
Apr 29, 2026Mahiro Nakao, Kazuhiro TakemotoLanguage Model-Based Control Healthcare
Kyushu Institute of Technology Iizuka, Fukuoka, Japan
A Comparative Evaluation of AI Agent Security Guardrails
Apr 27, 2026Qi Li, Jiu Li, Pingtao Wei +8LLM Guardrails AI Agent Security
Beijing Caizhi Tech, Beijing, China
ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
Apr 25, 2026Yizheng Huang, Wenjun Zeng, Aditi Kumaresan +1Active Learning AI Safety Evaluation
Google DeepMind
Ethics Testing: Proactive Identification of Generative AI System Harms
Apr 23, 2026Shin Hwei Tan, Haibo Wang, Heng LiAI Safety Evaluation Automated Test Generation
Concordia University, Montreal, Canada · Polytechnique Montreal, Montreal, Canada
How VLAs (Really) Work In Open-World Environments
Apr 23, 2026Amir Rasouli, Yangzheng Wu, Zhiyuan Li +4VLM Robustness Long-Horizon Robotic Manipulation
Noahs Ark Laboratory, Huawei Technologies Canada. · while at Huawei Canada.
When Safety Fails Before the Answer: Benchmarking Harmful Behavior Detection in Reasoning Chains
Apr 21, 2026Ishita Kakkar, Enze Zhang, Rheeya Uppaal +1AI Safety Evaluation LLM Safety Evaluation
Department of Computer Sciences, University of Wisconsin-Madison
HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models
Apr 14, 2026Zixing Chen, Yifeng Gao, Li Wang +8Robotics AI Safety Evaluation
Fudan University · City University of Hong Kong · Shanghai Innovation Institute
SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems
Mar 18, 2026Rima Hazra, Bikram Ghuku, Ilona Marchenko +5Intelligent Tutoring Systems AI Safety Evaluation
Eindhoven University of Technology, Netherlands (TU/e) · Indian Institute of Technology Kharagpur · Taras Shevchenko National University of Kyiv +3
First, do NOHARM: a medical safety benchmark and randomized study of physician and AI teaming on clinical consultations
Dec 1, 2025David Wu, Fateme Nateghi Haredasht, Saloni Kumar Maharaj +54Healthcare Clinical Decision Support

May 17, 2026, cs.CRJ/K move · Enter open · S save

AI Agents May Always Fall for Prompt Injections

Sahar Abdelnabi, Eugene Bagdasarian

ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center · University of Massachusetts Amherst

Contextual Integrity AI Agent Security AI Safety Evaluation Prompt Injection Defense Prompt Injection Attacks Prompt Injection Attacks on LLMs+1

Open paper AskPDF arXiv

Prompt injection is the most critical vulnerability in deployed AI agents. Despite recent progress, we show that the prevailing defense paradigm (data-instruction separation) both fails to detect attacks that operate through contextual manipulation and degrades contextually appropriate behavior. We then recast prompt injection via the lens of Contextual Integrity (CI), a privacy theory that judges information flow compliance with contextual norms. This explains types of attacks that current defenses attempt to patch and predict advanced ones future agents will face. We develop unique benign and attack scenarios that force an agent to violate the norms by (1) misrepresenting the flow, (2) manipulating norms, or (3) mixing multiple flows. This reframing suggests an impossibility result: an adversary can always construct a context under which a blocked flow appears legitimate, or a defender who tightens norms will block genuinely legitimate flows. Our findings suggest that current research addresses a shrinking fraction of future attack surfaces. Instead, through CI, we offer a principled framework for evaluating context-sensitive failures, and designing CI-aware alignment for the frontier autonomous agents.

Momentum

Related topics

Latest papers 91