AI Safety Evaluation

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 91

All topics

Warning labels shift perceptions of sycophantic AI, but not its influence
Jun 19, 2026Lujain Ibrahim, Myra Cheng, Cinoo Lee +4LLM Sycophancy AI Safety Evaluation
University of Oxford. · 2Stanford University. · 3Microsoft. +1
AI Sandboxes: A Threat Model, Taxonomy, and Measurement Framework
Jun 16, 2026Inderjeet Singh, Haitham Mahmoud, Andrés MurilloCyber-Physical Systems AI Assurance
Fujitsu Research of Europe, United Kingdom
SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior
Jun 16, 2026Mingyue Cui, Linghui Shen, Xingyi YangSparse Autoencoders Mechanistic Interpretability
The Hong Kong Polytechnic University
When and How Severely: Scenario-Specific Safety Envelopes for Driving VLAs
Jun 12, 2026Abhinaw Priyadershi, Jelena FrtunikjVLMs for Autonomous Driving Language Model Safety Evaluation
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA GmbH, Munich, Germany
Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistants
Jun 12, 2026Dipesh Tharu MahatoAI Safety Evaluation LLM Safety Evaluation
New York University
A Virtuous AI is an Existential Risk
Jun 11, 2026Guillermo Del Pinal, Youngchan Lee, Min OhnLLM Safety Alignment AI Agent Safety
University of Massachusetts Amherst
Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers
Jun 10, 2026Jun Wen LeongDistribution Shift Conformal Prediction
DrugBench: Evaluating AI Control Protocols for Medication Harm Mitigation
Jun 10, 2026Guido Freire, Agustín Martínez-Suñé, Viviana CotikHealthcare AI Control
Universidad de Buenos Aires, Facultad de Ciencias Exactas y Naturales, Departamento de Computación, Argentina. · AI Safety Argentina (AISAR). · Department of Computer Science, University of Oxford, United Kingdom. +1
Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety
Jun 9, 2026Neil Kale, Rebecca Portnoff, Pratiksha Thaker +5AI Safety Evaluation AI Safety
1Carnegie Mellon University · 2Thorn
Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis
Jun 8, 2026Hyeji Choi, Yongtaek Lim, Minwoo KimMultilingual Language Model Evaluation Language Model Safety Evaluation
AI Safety Team, DATUMO.INC, Seoul, South Korea.
Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models
Jun 5, 2026Dewi Gould, Francis Rhys Ward, Anders Cairns Woodruff +18LLM Evaluation AI Safety Evaluation
Redwood Research · Astra Fellows Program · Aether Research +10
Coding with "Enemy": Can Human Developers Detect AI Agent Sabotage?
Jun 4, 2026Jingheng Ye, Huiqi Zou, Simon Yu +1AI Coding Agents AI Agent Monitoring
Northeastern University
AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
Jun 3, 2026Yanjing Ren, Reza Ebrahimi, TengTeng MaLLM Safety Benchmarks LLM-as-a-Judge
The School of Information Systems University of South Florida Tampa, US
Attack Selection in Agentic AI Control Evaluations Meaningfully Decreases Safety
Jun 3, 2026Catherine Ge-Wang, Tyler Crosse, Benjamin Hadad +3Adversarial Attacks AI Agent Safety
University of Oxford · Mentorship for Alignment Research Students (MARS), CAISH · Georgia Institute of Technology +3
Safety Measurements for Fine-tuned LLMs Should be Grounded in Capability
Jun 2, 2026Krishnapriya Vishnubhotla, Hillary Dawkins, Isar Nejadgholi +1Language Model Safety Evaluation LLM Fine-Tuning
National Research Council, Canada
RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulation
Jun 1, 2026Huiqiong Li, Jiayu Wang, Zhiting Mei +3World Models AI Safety Evaluation
Singapore Management University · Fudan University · Princeton University
Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment
May 29, 2026Arkadiy Saakyan, Charvi Rastogi, Lora AroyoLanguage Model Safety Evaluation Cultural Alignment
Columbia University, New York, NY, USA. · Google DeepMind, New York, NY, USA.
Safe2Drive: Evaluating Safe Driving Behaviors of E2E Autonomous Driving Models
May 29, 2026Nishad Sahu, Kalpana Panda, Congyuan Yu +3Autonomous Driving Benchmarks End-to-End Autonomous Driving
1Carnegie Mellon University · 2Birla Institute of Technology and Science Pilani
EUDAIMONIA: Evaluating Undesirable Dynamics in AI
May 28, 2026Jun Rui Huang, Wang Bill Zhu, Ziyi Liu +3LLM Alignment Language Model Safety Evaluation
University of Southern California
A Study on Question-Answer Dataset for LLM Safety Evaluation with a Focus on Illegal Activities
May 28, 2026Kenji Imamura, Masao Ideuchi, Atsushi FujitaAI Safety Evaluation LLM Safety Evaluation
National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, 619-0289, Japan
Models That Know How Evaluations Are Designed Score Safer
May 27, 2026Katharina Deckenbach, Haritz Puerto, Jonas Geiping +1Language Model Safety Evaluation AI Safety Evaluation
ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center
Retrying vs Resampling in AI Control
May 25, 2026James Lucassen, Adam KaufmanAI Agent Safety AI Agent Monitoring
Redwood Research
Referential Security as a New Paradigm for AI Evaluations
May 25, 2026Dan Ristea, Vasilios MavroudisAI Assurance Algorithmic Auditing
University College London · Alan Turing Institute · King's College London
Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems
May 23, 2026Barak OrRobot Safety AI Safety Evaluation
STATE16
AI Assurance: A Comprehensive Testing Strategy for Enterprise AI Systems
May 22, 2026Chitra Badagi, Divye Singh, Animesh Sen +1AI Assurance AI Safety Evaluation
Thoughtworks Technologies · Pune, MH, India · Hyderabad, TG, India
General Hazard Detection
May 22, 2026Stephanie Ng, CP Lim, SueJen Looi +5VLM Evaluation AI Safety Evaluation
Swinburne University of Technology, VIC, Australia · National Transport Research Organisation (NTRO), Australia · Google Cloud, California, USA +1
Can AI Make Conflicts Worse? An Alignment Failure in LLM Deployment Across Conflict Contexts
May 21, 2026Andrii KryshtalLLM Alignment Language Model Safety Evaluation
BlueDot Impact Technical AI Safety Sprint · Independent Researcher
RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents
May 19, 2026Doguhuan Yeke, Yanming Zhou, Leo Y. Lin +3Jailbreak Attacks AI Agent Security Benchmarks
Purdue University
Adversarial Stress Testing of SPARK Humanoid Safety Filters
May 18, 2026Saurav Ghosh, Abdou Sow, Luke ZhangAdversarial Robustness Robotics
aDepartment of Computer Science and Engineering, Washington University in St. Louis, Missouri, United States
REBAR: Reference Ethical Benchmark for Autonomy Readiness
May 18, 2026Jonathan Diller, David Barnes, Rebekah Bogdanoff +14AI Accountability AI Safety Evaluation
University of Pennsylvania · David Barnes, LLC · Duality Robotics, Inc. +3

Jun 19, 2026, cs.HCJ/K move · Enter open · S save

Warning labels shift perceptions of sycophantic AI, but not its influence

Lujain Ibrahim, Myra Cheng, Cinoo Lee, Pranav Khadpe+3

University of Oxford. · 2Stanford University. · 3Microsoft.+1

LLM Sycophancy AI Safety Evaluation Trust in AI

Open paper AskPDF arXiv

Recent work has raised concerns about the influence of sycophantic AI on user judgment and relationships. One proposed mitigation, which has received regulatory attention, is to warn users about potentially harmful AI behaviors such as sycophancy. In a preregistered experiment in which participants (N = 2,610) discussed real interpersonal conflicts with an AI system, we test whether warning labels mitigate sycophancy's influence. We find that a basic AI disclosure (This chatbot is AI'') has no detectable effect. Labeling the system as sycophantic (...may agree with you and validate you even when you are wrong...'') does shift users' perceptions, reducing perceived objectivity and trust, but it does not reliably reduce sycophancy's influence on users' self-perceived rightness or their willingness to repair the conflict. Our results reveal a gap between AI perception and AI influence: by shifting perception without reducing influence, warning-based interventions may offer a false sense of protection. Addressing the harms of sycophancy will therefore require understanding the specific mechanisms through which it shapes judgment, and improving model behavior itself.

Momentum

Related topics

Latest papers 91