AI Agent Evaluation

Follow this topic Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 341

All topics

Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?
May 9, 2026Filippo Ziliotto, Ciro Beneduce, Bruno Lepri +3AI Agent Evaluation Vision-Language Grounding
University of Padova · FondazioneLucianoBrunoSerafiniKessler · Fondazione Bruno Kessler
Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents
May 9, 2026Ying Chen, Lihuang Fang, Rui Jiang +4AI Agent Evaluation Agent Evaluation
XPENG ROBOTICS
Log analysis is necessary for credible evaluation of AI agents
May 8, 2026Peter Kirgis, Sayash Kapoor, Stephan Rabanser +8Agent Failure Analysis AI Agent Evaluation
Princeton University · Independent · UK AISI +4
Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
May 8, 2026Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula +4Competitive Analysis Multi-Agent Orchestration
1IBM · 2Artificial Intelligence Institute at University of South Carolina · 3Steel Authority of India Limited +2
Behavioral Determinants of Deployed AI Agents in Social Networks: A Multi-Factor Study of Personality, Model, and Guardrail Specification
May 8, 2026Sarah Wilson, Diem Linh Dang, Usman Ali Moazzam +2AI Agent Evaluation LLM Agents
Columbia University
Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare
May 8, 2026Prasanna Desikan, Harshit Rajgarhia, Shivali Dalmia +1Healthcare Benchmark Design
Centific AI Research Redmond, WA, USA
When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory
May 8, 2026Jiaqi Shao, Yiyi Lu, Yunzhen Zhang +1Agent Memory AI Agent Evaluation
Hong Kong University of Science and Technology, Hong Kong · Duke Kunshan University, China
EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation
May 8, 2026Yi Liu, TingFeng Hui, Wei Zhang +4AI Agent Evaluation AI Agent Benchmarks
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology · Chongqing University
Computer Use at the Edge of the Statistical Precipice
May 7, 2026Pierluca D'Oro, Sneha Silwal, William Wong +6Computer-Use Agent Benchmarks Computer-Use Agents
Meta Superintelligence Labs
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
May 7, 2026Roger Creus Castanyer, Pablo Samuel Castro, Glen BersethAI Agent Evaluation AI Agent Benchmarks
Mila Quebec AI Institute · Université de Montréal · Mila Quebec AI Institute Université de Montréal +1
When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoning
May 7, 2026Vasilis Niarchos, Constantinos Papageorgakis, Alexander G. Stapleton +1AI for Science AI Agent Evaluation
Department of Physics, CCTP and ITCP, University of Crete, 71303, Greece · Centre for Theoretical Physics, Department of Physics and Astronomy, Queen Mary University of London, London E1 4NS, United Kingdom · Theoretical Physics Department, CERN, Geneva, Switzerland +1
Process Matters more than Output for Distinguishing Humans from Machines
May 7, 2026Milena Rmus, Mathew D. Hardy, Thomas L. Griffiths +1AI Agent Evaluation Imitation Learning
Roundtable Technologies Inc. · Princeton University
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
May 7, 2026Jonas Wiedermann-Möller, Leonard Dung, Maksym AndriushchenkoAI Agent Evaluation AI Agent Safety
Universität Bielefeld · Ruhr-Universität Bochum · ELLIS Institute Tübingen +2
BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents
May 7, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +8AI Agents for Scientific Discovery Deep Research Agents
University of Oxford · University College London · Technical University of Munich +1
Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback
May 7, 2026Mohammad Al Ridhawi, Mahtab Haj Ali, Hussein Al OsmanTime Series Forecasting AI Agent Evaluation
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
Intentionality is a Design Decision: Measuring Functional Intentionality for Accountable AI Systems
May 6, 2026Allessia Chiappetta, Robert MahariAI Accountability AI Agent Evaluation
CodeX, The Stanford Center for Legal Informatics, Palo Alto, California, USA
Executable World Models for ARC-AGI-3 in the Era of Coding Agents
May 6, 2026Sergey RodionovWorld Models Coding Agents
SingularityNET
DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents
May 6, 2026Zhaorun Chen, Xun Liu, Haibo Tong +14AI Agent Evaluation AI Agent Security
Virtue AI · University of Chicago · University of Illinois, Urbana-Champaign +4
Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies
May 5, 2026Zirui Tang, Xuanhe Zhou, Yumou Liu +19Computer-Use Agent Benchmarks AI Agent Evaluation
1Shanghai Jiao Tong University · 2ByteDance · 5Independent Researher +2
AcademiClaw: When Students Set Challenges for AI Agents
May 4, 2026Junjie Yu, Pengrui Lu, Weiye Si +75AI Agent Evaluation AI Agent Benchmarks
Foundation-Model-Based Agents in Industrial Automation: Purposes, Capabilities, and Open Challenges
May 4, 2026Vincent Henkel, Felix Gehlhoff, David Kube +13AI Agent Evaluation LLM Agent Reliability
Institute of Automation Technology, Helmut Schmidt University / University of the Federal Armed Forces Hamburg, Hamburg, Germany. · Siemens AG, Nuremberg, Germany; Institute for Technologies and Management of Digital Transformation, Bergische Universität Wuppertal, Germany. · Artiquare GmbH, Ingolstadt, Germany. +11
When Stress Becomes Signal: Detecting Antifragility-Compatible Regimes in Multi-Agent LLM Systems
May 4, 2026Jose Manuel de la Chica, Juan Manuel Vera, Jairo RodríguezMulti-Agent LLM Systems AI Agent Evaluation
Santander AI Lab
STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems
May 4, 2026Akash Bonagiri, Gerard Janno Anderias, Saee Patil +6Annotator Disagreement AI Agent Evaluation
Department of Computer Science, University of California, Davis, Davis, CA, USA.
Preregistration for Experiments with AI Agents
May 3, 2026Michelle VaccaroAI Agent Evaluation
Institute for Data, Systems & Society (IDSS), Massachusetts Institute of Technology, Cambridge, MA, United States.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
May 2, 2026Mukund PandeyAI Agent Reliability AI Agent Evaluation
Independent Researcher
Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents
May 1, 2026Maksym Nechepurenko, Pavel ShuvalovAI Agent Evaluation Forecasting Benchmarks
Director of Research, Devnull, Dubai, UAE · Chief Technology Officer, Devnull, Dubai, UAE
Agentic AI for Substance Use Education: Integrating Regulatory and Scientific Knowledge Sources
May 1, 2026Kosar Haghani, Zahra Kolagar, Mohammed AtiquzzamanAI Agent Evaluation Generative AI in Education
Department of Social Science, Texas Woman's University, Denton, Texas, USA. · Department of Computer Science, Augsburg University of Applied Sciences, Augsburg, Germany · School of Computer Science, University of Oklahoma, Norman, Oklahoma, USA
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Apr 30, 2026Jinchao Li, Yunxin Li, Chenrui Zhao +3Computer-Use Agent Benchmarks Computer-Use Agents
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
End-to-End Evaluation and Governance of an EHR-Embedded AI Agent for Clinicians
Apr 30, 2026Aaryan Shah, Andrew Hines, Alexia Downs +6Healthcare AI Agent Evaluation
1Canvas Medical, San Francisco, CA, USA · Department of Biomedical Data Science, Stanford University, Stanford, CA, USA · 3XPC (X Primary Care) +2
The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate
Apr 29, 2026Blaž Bertalanič, Carolina FortunaLLM Self-Correction AI Agent Evaluation
Jožef Stefan Institute Ljubljana, Slovenia

May 9, 2026, cs.AIJ/K move · Enter open · S save

Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?

Filippo Ziliotto, Ciro Beneduce, Bruno Lepri, Luciano Serafini+2

University of Padova · FondazioneLucianoBrunoSerafiniKessler · Fondazione Bruno Kessler

AI Agent Evaluation Vision-Language Grounding Embodied AI Vision-Language Agents

Open paper AskPDF arXiv

In the animal kingdom, mirror self-recognition is a canonical probe of higher-order cognition, emerging only in some species. We ask whether an analogous functional capability emerges in embodied vision-language model (VLM) agents: can they recognize themselves in a mirror? We introduce a controlled 3D benchmark where a first-person VLM agent must infer a hidden body attribute from its reflection and select the matching target, while avoiding self-other misattribution. To separate mirror-grounded self-identification from shortcuts, we test mirror removal, misleading cues, and occluded reflections. We also evaluate the decision process through mirror seeking, temporal ordering, self-attribution, and reasoning-action consistency. Our experiments show that mirror-based self-identification emerges mainly in stronger VLMs. These models can use reflected evidence for action, whereas weaker models often inspect the mirror but fail to extract self-relevant information or misattribute their reflection. Language-vision conflict further shows that self-referential language alone is not evidence of grounded self-identification. Overall, mirror-based evaluation provides a diagnostic for whether embodied self-grounding is causally rooted in perception and action rather than priors, prompt compliance, or confabulation.

Momentum

Related topics

Latest papers 341