AI Agent Evaluation

Follow this topic Ask about this topic

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

525 papers77 papers in the last four weeks, up 4% on the four weeks before. 0.4% of all new papers.

ClawArena: Benchmarking AI Agents in Evolving Information Environments
Apr 5, 2026Haonian Ji, Kaiwen Xiong, Siwei Han +9AI Agent Evaluation AI Agent Benchmarks
UNC-Chapel Hill · University of California, Berkeley · University of California, Santa Cruz
The Art of Building Verifiers for Computer Use Agents
Apr 5, 2026Corby Rosset, Pratyusha Sharma, Andrew Zhao +2LLM Agent Verification AI Agent Evaluation
Microsoft Research · Browserbase
MTI: A Behavior-Based Temperament Profiling System for AI Agents
Apr 2, 2026Jihoon JeongPersonality Modeling in Language Models AI Agent Evaluation
Department of Electrical Engineering and Computer Science Daegu Gyeongbuk Institute of Science and Technology (DGIST) ModuLabs
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
Apr 1, 2026Jingyue Li, André StorhaugAI Agent Evaluation Agent Evaluation
Norwegian University of Science and Technology, Trondheim, Norway
Paper Reconstruction Evaluation: Evaluating Presentation and Hallucination in AI-written Papers
Apr 1, 2026Atsuyuki Miyai, Mashiro Toyooka, Zaiying Zhao +3Generative AI Evaluation Language Model Generation Evaluation
The University of Tokyo
Agent psychometrics: Task-level performance prediction in agentic coding benchmarks
Apr 1, 2026Chris Ge, Daria Kryvosheieva, Daniel Fried +2LLM Agent Evaluation AI Agent Evaluation
MIT · Fulcrum · Carnegie Mellon University
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
Apr 1, 2026HyunJoon Jung, William NaLLM-as-a-Judge AI Agent Evaluation
Reward Hacking as Equilibrium under Finite Evaluation
Mar 30, 2026Jiacheng Wang, Jinbin HuangReward Hacking AI Agent Evaluation
Prediction Arena: Benchmarking AI Models on Real-World Prediction Markets
Mar 28, 2026Jaden Zhang, Gardenia Liu, Oliver Johansson +3AI Agent Evaluation Forecasting Benchmarks
Arcada Labs · Harvard University
AIRA_2: Overcoming Bottlenecks in AI Research Agents
Mar 27, 2026Karen Hambardzumyan, Nicolas Baldwin, Edan Toledo +22Autonomous Research Agents AI Agent Architecture
FAIR at Meta · University College London · University of Oxford
M-CARE: Standardized Clinical Case Reporting for AI Model Behavioral Disorders, with a 20-Case Atlas and Experimental Validation
Mar 27, 2026Jihoon JeongAI Agent Evaluation AI Safety Evaluation
Department of Electrical Engineering and Computer Science, Daegu Gyeongbuk Institute of Science and Technology (DGIST) · ModuLabs
MolQuest: A Benchmark for Agentic Evaluation of Abductive Reasoning in Chemical Structure Elucidation
Mar 26, 2026Taolin Han, Shuang Wu, Jinghang Wang +4Abductive Reasoning in Language Models Scientific Reasoning in Language Models
Alibaba Group, Hangzhou, China
Willful Disobedience: Automatically Detecting Failures in Agentic Traces
Mar 25, 2026Reshabh K Sharma, Shraddha Barke, Benjamin ZornAI Agent Evaluation AI Agent Auditing
University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA
Efficient Benchmarking of AI Agents
Mar 24, 2026Franck NdzomgaAI Agent Evaluation AI Agent Benchmarks
ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence
Mar 24, 2026ARC Prize FoundationARC-AGI AI Agent Benchmarks
ARC Prize Foundation
Code Review Agent Benchmark
Mar 24, 2026Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf +3Automated Code Review Software Engineering Benchmarks
National University of Singapore · Zhejiang University · SonarSource
UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities
Mar 24, 2026Qi Jia, Haodong Zhao, Dun Pei +7AI Agent Evaluation LLM Evaluation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
Mar 23, 2026Shoubin Yu, Lei Shu, Antoine Yang +6Agent Benchmarks Web Agent Benchmarks
Google DeepMind · UNC Chapel Hill
AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science
Mar 19, 2026An Luo, Jin Du, Xun Xian +12AI Agent Evaluation AI Agent Benchmarks
School of Statistics, University of Minnesota · AIScientists, Inc. · Data Science Institute, University of Chicago +4
A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance
Mar 18, 2026Ciprian Paduraru, Petru-Liviu Bouruc, Alin StefanescuAI Agent Reliability LLM Agent Orchestration
University of Bucharest Romania · University of Bucharest & Institute for Logic and Data Science Romania
Nonstandard Errors in AI Agents
Mar 17, 2026Ruijiang Gao, Steven Chong XiaoAI Agent Reliability Autonomous Research Agents
Naveen Jindal School of Management, University of Texas at Dallas
CoMAI: A Collaborative Multi-Agent Framework for Robust and Equitable Interview Evaluation
Mar 17, 2026Gengxin Sun, Ruihao Yu, Liangyi Yin +3Automated Evaluation Multi-Agent Systems
Shandong University, Qingdao, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Shandong University, Jinan, China
An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
Mar 16, 2026Hong Zhang, Barry Smith, Satish Balay +4High-Performance Computing Code Generation
Argonne National Laboratory, Lemont, IL, USA
CUBE: A Standard for Unifying Agent Benchmarks
Mar 16, 2026Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko +23AI Agent Benchmarks AI Agent Evaluation
ServiceNow AI Research · Dalhousie · McGill +7
Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents
Mar 16, 2026Simone Aonzo, Merve Sahin, Aurélien Francillon +1AI Agent Evaluation AI Agent Safety
Eurecom, Sophia Antipolis, France · SAP Labs, Sophia Antipolis, France · Depthfirst, San Francisco, United States
Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI
Mar 16, 2026Jinhu Qi, Yifan Li, Minghao Zhao +4AI Agent Evaluation AI Agent Safety
The Chinese University of Hong Kong, Hong Kong SAR, China · Macao Polytechnic University, Macao SAR, China · Jilin University, Changchun, China
TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents
Mar 16, 2026Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli +3AI Agent Evaluation AI Agent Benchmarks
Research Institute for Digital Future, Khalifa University · AT&T CDO · GSMA
Detecting Intrinsic and Instrumental Self-Preservation in Autonomous Agents: The Unified Continuation-Interest Protocol
Mar 11, 2026Christopher AltmanAI Agent Evaluation AI Safety Evaluation
Chief Scientist, Quantum Technology & Artificial Intelligence, Astradyne.
Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios
Mar 11, 2026Linus Folkerts, Will Payne, Simon Inman +11LLMs for Cybersecurity AI Agent Evaluation
AI Security Institute, United Kingdom.
CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
Mar 10, 2026Kristen Pereira, Neelabh Sinha, Rajat Ghosh +1Automated Code Review AI Agent Evaluation
Nutanix, Inc.

Apr 5, 2026J/K move · Enter open · S save

ClawArena: Benchmarking AI Agents in Evolving Information Environments

Haonian Ji, Kaiwen Xiong, Siwei Han, Peng Xia+8

UNC-Chapel Hill · University of California, Berkeley · University of California, Santa Cruz

AI Agent Evaluation AI Agent Benchmarks Knowledge Conflicts in Language Models Belief Revision

Open paper AskPDF arXiv

AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.