Overview

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

CardsList

563 papers47 papers in the last four weeks, down 48% on the four weeks before. 0.4% of all new papers.

  1. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

    Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark AuditingAI Agent Evaluation

  2. A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines

    Feb 25, 2026Gaoyuan Du, Amit Ahlawat, Xiaoyang Liu +1AI Agent EvaluationAI Agent Auditing

  3. Aletheia tackles FirstProof autonomously

    Feb 24, 2026Tony Feng, Junehyuk Jung, Sang-hyun Kim +14LLM Mathematical ReasoningAutonomous Research Agents

  4. Implicit Intelligence -- Evaluating Agents on What Users Don't Say

    Feb 23, 2026Ved Sirdeshmukh, Marc WetterAI Agent EvaluationAI Agent Benchmarks

  5. Quantifying the Expectation-Realisation Gap for Agentic AI Systems

    Feb 23, 2026Sebastian LobentanzerAI Agent EvaluationHuman Oversight of AI

  6. MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems

    Feb 23, 2026Jin Jia, Zhiling Deng, Zhuangbin Chen +2Multi-Agent LLM SystemsLLM Agent Reliability

  7. Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

    Feb 22, 2026Dongming Jiang, Yi Li, Songtao Wei +8Agent MemoryAgent Memory Management

  8. DREAM: Deep Research Evaluation with Agentic Metrics

    Feb 21, 2026Elad Ben Avraham, Changhao Li, Ron Dorfman +8AI Agent EvaluationDeep Research Agents

  9. Towards a Science of AI Agent Reliability

    Feb 18, 2026Stephan Rabanser, Sayash Kapoor, Peter Kirgis +3AI Agent ReliabilityAI Agent Evaluation

  10. JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

    Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-JudgeAI Agent Evaluation

  11. Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

    Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM EvaluationAI Agent Evaluation

  12. Persuasion Propagation: Does Persuasion Change AI Agent Behavior?

    Jan 31, 2026Hyejun Jeong, Amir Houmansadr, Shlomo Zilberstein +1Persuasion in Language ModelsAI Agent Evaluation

  13. LogicEnvGen: Task-Logic Driven Generation of Diverse Simulated Environments for Embodied AI

    Jan 20, 2026Jianan Wang, Siyang Zhang, Bin Li +4Robotics SimulationAI Agent Evaluation

  14. DR-Arena: an Automated Evaluation Framework for Deep Research Agents

    Jan 15, 2026Yiwen Gao, Ruochen Zhao, Yang Deng +1Deep Research AgentsBenchmark Design

  15. Monte Carlo Query Search: Active Capability Assessment of AI Agents

    Dec 18, 2025Daniel Bramblett, Rushang Karia, Adrian Ciotinga +3AI Agent EvaluationActive Learning

  16. Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

    Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent EvaluationMultimodal Large Language Models

  17. TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward

    Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Reward ModelingLLM Planning

  18. AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance

    Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent OrchestrationAI Agent Evaluation

  19. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments

    Jul 15, 2024Chong Zhang, Xinyi Liu, Zhongmou Zhang +10Multi-Agent LLM SystemsHuman Behavior Simulation

  20. Towards Shutdownable Agents via Stochastic Choice

    Jun 30, 2024Elliott Thornley, Alexander Roman, Christos Ziakas +2CorrigibilityAI Agent Safety