Overview

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

CardsList

1,598 papers130 papers in the last four weeks, down 53% on the four weeks before. 1.0% of all new papers.

  1. AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance

    Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent OrchestrationAI Agent Evaluation

  2. SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments

    May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM EvaluationAI Agent Benchmarks

  3. LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

    May 17, 2025Idriss Malek, Omar Choukrani, Daniil Orel +5LLM PlanningLLM Agent Evaluation

  4. Towards LLM Agents for Earth Observation

    Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code GenerationLLM Agent Evaluation

  5. MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents

    Mar 3, 2025Kunlun Zhu, Hongyi Du, Zhaochen Hong +8Multi-Agent CollaborationAI Agent Benchmarks

  6. GraphChase: A Platform and Benchmark for Urban Network Security Games

    Jan 29, 2025Shuxin Zhuang, Shuxin Li, Tianji Yang +4Benchmark DesignAI Agent Benchmarks

  7. How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?

    Dec 12, 2024Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5HealthcareLLM Agent Evaluation