Overview

Agent benchmarks test whether AI systems can complete tasks through sequences of decisions and actions, from web browsing and coding to desktop use, games, and professional workflows. They use simulated or real-world environments, tool-use tasks, long-horizon scenarios, and memory or multimodal challenges, with scoring based on verified outcomes, intermediate steps, costs, and reliability across repeated runs. Common approaches include executable state-based checks, trajectory review, human or rubric-based judging, and controlled comparisons of models, tools, and agent harnesses. Results show substantial gaps on complex tasks, while realism, reproducibility, verifier quality, and the influence of the harness make scores difficult to compare or generalize to deployment.

Latest papers

CardsList

130 papers14 papers in the last four weeks, down 18% on the four weeks before. 0.1% of all new papers.

  1. Agent Benchmarks Stop Ranking Before They Run Out of Tasks

    Nov 10, 2026Zihan Dong, Qishi Zhan, Minxuan HuAgent BenchmarksAutomated Agent Evaluation

  2. GTAGym: AAA Game Tasks Environment for Evaluation and Reinforcement Learning

    Nov 9, 2026Jiajun Chen, Shicheng Fang, Jiahao Lu +2AI Agent BenchmarksAgent Benchmarks

  3. SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents

    Nov 2, 2026Bangrui Xu, Jie Yang, Sidi Chen +6Self-Improving AgentsAI Agent Evaluation

  4. MAC-Bench: Benchmarking Multi-Agent Collaboration via Interconnected Tasks

    Nov 1, 2026Zerui Chen, Qinggang Zhang, Qichuan Liu +6Multi-Agent CollaborationAgent Benchmarks

  5. Benchmarking Task Takeover Across Agent Handoffs

    Oct 27, 2026Wenyuan Li, Guang Li, Dongyuan Li +3LLM Agent EvaluationAgent Evaluation

  6. AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning

    Oct 26, 2026Xiaofei Yu, Kailai Sun, Yanyu Mao +7Multimodal DatasetAgent Benchmarks

  7. Benchmarks Are Surprisingly Compaction-Friendly

    Oct 22, 2026Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2Context Compaction for LLM AgentsLLM Context Compression

  8. AgentVQA: A Unified Benchmark for Agentic Visual Understanding

    Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM EvaluationAgent Benchmarks

  9. ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents

    Oct 22, 2026Wenjie Zhang, Zhichao Shi, Xuhui Jiang +4Memory-Augmented AgentsPersonal AI Agents

  10. Did the Problem Actually Get Harder? Testing LLMs on Paired P and NP-Complete Problems Hardened by Adversarial Multi-Agent Rewriting

    Oct 21, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Zabir Al Nazi +2Language Model Reasoning EvaluationAgent Benchmarks

  11. VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement

    Oct 19, 2026Ruotian Peng, Yuanyi Ren, Shuaiqi Liu +7LLM Agent TrainingSelf-Improving Agents

  12. RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents

    Oct 15, 2026Yunqi Liu, Peiyu Wang, Bingli Wang +21Terminal-Agent BenchmarksLLM Agent Evaluation

  13. The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock

    Oct 14, 2026Ningxin Pan, Hanyu Li, Yeqing Shen +3LLM Agent Self-ImprovementLLM Decision-Making

  14. DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents

    Oct 10, 2026Jinlong Luo, Haoran Wu, Bo XuAgent BenchmarksText-to-SQL

  15. MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training

    Oct 4, 2026Yuxin Liu, Yuxuan Wang, Zhenxin Lei +9Multimodal Model EvaluationAgent Benchmarks

  16. GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation

    Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM EvaluationGraph Neural Networks

  17. HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

    Oct 2, 2026Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri +14Web Agent BenchmarksAgent Benchmarks

  18. Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents

    Oct 1, 2026Yekun Chai, Qiwei Peng, Haoyi XiongLLM Agent EvaluationAI Agent Evaluation

  19. DAYJOB: A Benchmark for Long-Horizon Professional Work

    Oct 1, 2026Stephanie Finley, Liudas Panavas, Thomas Mikkelson +12LLM Agent EvaluationLong-Horizon Agent Tasks

  20. AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

    Sep 30, 2026Ruifeng Yuan, Yizhi Li, Yaxin Du +9Agent BenchmarksAutonomous Research Agents

  21. EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses

    Sep 30, 2026Zhixuan Tan, Pengjie Gu, Zhao Li +4Software Engineering BenchmarksLong-Horizon Agent Evaluation

  22. AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search

    Sep 28, 2026Tongtong Feng, Xin Wang, Haoran Hou +9Aerial RoboticsObject-Goal Navigation

  23. GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games

    Sep 25, 2026Dhananjay Ashok, Adam Shen, Aslan Huo Feng +10Self-Improving AgentsGame-Playing Agents