AI Agent Benchmarks

Ask about this topic

Overview

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

CardsList

1,503 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

  1. When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents

    Mar 17, 2026Lu Yan, Xuan Chen, Xiangyu ZhangAI Coding AgentsCoding Agents

    Purdue University

  2. RetailBench: Evaluating Long-Horizon Autonomous Decision-Making and Strategy Stability of LLM Agents in Realistic Retail Environments

    Mar 17, 2026Linghua Zhang, Jun Wang, Jingtong Wu +1Long-Horizon Agent EvaluationAI Agent Benchmarks

    Ant Group · City University of Hong Kong

  3. VisBrowse-Bench: Benchmarking Visual-Native Search for Multimodal Browsing Agents

    Mar 17, 2026Zhengbo Zhang, Jinbo Su, Zhaowen Zhou +14Multimodal Search AgentsAI Agent Benchmarks

    CASIA · Ant Digital Technologies, Ant Group · RUC +4

  4. SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding

    Mar 17, 2026Songcheng Cai, Zhiheng Lyu, Yuansheng Ni +13AI Agent BenchmarksSoftware Engineering Benchmarks

    University of Waterloo · University of Toronto · The Hong Kong University of Science and Technology +2

  5. FormulaCode: Evaluating Agentic Optimization on Large Codebases

    Mar 16, 2026Atharva Sehgal, James Hou, Akanksha Sarkar +4Coding Agent BenchmarksCode Optimization

    The University of Texas at Austin · California Institute of Technology · Cornell University

  6. AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback

    Mar 16, 2026Andrea Tupini, Lars Liden, Reuben Tan +2AI Agent BenchmarksLLM Agent Planning

    Microsoft Research

  7. CUBE: A Standard for Unifying Agent Benchmarks

    Mar 16, 2026Alexandre Lacoste, Nicolas Gontier, Oleh Shliazhko +23AI Agent BenchmarksAI Agent Evaluation

    ServiceNow AI Research · Dalhousie · McGill +7

  8. The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

    Mar 16, 2026Seth Karten, Jake Grigsby, Tersoo Upaa +28AI Agent BenchmarksGame-Playing Agents

    Princeton · UT-Austin · Team Heatz +12

  9. Benchmarking Real-Time Question Answering via Executable Code Workflows

    Mar 16, 2026Wenjie Zhou, Yuan Gao, Xin Zhou +5AI Agent BenchmarksTemporal IR

    Li Auto Inc. · Minzu University of China

  10. Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

    Mar 16, 2026Penny Chong, Harshavardhan Abichandani, Jiyuan Shen +4LLM Agent EvaluationAgent Evaluation

    SAP · Stanford University

  11. Gym-V: A Unified Vision Environment System for Agentic Vision Research

    Mar 16, 2026Fanqing Meng, Lingxiao Du, Jiawei Gu +9AI Agent BenchmarksRL for VLMs

    National University of Singapore. · University of Washington. · The Hong Kong Polytechnic University. +1

  12. CCTU: A Benchmark for Tool Use under Complex Constraints

    Mar 16, 2026Junjie Ye, Guoqiang Zhang, Wenjie Fu +4Tool-Use EvaluationAI Agent Benchmarks

    College of Computer Science and Artificial Intelligence, Fudan University

  13. GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents

    Mar 16, 2026Yang Li, Yuchen Liu, Haoyu Lu +8Computer-Use Agent BenchmarksGUI Agents

    HyperAI Team, Xiaomi Corporation

  14. VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining

    Mar 16, 2026Xuanyu Zhu, Yuhao Dong, Rundong Wang +9Compositional Reasoning in VLMsTool Use in VLMs

    PKU · Meituan · NTU +4

  15. TelcoAgent-Bench: A Multilingual Benchmark for Telecom AI Agents

    Mar 16, 2026Lina Bariah, Brahim Mefgouda, Farbod Tavakkoli +3AI Agent EvaluationAI Agent Benchmarks

    Research Institute for Digital Future, Khalifa University · AT&T CDO · GSMA

  16. Shopping Companion: Benchmarking and Training LLM Agents for Long-Horizon Preference-Grounded E-Commerce Tasks

    Mar 16, 2026Zijian Yu, Kejun Xiao, Huaipeng Zhao +2LLM Agent TrainingAI Agent Benchmarks

    Alibaba International Digital Commercial Group

  17. OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence

    Mar 16, 2026Peigen Liu, Rui Ding, Yuren Mao +7LLM AgentsAI Agent Benchmarks

    School of Software Technology, Zhejiang University · Laboratory for Statistical Monitoring and Intelligent Governance of Common Prosperity, Zhejiang Gongshang University · Tongyi Lab, Alibaba Group +1

  18. LongVidSearch: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videos

    Mar 15, 2026Rongyi Yu, Chenyuan Duan, Wentao ZhangAI Agent BenchmarksMulti-Hop QA

    Peking University & Zhongguancun Academy Beijing, China

  19. ττ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

    Mar 14, 2026Soham Ray, Keshav Dhandhania, Victor Barres +1Voice Agent EvaluationAI Agent Benchmarks

    Sierra.ai, San Francisco, USA · Princeton Language and Intelligence, Princeton University

  20. EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings

    Mar 13, 2026Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6AI Agent BenchmarksLLM Agent Planning

    ServiceNow Research · Mila - Quebec AI Institute · Université de Montréal

  21. Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

    Mar 12, 2026Aadyaa Maddi, Prakhar Naval, Deepti Mande +3LLM Agent EvaluationData Analysis Agents

    Rockfish Data and Carnegie Mellon University

  22. Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

    Mar 12, 2026Łukasz Borchmann, Jordy Van Landeghem, Michał Turski +12Multi-Document QALLM Agent Evaluation

  23. SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

    Mar 11, 2026Yadi Cao, Sicheng Lai, Jiahe Huang +12AI Agent BenchmarksLarge Language Model-Guided Optimization

    University of California San Diego · The Chinese University of Hong Kong, Shenzhen · Peking University +3

  24. MedMASLab: A Unified Orchestration Framework for Benchmarking Multimodal Medical Multi-Agent Systems

    Mar 10, 2026Yunhang Qian, Xiaobin Hu, Jiaquan Yu +6Multi-Agent SystemsMultimodal Agents

    National University of Singapore · University of Science and Technology of China · Fudan University +3

  25. MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

    Mar 10, 2026Kangsan Kim, Yanlai Yang, Suji Kim +4Egocentric Video QAEmbodied QA

    KAIST · New York University · Samsung Electronics +1

  26. MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistants

    Mar 10, 2026Zuhao Zhang, Chengyue Yu, Yuante Li +3Web Application GenerationLLM Evaluation

    Inclusion AI, Ant Group, Hangzhou, Zhejiang, China · Shanghai Jiao Tong University, Shanghai, China · Carnegie Mellon University, Pittsburgh, PA, USA

  27. MM-tau-p2^2: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

    Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent EvaluationAgent Evaluation

    Sprinklr AI, Gurugram, India

  28. LLMs for Text-Based Exploration and Navigation Under Partial Observability

    Mar 10, 2026Stephan Sandfuchs, Maximilian Melchert, Jörg FrochteLanguage Model-Based ControlRobot Navigation

    AKIS – Interdisciplinary Institute for Applied AI and Data Science Ruhr, Bochum University of Applied Sciences, Bochum, Germany

  29. Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments

    Mar 10, 2026Yang Li, Xing Chen, Yutao Liu +5AI Agent EvaluationAI Agent Benchmarks

    Tsinghua University · Ocean University of China (OUC) · CAICT +4

  30. MASEval: Extending Multi-Agent Evaluation from Models to Systems

    Mar 9, 2026Cornelius Emde, Alexander Rubinstein, Anmol Goel +4AI Agent EvaluationAI Agent Benchmarks

    Parameter Lab · University of Oxford · University of Tübingen +4