AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,503 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

CocoaBench: Evaluating Unified Digital Agents in the Wild
Apr 13, 2026CocoaBench Team, Shibo Hao, Zhining Zhang +29AI Agent Benchmarks Computer-Use Agent Evaluation
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
Apr 13, 2026Xue Qin, Simin Luan, John See +2AI Agent Governance AI Agent Safety Benchmarks
School of Software, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus, Putrajaya, Malaysia +1
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
Apr 13, 2026Xiaomeng Hu, Yinger Zhang, Fei Huang +7LLM Agent Evaluation AI Agent Benchmarks
Qwen Team, Alibaba Group · The Chinese University of Hong Kong
CheeseBench: Evaluating Large Language Models on Rodent Behavioral Neuroscience Paradigms
Apr 12, 2026Zacharie BugaudLLM Evaluation LLM Agents
Astera Institute
Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
The Amazing Agent Race: Strong Tool Users, Weak Navigators
Apr 11, 2026Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2LLM Agent Evaluation AI Agent Benchmarks
University of Minnesota Twin Cities · Yonsei University · Grammarly
Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision
Apr 11, 2026Lingfeng Zhou, Junhao Shi, Jin Gao +1Software Engineering Agents AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai Innovation Institute
COMPOSITE-Stem
Apr 10, 2026Kyle Waters, Lucas Nuzzi, Tadhg Looram +20Scientific Agent Benchmarks AI Agent Benchmarks
PortexAI · University of Milano-Bicocca · University of Calgary +17
NetForge RL: A Multi-Agent Simulation Environment for Cyber Defense with Durative Actions
Apr 10, 2026Igor JankowskiAutonomous Cyber Defense Cybersecurity
HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?
Apr 10, 2026Tu Trinh, Mohamed Elfeki, Guangze Luo +9LLM Agents AI Agent Benchmarks
Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym
Apr 10, 2026Lars Benedikt Kaesberg, Tianyu Yang, Niklas Bauer +3Spatial Reasoning Benchmarks Spatial Reasoning
University of Göttingen, Germany
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Apr 10, 2026Ling Shi, Yuqin Dai, Ziyin Wang +7LLM Agent Evaluation Agent Evaluation
Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China +3
DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?
Apr 10, 2026Young-Suk Lee, Ramon Fernandez Astudillo, Radu FlorianAI Agent Benchmarks Scientific Agent Benchmarks
IBM Research
SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment
Apr 10, 2026Sihang Jiang, Lipeng Ma, Zhonghua Hong +9Self-Evolving Agents AI Agent Benchmarks
Fudan University
ParseBench: A Document Parsing Benchmark for AI Agents
Apr 9, 2026Boyang Zhang, Sebastián G. Acosta, Preston Carlson +4Document Parsing Benchmarks Document Parsing
ClawBench: Can AI Agents Complete Everyday Online Tasks?
Apr 9, 2026Yuxuan Zhang, Yubo Wang, Yipeng Zhu +25Web Agent Benchmarks AI Agent Evaluation
University of British Columbia · Vector Institute · Etude AI +9
KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Apr 9, 2026Tongbo Chen, Zhengxi Lu, Zhan Xu +13Mobile Agent Benchmarks AI Agent Benchmarks
Zhejiang University · Apple · Tencent
Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time
Apr 9, 2026Ruizhi Zhang, Ye Huang, Yuangang Pan +5Game-Playing Agents Multimodal Agents
Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China · A*STAR · College of Computer Science and Software Engineering, Shenzhen University
Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
Apr 9, 2026Jiaxuan Wang, Yulan Hu, Wenjin Yang +3Reward Model Evaluation AI Agent Benchmarks
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · AMAP, Alibaba Group
CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V
Apr 9, 2026John Chen, Sihan Cheng, Can Gurkan +1LLM Agent Evaluation AI Agent Benchmarks
Northwestern University
Towards Knowledgeable Deep Research: Framework and Benchmark
Apr 9, 2026Wenxuan Liu, Zixuan Li, Long Bai +13Deep Research Agents AI Agent Benchmarks
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · Also with University of Chinese Academy of Sciences. · National University of Singapore, Singapore, Singapore
GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
Apr 8, 2026Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin +2AI Agent Evaluation AI Agent Benchmarks
National University of Singapore · University of Oxford
$k$ -server-bench: Automating Potential Discovery for the $k$ -Server Conjecture
Apr 8, 2026Kirill Brilliantov, Etienne Bamas, Emmanuel AbbéAI-Assisted Mathematical Discovery Autonomous Scientific Discovery
MDS Lab, EPFL
EmoMAS: Emotion-Aware Multi-Agent System for High-Stakes Edge-Deployable Negotiation with Bayesian Orchestration
Apr 8, 2026Yunbo Long, Yuhan Liu, Liming XuMulti-Agent Negotiation Automated Negotiation
University of Cambridge, UK · University of Toronto, Canada
Riemann-Bench: A Benchmark for Moonshot Mathematics
Apr 8, 2026Suhaas Garre, Erik Knutsen, Sushant Mehta +1Mathematical Reasoning Benchmarks Benchmark Design
Surge AI
ATANT: An Evaluation Framework for AI Continuity
Apr 8, 2026Samuel Sameer TanguturiLLM Memory AI Agent Evaluation
Kenotic Labs
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Apr 7, 2026Bowen Ye, Rang Li, Qibin Yang +10AI Agent Evaluation AI Agent Benchmarks
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong
AgentCE-Bench: Agent Configurable Evaluation with Scalable Horizons and Controllable Difficulty under Lightweight Environments
Apr 7, 2026Wang Yang, Chaoda Song, Xinpeng Li +7AI Agent Benchmarks Agent Benchmarks
Case Western Reserve University · NII LLMC (Japan) · University of Zurich
FrontierFinance: A Long-Horizon Computer-Use Benchmark of Real-World Financial Tasks
Apr 7, 2026Michael Krumdick, Varshini Reddy, Shivani Chaudhary +8Computer-Use Agent Benchmarks LLM Evaluation
Kensho Technologies · S&P Global · MIT

Apr 13, 2026J/K move · Enter open · S save

CocoaBench: Evaluating Unified Digital Agents in the Wild

CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang+28

AI Agent Benchmarks Computer-Use Agent Evaluation Tool-Using Agents Long-Horizon Agent Evaluation Automated Agent Evaluation LLM Agent Planning

Open paper AskPDF arXiv

LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.