AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,505 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

MarketBench: Evaluating AI Agents as Market Participants
Apr 26, 2026Andrey Fradkin, Rohit KrishnanMulti-Agent Coordination LLM Agent Evaluation
Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation
SoccerRef-Agents: Multi-Agent System for Automated Soccer Refereeing
Apr 25, 2026Zi Meng, Wanli Song, Yi Hu +2Multi-Agent LLM Systems Sports Video Analysis
University of Michigan, Michigan, USA · Shanghai Jiao Tong University, Shanghai, China
Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Apr 24, 2026Mengzhuo Chen, Junjie Wang, Fangwen Mu +4Agent Failure Analysis Multi-Agent LLM Systems
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents
Apr 24, 2026Xirui Li, Ming Li, Yunze Xiao +4Multi-Agent Coordination AI Agent Evaluation
University of Maryland · 2Mohamed bin Zayed University of Artificial Intelligence · 3Carnegie Mellon University
AgentSearchBench: A Benchmark for AI Agent Search in the Wild
Apr 24, 2026Bin Wu, Arastun Mammadli, Xiaoyu Zhang +1Agentic Search AI Agent Evaluation
Centre for Artificial Intelligence, University College London
When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors
Apr 23, 2026Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4Tool-Augmented Language Model Agents LLM Agent Evaluation
School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1
How VLAs (Really) Work In Open-World Environments
Apr 23, 2026Amir Rasouli, Yangzheng Wu, Zhiyuan Li +4VLM Robustness Long-Horizon Robotic Manipulation
Noahs Ark Laboratory, Huawei Technologies Canada. · while at Huawei Canada.
Deep FinResearch Bench: Evaluating AI's Ability to Conduct Professional Financial Investment Research
Apr 22, 2026Mirazul Haque, Antony Papadimitriou, Samuel Mensah +6Quantitative Finance Deep Research Agents
AI Research, JPMorganChase
Cooperative Profiles Predict Multi-Agent LLM Team Performance in AI for Science Workflows
Apr 22, 2026Shivani Kumar, Adarsh Bharathwaj, David JurgensMulti-Agent LLM Systems AI for Science
University of Michigan
Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows
Apr 22, 2026Hardy Chen, Nancy Lau, Haoqin Tu +8Coding Agents LLM Agent Evaluation
UC Santa Cruz · UT Dallas · NUS
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Apr 22, 2026Shanshan Zhong, Yi Lu, Jingjie Ning +7Continual Learning for LLM Agents LLM Agent Evaluation
Carnegie Mellon University · Amazon AGI
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Apr 21, 2026Md Nayem Uddin, Kumar Shubham, Eduardo Blanco +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Arizona State University · Genies · University of Arizona
Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
Apr 21, 2026Yi Zhong, Buqiang Xu, Yijun Wang +4Agentic Workflows Business Process Automation
Zhejiang University · Tencent
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models
Apr 21, 2026Josue Torres-Fonseca, Naihao Deng, Yinpei Dai +5Safety Evaluation of VLA Models LLM Safety Benchmarks
University of Michigan · Boise State University
Time Series Augmented Generation for Financial Applications
Apr 21, 2026Anton Kolonin, Alexey Glushchenko, Evgeny Bochkov +1LLM Agent Evaluation AI Agent Benchmarks
SingularityNET Foundation
Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment
Apr 21, 2026Bobo Li, Rui Wu, Zibo Ji +5AI Agent Benchmarks Multi-Agent Reasoning
National University of Singapore · Sichuan University · University of Minnesota Twin Cities +2
Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents
Apr 21, 2026Vasundra SrininvasanLong-Horizon Agent Evaluation AI Agent Benchmarks
Stanford School of Engineering
AutomationBench
Apr 21, 2026Daniel Shepard, Robin SalimansLLM Agent Evaluation Business Process Automation
ClawEnvKit: Automatic Environment Generation for Claw-Like Agents
Apr 20, 2026Xirui Li, Ming Li, Ion Stoica +2Agent Evaluation AI Agent Benchmarks
University of Maryland · 5Mohamed bin Zayed University of Artificial Intelligence · 2Arena +2
MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline
Apr 20, 2026Jiyao Liu, Jianghan Shen, Sida Song +19Healthcare AI Agent Benchmarks
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Nanjing University, Nanjing, China +1
ComPASS: Towards Personalized Agentic Social Support via Tool-Augmented Companionship
Apr 20, 2026Zhaopei Huang, Yanfeng Jia, Jiayi Zhao +3AI Companions AI Agent Benchmarks
Renmin University of China · Beijing, China · Beihang University
Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence
Apr 20, 2026Guanting Dong, Junting Lu, Junjie Huang +17Lifelong Learning Agents AI Agent Benchmarks
Renmin University of China · ByteDance Seed
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering
Apr 20, 2026Yindong Zhang, Wenmian Yang, Yiquan Zhang +1LLM Agent Orchestration AI Agent Benchmarks
Hong Kong Baptist University · Beijing Normal-Hong Kong Baptist University · Beijing Normal University
Latent Preference Modeling for Multi-Session Personalized Tool Calling
Apr 20, 2026Yejin Yoon, Minseo Kim, Taeuk KimLLM Personalization LLM Agent Memory
Hanyang University, Seoul, Republic of Korea
Scaling Human-AI Coding Collaboration Requires a Governable Consensus Layer
Apr 20, 2026Tianfu Wang, Zhezheng Hao, Yin Wu +7AI Agent Benchmarks AI Agent Governance
HKUST (GZ) Guangzhou, China · ZJU Hangzhou, China · USTC Hefei, China +1
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
Apr 20, 2026Shiquan Zhang, Tianyi Zhang, Le Fang +3Mobile GUI Automation LLM Agent Evaluation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · School of Computer Science, University of Auckland, Auckland, New Zealand
SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Apr 19, 2026Ziao Zhang, Kou Shi, Shiting Huang +13Lifelong Learning Agents Long-Horizon Agent Evaluation
University of Science and Technology of China · University of Toronto · University of Sydney
HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
Apr 19, 2026Chao Jin, Wenkui Yang, Hao Sun +6GUI Agents Hallucination in Language Models
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences

Apr 26, 2026J/K move · Enter open · S save

MarketBench: Evaluating AI Agents as Market Participants

Andrey Fradkin, Rohit Krishnan

Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.

Multi-Agent Coordination LLM Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Markets are a promising way to coordinate AI agent activity for similar reasons to those used to justify markets more broadly. In order to effectively participate in markets, agents need to have informative signals of their own ability to successfully complete a task and the cost of doing so. We propose MarketBench, a benchmark for assessing whether AI agents have these capabilities. We use a 93-task subset of SWE-bench Lite, a software engineering benchmark, with six recently released LLMs as a demonstration. These LLMs are miscalibrated on both success probability and token usage, and auctions built from these self-reports diverge from a full-information allocation. A follow-up intervention where we add information about capabilities from prior experiments to the context improves calibration, but only modestly narrows the gap to a full-information benchmark. We also document the performance of a market-based scaffolding with these LLMs. Our results point to self-assessment as a key bottleneck for market-style coordination of AI agents.