AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,503 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies
Jan 18, 2026Ming Zhang, Jiabao Zhuang, Wenqing Jing +15AI Agent Benchmarks Deep Research Agents
Fudan University · Hunyuan Team, Tencent
AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks
Jan 16, 2026Weiyi Wang, Xinchi Chen, Jingjing Gong +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · OpenMOSS Team · Shanghai Innovation Institute
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
Jan 10, 2026Yingchaojie Feng, Qiang Huang, Xiaoya Xie +4Deep Research Agents LLM Agent Evaluation
School of Computing, National University of Singapore · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · Zhejiang University +1
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Dec 31, 2025Zixiang Liu, Wenrui Liu, Elsie Dai +5LLM Agent Evaluation AI Agent Benchmarks
Peking University · ZTE
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Dec 19, 2025Janghoon Han, Heegyu Kim, Changho Lee +6Deep Research Agents LLM Agent Evaluation
LG AI Research · University of Illinois Chicago · University of Michigan, Ann Arbor
PPTArena: A Benchmark for PowerPoint Editing
Dec 2, 2025Michael Ofengenden, Yunze Man, Ziqi Pang +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · University of California, Berkeley
CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflows
Nov 25, 2025Chenyue Li, Hyeonjae Kim, Wen Deng +4AI for Science Multi-Agent Orchestration
1The Hong Kong University of Science and Technology
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent Evaluation Multimodal Large Language Models
Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2
IndustryNav: Evaluating Spatial Reasoning and Safety of Embodied Agents in Dynamic Industrial Navigation
Nov 21, 2025Yifan Li, Lichi Li, Anh Dao +15Visual Spatial Reasoning Embodied Navigation
Michigan State University · Independent Researcher · University of California, Irvine +7
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
Nov 20, 2025Juntao Cheng, Wanyue Zhang, Zhiwei Yu +7Long-Horizon Agent Evaluation Egocentric Video Understanding
Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
Nov 4, 2025Jiayu Liu, Cheng Qian, Zhaochen Su +4LLM Planning LLM Agent Evaluation
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign · Tsinghua University
DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents
Oct 29, 2025Yun-Shiuan Chuang, Ruixuan Tu, Chengtao Dai +9Opinion Dynamics Large Language Model-Based Social Simulation
University of Wisconsin–Madison · Google DeepMind · Stanford University
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
Oct 22, 2025Gil Pasternak, Dheeraj Rajagopal, Julia White +4LLM Agent Evaluation AI Agent Benchmarks
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use Evaluation AI Agent Benchmarks
AWS AI Labs · University of Pittsburgh
Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search
Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Agentic Search Long-Horizon Agent Evaluation
Princeton Language and Intelligence, Princeton University · Work done as an intern at Samaya AI. · Samaya AI
HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning
Oct 16, 2025Chance Jiajie Li, Zhenze Mo, Yuhan Tang +11Cognitive Modeling Human Behavior Simulation
1MIT Media Lab · 8Northeastern University · 5MIT CEE +7
Compositional Machine Design as Program Synthesis with LLMs
Oct 16, 2025Wenqian Zhang, Yangyi Huang, Weiyang Liu +1LLM-Based Program Synthesis Program Synthesis
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent Evaluation Human-AI Interaction
Apart Research · AI Safety Cape Town · University of Chicago +2
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
Aug 14, 2025Shilong Li, Xingyuan Bu, Wenjie Wang +22Multimodal IR Multimodal Reasoning
1ByteDance · 2Nanjing University · 5Zhejiang University +2
Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Aug 3, 2025Yuhan Guo, Cong Guo, Aiwen Sun +12Cognitive Modeling Multimodal Reasoning
Southwestern University of Finance and Economics · Shanghai Jiao Tong University · Central South University +9
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing Agents Agent Evaluation
Nanyang Technological University, Singapore · Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China +1
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Jul 7, 2025Yuanzhe Hu, Yu Wang, Julian McAuleyLLM Agent Memory LLM Agent Evaluation
University of California, San Diego
HiMA-Ecom: A Long-Horizon Memory Benchmark for Hierarchical Multi-Agent E-commerce Assistants
Jun 24, 2025Junxing Hu, Ai Han, Haolan Zhan +8Long-Horizon Agent Evaluation AI Agent Benchmarks
JD Retail, JD.com, Inc., Beijing, 101111, China · Faculty of Engineering and Information Technology, The University of Melbourne, Parkville, Victoria, 3082, Australia · School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing, 100049, China
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent Orchestration AI Agent Evaluation
IBM · University of South Carolina · Steel Authority of India +1
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
May 17, 2025Idriss Malek, Omar Choukrani, Daniil Orel +5LLM Planning LLM Agent Evaluation
Tandon School of Engineering New York University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Abu Dhabi, UAE
Towards LLM Agents for Earth Observation
Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code Generation LLM Agent Evaluation
Cornell University · Columbia University

Jan 18, 2026J/K move · Enter open · S save

TaxoBench: Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies

Ming Zhang, Jiabao Zhuang, Wenqing Jing, Kexin Tan+14

Fudan University · Hunyuan Team, Tencent

AI Agent Benchmarks Deep Research Agents IR Evaluation Taxonomy Induction

Open paper AskPDF arXiv

Deep Research Agents increasingly automate survey writing, yet existing benchmarks do not jointly test whether they retrieve the papers experts consider essential and organize those papers into paper-grounded taxonomies. We introduce TaxoBench, a benchmark built from 72 highly cited LLM surveys, 3,815 cited papers, and their expert-authored taxonomies. TaxoBench evaluates systems in two settings: Deep Research mode measures end-to-end retrieval and organization from a topic, while Bottom-Up mode provides the expert paper set and isolates organization. We evaluate leaf-level assignments with ARI and V-Measure and hierarchy-level structure with US-TED, US-NTED, and Sem-Path. Across 7 Deep Research Agents and 16 LLM configurations, the best agent retrieves only 20.92% of expert-cited papers, and none of 70 standard Bottom-Up runs reaches the experts' average depth of 4.86. A controlled probe shows that models which match this depth do so by fragmenting the taxonomy, reducing alignment with the expert reference. We further find that raw Sem-Path remains near a no-organization floor even when a newer model generation gains 3.68 pp ARI; after depth matching, humans lead on all 10 matched surveys by 13.27 pp. These results identify retrieval and hierarchical organization as separate bottlenecks and show why hierarchy metrics must be calibrated before they are used to compare models.