Overview

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

CardsList

1,598 papers130 papers in the last four weeks, down 53% on the four weeks before. 1.0% of all new papers.

  1. Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision

    Jan 19, 2026Bingsen Chen, Boyan Li, Ping Nie +3Deep Research AgentsLLM Agent Reliability

  2. AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks

    Jan 16, 2026Weiyi Wang, Xinchi Chen, Jingjing Gong +2LLM Agent EvaluationAI Agent Benchmarks

  3. Advancing ESG Intelligence: An Expert-level Agent and Comprehensive Benchmark for Sustainable Finance

    Jan 13, 2026Yilei Zhao, Wentao Zhang, Lei Xiao +3AI Agent BenchmarksAgent Benchmarks

  4. DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports

    Jan 13, 2026Ruizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research AgentsLLM Agent Evaluation

  5. IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents

    Jan 10, 2026Yingchaojie Feng, Qiang Huang, Xiaoya Xie +4Deep Research AgentsLLM Agent Evaluation

  6. DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation

    Dec 19, 2025Janghoon Han, Heegyu Kim, Changho Lee +6Deep Research AgentsLLM Agent Evaluation

  7. PPTArena: A Benchmark for PowerPoint Editing

    Dec 2, 2025Michael Ofengenden, Yunze Man, Ziqi Pang +2LLM Agent EvaluationAI Agent Benchmarks

  8. CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflows

    Nov 25, 2025Chenyue Li, Hyeonjae Kim, Wen Deng +4AI for ScienceMulti-Agent Orchestration

  9. Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

    Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent EvaluationMultimodal Large Language Models

  10. IndustryNav: Evaluating Spatial Reasoning and Safety of Embodied Agents in Dynamic Industrial Navigation

    Nov 21, 2025Yifan Li, Lichi Li, Anh Dao +15Visual Spatial ReasoningEmbodied Navigation

  11. SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

    Nov 20, 2025Juntao Cheng, Wanyue Zhang, Zhiwei Yu +7Long-Horizon Agent EvaluationEgocentric Video Understanding

  12. CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents

    Nov 4, 2025Jiayu Liu, Cheng Qian, Zhaochen Su +4LLM PlanningLLM Agent Evaluation

  13. DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents

    Oct 29, 2025Yun-Shiuan Chuang, Ruixuan Tu, Chengtao Dai +9Opinion DynamicsLarge Language Model-Based Social Simulation

  14. When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

    Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use EvaluationAI Agent Benchmarks

  15. Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search

    Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Agentic SearchLong-Horizon Agent Evaluation

  16. HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning

    Oct 16, 2025Chance Jiajie Li, Zhenze Mo, Yuhan Tang +11Cognitive ModelingHuman Behavior Simulation

  17. Compositional Machine Design as Program Synthesis with LLMs

    Oct 16, 2025Wenqian Zhang, Yangyi Huang, Weiyang Liu +1LLM-Based Program SynthesisProgram Synthesis

  18. LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

    Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language ModelsAI Agent Benchmarks

  19. HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants

    Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent EvaluationHuman-AI Interaction

  20. Training Language Model Agents to Find Vulnerabilities with CTF-Dojo

    Aug 25, 2025Terry Yue Zhuo, Dingmin Wang, Hantian Ding +2LLM Agent TrainingLLMs for Cybersecurity

  21. MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

    Aug 14, 2025Shilong Li, Xingyuan Bu, Wenjie Wang +22Multimodal IRMultimodal Reasoning

  22. Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents

    Aug 3, 2025Yuhan Guo, Cong Guo, Aiwen Sun +12Cognitive ModelingMultimodal Reasoning

  23. StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

    Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing AgentsAgent Evaluation

  24. AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

    Jul 9, 2025Talor Abramovich, Gal ChechikLarge Language Model-Guided Scientific DiscoveryAI Agent Benchmarks

  25. Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

    Jul 7, 2025Yuanzhe Hu, Yu Wang, Julian McAuleyLLM Agent MemoryLLM Agent Evaluation

  26. HiMA-Ecom: A Long-Horizon Memory Benchmark for Hierarchical Multi-Agent E-commerce Assistants

    Jun 24, 2025Junxing Hu, Ai Han, Haolan Zhan +8Hierarchical MARLLong-Horizon Agent Evaluation

  27. AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents

    Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI AlignmentLLM Agent Evaluation