AI Agent Benchmarks

Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 1,013

All topics
CardsList

  1. Training AI Scientists to Replicate Research

    Aug 13, 2026Damon Falck, Samer Sabri, Anja Surina +8AI Agents for Scientific DiscoveryAI Agent Benchmarks

    1Inherent

  2. ARAC: Benchmarking Auto-Research's Alignment and Completeness on End-to-End Researchs

    Aug 13, 2026Jiale Cui, Yueyao Yuan, Kaixi Zhong +3Human Preference EvaluationAI Agent Benchmarks

    School of Software Technology, Zhejiang University

  3. DiG-bench: Discovery in Games

    Aug 12, 2026Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan +13Benchmark DesignAI Agent Evaluation

    1Thinking About Thinking · 2Independent · University of Oxford +5

  4. Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Games

    Aug 12, 2026Deborah Sinishaw, Qile Zhu, Edwin Meriaux +1Multi-Agent CoordinationGame Theory

    Department of Electrical and Computer Engineering, McGill University · School of Computer Science, McGill University · Centre for Intelligent Machines (CIM), McGill University

  5. VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

    Aug 12, 2026Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder +6Tool-Use EvaluationAI Agent Benchmarks

    IBM, Yorktown Heights, NY, USA

  6. CTBench: Evaluating Troubleshooting Capabilities of AI Agents in Realistic Telecom Network Operations

    Aug 12, 2026Xingyu Yan, Tingting Dai, Antonio De Domenico +16AI Agent Benchmarks

    *Huawei Technologies, China · †Paris Research Center, Huawei Technologies, Boulogne-Billancourt, France · +Queen Mary University of London

  7. Retry, Switch, or Abstain? Learning Strategy-Aware Tool-Use Policies via Controlled Error Injection

    Aug 12, 2026Chaoran Chen, Vy Nguyen, Ziji Zhang +7Curriculum RLAI Agent Benchmarks

    Amazon

  8. Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agents

    Aug 12, 2026Zining Huang, Haoran Que, Hong Zeng +8AI Coding AgentsLLM Agent Evaluation

    ByteDance Seed · Tsinghua University · Peking University

  9. FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

    Aug 12, 2026Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh +4AI Agent EvaluationAI Agent Benchmarks

    Samaya AI

  10. MBA: Multimodal Benchmark and Agents for Real-World Business Ideation

    Aug 12, 2026Hojun Choi, Jaeyo Shin, Suin Lee +1AI Agent BenchmarksComputational Creativity

    KAIST AI, Republic of Korea

  11. Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

    Aug 11, 2026Brian Wang, Bin Feng, Xiaoman Pan +26AI for ScienceAI Agent Evaluation

  12. VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

    Aug 11, 2026Xiaohongshu IncLong-Horizon Agent EvaluationAI Agent Benchmarks

    Xiaohongshu Dots Studio · Evolvent AI

  13. DuplexWorld: Can voice agents help you get through the day?

    Aug 11, 2026Aryan Vijay Bhosale, Harshit Rajgarhia, Akhil Pothanapalli +3Voice Agent EvaluationSpoken Dialogue Systems

    Centific Global Solutions Inc. · University of Maryland · Work done during an internship at Centific.

  14. SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

    Aug 11, 2026Junjie Ye, Zhuohui Sheng, Shaofan Liu +12LLM EvaluationAI Agent Benchmarks

    1Fudan University · 2Tencent Hunyuan Team

  15. DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation

    Aug 11, 2026Xiaotong Wang, Dazhen DengData VisualizationLLM-as-a-Judge

    State Key Lab of CAD&CG, Zhejiang University, China · School of Software Technology, Zhejiang University, China

  16. Evaluating Rational Contracting in Natural Language

    Aug 11, 2026Bhavyesh Sajja, Max Kleiman-Weiner, Roger Zimmermann +1Multi-Agent NegotiationAutomated Negotiation

    Department of Computer Science, National University of Singapore · Department of Computer Science, University of Washington · A*STAR Institute of Advanced Intelligence and Computing

  17. ComboShoppingBench: Evaluating LLM Agents for Budget-Constrained Basket Shopping with Coupons

    Aug 10, 2026Adrian Li, Kelong Mao, Yudong Guo +7Benchmark DesignLLM Agent Evaluation

  18. TRACE: TRajectory Attribution for Automated Context Engineering

    Aug 10, 2026Yikai Zhao, Pradeep Kumar Misra, Saurabh PandeyAgent Failure AnalysisAI Agent Benchmarks

    Amazon Dallas, TX, USA · Amazon Seattle, WA, USA

  19. Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

    Aug 10, 2026Keyu He, Xuhui Zhou, Maarten SapGame-Playing AgentsLLM Agent Evaluation

    Carnegie Mellon University

  20. Evo-Bench: Can Language Models Improve Agent Harness?

    Aug 10, 2026Lisheng Huang, Chen Yang, Hao Zhou +6AI Agent EvaluationLong-Horizon Agent Evaluation

    1Gaoling School of Artificial Intelligence, Renmin University of China · 2BOSS Zhipin, Beijing, China

  21. 360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

    Aug 9, 2026Kenta Watanabe, Atsuyuki Miyai, Mizuki Takenawa +2Spatial Reasoning BenchmarksEmbodied Navigation

    The University of Tokyo, Tokyo, Japan

  22. OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents

    Aug 9, 2026Andrea Caciolai, Pere-Lluís Huguet Cabot, Chierh Cheng +11AI Agent EvaluationAI Agent Benchmarks

    Meta Superintelligence Labs

  23. PluginEval: A Diagnostic Benchmark for Fine-Grained Error Attribution in Function Calling

    Aug 9, 2026Dongjie Xu, Julius, Hanchi Dong +6LLM EvaluationTool-Use Evaluation

    Soochow University · Tencent Inc. · Beijing University of Posts and Telecommunications +4

  24. SkillReason: Reasoning-Enhanced Agent Skill Retrieval for Implicit User Requests

    Aug 9, 2026Donghong Jiang, Endian Lin, Luoping Cui +6AI Agent BenchmarksLLM Agent Skill Retrieval

    Beijing University of Posts and Telecommunications · Peking University · Beijing ZOYEN Technology Co., Ltd. +1

  25. Business Arena: Benchmarking LLM Agents in a Realistic Marketplace

    Aug 9, 2026Yijun Pan, Yukun Lian, Kunyu Shi +5LLM Agent EvaluationAI Agent Benchmarks

    1Accio Team, Alibaba Group · 2Yale University

  26. ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration

    Aug 9, 2026Guo Chen, Ziwen Li, Reed Li +4Multi-Agent LLM SystemsAI Agent Evaluation

    Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China

  27. Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives

    Aug 8, 2026Yingpeng Ma, Jianhao Yan, Bei Shi +6Interactive StorytellingAI Agent Benchmarks

    NLP2CT Lab, University of Macau, Macau, China · Westlake University, Hangzhou, China · Harbin Institute of Technology, Shenzhen, China +1

  28. Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Cities

    Aug 8, 2026Xiaohe Li, Yiru Wang, Junhao Fan +6Multi-Agent CoordinationRobotics Simulation

    the Key Laboratory of Target Cognition and Application Technology, Aerospace Information Research Institute, Chinese Academy of Sciences · School of Artificial Intelligence, Shanghai Jiaotong University

  29. TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?

    Aug 8, 2026Yuxuan Zhu, Peng PuSelective PredictionFault Detection

    School of Data Science and Engineering East China Normal University Shanghai, China

  30. Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing

    Aug 7, 2026Jiacheng Miao, Jin Mu, Guanhua Chen +1AI Agent BenchmarksLLM Agent Training

    Department of Biomedical Data Science, Stanford University · Department of Genetics, Stanford University · Department of Biostatistics and Medical Informatics, University of Wisconsin–Madison +3