AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

146 papers in the last four weeks, up 161% on the four weeks before. 1.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Jun 21, 2026Jiayu Liu, Qihan Lin, Cheng Qian +8Long-Horizon Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign
CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents
Jun 20, 2026Rishi SrivastavaBenchmark Construction LLM Agent Evaluation
Beiing Human
OpenBioRQ: Unsolved Biomedical Research Questions for Agents
Jun 20, 2026Minbyul JeongAI Agent Reliability Citation Verification
Upstage AI
AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
Jun 20, 2026Weiyuan Zhou, Haiping Ma, Xiaoshan Yu +3Multi-Agent LLM Systems Educational Assessment
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University, China · School of Artificial Intelligence, Anhui University, China · School of Computer Science and Technology, Dalian University of Technology, China +1
ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks
Jun 19, 2026Vincent Siu, Manasi Sharma, Dawn Song +2Computer-Use Agent Benchmarks Computer-Use Agents
1Scale AI · University of California, Santa Cruz · *Work done during an internship at Scale AI. +1
Trip+: Benchmarking Agents in Personalized Interactive Travel Planning
Jun 19, 2026Junle Chen, Wei Chen, Yehong Xu +6LLM Agent Evaluation AI Agent Benchmarks
HKUST · Tencent Hy · HKUST(GZ)
AgentMeter: Evaluating Model-CLI Matching for CLI-Based Local Task-Solving Agents
Jun 19, 2026Han Chi, Jiaxin Qi, Yan Cui +2LLM Agent Evaluation AI Agent Benchmarks
Computer Network Information Center, Chinese Academy of Sciences · Hangzhou Institute for Advanced Study, Chinese Academy of Sciences
Agentic Time Machine as an Infrastructure for Future-Event Forecasting
Jun 19, 2026Jingyi Chai, Bingyang Zheng, Xiangrui Liu +5Multi-Agent LLM Systems LLM Agent Evaluation
Shanghai Jiao Tong University
Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineering
Jun 18, 2026Sergei TrashchenkovLLM Agent Evaluation AI Agent Benchmarks
ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
Jun 18, 2026Jiajun Li, Mingshu Cai, Yixuan Li +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science and Engineering, Southeast University · Waseda University · Nanyang Technological University +1
Benchmarking Agentic Review Systems
Jun 18, 2026Dang Nguyen, Wanqing Hao, Yanai Elazar +1LLM Agent Evaluation AI Agent Benchmarks
University of Chicago · Bar-Ilan University
Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents
Jun 18, 2026Dhaval C. Patel, Kaoutar El Maghraoui, Shuxin Lin +58OOD Generalization LLM Agent Evaluation
IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows
Jun 17, 2026Ahmad Salimi, Wentao Ma, Yuzhi Tang +3Audio-Language Model Evaluation AI Agent Benchmarks
Boson AI Toronto, ON, Canada · Boson AI Santa Clara, CA, USA
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
Jun 17, 2026Shengyuan Ding, Xilin Wei, Xinyu Fang +4Multimodal Memory Sequential Decision Making
1Fudan University · 2Shanghai Innovation Institute · 3Shanghai Artificial Intelligence Laboratory +2
TxBench-PP: Analyzing AI Agent Performance on Small-Molecule Preclinical Pharmacology
Jun 17, 2026Hannah Le, Ramesh Ramasamy, Alex Urrutia +3AI Agent Evaluation AI Agent Benchmarks
1LatchBio, San Francisco, CA, USA
WorldLines: Benchmarking and Modeling Long-Horizon Stateful Embodied Agents
Jun 17, 2026Yehang Zhang, Jianchong Su, Haojian Huang +7Embodied QA Long-Horizon Agent Evaluation
HKUST(GZ) · Knowin · HKUST
Are LLMs Ready to Assist Physicians? PhysAssistBench for Interactive Doctor-Patient-EHR Assistance
Jun 17, 2026Tianming Du, Peijie Yu, Sihan Shang +12AI Agent Benchmarks Clinical Language Model Evaluation
1ELLIS Institute Finland · 2Aalto University · 3Tencent +7
CEO-Bench: Can Agents Play the Long Game?
Jun 16, 2026Haozhe Chen, Karthik Narasimhan, Zhuang LiuLong-Horizon Agent Evaluation AI Agent Benchmarks
Princeton University
ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues
Jun 16, 2026Shanda Li, Qiuhong Anna Wei, Jingwu Tang +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science, Carnegie Mellon University · Datadog
Compositional Skill Routing for LLM Agents: Decompose, Retrieve, and Compose
Jun 16, 2026Xueping GaoTool-Augmented Language Model Agents LLM Planning
Alibaba Cloud Hangzhou, China
GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?
Jun 16, 2026Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22AI Agent Benchmarks Agentic Code Generation
1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Jun 16, 2026Maria I. Gorinova, Macey Baker, Amy Heineike +3AI Coding Agents Benchmark Design
Tessl London, UK
EComAgentBench: Benchmarking Shopping Agents on Long-Horizon Tasks with Distributed Hidden Intent
Jun 16, 2026Zeyao Du, Tong Li, Haibo ZhangTool-Use Evaluation Long-Horizon Agent Evaluation
Shopee
ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
Jun 16, 2026Hong Yang, Basura FernandoVisual Spatial Reasoning Embodied QA
Centre for Frontier AI Research Agency for Science, Technology and Research Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
Jun 16, 2026Congjie Zheng, Chuanyi Xue, Bin Liang +2LLM Agent Evaluation AI Agent Benchmarks
Department of Automation, Tsinghua University, Beijing, 100084, China · Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, China
Offline Preference-Based Trajectory Evaluation
Jun 16, 2026Fernando DiazAI Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University Pittsburgh, PA
Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation
Jun 16, 2026Yuyang Dai, Xueqing Peng, Lingfei Qian +1LLM Agent Evaluation AI Agent Benchmarks
1MBZUAI · 2Yale University
Dissecting model behavior through agent trajectories
Jun 16, 2026Gaurav Gupta, Vatshank Chaturvedi, Jun Huan +1Software Engineering Agents LLM Agent Evaluation
AWS AI Labs
Beyond the GUI Paradigm: Do Mobile Agents Need the Phone Screen?
Jun 16, 2026Li Gu, Zihuan Jiang, Linqiang Guo +6Mobile GUI Automation AI Coding Agents
1Mila – Québec AI Institute · 2Concordia University · University of Toronto +1
MemTrace: Probing What Final Accuracy Misses in Long-Term Memory
Jun 15, 2026Xianxuan Long, Zhikai Chen, Shenglai Zeng +3Temporal Reasoning in Language Models LLM Agent Memory
Michigan State University · Case Western Reserve University

Jun 21, 2026, cs.AIJ/K move · Enter open · S save

PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems

Jiayu Liu, Qihan Lin, Cheng Qian, Rui Wang+7

University of Illinois Urbana-Champaign

Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Agent Planning LLM Agent Reliability

Open paper AskPDF arXiv

LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.

Momentum

Related topics

Latest papers 1,013