AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
May 21, 2026Asaf Yehudai, Lilach Eden, Michal Shmueli-ScheuerLLM Agent Evaluation AI Agent Benchmarks
IIBM Research
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
May 21, 2026Zhaoyang Chu, Jiarui Hu, Xingyu Jiang +8Computer-Use Agent Benchmarks Terminal Agents
University College London · Nanjing University · Tencent
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
May 21, 2026Kai Tzu-iunn Ong, Minseok Kang, Dongwook Choi +11AI Agent Evaluation Agent Harness Optimization
Yonsei University · Microsoft Research Asia · Texas A&M University
AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture
May 21, 2026Zi Ye, Yibin Wen, Xiaoya Fan +10AI Agent Benchmarks Tool-Using Vision-Language Agents
1Sun Yat-Sen University · 2Southwest University · 3Northeastern University +4
Cross-domain benchmarks reveal when coordinated AI agents improve scientific inference from partial evidence
May 21, 2026Fiona Y. Wong, Markus J. BuehlerAI for Science Multi-Agent Coordination
Laboratory for Atomistic and Molecular Mechanics (LAMM) · Department of Biological Engineering · Department of Mechanical Engineering +2
Evaluating Large Language Models as Live Strategic Agents: Provider Performance, Hybrid Decomposition, and Operational Gaps in Timed Risk Play
May 21, 2026H. C. EkneLLM Agent Evaluation AI Agent Benchmarks
SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
May 21, 2026Ningyuan Li, Haiyang Shen, Mugeng Liu +4Web Search Agents AI Agent Benchmarks
2Beijing University of Technology · 1Peking University
Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems
May 20, 2026Deepak Panigrahy, Aakash TyagiLLM Agent Orchestration Agentic Workflows
Independent Researcher, USA · Texas A&M University, Department of Computer Science and Engineering, USA
SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?
May 20, 2026Kevin Han, Renfei Zhang, Kathy Wei +3LLM Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University · Stealth · Pennsylvania State University
What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema
May 20, 2026Mahdi Naser Moghadasi, Faezeh GhaderiBenchmark Auditing LLM Auditing
BrightMind AI, Texas Tech University · Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX +2
Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
May 20, 2026Amit Roth, Ankur Samanta, Matan Halevy +2Reward Hacking AI Agent Benchmarks
1Tel Aviv University · 2Columbia University · 3Taso Labs
Evaluating Temporal Semantic Caching and Workflow Optimization in Agentic Plan-Execute Pipelines
May 20, 2026Alimurtaza Mustafa Merchant, Krish Veera, Sajal Kumar Goyla +3LLM Agent Orchestration Agentic Workflows
Columbia University · IBM · IBM Research
The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents
May 19, 2026Doguhan Yeke, Elif Su Temirel, Ananth Shreekumar +3VLM Robustness Large Language Model-Based Robot Planning
Purdue University · Bilkent University · Work performed while an intern at Purdue University.
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
May 19, 2026Parsa Mazaheri, Kasra MazaheriAgent Failure Analysis Computer-Use Agent Benchmarks
University of California, Santa Cruz · Massachusetts Institute of Technology
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
May 19, 2026Priyansh Trivedi, Olivier SchmittCoding Agents AI Agent Evaluation
SonarSource
WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents
May 19, 2026Bingnan Liu, Chenhang Cui, Rui Huang +7VLM Evaluation AI Agent Benchmarks
University of Electronic Science and Technology of China · National University of Singapore · De Artificial Intelligence Lab +2
Synthesis and Evaluation of Long-term History-aware Medical Dialogue
May 19, 2026Hebin Hu, Renke Dai, Ah-Hwee Tan +1Healthcare AI Agent Benchmarks
South-Central Minzu University Wuhan, China · Singapore Management University Singapore, Singapore
CutVerse: A Compositional GUI Agents Benchmark for Media Post-Production Editing
May 19, 2026Haobo Hu, Xiangwu Guo, Zhiheng Chen +4GUI Agents Video Editing
MIPG, Communication University of China · National University of Singapore · USEIT AI
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
May 19, 2026Yingjie LeiAutomated Negotiation LLM Agent Evaluation
University of Aberdeen
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
May 18, 2026Leyao Wang, Yanan He, Peng Chen +5AI Agent Reliability LLM-as-a-Judge
Yale University · IBM Research
How Far Are We From True Auto-Research?
May 18, 2026Zhengxin Zhang, Ning Wang, Sainyam Galhotra +1LLM Agent Evaluation AI Agent Benchmarks
Cornell University
DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows
May 18, 2026Yuxuan Gao, Megan Wang, Yi Ling Yu +2LLM Agent Orchestration Long-Horizon Agent Evaluation
1OpenMesh AI · University of Pennsylvania · 3Columbia University +2
ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
May 18, 2026Yining Hong, Jiageng Liu, Han Yin +5Active Perception Spatial Reasoning Benchmarks
1Stanford University · 2UCLA · 3Northwestern University
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
May 18, 2026Yifan Zhou, Zhentao Zhang, Ziming Cheng +8LLM Agent Evaluation LLM Agent Skill Learning
1SJTU · 2XJTU · 3NUS +6
MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems
May 18, 2026Hyunji Lee, Justin Chih-Yao Chen, Joykirat Singh +3Agent Memory Long-Horizon Agent Evaluation
UNC Chapel Hill · The University of Texas at Austin
STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics
May 18, 2026Tingfeng Hui, Hao Xu, Pengyu Zhu +5Spatiotemporal Reasoning LLM Planning
Hong Kong Institute of AI for Science, City University of Hong Kong · Beijing University of Posts and Telecommunications · Li Auto Inc. +2
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
Causely: A Causal Intelligence Layer for Enterprise AI A Benchmark Study on SRE and Reliability Workflows
May 18, 2026Dhairya Dalal, Endre Sara, Ben Yemini +2Software Engineering AI Agent Benchmarks
Causely
DocOS: Towards Proactive Document-Guided Actions in GUI Agents
May 18, 2026Jingjing Liu, Ziye Huang, Zihao Cheng +6GUI Agents Web Search Agents
School of Computer Science and Engineering, Beihang University, Beijing, China · School of Computer Science, Peking University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing +2
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
May 18, 2026Jiawei He, Jie Jia, Chenbo Liu +4AI Coding Agents LLM Agent Evaluation
Amap, Alibaba Group

May 21, 2026, cs.CLJ/K move · Enter open · S save

Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents

Asaf Yehudai, Lilach Eden, Michal Shmueli-Scheuer

IIBM Research

LLM Agent Evaluation AI Agent Benchmarks Automated Evaluation

Open paper AskPDF arXiv

Agentic systems are becoming more capable: agents define strategies, take actions, and interact with different environments. This autonomy poses serious challenges for overseeing and assessing agent behavior. Most current tools are limited, focusing on observability with basic evaluation capabilities or imposing static, hand-crafted error taxonomies that cannot adapt to new domains. To address this gap, we present Agentic CLEAR, an automatic, dynamic, and easy-to-use evaluation framework. It produces textual insights into the agent behavior on three levels of granularity: system, trace, and node. Agentic CLEAR operates above the observability layer, enabling seamless integration and featuring an intuitive UI that makes agent evaluation highly accessible. In our experiments on four benchmarks, seven agentic settings, and tens of thousands of LLM calls, we show that Agentic CLEAR produces high-quality, data-driven, insightful feedback. Our analysis shows strong alignment with human-annotated errors and the ability to predict task success rate.

Momentum

Related topics

Latest papers 1,013