AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
May 13, 2026Yuxiang Lai, Peng Xia, Haonian Ji +8Computer-Use Agent Benchmarks LLM Agent Evaluation
University of North Carolina at Chapel Hill · Stanford University · University of Southern California +2
Reinforcement Learning for Tool-Calling Agents in Fast Healthcare Interoperability Resources (FHIR)
May 13, 2026Marius S. Knorr, Robert Müller, Jan P. Bremer +1Healthcare RL for Language Model Reasoning
IDM gGmbH, University Medical Center Hamburg-Eppendorf, Hamburg, Germany · Aganthos
PolitNuggets: Benchmarking Agentic Discovery of Long-Tail Political Facts
May 13, 2026Yifei ZhuAI Agent Benchmarks Multilingual IR
The University of Hong Kong
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
May 13, 2026Lucas Jing, Xinqi Wang, Liao Zhang +1Automated Software Testing Software Engineering Agents
Tsinghua University · University of Washington · Beneficial AI Foundation +1
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
May 13, 2026Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang +2AI Agents for Scientific Discovery AI Agent Evaluation
New High Energy Theory Center Department of Physics & Astronomy Rutgers University · Faculty of Computing & Data Sciences Boston University
Turning Intent into Specifications: A Benchmark and an Interactive User-Assistant Agent
May 13, 2026Hao Wang, Ligong Han, Kai Xu +1Requirements Engineering Software Engineering Agents
Red Hat AI Innovation · MIT-IBM Watson AI Lab · IBM Core AI
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
May 13, 2026Qingyun Zou, Feng Yu, Hongshi Tan +2Electronic Design Automation LLM Agent Evaluation
National University of Singapore
Evaluating Memory Condensation Strategies for Coding Agents in Data-Driven Scientific Discovery
May 13, 2026Renuka Chintalapati, Sid Raskar, Anurag Acharya +3AI Coding Agents AI Agent Benchmarks
Pacific Northwest National Laboratory · National Laboratory of the Rockies
TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
May 13, 2026Zabir Al Nazi, Shubhashis Roy DiptaLLM Agent Evaluation AI Agent Benchmarks
Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning
May 13, 2026Qinchuan Cheng, Zhantao Gong, Pengzhan Sun +3Belief-Space Planning Long-Horizon Agent Evaluation
Xi’an Jiaotong University · Nankai University · National University of Singapore +1
Do Biological Structural Guarantees Earn Their Complexity?
May 13, 2026Bogdan BanuAI Agent Reliability AI Agent Benchmarks
Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue
May 13, 2026Vardhan Dongre, Dilek Hakkani-TürMulti-Agent Coordination AI Agent Benchmarks
Siebel School of Computing & Data Science University of Illinois Urbana-Champaign
Neurodata Without Boredom: Benchmarking Agentic AI for Data Reuse
May 12, 2026Ling-Qi Zhang, Kristin BransonAI Coding Agents AI Agent Benchmarks
HHMI Janelia Research Campus · Ashburn, VA 20147
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
No More, No Less: Task Alignment in Terminal Agents
May 12, 2026Sina Mavali, David Pape, Jonathan Evertz +5Computer-Use Agent Benchmarks AI Alignment
CISPA Helmholtz Center for Information Security · ELLIS Institute Tübingen and MPI for Intelligent Systems · Tübingen AI Center
Rollout Cards: A Reproducibility Standard for Agent Research
May 12, 2026Charlie Masters, Ziyuan Liu, Stefano V. AlbrechtAI Agent Evaluation Agent Evaluation
Deepflow · Nanyang Technological University
When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
May 12, 2026Xiaolin Zhou, Aojie Yuan, Zheng Luo +12Domain Randomization AI Agent Benchmarks
Arizona State University · University of Southern California · University of Pennsylvania +2
AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents
May 12, 2026Jiarui Jin, Zexuan Yan, Shijian Wang +2Deep Research Agents Multi-Agent Collaboration
Xiaohongshu Inc.
Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
May 12, 2026Junjue Wang, Weihao Xuan, Heli Qi +7LLM Agent Evaluation Geospatial Reasoning
1The University of Tokyo · 2RIKEN AIP · 3Waseda University +1
PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments
May 12, 2026Yunn Kang Lim, Pengzhan Sun, Ziyi Bai +4Long-Horizon Agent Evaluation AI Agent Benchmarks
1A*STAR · 2National University of Singapore · 3BAAI
An Empirical Study of Automating Agent Evaluation
May 12, 2026Kang Zhou, Sangmin Woo, Haibo Ding +15Agent Evaluation AI Agent Benchmarks
AWS AI Labs
gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy
May 11, 2026Tousif Islam, Digvijay Wadekar, Zihan ZhouGravitational-Wave Astronomy AI Coding Agents
Kavli Institute for Theoretical Physics, University of California Santa Barbara, Kohn Hall, Lagoon Rd, Santa Barbara, CA 93106 · Center for Gravitational Physics, University of Texas at Austin, Austin, TX 78712, USA · Department of Physics, Princeton University, Princeton, NJ 08540, USA
ABRA: Agent Benchmark for Radiology Applications
May 11, 2026Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran +1Radiology Report Generation Healthcare
School of Computing Dublin City University Dublin, Ireland · AXXX · School of Medicine University College Dublin Dublin, Ireland
Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games?
May 11, 2026Maximilian Triebel, Marco Menner, Dominik HelfensteinVLM Evaluation Physical Reasoning
Institute of Artificial Intelligence, University of Stuttgart, Stuttgart, Germany.
Strategic Coercion Within Alliances: The Greenland Sovereignty Game as an AI Stress Test
May 11, 2026Rommin Adl, Peyton WilliamsMulti-Agent LLM Systems AI Agent Benchmarks
Grinnell College
RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
May 11, 2026Imad Aouali, Flavian Vasile, Otmane Sakhi +2Tool-Use Evaluation LLM Agent Evaluation
Criteo AI Lab
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14Computer-Use Agent Benchmarks Tool-Augmented Language Model Agents
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents
May 11, 2026Edward De Brouwer, Carl Edwards, Alexander Wu +9LLM Evaluation AI Agent Benchmarks
Genentech
MaD Physics: Evaluating information seeking under constraints in physical environments
May 11, 2026Moksh Jain, Mehdi Bennani, Johannes Bausch +4Scientific Reasoning AI Agent Benchmarks
Google DeepMind · Mila – Quebec AI Institute, Université de Montréal · Work done during an internship at Google DeepMind
ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox
May 11, 2026Yuanyang Li, Xue Yang, Longyue Wang +2LLM Agent Evaluation AI Agent Benchmarks
1Zhejiang University · 2Zhejiang Lab · 3Alibaba Group

May 13, 2026, cs.AIJ/K move · Enter open · S save

ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

Yuxiang Lai, Peng Xia, Haonian Ji, Kaiwen Xiong+7

University of North Carolina at Chapel Hill · Stanford University · University of Southern California+2

Computer-Use Agent Benchmarks LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks Agent Benchmarks

Open paper AskPDF arXiv

Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend and revise, while static prompt evaluation misses failures that only appear when agents operate over persistent state. Existing interactive benchmarks have advanced agent evaluation significantly, but most initialize tasks from clean state and do not systematically test how agents handle pre-existing partial, stale, or conflicting artifacts. We present \textbf{ClawForge}, a generator-backed benchmark framework for executable command-line workflows under state conflict. The framework compiles scenario templates, grounded slots, initialized state, reference trajectories, and validators into reproducible task specifications, and evaluates agents step by step over persistent workflow surfaces using normalized end state and observable side effects rather than exact trajectory matching. We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories). Results across seven frontier models show that the best model reaches only 45.3% strict accuracy, wrong-state replacement remains below 17% for all models, and the widest model separation (17% to 90%) is driven by whether agents inspect existing state before acting. Partial-credit and step-efficiency analyses further reveal that many failures are near-miss closures rather than early breakdowns, and that models exhibit qualitatively different failure styles under state conflict.

Momentum

Related topics

Latest papers 1,013