AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,505 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

AcademiClaw: When Students Set Challenges for AI Agents
May 4, 2026Junjie Yu, Pengrui Lu, Weiye Si +75AI Agent Evaluation AI Agent Benchmarks
SOTOPIA-TOM: Evaluating Privacy and Information Management in Multi-Agent Interaction with Theory of Mind
May 4, 2026Yashwanth YS, Ruichen Wang, Shihua Zeng +4Multi-Agent Coordination Theory of Mind
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, USA · Fujitsu Research of America, Pittsburgh, USA
Towards Understanding Specification Gaming in Reasoning Models
May 4, 2026Kei Nishimura-Gasparian, Robert McCarthy, David LindnerReward Hacking RL for Language Model Reasoning
1MATS · University College London, London, UK · 3Google Deep- Mind.
PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments
May 4, 2026Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler +10AI Agent Benchmarks Electronic Health Records
Stanford University
NeuroState-Bench: A Human-Calibrated Benchmark for Commitment Integrity in LLM Agent Profiles
May 3, 2026Xiao JiaAI Agent Reliability LLM Agent Evaluation
School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong, Shenzhen
The Compliance Gap: Why AI Systems Promise to Follow Process Instructions but Don't
May 3, 2026Kwan Soo ShinAI Agent Benchmarks LLM Honesty
PolymathMinds AI Lab
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation
May 2, 2026Dong Xu, Jialun Cao, Guozhao Mo +9LLM Agent Evaluation Formal Verification
Institute of Software, Chinese Academy of Sciences · The Hong Kong University of Science and Technology · Guangzhou Institute of Technology, Xidian University
ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescue
May 2, 2026Daoxuan Zhang, Ping Chen, Jianyi Zhou +1Aerial Robotics Robotics Simulation
Harbin Institute of Technology, Shenzhen
Can Coding Agents Reproduce Findings in Computational Materials Science?
May 1, 2026Ziyang Huang, Yi Cao, Ali K. Shargh +15Coding Agents AI Agent Benchmarks
Johns Hopkins University
AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
May 1, 2026Ranit Karmakar, Jayita ChatterjeeComputer-Use Agent Benchmarks LLM Agent Evaluation
Harvard University, Boston, MA 02115 · Boston, MA 02115
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent Evaluation AI Agent Benchmarks
1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery
Apr 30, 2026Hanane Nour Moussa, Yifei Li, Zhuoyang Li +7AI Agents for Scientific Discovery AI Agent Benchmarks
1The Ohio State University · 2Cisco Research
KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
Apr 30, 2026Thomas Grady, Kip Parker, Iliyan Zarov +3Sequential Decision Making Long-Horizon Agent Evaluation
General Reasoning, Inc.
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Apr 30, 2026Jinchao Li, Yunxin Li, Chenrui Zhao +3Computer-Use Agent Benchmarks Computer-Use Agents
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
RoadMapper: A Multi-Agent System for Roadmap Generation of Solving Complex Research Problems
Apr 30, 2026Jiacheng Liu, Zichen Tang, Zhongjun Yang +8Multi-Agent LLM Systems LLM Planning
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology (Guangzhou) · IDEA Research +1
ClawGym: A Scalable Framework for Building Effective Claw Agents
Apr 29, 2026Fei Bai, Huatong Song, Shuang Sun +11Synthetic Data Generation AI Agent Benchmarks
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
Apr 29, 2026Yerong Wu, Tianxing Wu, Minghao Zhu +2Conversational Memory AI Agent Benchmarks
School of Computer Science and Engineering, Southeast University, China · 2Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China · 3Independent Researcher +1
LATTICE: Evaluating Decision Support Utility of Crypto Agents
Apr 29, 2026Aaron Chan, Tengfei Li, Tianyi Xiao +3AI-Assisted Decision Making AI Agent Benchmarks
Sahara AI · University of Southern California
DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios
Apr 28, 2026Jinxiang Meng, Shaoping Huang, Fangyu Lei +17Data Visualization AI Agent Benchmarks
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*
ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable
Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-Judge AI Agent Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA
Cooperate to Compete: Strategic Coordination in Multi-Agent Conquest
Apr 28, 2026Abigail O'Neill, Alan Zhu, Mihran Miroyan +2Multi-Agent Coordination Multi-Agent Negotiation
University of California, Berkeley
Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
Apr 27, 2026Joshua Sherwood, Ben Aybar, Benjamin KaplanAI Coding Agents AI Agent Evaluation
Department of Computer Science University of Chicago Chicago, IL 60637 · Independent Researcher
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark Auditing AI Agent Benchmarks
Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1
GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
Apr 27, 2026Yunsu Kim, Kaden Uhlig, Joern WuebkerAI Agent Evaluation AI Agent Benchmarks
LILT
Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft
Apr 27, 2026Zhou Ziheng, Huacong Tang, Jinyuan Zhang +9AI Agents for Scientific Discovery Circuit Discovery
University of California, Los Angeles · Peking University · Amazon
NeuroClaw Technical Report
Apr 27, 2026Cheng Wang, Zhibin He, Zhihao Peng +7Multi-Agent LLM Systems AI Agent Benchmarks
Electronic Engineering Department, The Chinese University of Hong Kong, China · School of Electronic Information, Northwest University, China · Department of Computer Science, Emory University, Atlanta, GA, USA +3
Skill Retrieval Augmentation for Agentic AI
Apr 27, 2026Weihang Su, Jianming Long, Qingyao Ai +6AI Agent Benchmarks LLM Agent Skill Retrieval
Department of Computer Science and Technology, Tsinghua University · ByteDance Inc.

May 4, 2026J/K move · Enter open · S save

AcademiClaw: When Students Set Challenges for AI Agents

Junjie Yu, Pengrui Lu, Weiye Si, Hongliang Lu+74

AI Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw largely unexamined. We introduce AcademiClaw, a bilingual benchmark of 80 complex, long-horizon tasks sourced directly from university students' real academic workflows -- homework, research projects, competitions, and personal projects -- that they found current AI agents unable to solve effectively. Curated from 230 student-submitted candidates through rigorous expert review, the final task set spans 25+ professional domains, ranging from olympiad-level mathematics and linguistics problems to GPU-intensive reinforcement learning and full-stack system debugging, with 16 tasks requiring CUDA GPU execution. Each task executes in an isolated Docker sandbox and is scored on task completion by multi-dimensional rubrics combining six complementary techniques, with an independent five-category safety audit providing additional behavioral analysis. Experiments on six frontier models show that even the best achieves only a 55% pass rate. Further analysis uncovers sharp capability boundaries across task domains, divergent behavioral strategies among models, and a disconnect between token consumption and output quality, providing fine-grained diagnostic signals beyond what aggregate metrics reveal. We hope that AcademiClaw and its open-sourced data and code can serve as a useful resource for the OpenClaw community, driving progress toward agents that are more capable and versatile across the full breadth of real-world academic demands. All data and code are available at https://github.com/GAIR-NLP/AcademiClaw.