AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

Can Coding Agents Reproduce Findings in Computational Materials Science?
May 1, 2026Ziyang Huang, Yi Cao, Ali K. Shargh +15Coding Agents AI Agent Benchmarks
Johns Hopkins University
AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?
May 1, 2026Ranit Karmakar, Jayita ChatterjeeComputer-Use Agent Benchmarks LLM Agent Evaluation
Harvard University, Boston, MA 02115 · Boston, MA 02115
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent Evaluation AI Agent Benchmarks
1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery
Apr 30, 2026Hanane Nour Moussa, Yifei Li, Zhuoyang Li +7AI Agents for Scientific Discovery AI Agent Benchmarks
1The Ohio State University · 2Cisco Research
KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
Apr 30, 2026Thomas Grady, Kip Parker, Iliyan Zarov +3Sequential Decision Making Long-Horizon Agent Evaluation
General Reasoning, Inc.
WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments
Apr 30, 2026Jinchao Li, Yunxin Li, Chenrui Zhao +3Computer-Use Agent Benchmarks Computer-Use Agents
Harbin Institute of Technology, Shenzhen · Shenzhen Loop Area Institute
RoadMapper: A Multi-Agent System for Roadmap Generation of Solving Complex Research Problems
Apr 30, 2026Jiacheng Liu, Zichen Tang, Zhongjun Yang +8Multi-Agent LLM Systems LLM Planning
Beijing University of Posts and Telecommunications · The Hong Kong University of Science and Technology (Guangzhou) · IDEA Research +1
ClawGym: A Scalable Framework for Building Effective Claw Agents
Apr 29, 2026Fei Bai, Huatong Song, Shuang Sun +11Synthetic Data Generation AI Agent Benchmarks
Gaoling School of Artificial Intelligence, Renmin University of China · IQuest Research · Beihang University
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
Apr 29, 2026Yerong Wu, Tianxing Wu, Minghao Zhu +2Conversational Memory AI Agent Benchmarks
School of Computer Science and Engineering, Southeast University, China · 2Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China · 3Independent Researcher +1
LATTICE: Evaluating Decision Support Utility of Crypto Agents
Apr 29, 2026Aaron Chan, Tengfei Li, Tianyi Xiao +3AI-Assisted Decision Making AI Agent Benchmarks
Sahara AI · University of Southern California
DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios
Apr 28, 2026Jinxiang Meng, Shaoping Huang, Fangyu Lei +17Data Visualization AI Agent Benchmarks
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · National University of Singapore +1
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*
ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable
Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-Judge AI Agent Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA
Cooperate to Compete: Strategic Coordination in Multi-Agent Conquest
Apr 28, 2026Abigail O'Neill, Alan Zhu, Mihran Miroyan +2Multi-Agent Coordination Multi-Agent Negotiation
University of California, Berkeley
Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver
Apr 27, 2026Joshua Sherwood, Ben Aybar, Benjamin KaplanAI Coding Agents AI Agent Evaluation
Department of Computer Science University of Chicago Chicago, IL 60637 · Independent Researcher
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
Apr 27, 2026Xinming Tu, Tianze Wang, Yingzhou +4Benchmark Auditing AI Agent Benchmarks
Minta · Allen School, University of Washington, Seattle, WA, USA · Phylo, Inc., South San Francisco, CA, USA +1
GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
Apr 27, 2026Yunsu Kim, Kaden Uhlig, Joern WuebkerAI Agent Evaluation AI Agent Benchmarks
LILT
Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft
Apr 27, 2026Zhou Ziheng, Huacong Tang, Jinyuan Zhang +9AI Agents for Scientific Discovery Circuit Discovery
University of California, Los Angeles · Peking University · Amazon
NeuroClaw Technical Report
Apr 27, 2026Cheng Wang, Zhibin He, Zhihao Peng +7Multi-Agent LLM Systems AI Agent Benchmarks
Electronic Engineering Department, The Chinese University of Hong Kong, China · School of Electronic Information, Northwest University, China · Department of Computer Science, Emory University, Atlanta, GA, USA +3
Skill Retrieval Augmentation for Agentic AI
Apr 27, 2026Weihang Su, Jianming Long, Qingyao Ai +6AI Agent Benchmarks LLM Agent Skill Retrieval
Department of Computer Science and Technology, Tsinghua University · ByteDance Inc.
AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark
Apr 27, 2026Hongxin Li, Xiping Wang, Jingran Su +4VLM Evaluation GUI Agents
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents
Apr 27, 2026Zheng Wu, Yi Hua, Zhaoyuan Huang +8Computer-Use Agents AI Agent Evaluation
School of Computer Science, Shanghai Jiao Tong University
AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment
Apr 27, 2026Yuxuan Gao, Megan Wang, Yi Ling YuAI Agent Evaluation AI Agent Benchmarks
University of Pennsylvania · 2Columbia University · 3OpenMesh AI
MarketBench: Evaluating AI Agents as Market Participants
Apr 26, 2026Andrey Fradkin, Rohit KrishnanMulti-Agent Coordination LLM Agent Evaluation
Boston University and the MIT Initiative on the Digital Economy. · Independent Researcher.
ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation
SoccerRef-Agents: Multi-Agent System for Automated Soccer Refereeing
Apr 25, 2026Zi Meng, Wanli Song, Yi Hu +2Multi-Agent LLM Systems Sports Video Analysis
University of Michigan, Michigan, USA · Shanghai Jiao Tong University, Shanghai, China
Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
Apr 24, 2026Mengzhuo Chen, Junjie Wang, Fangwen Mu +4Agent Failure Analysis Multi-Agent LLM Systems
State Key Laboratory of Complex System Modeling and Simulation Technology · Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents
Apr 24, 2026Xirui Li, Ming Li, Yunze Xiao +4Multi-Agent Coordination AI Agent Evaluation
University of Maryland · 2Mohamed bin Zayed University of Artificial Intelligence · 3Carnegie Mellon University
AgentSearchBench: A Benchmark for AI Agent Search in the Wild
Apr 24, 2026Bin Wu, Arastun Mammadli, Xiaoyu Zhang +1Agentic Search AI Agent Evaluation
Centre for Artificial Intelligence, University College London

May 1, 2026, cs.CLJ/K move · Enter open · S save

Can Coding Agents Reproduce Findings in Computational Materials Science?

Ziyang Huang, Yi Cao, Ali K. Shargh, Jing Luo+14

Johns Hopkins University

Coding Agents AI Agent Benchmarks Scientific Reproducibility Computational Materials Science

Open paper AskPDF arXiv

Large language models are increasingly deployed as autonomous coding agents and have achieved remarkably strong performance on software engineering benchmarks. However, it is unclear whether such success transfers to computational scientific workflows, where tasks require not only strong coding ability, but also the ability to navigate complex, domain-specific procedures and to interpret results in the context of scientific claims. To address this question, we present AutoMat, a benchmark for evaluating LLM-based agents' ability to reproduce claims from computational materials science. AutoMat poses three interrelated challenges: recovering underspecified computational procedures, navigating specialized toolchains, and determining whether the resulting evidence supports a claim. By working closely with subject matter experts, we curate a set of claims from real materials science papers to test whether coding agents can recover and execute the end-to-end workflow needed to support (or undermine) such claims. We then evaluate multiple representative coding agent settings across several foundation models. Our results show that current LLM-based agents obtain low overall success rates on AutoMat, with the best-performing setting achieving a success rate of only 53%. Error analysis further reveals that agents perform worst when workflows must be reconstructed from paper text alone and that they fail primarily due to incomplete procedures, methodological deviations, and execution fragility. Taken together, these findings position AutoMat as both a benchmark for computational scientific reproducibility and a tool for diagnosing the current limitations of agentic systems in AI-for-science settings.

Momentum

Related topics

Latest papers 1,013