Agent Benchmarks

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 50% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 87

All topics

Finding Blind Spots in AppWorld and WorkArena Task Verifiers
Oct 6, 2026Richard AbrichBenchmark Auditing Agent Benchmarks
OpenAdapt.AI (MLDSAI Inc.)
MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks
Oct 5, 2026Ziqing Wang, Lili Zhao, Kaize DingLLM Agent Evaluation AI Agent Benchmarks
Northwestern University
MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
Oct 4, 2026Yuxin Liu, Yuxuan Wang, Zhenxin Lei +9Multimodal Model Evaluation Agent Benchmarks
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · University of the Chinese Academy of Sciences +2
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM Evaluation Graph Neural Networks
University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt +23
DAYJOB: A Benchmark for Long-Horizon Professional Work
Oct 1, 2026Stephanie Finley, Liudas Panavas, Thomas Mikkelson +12LLM Agent Evaluation Long-Horizon Agent Tasks
Surge AI
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses
Sep 30, 2026Zhixuan Tan, Pengjie Gu, Zhao Li +4Software Engineering Benchmarks Long-Horizon Agent Evaluation
The Chinese University of Hong Kong, Shenzhen · MemoraX AI
AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search
Sep 28, 2026Tongtong Feng, Xin Wang, Haoran Hou +9Aerial Robotics Object-Goal Navigation
Department of Computer Science and Technology, BNRist, Tsinghua University · School of Electrical & Electronic Engineering, University College Dublin · School of Electronics Engineering and Computer Science, Peking University +1
GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
Sep 21, 2026Yiran Wang, Xingyilang Yin, Junfu Pu +12Game-Playing Agents Long-Horizon Agent Evaluation
ARC Lab, Tencent · GVC Lab, Great Bay University · University of Macau +3
Quantifying Overclaiming Propensity in Frontier LLM Agents
Sep 17, 2026Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo +6AI Agent Reliability AI Coding Agents
Tara Research · Mila – Quebec AI Institute · Cohere
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Sep 14, 2026Umesh Bodhwani, Thanh Tran, Kai WeiLLM-as-a-Judge LLM Agent Evaluation
Amazon
Benchmarking Hybrid Deep Research Across Database Querying and Web Search
Sep 8, 2026Ruofan Wu, Peiran Xu, Xiaolong Li +9Deep Research Agents Agentic Search
University of Houston · University of California, Los Angeles · The University of Hong Kong +2
InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
Sep 1, 2026Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan +3VLM Evaluation Claim Verification
City, University of London · Imperial College London · The Alan Turing Institute
GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space
Aug 31, 2026Boqi Chen, Xudong Liu, Yunke Ao +2Healthcare Constrained RL
ETH Zurich · MBZUAI · University of Toronto
You Know What I Mean: A Benchmark for Agentic Conversational Reference Grounding
Aug 30, 2026Karen Fuchs, Uri Katz, Yoav GoldbergSoftware Engineering Agents Agentic Retrieval
Bar-Ilan University · Allen Institute for AI
BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
Aug 26, 2026Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman +5AI for Science AI Agent Benchmarks
Edison Scientific, Inc., San Francisco, CA, USA
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Aug 20, 2026Zhuochun Li, Youngmin Ko, Ali Keramati +11Business Process Automation AI Agent Benchmarks
University of Pittsburgh · Northwestern University · University of California, Irvine +1
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Aug 11, 2026Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3Computer-Use Agent Benchmarks AI Agent Evaluation
York University · Nanyang Technological University · Salesforce AI Research
CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
Aug 7, 2026Veronica Chatrath, Bryan Zhu, George Pu +16Evidence-Grounded Reasoning Clinical Reasoning
Scale AI · Emory University · University of California, Santa Cruz +4
CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
Aug 6, 2026Fanzhe Meng, Guoxin Chen, Jiale Zhao +6LLM Agent Training Agent Benchmarks
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Independent Researcher · 3AweAI Team
Agentic self-driving microscopy benchmarks support qualification but do not necessarily generalize to unseen tasks
Aug 5, 2026Nathan S Johnson, Ian AbshireAI Agents for Scientific Discovery LLM Agent Orchestration
Carl Zeiss Research Microscopy Solutions, 5300 Central Boulevard, Dublin, CA 94568
OrchestraBench: Evaluating Multi-Agent Orchestration Failure Modes, Recovery, and Decomposition Quality
Aug 5, 2026Yidian Chen, Yingzi Gu, Natan Vidra +2Agent Failure Analysis AI Agent Reliability
Anote
DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces
Aug 4, 2026Boyan Li, Zhuowen Liang, Yupeng Xie +11LLM Agent Evaluation Multimodal Agents
HKUST(GZ) · Tsinghua University
SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
Aug 3, 2026Zelin Tan, Yiqun Zhang, Hao Li +11LLM Agent Skill Learning LLM Agent Trajectory Synthesis
Shanghai Artificial Intelligence Laboratory
PATH-Bench: Path-Dependent Evaluation of Lifelong Agents
Aug 2, 2026Xidong Yang, Xingyi Zhang, Wenhao Li +7Continual Learning for LLM Agents Lifelong Learning Agents
East China Normal University · Tongji University · Ant Group +5
MetaRoute-Bench: Evaluating Meta-Decision Policies for Agentic Workflow Routing
Jul 31, 2026Natan Vidra, Alina Kapanova, Arun Kanhai +1Agentic Workflows Agent Evaluation
Anote AI · New York, United States · Anote AI, Cornell University +2
Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments
Jul 30, 2026Haomin Qi, Xingliang Wang, Xuanqi Gao +9Code Generation Benchmarks Agent Benchmarks
1Microsoft · 3Zhejiang University · 4Xi’an Jiaotong University +1
DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
Jul 30, 2026Debin Meng, Jiaming Yang, Zefang Zong +4LLM Agent Evaluation Automated Evaluation
Tencent Shenzhen, China · Xidian University Xi’an, China · Tencent Beijing, China
Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions?
Jul 28, 2026Abhishek Pillai, Samir Kumar Nayak, Yuan ChenComputer-Use Agent Benchmarks Computer-Use Agents
NVIDIA Austin, TX, USA · NVIDIA Bangalore, Karnataka, IND
WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting
Jul 20, 2026Zhaokai Wang, Tianlin Gui, Jiayuan Rao +3LLM Evaluation Forecasting Benchmarks
Shanghai Jiao Tong University · Shanghai, China · Nanjing University +4
RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contexts
Jul 18, 2026Mihir Shriniwas AryaCompositional Reasoning LLM Agent Memory
Department of Computer Science and Engineering RV College of Engineering

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

Finding Blind Spots in AppWorld and WorkArena Task Verifiers

Richard Abrich

OpenAdapt.AI (MLDSAI Inc.)

Benchmark Auditing Agent Benchmarks

Open paper AskPDF arXiv

Execution-based task verifiers decide whether an agent succeeded. We audit shipped AppWorld and WorkArena verifiers with source-informed mutation tests. The main audit never modifies a shipped checker. In AppWorld, duplicating a non-idempotent write creates an extra record while preserving every checked field value. The verifier accepts all three task variants from two of five eligible generators: 6/15 constructed effects. A cardinality patch applied to checker copies after the census makes all six cells fail while preserving valid controls. In WorkArena, we prospectively rerun 23 extra-field candidates selected for earlier checker-PASS outcomes. Independent Table API readback confirms nondefault persisted values in 21, while all 23 receive PASS. Two requested strings are aliases of stored defaults. The 21 confirmed wrong effects span three form templates. These selected cases confirm wrong effects under the audit's protocol; they do not estimate a population rate. No other construction produces an independently confirmed false accept. Other checker-PASS cases are effect-correct degeneracies. We report zero-PASS families separately because retained evidence differs. In fixed intent-swap grids, the checkers return no PASS on 2,689 off-diagonal executions. This is a rejection census: 57 WorkArena cells use session-scoped evidence; the other 2,632 lack classified rejection causes and independent target ground truth. Each increment is specified before its own cells are scored. A supplement accompanies the OpenReview submission with the construction grammar, evidence, content-bound stage lineage and count reproducer.

Momentum

Related topics

Latest papers 87