AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

Interface-Induced Trajectory Censoring
Sep 3, 2026Wenbo WangTool-Use Evaluation AI Agent Evaluation
City University of Hong Kong
KC-Bench: A Dynamic Interactive Benchmark for Evaluating Knowledge Conflicts in LLM Agents
Sep 3, 2026Yaxing Lyu, Shengjie Zhou, Binbin Toh +2Knowledge Conflicts in Language Models LLM Agent Evaluation
1Shanghai Artificial Intelligence Laboratory · 2The University of Hong Kong · 3Xiamen University Malaysia +1
GPS-Bench: A Governance Policy Benchmark for Automating Policy Analysis
Sep 3, 2026Linh Le, Melanie Bui, My Chiffon Nguyen +2Human Behavior Simulation AI Agent Benchmarks
1Lida Safety · 2ERA
DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
Sep 3, 2026Puneet Mathur, Dinesh ManochaAI Agent Benchmarks Instruction Following
University of Maryland College Park, USA
CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
Sep 1, 2026Damien Sileo, Dimitri KachlerTemporal Reasoning in Language Models AI Agent Benchmarks
Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Sep 1, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +16LLM Agent Evaluation Agent Harness Optimization
ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning
Sep 1, 2026Fanrui Zhang, Ruixue Ding, Qiang Zhang +13AI Agent Benchmarks Rubric-Based RL
Alibaba ATH Token Foundry · Hema, Alibaba Group
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
Sep 1, 2026Leonardo Ranaldi, Sherrie Shen, Jushi Kai +1Cross-Cultural Language Model Evaluation LLM Agent Evaluation
(•) ILCC, School of Informatics, University of Edinburgh · (•,◦) ILCC, School of Informatics, University of Edinburgh · (◦) School of Artificial Intelligence, Shanghai Jiao Tong University
RestoreBench: Can AI Agents Restore Power Flow Convergence?
Aug 31, 2026Riccardo Mansutti, Andrea Pomarico, Robert Jakob +3Multi-Agent LLM Systems LLM Agent Evaluation
1ETH Zürich · 2Politecnico di Milano · 3Harvard University
Toward Workflow-Aware Benchmarking for Healthcare NLP Agents
Aug 31, 2026Junyi Yao, Baichuan Li, Zihao Zheng +1Benchmark Design LLM Agent Evaluation
Washington University in St. Louis, USA · Southern Methodist University, USA
Aspire: Can Models Self-Evolve from Vague Goals?
Aug 31, 2026Yuhao Wu, Jingyuan Zhang, Jiajun Shi +18LLM Agent Self-Improvement AI Agent Benchmarks
S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?
Aug 31, 2026Jiajun Shi, Siyuan Tao, Yuhao Wu +18LLM Agent Evaluation Language Model Self-Assessment
ByteDance Seed
MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
Aug 31, 2026Vernon Toh, Navonil Majumder, Zhengyuan Liu +2Active Perception AI Agent Evaluation
DeCLaRe Lab, Nanyang Technological University, Singapore · Agency for Science, Technology, and Research (A*STAR), Singapore
E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation
Aug 31, 2026Wei Fan, Xinjie Shen, Xudong Guo +8Long-Horizon Agent Evaluation AI Agent Benchmarks
1Qwen Team, Alibaba Group · Department of Computer Science and Engineering, HKUST · 2Taobao & Tmall Group, Alibaba Group
BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
Aug 31, 2026Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs +3Reward Hacking AI Agent Evaluation
National University of Singapore · MIT · Vrije Universiteit Amsterdam +3
SwarmBench: Can Large Language Models Act as Agent Swarm Orchestrators?
Aug 31, 2026Jinshan Gao, Zhuoran Jin, Tianyi Men +2LLM Agent Orchestration AI Agent Benchmarks
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents
Aug 31, 2026Hanlin Tian, Minhao Li, Yu Mi +5LLM Agent Evaluation AI Agent Benchmarks
1DataGrids · 2Shanghai University · 3Peking University +3
Science sandboxes measure the scientific capability of AI agents
Aug 31, 2026Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai +8Protein Fitness Prediction AI Agent Evaluation
The Broad Institute of MIT and Harvard; Cambridge, MA 02142, USA. · Department of Biomedical Informatics, Harvard Medical School; Boston, MA 02115, USA. · The Jackson Laboratory; Bar Harbor, ME 04609, USA. +10
Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment
Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific Discovery LLM-as-a-Judge
1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1
Cost-Effective Repository Exploration for Agentic Issue Localization
Aug 30, 2026Mohammad Nour Al Awad, Sergey IvanovAI Coding Agents Agentic Search
ITMO University Saint Petersburg, Russian Federation
EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants
Aug 29, 2026Yue Peng, Lanke Xia, Zihan Wang +3Web Application Generation AI Agent Benchmarks
New York University Shanghai
TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development
Aug 26, 2026Jiarui Yan, Weiwei Sun, Sijie Li +2Long-Horizon Agent Evaluation AI Agent Benchmarks
Carnegie Mellon University
BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
Aug 26, 2026Zane Koch, Asmamaw T. Wassie, Javier Valdes-Aleman +5AI for Science AI Agent Benchmarks
Edison Scientific, Inc., San Francisco, CA, USA
LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks
Aug 24, 2026Xiao Zhang, Qumeng Sun, Jiahao Li +3Agent Memory LLM Evaluation
Infinite Evolution Lab, EvoMap
TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
Aug 24, 2026Wenhao Wu, Menghao Zhang, Xin Wang +3Tool-Augmented Language Model Agents LLM Agent Orchestration
Xiaomi Inc. · Nanjing University · Tsinghua University
K-Bench: measuring model performance on real scientific agent requests
Aug 21, 2026Aubrey M. Brueckner, Darshil Patel, Yuhuan He +1AI for Science LLM Agent Evaluation
1K-Dense, Inc.
One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows
Aug 20, 2026Zhuochun Li, Youngmin Ko, Ali Keramati +11Business Process Automation AI Agent Benchmarks
University of Pittsburgh · Northwestern University · University of California, Irvine +1
QuoteBench: How Matched Scores Can Hide Command-Path Failures
Aug 13, 2026Shangao Li, Yao Zhang, Volker Tresp +1LLM Agent Evaluation Agent Evaluation
1Stony Brook University · 2LMU Munich · 3Munich Center for Machine Learning
Vero: Can AI Agents Build Formally Verified Software Repositories?
Aug 13, 2026Zhe Ye, Hantao Lou, Yuechun Sun +8Code Generation Formal Verification
1UC Berkeley · University of Chicago · 3California Institute of Technology +3
Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development
Aug 13, 2026Yiwei Li, Wanli Yang, Hexiang Tan +10Long-Horizon Agent Evaluation Agent Harness Optimization
1Meituan · University of Chinese Academy of Sciences

Sep 3, 2026, cs.AIJ/K move · Enter open · S save

Interface-Induced Trajectory Censoring

Wenbo Wang

City University of Hong Kong

Tool-Use Evaluation AI Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses

Open paper AskPDF arXiv

Agent evaluations report a tool-call rate read off the serving stack. That number can be zero while the model is emitting well-formed calls: the interface censors the trajectory before anything downstream sees it. On BFCL v4's own data, executor and scorer, holding weights, cases, decoding and seeds fixed and changing only the serving adapter, the same model scores 0.00 or 0.96 / 0.19. A 2x2 over chat template and parser locates the effect exactly: both main effects are exactly zero and all of it sits in the interaction -- no component is defective, and repairing one side of the contract buys precisely nothing. On tau-bench's 115 interactive retail tasks the same swap moves server-parsed calls from 0 to 636 and tasks reaching any tool execution from 0 to 103. Our probe reproduces the funnel across a 21x scale range of Qwen2.5-Coder: the server parses 0/100 at every size while well-formed emitted calls rise to 80/100 at 32B (~72 after calibration against an adjudicated gold standard). Under a matched envelope, across a comparable scale span, the silent fraction stays at 0-2, a prediction committed to the repository before the run. Llama-3.1-8B's 23% rate of calling the task function itself as a tool falls to 0 under one strict:true flag. The mismatch reaches inside the training loop, and its consequence is scale-dependent: in verl's AgentLoop at 7B, 45 of 115 generations carry a complete call; 0 are accepted, 0 execute, 0 return an observation. At 1.5B the same zero is over-determined, so we report the two scales separately. At evaluation time, repairing the adapter restores the mechanism but not a significant outcome gain: parsing 0->84, rescues 0->9, pass rate 53->62 (n.s.). We release a 98-line preflight check that catches every silent failure here. The observed tool-call rate is not a property of the model alone; it is a property of the model-interface stack that measures it.

Momentum

Related topics

Latest papers 1,013