Agent Benchmarks

Follow this topic Ask about this topic

Agent benchmarks test whether AI systems can complete tasks through sequences of decisions and actions, from web browsing and coding to desktop use, games, and professional workflows. They use simulated or real-world environments, tool-use tasks, long-horizon scenarios, and memory or multimodal challenges, with scoring based on verified outcomes, intermediate steps, costs, and reliability across repeated runs. Common approaches include executable state-based checks, trajectory review, human or rubric-based judging, and controlled comparisons of models, tools, and agent harnesses. Results show substantial gaps on complex tasks, while realism, reproducibility, verifier quality, and the influence of the harness make scores difficult to compare or generalize to deployment.

Latest papers

122 papers20 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Benchmarking Task Takeover Across Agent Handoffs
Oct 27, 2026Wenyuan Li, Guang Li, Dongyuan Li +3LLM Agent Evaluation Agent Evaluation
Hokkaido University Sapporo, Japan · The University of Tokyo Tokyo, Japan
AgenticCity-Bench: Longitudinal Multimodal Observations of Urban Dynamics with Agentic Reasoning
Oct 26, 2026Xiaofei Yu, Kailai Sun, Yanyu Mao +7Multimodal Dataset Agent Benchmarks
Renmin University of China · Singapore-MIT Alliance for Research and Technology, MIT · Massachusetts Institute of Technology +4
Benchmarks Are Surprisingly Compaction-Friendly
Oct 22, 2026Jeanne Salle, Hardik Bhatnagar, Matthias Bethge +2Context Compaction for LLM Agents LLM Context Compression
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +1
AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University
ColaBench: Evaluating Memory-Guided Knowledge Acquisition in Personal Agents
Oct 22, 2026Wenjie Zhang, Zhichao Shi, Xuhui Jiang +4Memory-Augmented Agents Personal AI Agents
Jiangnan University, Wuxi, China · School of Advanced Interdisciplinary Sciences, UCAS, Beijing, China · State Key Lab of AI Safety, Institute of Computing Technology, CAS, Beijing, China +3
Did the Problem Actually Get Harder? Testing LLMs on Paired P and NP-Complete Problems Hardened by Adversarial Multi-Agent Rewriting
Oct 21, 2026Nurul Labib Sayeedi, Md. Faiyaz Abdullah Sayeedi, Zabir Al Nazi +2Language Model Reasoning Evaluation Agent Benchmarks
United International University · BRAC University · University of California, Riverside +1
VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement
Oct 19, 2026Ruotian Peng, Yuanyi Ren, Shuaiqi Liu +7LLM Agent Training Self-Improving Agents
Zhejiang University · Westlake University · Alibaba Token Hub, Alibaba Group +2
RealCLI: Toward a Unified Evaluation of Human-Agent Collaboration for CLI Agents
Oct 15, 2026Yunqi Liu, Peiyu Wang, Bingli Wang +21Terminal-Agent Benchmarks LLM Agent Evaluation
IQuest Research · Beihang University · Renmin University
ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing
Oct 15, 2026Mushir AkhtarLLM Mathematical Reasoning Agent Benchmarks
The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock
Oct 14, 2026Ningxin Pan, Hanyu Li, Yeqing Shen +3LLM Agent Self-Improvement LLM Decision-Making
Peking University · Samsung Research, Beijing, China
DSABench: Benchmarking Source Discovery, Coordination, and Execution for Multi-Database Agents
Oct 10, 2026Jinlong Luo, Haoran Wu, Bo XuAgent Benchmarks Text-to-SQL
Institute of Automation, Chinese Academy of Sciences, Beijing, China
Finding Side-Effect Loopholes in Verifiable Rewards
Oct 6, 2026Richard AbrichBenchmark Auditing Agent Benchmarks
OpenAdapt.AI (MLDSAI Inc.)
MedicalHarness: A Controlled Evaluation of LLMs and Agent Harnesses on Medical Tasks
Oct 5, 2026Ziqing Wang, Lili Zhao, Kaize DingLLM Agent Evaluation AI Agent Benchmarks
Northwestern University
MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
Oct 4, 2026Yuxin Liu, Yuxuan Wang, Zhenxin Lei +9Multimodal Model Evaluation Agent Benchmarks
University of Science and Technology of China · Alibaba Token Hub, Alibaba Group · University of the Chinese Academy of Sciences +2
GNN-CB: A Graph Neural Network Competition Benchmark for Human and LLM Evaluation
Oct 4, 2026Murad Hossen, Tasneem Selim, Gurur Gamgam +22LLM Evaluation Graph Neural Networks
University of Houston, Texas,USA · BASIRA Lab, Department of Computing, Imperial College London, United Kingdom · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt +23
HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents
Oct 2, 2026Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri +14Web Agent Benchmarks Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · Mila – Quebec Artificial Intelligence Institute · Inception AI +2
Finding the Move Is Not Winning the Game: XiangqiBench for Closed-Loop Evaluation of LLM Agents
Oct 1, 2026Yekun Chai, Qiwei Peng, Haoyi XiongLLM Agent Evaluation AI Agent Evaluation
FloatAI · ETH Zurich · University of Copenhagen +1
VisAudit: Evaluating Multimodal Agents for Visual Diagnosis and Repair
Oct 1, 2026Shicheng Liu, Adam Kahirov, Qi Zhang +5Multimodal Model Evaluation Multimodal Agents
DAYJOB: A Benchmark for Long-Horizon Professional Work
Oct 1, 2026Stephanie Finley, Liudas Panavas, Thomas Mikkelson +12LLM Agent Evaluation Long-Horizon Agent Tasks
Surge AI
AutoDataBench: A Data-centric Testbed for Accelerating Auto Research
Sep 30, 2026Ruifeng Yuan, Yizhi Li, Yaxin Du +9Agent Benchmarks Autonomous Research Agents
The Hong Kong Polytechnic University · IQuest Research · Shanghai Jiaotong University +5
EngramBench: A Capability-Grounded Benchmark for Skill-Evolution Harnesses
Sep 30, 2026Zhixuan Tan, Pengjie Gu, Zhao Li +4Software Engineering Benchmarks Long-Horizon Agent Evaluation
The Chinese University of Hong Kong, Shenzhen · MemoraX AI
AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search
Sep 28, 2026Tongtong Feng, Xin Wang, Haoran Hou +9Aerial Robotics Object-Goal Navigation
Department of Computer Science and Technology, BNRist, Tsinghua University · School of Electrical & Electronic Engineering, University College Dublin · School of Electronics Engineering and Computer Science, Peking University +1
GameBoyWorlds: A Testbed for Self-Improvement in Embodied Video Games
Sep 25, 2026Dhananjay Ashok, Adam Shen, Aslan Huo Feng +10Self-Improving Agents Game-Playing Agents
Information Sciences Institute, University of Southern California · Columbia University · University of Chicago +2
GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
Sep 21, 2026Yiran Wang, Xingyilang Yin, Junfu Pu +12Game-Playing Agents Long-Horizon Agent Evaluation
ARC Lab, Tencent · GVC Lab, Great Bay University · University of Macau +3
What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus
Sep 20, 2026Edward Lue Chee Lip, Boden Moraski, Tim Knappe +4Benchmark Validity Agent Benchmarks
Quantifying Overclaiming Propensity in Frontier LLM Agents
Sep 17, 2026Nolan Smyth, Yorguin-Jose Mantilla-Ramos, Pascal Jr Tikeng Notsawo +6AI Agent Reliability AI Coding Agents
Tara Research · Mila – Quebec AI Institute · Cohere
GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Sep 10, 2026Umesh Bodhwani, Thanh Tran, Kai WeiLLM-as-a-Judge LLM Agent Evaluation
Amazon
Benchmarking Hybrid Deep Research Across Database Querying and Web Search
Sep 8, 2026Ruofan Wu, Peiran Xu, Xiaolong Li +9Deep Research Agents Agentic Search
University of Houston · University of California, Los Angeles · The University of Hong Kong +2
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Sep 3, 2026Lin Shi, Haowei Lin, Zixuan Zhu +123LLM Agent Evaluation AI Agent Evaluation
Audrey
EvoHarnessBench: Can Your Agents Keep Pace with an Evolving Harness?
Sep 3, 2026Zixuan Ke, Vaidehi Patil, Haizhou Shi +9Continual Learning for LLM Agents LLM Agent Harnesses
Salesforce Research · University of North Carolina at Chapel Hill, conducted this work during an internship at Salesforce Research · University of Wisconsin–Madison +1

Oct 27, 2026J/K move · Enter open · S save

Benchmarking Task Takeover Across Agent Handoffs

Wenyuan Li, Guang Li, Dongyuan Li, Keisuke Maeda+2

Hokkaido University Sapporo, Japan · The University of Tokyo Tokyo, Japan

LLM Agent Evaluation Agent Evaluation Agent Benchmarks

Open paper AskPDF arXiv

As agents collaborate on longer tasks, responsibility may pass between models before the work is complete. The receiving model must build on prior decisions, observations, and actions while addressing the remaining requirements. Evaluating this transition requires connecting the information preserved at handoff to how the receiver continues the work, a relationship that task-success and memory-retention scores alone do not capture. We introduce the Handoff State Continuity Benchmark (HSC-Bench), comprising 1,039 tasks that evaluate two complementary aspects of task takeover: recovering relevant repository evidence and continuing from established conversational, tool, and file-work state. Comparisons across six receiving models show that structured handoff state improves repository evidence recovery for every model and continuation performance for most models. The strongest receivers differ between the two tracks, showing that evidence-recovery rankings alone do not characterize performance when continuing from inherited state. By combining these comparisons with checks of subsequent responses and actions, HSC-Bench evaluates both access to prior evidence and adherence to established commitments and completed work. This provides a basis for selecting receiving models and assessing handoff representations according to the work that must be carried forward.