AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,503 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads
Oct 29, 2026Shengxuan Qiu, Haochen Huang, Shuzhang Zhong +4LLM Agent Serving LLM Serving
Peking University · Georgia Institute of Technology
Bank-World: A Benchmark for Evaluating Agents on Banking Workflows
Oct 29, 2026Daniel RupawallaAI Agent Benchmarks Process-Based Agent Evaluation
Arcanum Labs
AgentDecept: Benchmarking Deceptive Behavior in Tool-Using LLM Agents under Pressure
Oct 29, 2026Yehan Zhang, Qianqi Yan, Hang Yin +5Deception in Language Models LLM Agent Evaluation
University of California, Santa Barbara · eBay
Can Image Agents Complete Visual Commissions? A Benchmark and Modular Framework
Oct 29, 2026Tong Li, Lishuai Gao, Qi Zeng +4AI Agent Benchmarks LLM Agent Orchestration
Meituan · University of Science and Technology of China
TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?
Oct 28, 2026Ritiz Tambi, Sahar Shayegan, Joshua Meyer +5AI Agent Benchmarks Document QA
Veris AI
Aligning Autoresearch: Co-Improvement between Agents and Humans
Oct 28, 2026Tianjian Li, Chuanyang Jin, Seungone Kim +10Autonomous Research Agents Human-AI Collaboration
FAIR at Meta · Johns Hopkins University · Carnegie Mellon University +1
TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information
Oct 27, 2026Chanwoo Park, Jaywoong Jeong, Zachary Wojtowicz +4Multi-Agent Collaboration Multi-Agent Systems
MIT · KAIST · UT Austin +1
Gym-Anything-RT: Scaling Real-Time Environments for Vision-Language Agents
Oct 27, 2026Pranjal Aggarwal, Sean WelleckAI Agent Benchmarks Agentic AI
Carnegie Mellon University
OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
Oct 27, 2026Humza Nusrat, Kundan ThindMedical AI Agents LLM Agent Evaluation
Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA · Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Oct 27, 2026Qiming Shi, Yikun Li, Beidi Luan +5Long-Horizon LLM Agents LLM Agent Evaluation
StepFun · Zhejiang University · Xi’an Jiaotong-Liverpool University +1
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Oct 27, 2026Binhang Qi, Yiming Liu, Yun Lin +3Automated Test Generation LLM Agent Evaluation
National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1
NextEval: An Agent-Based Solver and Benchmark for Derivative-Free Optimization
Oct 27, 2026Shuo Chen, Li Cui, Cunxin Huang +4Derivative-Free Optimization Large Language Model-Guided Optimization
Independent Researcher · Beijing Normal University · The Hong Kong Polytechnic University +3
kw-bench: Breaking Frontier Harnesses on Real-World Tasks
Oct 26, 2026George Juraj SalapaAI Agent Benchmarks LLM Agent Harnesses
G.S. s.r.o. · PwC Austria
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
Oct 26, 2026Muxi Diao, Wenbin Xu, Lele Yang +21Video Generation Evaluation Multimodal Agent Evaluation
A Unified Arena for LLM Agents across Diverse Eurogame
Oct 26, 2026Fengrui Zhang, Xiang Li, Ruomeng Ding +6LLM Agent Evaluation Game-Playing Agents
Alibaba Group · Nanjing University · National University of Singapore +3
PatentAgentBench: Benchmarking LLM Agents with Professional Patent Task Suites
Oct 26, 2026Yijie Hu, Wei Li, Junjie Zhou +6LLM Agent Evaluation AI Agent Benchmarks
AI Eval & Alignment, PatSnap Ltd. Co.
SearchHorizonBench: A Benchmark for Search Value in AI Agents
Oct 26, 2026Jingjie Ning, Xueqi Li, Yibo KongAI Agent Benchmarks Inference-Time Search
Carnegie Mellon University, Pittsburgh, PA, USA
How Humans Learn Unfamiliar Worlds: A Think-Aloud Dataset on ARC-AGI-3
Oct 25, 2026Shuzhi Liu, Archana Warrier, Dat Nguyen +3World Model Learning Human Behavior Modeling
Nanyang Technological University · TU Darmstadt · Basis Research Institute & Harvard University +1
ParetoRSI-Gym: Can AI Agents Improve Language Models Across Multiple Domains?
Oct 25, 2026Haojin Wang, Yangzhen Wu, Zeyi Chen +6LLM Agent Self-Improvement Language Model Post-Training
University of Illinois Urbana-Champaign · University of California, Berkeley · The Ohio State University +2
$Δτ$ -Bench: Pushing Agents Toward a Moving Capability
Oct 25, 2026Shuofei Qiao, Xuanguang Pan, Yijun Chen +6AI Agent Benchmarks LLM Agent Evaluation
Zhejiang University · University College London
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Oct 25, 2026Haoyu Zhao, Sitian Tu, Yuxuan Huang +16Continual Learning for LLM Agents Long-Horizon Agent Evaluation
Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players
Oct 24, 2026Changlian Ma, Limin WangLLM Agent Self-Improvement Autonomous Research Agents
Nanjing University · Shanghai AI Laboratory
IntentDelta: A Human-Constructed Benchmark for Evolving User Intent
Oct 24, 2026Jieyu Zhou, Zijian Wang, Yun WangLLM Agent Evaluation AI Agent Benchmarks
Georgia Institute of Technology, Atlanta, USA · Microsoft Research, Hong Kong SAR, China
AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment
Oct 24, 2026Shunchao Zhou, Chang Yang, Jingrui Wang +4Research Idea Generation AI Agent Benchmarks
Singapore Management University · The Hong Kong Polytechnic University · Hangzhou Dianzi University +3
Novel or Not? PatEx-Gym: Agentic Prior-Art Search for Novelty Assessment in Patent Examination
Oct 24, 2026Qiyao Wang, Xinyi Chen, Hongbo Wang +5Novelty Search AI Agent Benchmarks
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Dalian University of Technology +1
AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?
Oct 24, 2026Zijie Liu, Qi Yang Huang, Jinhao Duan +6LLM Agent Orchestration Enterprise AI Agents
University of North Carolina at Chapel Hill · Workato
Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents
Oct 23, 2026Anish Laddha, Nitesh Pradhan, Gaurav SrivastavaLLM Agents LLM Agent Evaluation
Department of Computer Science and Engineering, LNMIIT, India · Department of Computer Science, Virginia Tech, USA
SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks
Oct 23, 2026Tianyun Zhong, Wei Wang, Jialin Li +17Scientific Agent Benchmarks LLM Agent Evaluation
University of Chinese Academy of Sciences · Institute of Software, Chinese Academy of Sciences · AI Data, Alibaba Group +4
Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents
Oct 23, 2026Detao Zeng, Guilin Ou, Nuo Chen +5LLM Agent Evaluation AI Agent Benchmarks
University of Electronic Science and Technology of China
Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement
Oct 23, 2026Yanze Wang, Pengfei Yao, Tianyi Sun +5LLM Agent Skill Evaluation AI Agent Benchmarks
EverMind · Shanda Group · Peking University +1

Oct 29, 2026J/K move · Enter open · S save

MAServeBench: Benchmarking Multi-Agent LLM Serving Systems with Hierarchical Collaboration Workloads

Shengxuan Qiu, Haochen Huang, Shuzhang Zhong, Zizhuo Fu+3

Peking University · Georgia Institute of Technology

LLM Agent Serving LLM Serving AI Agent Benchmarks Multi-Agent Collaboration

Open paper AskPDF arXiv

Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible comparisons of legal release constraints at fixed work. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.