AI Agent Benchmarks

Follow this topic Ask about this topic

AI agent benchmarks test systems that act over multiple steps, use tools, and respond to changing information, rather than only answer isolated questions. Researchers build executable environments for settings such as web search, coding, graphical interfaces, games, scientific workflows, and multi-agent coordination, then score outcomes with programmatic checks, expert rubrics, or simulated users. Increasingly, evaluations also inspect trajectories, evidence use, efficiency, safety, and recovery from errors. Agents can perform well on narrow tasks yet remain unreliable on long-horizon work, and results often depend strongly on the model, harness, environment, and grading method.

Latest papers

1,503 papers271 papers in the last four weeks, up 91% on the four weeks before. 1.6% of all new papers.

DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent
Mar 1, 2026Tongzhou Wu, Yuhao Wang, Xinyu Ma +4Deep Research Agents AI Agent Benchmarks
Shenzhen Technology University Shenzhen, China · City University of Hong Kong Hong Kong, China · Baidu Inc. Beijing, China
HVR-Met: A Hypothesis-Verification-Replanning Agentic System for Extreme Weather Diagnosis
Mar 1, 2026Shuo Tang, Jiadong Zhang, Gengxian Zhou +11Multi-Agent LLM Systems AI Agent Benchmarks
MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy, Beijing +3
MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains
Mar 1, 2026Xuying Ning, Dongqi Fu, Tianxin Wei +7Multimodal RAG Multimodal Search Agents
University of Illinois Urbana-Champaign · Meta · IBM Research +1
The Synthetic Web: Adversarially-Curated Mini-Internets for Diagnosing Epistemic Weaknesses of Language Agents
Feb 28, 2026Shrey Shah, Levent OzgurAdversarial Robustness of Language Models AI Agent Benchmarks
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
Feb 28, 2026Shiqi Chen, Jingze Gai, Ruochen Zhou +13Tool-Using Agents Tool Use
University of Oxford · City University of Hong Kong · Hong Kong University of Science and Technology +3
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
Feb 28, 2026Shu-Xun Yang, Cunxiang Wang, Haoke Zhang +12Agent Failure Analysis Reasoning Trace Analysis
Beijing Institute of Technology, Beijing, China · Zhipu AI, Beijing, China · Tsinghua University, Beijing, China +1
Theory of Code Space: Do Code Agents Understand Software Architecture?
Feb 28, 2026Grigory SapunovSoftware Engineering AI Agent Benchmarks
Intento
WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networks
Feb 28, 2026Jingwen Tong, Zijian Li, Fang Liu +2Wireless Local Area Networks Agentic Workflows
College of Electronics and Information Engineering, Shenzhen University, Shenzhen, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR
LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks
Feb 28, 2026Hengjian Gao, Kaiwei Zhang, Shibo Wang +8AI Agent Benchmarks Multimodal Model Evaluation
Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University · Shanghai University of Electric Power +1
TraderBench: How Robust Are AI Agents in Adversarial Capital Markets?
Feb 27, 2026Xiaochuang Yuan, Hui Xu, Silvia Xu +2AI Agent Evaluation AI Agent Benchmarks
UC Santa Cruz
HotelQuEST: Balancing Quality and Efficiency in Agentic Search
Feb 27, 2026Guy Hadad, Shadi Iskander, Oren Kalinsky +3LLM Agent Efficiency Agentic Search
Ben-Gurion University · Amazon
From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
Feb 27, 2026Seungdong Yoa, Sanghyu Yoon, Suhee Yoon +4Language Model Reasoning Evaluation LLM Evaluation
LG AI Research · Division of Computer Engineering, Hankuk University of Foreign Studies
Perfect score on IPhO 2025 theory by Gemini agent
Feb 26, 2026Yichen HuangAI Agent Evaluation AI Agent Benchmarks
Santa Clarita, California 91350, USA
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Feb 26, 2026Zhaochen Su, Jincheng Gao, Hangyu Guo +10Multimodal Agent Evaluation AI Agent Benchmarks
Hong Kong University of Science and Technology · University of North Carolina at Chapel Hill · Zhejiang University +1
SkillNet: Create, Evaluate, and Connect AI Skills
Feb 26, 2026Yuan Liang, Ruobin Zhong, Haoming Xu +47LLM Agent Skill Evaluation Agent Skill Learning
General Agent Evaluation
Feb 26, 2026Elron Bandel, Asaf Yehudai, Lilach Eden +12Agent Evaluation Automated Agent Evaluation
IBM Research · MIT
OmniGAIA: Towards Native Omni-Modal AI Agents
Feb 26, 2026Xiaoxi Li, Wenxiang Jiao, Jiarui Jin +10Multimodal Reasoning AI Agent Benchmarks
1Renmin University of China · 2Xiaohongshu Inc. · 3Peking University +2
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
Feb 26, 2026Zhiheng Song, Jingshuai Zhang, Chuan Qin +6LLM Agent Evaluation AI Agent Benchmarks
Computer Network Information Center, Chinese Academy of Sciences · AMAP, Alibaba Group · Independent Researcher
VeRO: A Harness for Agents to Optimize Agents
Feb 25, 2026Varun Ursekar, Apaar Shanker, Veronica Chatrath +2Agent Harness Optimization LLM Agent Harnesses
Scale AI.
ProactiveMobile: A Comprehensive Benchmark for Boosting Proactive Intelligence on Mobile Devices
Feb 25, 2026Dezhi Kong, Zhengzhao Feng, Qiliang Liang +12Mobile Agent Benchmarks AI Agent Benchmarks
HyperAI Team, Xiaomi Corporation · Zhejiang University · Peking University +1
Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?
Feb 25, 2026Germán T. Eizaguirre, Lars Tissen, Marc Sánchez-ArtigasText-to-SQL LLM Evaluation
Universitat Rovira i Virgili Tarragona, Spain · RWTH Aachen University Aachen, Germany
A Benchmark for Deep Information Synthesis
Feb 24, 2026Debjit Paul, Daniel Murphy, Milan Gritta +14LLM Agent Evaluation AI Agent Benchmarks
Huawei Noah’s Ark Lab, UK · Imperial College London · University of Zurich +3
Probing Dec-POMDP Reasoning in Cooperative MARL
Feb 24, 2026Kale-ab Tessera, Leonard Hinckeldey, Riccardo Zamboni +2Cooperative MARL Multi-Agent Reasoning
University of Edinburgh, Edinburgh, United Kingdom · Politecnico di Milano, Milan, Italy
Turing Test on Screen: A Benchmark for Mobile GUI Agent Humanization
Feb 24, 2026Jiachen Zhu, Lingyu Yang, Rong Shan +6Mobile GUI Agents AI Agent Benchmarks
Shanghai Jiao Tong University Shanghai, China · Carnegie Mellon University Pittsburgh, Pennsylvania, USA
Implicit Intelligence -- Evaluating Agents on What Users Don't Say
Feb 23, 2026Ved Sirdeshmukh, Marc WetterAI Agent Evaluation AI Agent Benchmarks
Applied Machine Learning Research
Agentic AI for Scalable and Robust Optical Systems Control
Feb 23, 2026Zehao Wang, Mingzhe Han, Wei Cheng +12LLM Agent Orchestration AI Agent Benchmarks
Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA · NEC Laboratories America, Princeton, NJ 08540, USA · Duke Quantum Center and Department of Physics, Duke University, Durham, NC, USA 27708 +4
CodeCompass: Navigating the Navigation Paradox in Agentic Code Intelligence
Feb 23, 2026Tarakanath PaipuruSoftware Engineering Agents Tool-Using Agents
Independent Researcher
ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?
Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1Coding Agent Benchmarks AI Coding Agents
AgenticRAGTracer: A Hop-Aware Benchmark for Diagnosing Multi-Step Retrieval Reasoning in Agentic RAG
Feb 22, 2026Qijie You, Wenkai Yu, Wentao ZhangMulti-Hop QA LLM Agent Evaluation
University of Science and Technology Beijing · 2Peking University · 3Zhongguancun Academy +1
Evaluating Test-Time Scaling of General LLM Agents
Feb 22, 2026Xiaochuan Li, Ryan Ming, Pranav Setlur +6LLM Agent Evaluation Test-Time Scaling
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Meta

Mar 1, 2026J/K move · Enter open · S save

DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agent

Tongzhou Wu, Yuhao Wang, Xinyu Ma, Xiuqiang He+3

Shenzhen Technology University Shenzhen, China · City University of Hong Kong Hong Kong, China · Baidu Inc. Beijing, China

Deep Research Agents AI Agent Benchmarks LLM Agent Training RL for LLM Agents

Open paper AskPDF arXiv

Deep-research agents are capable of executing multi-step web exploration, targeted retrieval, and sophisticated question answering. Despite their powerful capabilities, deep-research agents face two critical bottlenecks: (1) the lack of large-scale, challenging datasets with real-world difficulty, and (2) the absence of accessible, open-source frameworks for data synthesis and agent training. To bridge these gaps, we first construct DeepResearch-9K, a large-scale challenging dataset specifically designed for deep-research scenarios built from open-source multi-hop question-answering (QA) datasets via a low-cost autonomous pipeline. Notably, it consists of (1) 9000 questions spanning three difficulty levels from L1 to L3 (2) high-quality search trajectories with reasoning chains from Tongyi-DeepResearch-30B-A3B, a state-of-the-art deep-research agent, and (3) verifiable answers. Furthermore, we develop an open-source training framework DeepResearch-R1 that supports (1) multi-turn web interactions, (2) different reinforcement learning (RL) approaches, and (3) different reward models such as rule-based outcome reward and LLM-as-judge feedback. Finally, empirical results demonstrate that agents trained on DeepResearch-9K under our DeepResearch-R1 achieve state-of-the-art results on challenging deep-research benchmarks. We release the DeepResearch-9K dataset on https://huggingface.co/datasets/artillerywu/DeepResearch-9K and the code of DeepResearch-R1 on https://github.com/Applied-Machine-Learning-Lab/DeepResearch-R1.