AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

209 papers in the last four weeks, up 61% on the four weeks before. 1.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,159

All topics

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Feb 15, 2026Lingxiang Hu, Yiding Sun, Tianle Xia +6Online Advertising LLM Agent Evaluation
Tencent
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences
Feb 11, 2026Bang Nguyen, Dominik Soós, Qian Ma +8LLM Agent Evaluation AI Agent Benchmarks
University of Notre Dame · Computer Science and Engineering · Notre Dame, IN, United States +8
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Software Engineering Agents Agent Evaluation
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA
Evaluating Memory Structure in LLM Agents
Feb 11, 2026Alina Shutova, Alexandra Olenina, Ivan Vinogradov +1LLM Agent Memory LLM Agent Evaluation
HSE University · Yandex · YSDA +1
ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response
Feb 3, 2026Xiaomeng Zhu, Fengming Zhu, Weijie Zhou +8AI Agent Benchmarks Proactive Assistance
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China · Tencent, Shenzhen, China · Futian Laboratory, Shenzhen, China +1
Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performance
Feb 3, 2026Abdelghny Orogat, Ana Rostam, Essam MansourMulti-Agent LLM Systems Multi-Agent Orchestration
Concordia University
Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability
Feb 3, 2026Shafiuddin Rehan Ahmed, Sourabh DeshpandeAI Agent Benchmarks Code Generation Evaluation
Center for Advanced AI, Accenture · Mountain View, California, USA
AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
Feb 2, 2026Shraddha Barke, Arnav Goyal, Alind Khare +3Agent Failure Analysis AI Agent Reliability
1Microsoft Research · 2Microsoft · 3UIUC
Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory
Jan 30, 2026Yuhao Zhan, Tianyu Fan, Linxuan Huang +2Deep Research Agents Hallucination Detection
Zhejiang University · The University of Hong Kong
EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents
Jan 23, 2026Xinze Li, Ziyue Zhu, Siyuan Liu +4Episodic Memory Long-Horizon Agent Evaluation
Nanyang Technological University · Shanghai AI Laboratory · Fudan University
Toward Efficient Agents: Memory, Tool learning, and Planning
Jan 20, 2026Xiaofang Yang, Lijun Li, Heng Zhou +12LLM Agent Memory AI Agent Benchmarks
Shanghai Artificial Intelligence Laboratory · Fudan University · University of Science and Technology of China +5
LifeAgentBench: Benchmarking LLMs for Long-Horizon, Cross-Dimensional Lifestyle Health Reasoning
Jan 20, 2026Ye Tian, Zihao Wang, Onat Gungor +2Healthcare Long-Horizon Agent Evaluation
University of California San Diego, La Jolla, CA, USA · Google Research, Mountain View, CA, USA
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5LLM Agent Evaluation Agent Evaluation
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks
Jan 16, 2026Weiyi Wang, Xinchi Chen, Jingjing Gong +2LLM Agent Evaluation AI Agent Benchmarks
Fudan University · OpenMOSS Team · Shanghai Innovation Institute
IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents
Jan 10, 2026Yingchaojie Feng, Qiang Huang, Xiaoya Xie +4Deep Research Agents LLM Agent Evaluation
School of Computing, National University of Singapore · School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen) · Zhejiang University +1
MCPAgentBench: A Real-world Task Benchmark for Evaluating LLM Agent MCP Tool Use
Dec 31, 2025Zixiang Liu, Wenrui Liu, Elsie Dai +5LLM Agent Evaluation AI Agent Benchmarks
Peking University · ZTE
DEER: A Benchmark for Evaluating Deep Research Agents on Expert Report Generation
Dec 19, 2025Janghoon Han, Heegyu Kim, Changho Lee +6Deep Research Agents LLM Agent Evaluation
LG AI Research · University of Illinois Chicago · University of Michigan, Ann Arbor
PPTArena: A Benchmark for PowerPoint Editing
Dec 2, 2025Michael Ofengenden, Yunze Man, Ziqi Pang +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · University of California, Berkeley
CLIMATEAGENT: Multi-Agent Orchestration for Complex Climate Data Science Workflows
Nov 25, 2025Chenyue Li, Hyeonjae Kim, Wen Deng +4AI for Science Multi-Agent Orchestration
1The Hong Kong University of Science and Technology
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent Evaluation Multimodal Large Language Models
Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2
IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation
Nov 21, 2025Yifan Li, Lichi Li, Anh Dao +15Visual Spatial Reasoning Embodied Navigation
Michigan State University · Independent Researcher · University of California, Irvine +7
SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios
Nov 20, 2025Juntao Cheng, Wanyue Zhang, Zhiwei Yu +7Long-Horizon Agent Evaluation Egocentric Video Understanding
Beijing Academy of Artificial Intelligence (BAAI) · Shanghai Jiao Tong University · School of Artificial Intelligence, University of Chinese Academy of Sciences +3
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
Nov 4, 2025Jiayu Liu, Cheng Qian, Zhaochen Su +4LLM Planning LLM Agent Evaluation
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign · Tsinghua University
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
Oct 22, 2025Gil Pasternak, Dheeraj Rajagopal, Julia White +4LLM Agent Evaluation AI Agent Benchmarks
When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue
Oct 22, 2025Tanya Shourya, Yingfan Wang, Zhaoyi Joey Hou +3Tool-Use Evaluation AI Agent Benchmarks
AWS AI Labs · University of Pittsburgh
Lost in the Maze: Overcoming Context Limitations in Long-Horizon Agentic Search
Oct 21, 2025Howard Yen, Yoonsang Lee, Ashwin Paranjape +5Agentic Search Long-Horizon Agent Evaluation
Princeton Language and Intelligence, Princeton University · Work done as an intern at Samaya AI. · Samaya AI
HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning
Oct 16, 2025Chance Jiajie Li, Zhenze Mo, Yuhan Tang +11Cognitive Modeling Human Behavior Simulation
1MIT Media Lab · 8Northeastern University · 5MIT CEE +7
Compositional Machine Design as Program Synthesis with LLMs
Oct 16, 2025Wenqian Zhang, Yangyi Huang, Weiyang Liu +1LLM-Based Program Synthesis Program Synthesis
1The Chinese University of Hong Kong (Shenzhen) · 2The Chinese University of Hong Kong
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent Evaluation Human-AI Interaction
Apart Research · AI Safety Cape Town · University of Chicago +2

Feb 15, 2026, cs.CLJ/K move · Enter open · S save

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

Lingxiang Hu, Yiding Sun, Tianle Xia, Wenwei Li+5

Tencent

Online Advertising LLM Agent Evaluation AI Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

While Large Language Model (LLM) agents have made remarkable progress on complex reasoning, evaluating them in real-world environments remains an open problem. Existing benchmarks are largely confined to idealized simulations and fail to capture specialized domains such as advertising and marketing analytics, where tasks require multi-round interaction with professional tools and where ground-truth answers quickly become obsolete as data and platform rules evolve. To address this, we propose AD-Bench, a benchmark built from real user marketing-analysis requests on a production advertising platform. AD-Bench introduces two key designs: (i) a dynamic ground-truth pipeline that replays expert tool-call trajectories to regenerate answers consistent with the current environment, mitigating answer obsolescence; and (ii) a trajectory-aware evaluation that jointly measures end-to-end answer correctness (Pass@k) and trajectory coverage. Requests are stratified into three difficulty levels (L1-L3) to probe multi-round, multi-tool collaboration. Experiments show that the best model, Claude-Opus-4.7, attains Pass@1 = 76.9% and Pass@3 = 80.4% with 82.7% trajectory coverage overall, yet drops sharply on L3 to Pass@1 = 61.4% and Pass@3 = 65.1%, revealing that even state-of-the-art agents have substantial gaps in complex advertising analytics.

Momentum

Related topics

Latest papers 1,159