AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

146 papers in the last four weeks, up 161% on the four weeks before. 1.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models
Jun 11, 2026Quan QuanIndustrial Anomaly Detection Multimodal Reasoning
Iterating Toward Better Search: A Two-Agent Simulation Framework for Evaluating Agentic Search Architectures in E-Commerce
Jun 11, 2026Jetlir Duraj, Jayanth Yetukuri, Shuang Zhou +4LLM Agent Memory E-Commerce Search
eBay Inc.
GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models
Jun 11, 2026Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +2LLM Agent Evaluation Geospatial Reasoning
Universidad Católica de Ávila (UCAV) Ávila, Spain · Johns Hopkins University Baltimore, MD, USA · Independent Researcher Madrid, Spain +1
Benchmarking AI Agents for Addressing Scientific Challenges Across Scales
Jun 10, 2026Tianyu Liu, Allen Xin Wang, Antonia Panescu +30AI for Science AI Agent Evaluation
Yale University, CT, USA · The Pennsylvania State University, PA, USA · Northeastern University, MA, USA +7
WorkBench Revisited: Workplace Agents Two Years On
Jun 10, 2026Olly Styles, Sam MillerComputer-Use Agent Benchmarks AI Agent Reliability
Independent researcher
Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks
Jun 10, 2026Mengyu Zheng, Kai Han, Boxun Li +13AI Coding Agents Software Engineering Benchmarks
TokenRhythm Technologies · Infinigence AI · Peking University +5
MedCTA: A Benchmark for Clinical Tool Agents
Jun 10, 2026Tajamul Ashraf, Hyewon Jeong, Fida Mohammad Thoker +1Agent Evaluation AI Agent Benchmarks
King Abdullah University of Science and Technology (KAUST), Saudi Arabia · Massachusetts Institute of Technology (MIT), USA
DFAH-Bench: Benchmarking Observable Agent Instability in Financial Decision-Making
Jun 10, 2026Raffi KhatchadourianAgent Evaluation AI Agent Benchmarks
IBM Financial Services Market New York, NY, USA
AI Coding Agents Can Reproduce Social Science Findings
Jun 9, 2026Meysam Alizadeh, Mohsen Mosleh, Fabrizio Gilardi +2AI Coding Agents AI Agent Benchmarks
University of Oxford · University of Zurich · Carnegie Mellon University +1
Can AI Agents Synthesize Scientific Conclusions?
Jun 9, 2026Hayoung Jung, Pedro Viana Diniz, José Reinaldo Corrêa Roveda +5AI Agent Evaluation AI Agent Benchmarks
Princeton University · Universidade Federal de Minas Gerais · Stony Brook University +1
From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents
Jun 9, 2026Yifan Li, Shengbin Yue, Boyu Feng +6LLM Agent Evaluation AI Agent Benchmarks
The Chinese University of Hong Kong · Fudan University · Tencent +2
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
Jun 9, 2026Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin +12Customer Support Automation LLM Agent Evaluation
AI Foundations, Capital One
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Jun 9, 2026Liya Zhu, Jingzhe Ding, Jian Zhang +63Computer-Use Agent Benchmarks GUI Agents
Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?
Jun 9, 2026Tengchao Lv, Dongdong Zhang, Jiayu Ding +10LLM Agent Evaluation AI Agent Benchmarks
Microsoft Research
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
Jun 9, 2026Haonan Wang, Jiaxiang Liu, Yurong Liu +11Multi-Hop QA AI Agent Benchmarks
Columbia University · New York University · Barnard College
STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios
Jun 9, 2026Sirui Liang, Bohan Yu, Peiyu Wang +8Computer-Use Agent Benchmarks LLM Agent Evaluation
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation · Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences · Zhongguancun Academy +3
$τ$ -Rec: A Verifiable Benchmark for Agentic Recommender Systems
Jun 8, 2026Bharath Sivaram Narasimhan, Karthik R NarasimhanAI Agent Reliability AI Agent Benchmarks
Independent Researcher Mountain View, CA, USA · Princeton University Princeton, NJ, USA
Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces
Jun 8, 2026Shardul Bansal, Seth Schilbe, Jarrod BarnesAI Agent Benchmarks LLM Agent Training
1ORO AI · 2Dynamical Systems
OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
Jun 8, 2026Mingxian Lin, Shengju Qian, Yuqi Liu +9VLM Evaluation Game-Playing Agents
1The University of Hong Kong · 2LIGHTSPEED · 3The Chinese University of Hong Kong +1
iOSWorld: A Benchmark for Personally Intelligent Phone Agents
Jun 8, 2026Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom +3Computer-Use Agent Benchmarks Computer-Use Agents
Carnegie Mellon University
SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
Jun 8, 2026Hongcheng Gao, Hailong Qu, Jingyi Tang +18Spatial Reasoning Benchmarks Visual Spatial Reasoning
Tsinghua University · Chongqing University · Peking University +6
H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions
Jun 8, 2026Shiping Zhu, Yibo Yang, Zhengyang Wang +3Multimodal Memory LLM Agent Memory
Jilin University · Shanghai Jiao Tong University · University of California at Merced
WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
Jun 8, 2026Wanli Li, Bowen Zhou, Yunyao Yu +4Computer-Use Agent Benchmarks Computer-Use Agents
Microsoft Research Asia · Tsinghua University
Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
Jun 8, 2026Ziqian Zhong, Ivgeni Segal, Ivan Bercovich +3Reward Hacking AI Agent Security Benchmarks
Carnegie Mellon University · Fewshot Corp · Independent Researcher
PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting
Jun 7, 2026Youran Sun, Xingyu Ren, Kejia Zhang +2Multi-Agent Orchestration LLM Agent Evaluation
University of Maryland · The Chinese University of Hong Kong · Stanford University
Scaffold Effects on GAIA: A Controlled Comparison
Jun 7, 2026Jason StaraceLLM Agent Evaluation AI Agent Benchmarks
Independent Researcher
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination LLM Agent Evaluation
University of Edinburgh · University of Oxford · University College London
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
Jun 5, 2026Jiayu Wang, Weijiang Lv, Bowen Fu +8AI-Assisted Scientific Research AI Agent Evaluation
Xi’an Jiaotong University · Xidian University
M $^3$ Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
Jun 5, 2026Zhengjun Huang, Wenxuan Liu, Zhoujin Tian +6Multimodal Grounding Efficient Multimodal Inference
The Hong Kong University of Science and Technology · Peng Cheng Laboratory · Beijing University of Chemical Technology +4

Jun 11, 2026, cs.AIJ/K move · Enter open · S save

Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models

Quan Quan

Industrial Anomaly Detection Multimodal Reasoning AI Agent Benchmarks Multimodal Agents Multimodal Model Evaluation Power Systems

Open paper AskPDF arXiv

The power distribution network is critical to reliable electricity delivery, yet traditional inspection methods face limitations in semantic understanding, generalization, and closed-loop automation. To address these challenges, this paper proposes a Multi-Modal Agent framework specifically for power distribution defect detection. Central to this study is the systematic evaluation of multimodal foundation models as unified cognitive engines. We rigorously assess their integrated performance across three critical capabilities: (1) Perception, where the model must accurately identify equipment and generate expert-level descriptions of defects; (2) Reasoning, where the model interprets visual findings to diagnose causes, assess severity, and plan maintenance strategies based on domain knowledge; and (3) Tool Usage, where the model acts as an autonomous operator to execute actions -- such as querying knowledge bases or generating work orders -- to achieve closed-loop maintenance. To support this evaluation, a domain-specific evaluation dataset and a comprehensive benchmark are developed. Experimental results demonstrate the strengths and limitations of current foundation models in these three dimensions, providing empirical evidence for deploying autonomous agents in high-stakes industrial environments.

Momentum

Related topics

Latest papers 1,013