AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

209 papers in the last four weeks, up 61% on the four weeks before. 1.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,159

All topics

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
Aug 14, 2025Shilong Li, Xingyuan Bu, Wenjie Wang +22Multimodal IR Multimodal Reasoning
1ByteDance · 2Nanjing University · 5Zhejiang University +2
Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Aug 3, 2025Yuhan Guo, Cong Guo, Aiwen Sun +12Cognitive Modeling Multimodal Reasoning
Southwestern University of Finance and Economics · Shanghai Jiao Tong University · Central South University +9
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing Agents Agent Evaluation
Nanyang Technological University, Singapore · Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China +1
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Jul 7, 2025Yuanzhe Hu, Yu Wang, Julian McAuleyLLM Agent Memory LLM Agent Evaluation
University of California, San Diego
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent Orchestration AI Agent Evaluation
IBM · University of South Carolina · Steel Authority of India +1
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
Towards LLM Agents for Earth Observation
Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code Generation LLM Agent Evaluation
Cornell University · Columbia University
GraphChase: A Platform and Benchmark for Urban Network Security Games
Jan 29, 2025Shuxin Zhuang, Shuxin Li, Tianji Yang +4Benchmark Design AI Agent Benchmarks
City University of Hong Kong CAIR, Hong Kong Institute of Science & Innovation, CAS Hong Kong, Hong Kong · Nanyang Technological University Singapore, Singapore · Georgia Institute of Technology Atlanta, Georgia, United States +2
How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?
Dec 12, 2024Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5Healthcare LLM Agent Evaluation
Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent Evaluation AI Agent Benchmarks
Princeton University
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
Date pendingIdriss Malek, Omar Choukrani, Daniil Orel +5LLM Planning LLM Agent Evaluation
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Date pendingRuizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China
WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
Date pendingMadhav Kanda, Sharad Agarwal, Rodrigo Fonseca +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · Microsoft
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Date pendingTara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science
Date pendingAleksandr Tsymbalov, Danis Zaripov, Artem Epifanov +1LLM Agent Evaluation AI Agent Benchmarks
AI Talent Hub, ITMO University, Saint Petersburg, Russia · HSE University
OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents
Date pendingKaicheng Zhang, Wen Ge, Lei Jiang +5Quantitative Finance LLM Agent Evaluation
University of Edinburgh · University College London · Alan Turing Institute +1
PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation
Date pendingHyungseok Song, Junseok Park, Won-Seok Choi +4Electronic Design Automation AI Agent Benchmarks
LG AI Research Seoul, Republic of Korea
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
Date pendingWilliam CabanInter-Rater Reliability AI Agent Evaluation
Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria

Aug 14, 2025, cs.CLJ/K move · Enter open · S save

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu+21

1ByteDance · 2Nanjing University · 5Zhejiang University+2

Multimodal IR Multimodal Reasoning Multimodal Search Agents AI Agent Benchmarks Multimodal Agents

Open paper AskPDF arXiv

AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However, existing benchmarks such as BrowseComp primarily focus on textual content, overlooking the prevalence of multimodal content. To bridge this gap, we introduce MM-BrowseComp, a novel benchmark comprising 400 challenging, hand-crafted questions designed to evaluate multimodal retrieval and reasoning capabilities. Unlike prior work, MM-BrowseComp incorporates visual prompts and necessitates the extraction of key evidence from web images and videos to complete questions, rendering text-only approaches insufficient. Additionally, we provide a verified checklist for each question, enabling fine-grained analysis of multimodal dependencies and reasoning paths. Our comprehensive evaluation of 27 state-of-the-art models reveals that even leading models like GPT-5-High with tools achieve only 24.25% accuracy, highlighting the suboptimal multimodal browsing capabilities, establishing MM-BrowseComp as a rigorous new standard for the field.

Momentum

Related topics

Latest papers 1,159