AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

148 papers in the last four weeks, up 185% on the four weeks before. 1.5% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,013

All topics

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition
Oct 10, 2025Yushuo Zheng, Tongrui Ye, Zicheng Zhang +3Multimodal Large Language Models AI Agent Benchmarks
Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent Evaluation Human-AI Interaction
Apart Research · AI Safety Cape Town · University of Chicago +2
MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
Aug 14, 2025Shilong Li, Xingyuan Bu, Wenjie Wang +22Multimodal IR Multimodal Reasoning
1ByteDance · 2Nanjing University · 5Zhejiang University +2
Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Aug 3, 2025Yuhan Guo, Cong Guo, Aiwen Sun +12Cognitive Modeling Multimodal Reasoning
Southwestern University of Finance and Economics · Shanghai Jiao Tong University · Central South University +9
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing Agents Agent Evaluation
Nanyang Technological University, Singapore · Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China +1
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Jul 7, 2025Yuanzhe Hu, Yu Wang, Julian McAuleyLLM Agent Memory LLM Agent Evaluation
University of California, San Diego
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent Orchestration AI Agent Evaluation
IBM · University of South Carolina · Steel Authority of India +1
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
Towards LLM Agents for Earth Observation
Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code Generation LLM Agent Evaluation
Cornell University · Columbia University
GraphChase: A Platform and Benchmark for Urban Network Security Games
Jan 29, 2025Shuxin Zhuang, Shuxin Li, Tianji Yang +4Benchmark Design AI Agent Benchmarks
City University of Hong Kong CAIR, Hong Kong Institute of Science & Innovation, CAS Hong Kong, Hong Kong · Nanyang Technological University Singapore, Singapore · Georgia Institute of Technology Atlanta, Georgia, United States +2
How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?
Dec 12, 2024Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5Healthcare LLM Agent Evaluation
Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent Evaluation AI Agent Benchmarks
Princeton University
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
Date pendingIdriss Malek, Omar Choukrani, Daniil Orel +5LLM Planning LLM Agent Evaluation
DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports
Date pendingRuizhe Li, Mingxuan Du, Benfeng Xu +3Deep Research Agents LLM Agent Evaluation
University of Science and Technology of China · Metastone Technology, Beijing, China
WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
Date pendingMadhav Kanda, Sharad Agarwal, Rodrigo Fonseca +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · Microsoft
EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Date pendingTara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent Evaluation Spoken Dialogue Systems
ServiceNow AI Research
GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science
Date pendingAleksandr Tsymbalov, Danis Zaripov, Artem Epifanov +1LLM Agent Evaluation AI Agent Benchmarks
AI Talent Hub, ITMO University, Saint Petersburg, Russia · HSE University
OpenFinGym: A Verifiable Multi-Task Gym Environment for Evaluating Quant Agents
Date pendingKaicheng Zhang, Wen Ge, Lei Jiang +5Quantitative Finance LLM Agent Evaluation
University of Edinburgh · University College London · Alan Turing Institute +1
PCBWorld: A Benchmark Environment for Engine-Grounded PCB Design Automation
Date pendingHyungseok Song, Junseok Park, Won-Seok Choi +4Electronic Design Automation AI Agent Benchmarks
LG AI Research Seoul, Republic of Korea
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
Date pendingWilliam CabanInter-Rater Reliability AI Agent Evaluation
Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria
SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?
Date pendingYuqiao Tan, Shizhu He, Jun Zhao +1AI Agent Evaluation Sparse Autoencoders
The Key Laboratory of Cognitive Intelligence, Institute of Automation, CAS · School of Artificial Intelligence, University of Chinese Academy of Sciences
Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
Date pendingMinghao Guo, Meng Cao, Sui Zhao +10Deep Research Agents Long-Horizon Agent Evaluation
Mohamed bin Zayed University of Artificial Intelligence · University of Science and Technology of China · Zhejiang University +1

Oct 10, 2025, cs.AIJ/K move · Enter open · S save

LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition

Yushuo Zheng, Tongrui Ye, Zicheng Zhang, Xiongkuo Min+2

Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · Tongji University, Shanghai, China

Multimodal Large Language Models AI Agent Benchmarks Multimodal Model Evaluation

Open paper AskPDF arXiv

Existing benchmarks for large multimodal models (LMMs) often fail to capture their performance in real-time, adversarial environments. We introduce LM Fight Arena (Large Model Fight Arena), a novel framework that evaluates LMMs by pitting them against each other in the classic fighting game Mortal Kombat II, a task requiring rapid visual understanding and tactical, sequential decision-making. In a controlled tournament, we test six leading open- and closed-source models, where each agent operates controlling the same character to ensure a fair comparison. The models are prompted to interpret game frames and state data to select their next actions. Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities in a dynamic setting. This work introduces a challenging and engaging benchmark that bridges the gap between AI evaluation and interactive entertainment.

Momentum

Related topics

Latest papers 1,013