AI Agent Benchmarks

Follow this topic Ask about this topic

Momentum

96 papers in the last four weeks, up 30% on the four weeks before. 1.2% of all new papers.

Jul 6Week of Sep 21

Latest papers 1,013

All topics

GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing
May 28, 2026Xiaoyi Chen, Yifei Gao, Yang Xu +3GUI Agents AI Agent Benchmarks
Beijing Jiaotong University · Independent Researcher
MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
May 28, 2026Kevin Wang, Anna Thöni, Benjamin Kempinski +50LLM Agent Evaluation Strategic Reasoning Risks in Language Models
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
May 28, 2026Chengzhi Liu, Yuzhe Yang, Sophia Xiao Pu +14Multimodal Memory LLM Agent Memory
University of California, Santa Barbara · Johns Hopkins University · Carnegie Mellon University +3
STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
May 28, 2026Junyang Wang, Haiyang Xu, Xi Zhang +4GUI Agents AI Agent Benchmarks
Beijing Jiaotong University · Tongyi AI Lab, Alibaba Group
OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories
May 28, 2026Yibing Liu, Yangze Liu, Xiaolong Yin +4Agent Failure Analysis Anomaly Localization
School of Software, Shandong University, Jinan, China · School of Artificial Intelligence, Nanjing University; State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · Center for Medical Artificial Intelligence; Qingdao Academy of Chinese Medical Sciences; Institute of Marine Traditional Chinese Medicine, Shandong University of Traditional Chinese Medicine,2026 Qingdao 266112, China +1
BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents
May 28, 2026Jiahao Huang, Fei Cheng, Junfeng Jiang +2LLM Agent Evaluation AI Agent Benchmarks
University of Tokyo · Kyoto University · National Institute of Informatics
unix-ctf: Procedural Environments for Unix-Competence Reinforcement Learning
May 27, 2026Geoffrey Bradway, Roger Creus Castanyer, Lorenz Wolf +3AI Agent Benchmarks LLM Agent Training
Vmax
Personal Visual Memory from Explicit and Implicit Evidence
May 27, 2026Viet Nguyen, Thao Nguyen, Vishal M. Patel +1Multimodal Memory AI Agent Benchmarks
Johns Hopkins University · University of Wisconsin-Madison · Adobe Research
LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?
May 27, 2026HuiMing Fan, Xiao Wang, Zheng Chu +5Computer-Use Agent Benchmarks LLM Agent Evaluation
1Harbin Institute of Technology · 2Xiaohongshu
VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora
May 27, 2026Yuting Xu, Jiayi Tian, Jian Liang +4Multimodal Reasoning AI Agent Benchmarks
Institute of Information Engineering, CAS · School of Cyber Security, UCAS · Amap, Alibaba Group +2
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
May 27, 2026Liang Cheng, Mingsheng Cai, Jiuming Jiang +1LLM Agent Evaluation AI Agent Benchmarks
†University of Edinburgh
Beyond One Path: Evaluating and Enhancing Divergent Thinking in Interactive LLM Agents
May 27, 2026Jihyeong Park, Ingeol Baek, Jeonghyun Park +1LLM Agent Evaluation AI Agent Benchmarks
Chung-Ang University, Seoul, Korea
From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets
May 27, 2026Taojie Zhu, Wentao Zhao, Rui Sun +7Benchmark Contamination LLM Agent Evaluation
1Tsinghua University, 2Stepfun · 2Stepfun · 4Shanghai Jiao Tong University +3
OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents
May 27, 2026Chenyu Zhou, Xinyun Lu, Jiangyue Zhao +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai University of Finance and Economics · Stanford University
Verifiable Benchmarking of Long-Horizon Spatial Biology
May 27, 2026Ian Diks, Harihara Muralidharan, Tim Proctor +1Spatial Transcriptomics Benchmark Design
LatchBio, San Francisco, CA, USA
MemCog: From Memory-as-Tool to Memory-as-Cognition in Conversational Agents
May 27, 2026Zihan Li, Xingyu Fan, Feifei Li +1Conversational Memory AI Agent Benchmarks
WeChat, Tencent Inc., Beijing, China
AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
May 27, 2026Kou Shi, Ziao Zhang, Shiting Huang +7Tool-Use Evaluation AI Agent Benchmarks
University of Science and Technology of China · University of Toronto
DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints
May 27, 2026Zhitong Chen, Kai Yin, Weifeng Zhang +7Tool-Use Planning AI Agent Benchmarks
Texas A&M University · University of Toronto
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9Computer-Use Agent Benchmarks AI Agent Reliability
1Peking University · 2Qiyuan Tech
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
May 27, 2026Pengyu Zhu, Lijun Li, Yaxing Lyu +8LLM Agent Evaluation AI Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2Shanghai Artificial Intelligence Laboratory · 4North China Electric Power University +1
VibeSearchBench: Benchmarking Long-horizon Proactive Search in the Wild
May 27, 2026Xiaohongshu IncAgentic Search Long-Horizon Agent Evaluation
Xiaohongshu Dots Studio & Unipat AI
Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness
May 27, 2026Jaechang Kim, Sunung Mun, Seungjoon Lee +2Explainable Artificial Intelligence AI Agent Benchmarks
Graduate School of AI, POSTECH · Department of Computer Science and Engineering, POSTECH · Krafton
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
May 27, 2026Yunqi Liu, Tong Niu, Zitong Wang +4AI Agent Evaluation Egocentric Video Understanding
Ant Group
AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications
May 26, 2026Yifan Sui, Xin Huang, Hongbing Li +14Mobile GUI Automation Computer-Use Agent Benchmarks
1Beijing University of Posts and Telecommunications · 2StepFun · 3Waseda University
DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents
May 26, 2026Shijie Cao, Yuan Yuan, Jing LiuLLM Agent Evaluation Combinatorial Optimization
School of Computer Science and Engineering, Beihang University, Beijing 100191, China · Shenzhen Loop Area Institute, Shenzhen, China · Qingdao Research Institute, Beihang University +3
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
May 26, 2026Yipeng Ouyang, Xin Huang, Bingjie Liu +3Software Engineering Agents LLM Agent Evaluation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
ENPMR-Bench: Benchmarking Proactive Memory Retrieval for Emotional Support Agents
May 26, 2026Xing Fu, Yulin Hu, Mengtong Ji +5Emotional Support Conversation AI Agent Benchmarks
Research Center for Social Computing and Interactive Robotics Harbin Institute of Technology, Heilongjiang, China
VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions
May 26, 2026Yuxin Chen, Yi Zhang, Zhengzhou Cai +11LLM Agent Memory AI Agent Evaluation
1National University of Singapore · 2Meituan · University of Science and Technology of China +2
IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams
May 26, 2026Jinzhao Li, Yinuo Chen, Wenxuan Song +5Streaming Video Understanding AI Agent Benchmarks
College of AI, Tsinghua University · 2ByteDance
ChartAct: A Benchmark for Dynamic Chart Understanding
May 26, 2026Muye Huang, Lin Wu, Lingling Zhang +5Data Visualization Chart QA
School of Computer Science and Technology, Xi’an Jiaotong University · 2MOE KLNN Lab, Xi’an Jiaotong University

May 28, 2026, cs.SEJ/K move · Enter open · S save

GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

Xiaoyi Chen, Yifei Gao, Yang Xu, Xingxing Song+2

Beijing Jiaotong University · Independent Researcher

GUI Agents AI Agent Benchmarks Mobile GUI Agents

Open paper AskPDF arXiv

Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.

Momentum

Related topics

Latest papers 1,013