Agent Evaluation

Follow this topic Ask about this topic

Agent evaluation studies whether AI systems can use tools, navigate changing environments, and complete multi-step tasks reliably—not just produce plausible answers. Benchmarks span software engineering, web and computer use, voice, and domain workflows, while methods include executable verifiers, human or model judges, and analysis of action traces and intermediate states. Researchers increasingly measure process quality, evidence use, safety, cost, latency, and recovery alongside task success. A central tension is making evaluations realistic and broad while keeping scores reproducible and trustworthy, especially when results depend on the agent’s harness, scoring rules, or evaluation channel.

Latest papers

144 papers30 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents
Apr 26, 2026Fanqing Meng, Lingxiao Du, Zijian Wu +46Computer-Use Agent Benchmarks LLM Agent Evaluation
AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuAgent Failure Analysis AI Agent Evaluation
The University of Hong Kong · Stellaris AI Limited
ClawEnvKit: Automatic Environment Generation for Claw-Like Agents
Apr 20, 2026Xirui Li, Ming Li, Ion Stoica +2Agent Evaluation AI Agent Benchmarks
University of Maryland · 5Mohamed bin Zayed University of Artificial Intelligence · 2Arena +2
From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
Apr 16, 2026Ke Xu, Yuhao Wang, Yu WangLLM Agent Evaluation Agent Evaluation
Shanghai Jiao Tong University
RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
Apr 13, 2026Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini +2Role-Playing Agent Evaluation LLM Agent Evaluation
Department of Political Sciences, Communication and International Relations, University of Macerata, Via Don Minzoni 22/A, Macerata, 62100, Italy · Deparment of Information Engineering, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60131, Italy · Department of Law, University of Macerata, Piaggia dell’Università 2, Macerata, 62100, Italy
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Apr 10, 2026Ling Shi, Yuqin Dai, Ziyin Wang +7LLM Agent Evaluation Agent Evaluation
Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China +3
Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
Apr 8, 2026Jiwan Chung, JiHyuk Byun, Vibhav Vineet +1Process-Based Agent Evaluation Agent Evaluation
Yonsei University · Microsoft Research
Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence?
Apr 3, 2026Qianshan Wei, Yishan Yang, Siyi Wang +12Multimodal Agent Evaluation Agent Evaluation
CASIA · UCAS · SEU +5
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
Apr 1, 2026Jingyue Li, André StorhaugAI Agent Evaluation Agent Evaluation
Norwegian University of Science and Technology, Trondheim, Norway
The Hitchhikers Guide to Rubric Quality Understanding and Enrichment
Apr 1, 2026Ankit Aich, Zhengyang Qi, Charles Dickens +7Agent Evaluation Psychometric Validation
Snorkel AI · Department of Computer Sciences, University of Wisconsin–Madison
Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild
Mar 30, 2026Deepak Akkil, Mowafak Allaham, Amal Raj +2Agent Evaluation Web Agent Benchmarks
Emergence AI · Northwestern University
Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
Mar 16, 2026Penny Chong, Harshavardhan Abichandani, Jiyuan Shen +4LLM Agent Evaluation Agent Evaluation
SAP · Stanford University
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
Mar 10, 2026Chengyu Shen, Yanheng Hou, Minghui Pan +8Automated Evaluation LLM Evaluation
Peking University · Beijing Institute of Technology · Beijing University of Posts and Telecommunications +1
MM-tau-p $^2$ : Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings
Mar 10, 2026Anupam Purwar, Aditya ChoudharyMultimodal Agent Evaluation Agent Evaluation
Sprinklr AI, Gurugram, India
ConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments
Mar 9, 2026Weixiang Zhao, Haozhen Li, Yanyan Zhao +5Multimodal Agent Evaluation AI Agent Safety Benchmarks
Harbin Institute of Technology · SERES
The World Won't Stay Still: Programmable Evolution for Agent Benchmarks
Mar 6, 2026Guangrui Li, Yaochen Xie, Yi Liu +11AI Agent Benchmarks Agent Evaluation
Amazon · UC Berkeley
$τ$ -Knowledge: Evaluating Conversational Agents over Unstructured Knowledge
Mar 4, 2026Quan Shi, Alexandra Zytek, Pedram Razavi +2LLM Agent Evaluation Agent Evaluation
Sierra · Princeton University
Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects
Mar 4, 2026Ji-Lun Peng, Yun-Nung ChenPersonality Modeling in Language Models OOD Generalization
National Taiwan University, Taipei, Taiwan · Academia Sinica, Taipei, Taiwan
MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaboration
Mar 1, 2026Abdulhamid M. Mousa, Jinhui Pang, Rakhmonberdi Khajiev +5AI Agent Evaluation Agent Evaluation
School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Mathematics Department, Faculty of Science, Ain Shams University, Cairo, Egypt +1
General Agent Evaluation
Feb 26, 2026Elron Bandel, Asaf Yehudai, Lilach Eden +12Agent Evaluation Automated Agent Evaluation
IBM Research · MIT
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Feb 24, 2026Zhenyu Wang, Xiaofen Xing, Yirong Chen +1Role-Playing Agent Evaluation Multimodal Agent Evaluation
South China University of Technology, Guangzhou 510641, China · Foshan University, Foshan 528225, China
GameDevBench: Evaluating Agentic Capabilities Through Game Development
Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Multimodal Agent Evaluation Software Engineering Agents
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA
DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems
Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5Multimodal Agent Evaluation LLM Agent Evaluation
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Department of Applied Mathematics, The Hong Kong Polytechnic University
StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley
Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing Agents Agent Evaluation
Nanyang Technological University, Singapore · Harbin Institute of Technology, Shenzhen, China · The Chinese University of Hong Kong, Shenzhen, China +1

Apr 26, 2026J/K move · Enter open · S save

ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen+45

Computer-Use Agent Benchmarks LLM Agent Evaluation Long-Horizon Agent Evaluation Agent Evaluation AI Agent Benchmarks Multimodal Agents

Open paper AskPDF arXiv

Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.