Overview

Agent evaluation studies whether AI systems can use tools, navigate changing environments, and complete multi-step tasks reliably—not just produce plausible answers. Benchmarks span software engineering, web and computer use, voice, and domain workflows, while methods include executable verifiers, human or model judges, and analysis of action traces and intermediate states. Researchers increasingly measure process quality, evidence use, safety, cost, latency, and recovery alongside task success. A central tension is making evaluations realistic and broad while keeping scores reproducible and trustworthy, especially when results depend on the agent’s harness, scoring rules, or evaluation channel.

Latest papers

CardsList

160 papers15 papers in the last four weeks, down 56% on the four weeks before. 0.1% of all new papers.

  1. ττ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

    Mar 4, 2026Quan Shi, Alexandra Zytek, Pedram Razavi +2LLM Agent EvaluationAgent Evaluation

  2. Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

    Mar 4, 2026Ji-Lun Peng, Yun-Nung ChenPersonality Modeling in Language ModelsOOD Generalization

  3. MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaboration

    Mar 1, 2026Abdulhamid M. Mousa, Jinhui Pang, Rakhmonberdi Khajiev +5AI Agent EvaluationAgent Evaluation

  4. General Agent Evaluation

    Feb 26, 2026Elron Bandel, Asaf Yehudai, Lilach Eden +12Agent EvaluationAutomated Agent Evaluation

  5. MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

    Feb 24, 2026Zhenyu Wang, Xiaofen Xing, Yirong Chen +1Role-Playing Agent EvaluationMultimodal Agent Evaluation

  6. GameDevBench: Evaluating Agentic Capabilities Through Game Development

    Feb 11, 2026Wayne Chi, Yixiong Fang, Arnav Yayavaram +8Multimodal Agent EvaluationSoftware Engineering Agents

  7. DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems

    Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5Multimodal Agent EvaluationLLM Agent Evaluation

  8. StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

    Jul 10, 2025Weihao Tan, Changjiu Jiang, Yu Duan +5Game-Playing AgentsAgent Evaluation