Agent Evaluation

Ask about this topic

Overview

Agent evaluation studies whether AI systems can use tools, navigate changing environments, and complete multi-step tasks reliably—not just produce plausible answers. Benchmarks span software engineering, web and computer use, voice, and domain workflows, while methods include executable verifiers, human or model judges, and analysis of action traces and intermediate states. Researchers increasingly measure process quality, evidence use, safety, cost, latency, and recovery alongside task success. A central tension is making evaluations realistic and broad while keeping scores reproducible and trustworthy, especially when results depend on the agent’s harness, scoring rules, or evaluation channel.

Latest papers

CardsList

144 papers30 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

  1. RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

    Oct 27, 2026Delin Wang, Sinuo Wang, Rui Sun +5Agent EvaluationRubric-Based Evaluation

    StepFun · Renmin University of China · University of Adelaide +2

  2. Benchmarking Task Takeover Across Agent Handoffs

    Oct 27, 2026Wenyuan Li, Guang Li, Dongyuan Li +3LLM Agent EvaluationAgent Evaluation

    Hokkaido University Sapporo, Japan · The University of Tokyo Tokyo, Japan

  3. PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers

    Oct 23, 2026Siyang Wu, Weizhi Zhang, Chen Yang +29Scientific Agent BenchmarksLLM Agent Evaluation

    University of Chicago · University of Illinois Chicago · Harvard University +15

  4. Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

    Oct 22, 2026Zhefan Rao, Meng Chu, Xuan Zhang +80Multimodal Agent EvaluationAgent Evaluation

  5. PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

    Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent EvaluationComputer-Use Agent Benchmarks

    The University of Texas at Dallas · University of Southern California · Carnegie Mellon University +14

  6. Recognizing Atari and Assessing Sente: Controlled Evaluation Tasks for Language-Based Go Agents

    Oct 16, 2026Taiyi Chen, Iskar Deng, Amandalynne PaulladaGame-Playing AgentsAI Agent Evaluation

    University of Washington

  7. Harness-Delta Attribution: Dissecting Gains From Harness Evolution

    Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness OptimizationAgent Evaluation

    Amazon · New York University

  8. Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

    Oct 15, 2026Sebastian-Antonio Toma, Jean de Kernier, Mike Ply +1Agent MemoryLLM Agent Memory

    Amazon Web Services · Technical University of Cluj-Napoca, Romania

  9. Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

    Oct 15, 2026Jingyu Tang, Xu Yang, Zhehao Zhang +7Agent EvaluationAI Agent Benchmarks

    University of Texas at Austin · The Ohio State University · University of Illinois at Urbana-Champaign +3

  10. AgentEval: Evidence-Bounded Assessment of Stateful Agents

    Oct 15, 2026Ruiqi GaoAI Agent EvaluationAgent Evaluation

    Alibaba Cloud, Alibaba Group

  11. OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics

    Oct 15, 2026Gideon Shaked, Siddharth NagisettyBioinformaticsAI Agent Benchmarks

    Department of Computational Medicine and Bioinformatics, University of Michigan · AfterQuery

  12. Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

    Oct 14, 2026Jiarui Qin, Junjie Huang, Yunjia Xi +5LLM Agent EvaluationAgent Evaluation

    Tencent Youtu Lab · Shanghai Jiao Tong University

  13. Smart Alone, Blind Together: Why Accuracy Fails to CertifyTheory of Mind in Multi-Agent Teams

    Oct 13, 2026Will Beaumaster, Paul SchraterAI Agent ReliabilityAgent Evaluation

    University of Minnesota, Minneapolis, MN, USA

  14. Sensitivity or Criterion? Measuring and Moving When Language Models Act

    Oct 11, 2026Sixun Dong, Yebowen Hu, Ming Yin +3Agent EvaluationLLM Tool Use

    University of Central Florida · Zoom Communications · Duke University +1

  15. SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications

    Oct 10, 2026Zhengxi Li, Junyi Zhou, Qiyuan Zhang +6Agent EvaluationLLM Agent Evaluation

    MBZUAI · City University of Hong Kong · WeChat AI, Tencent +1

  16. Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

    Oct 9, 2026Zongxia Li, Zhongzhi Li, Yucheng Shi +14Process-Based Agent EvaluationTerminal-Agent Benchmarks

    Tencent HY LLM Frontier · University of Maryland, College Park · University of Georgia +7

  17. Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer Validation

    Oct 5, 2026Dishu Yang, Qi Su, Hongbo Qin +1Benchmark AuditingAI Agent Auditing

    Khoury College of Computer Sciences Northeastern University San Jose, CA, USA · School of Engineering and Applied Science The George Washington University Bellevue, WA, USA · Khoury College of Computer Sciences Northeastern University Boston, MA, USA +1

  18. Separating Decision Time from Decision Quality in the Real-Time Gap of Distilled Deciders: Evidence from a Game and a Conveyor Simulator

    Oct 3, 2026Chihoon Shin, Junyeong Lee, Kihyeok Jeong +1Agent EvaluationGame-Playing Agents

    Myeongseongsimjae AX Institute, Yeongyang-gun, Gyeongsangbuk-do, Republic of Korea · Department of Computer Science and Software Engineering, Korea University, Seoul, Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea

  19. Label Agreement Does Not Measure Authorization

    Oct 3, 2026Amir Sabbaghziarani, Bradley Thomas Baker, Theodore J. LaGrow +1Agent EvaluationLLM Agent Reliability

    Joint Georgia State University / Georgia Institute of Technology / Emory University Center for Translational Research in Data Science and Neuroimaging · Georgia Institute of Technology

  20. Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn

    Oct 3, 2026Yangze Liu, Zhongyi HanAgent EvaluationAI Agent Benchmarks

    Shandong University

  21. InvestigationWorlds: An Agentic Environment for Legal Investigation

    Oct 2, 2026Albert Yu Sun, Andrew Benard, Sil Hamilton +8Legal ReasoningAgent Evaluation

    Epiq AI Labs · Cornell University

  22. Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case

    Oct 2, 2026Tingzhu Bi, Ping Wang, Meng MaLLM Decision-MakingLLM Reliability

    Peking University

  23. Open-Endedness Bench: Measuring Epistemic Process from Agent Records

    Oct 1, 2026Chengyang Shi, Xianglin Ji, Jintao Huang +3Agent EvaluationLLM Agent Evaluation

    University of Michigan, Ann Arbor · ARA Lab · Massachusetts Institute of Technology +2

  24. Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills

    Oct 1, 2026Ngoc Phuoc An Vo, Aarya Doshi, Vadim SheininProcess-Based Agent EvaluationTool-Use Evaluation

    IBM · Georgia Institute of Technology · IBM Research

  25. MCRI: A Four-Dimensional Framework for Analyzing and Evaluating Agent Skills

    Oct 1, 2026Zongrui Yang, Li Xintong, Runchen Xu +4LLM EvaluationAgent Evaluation

  26. Auditing Web Agent Evaluation on WebArena-Lite: Human Review of Outcomes and Trajectories

    Oct 1, 2026Chengguang Gan, Zimeng He, Yoshihiro Tsujii +3AI Agent AuditingWeb Agent Benchmarks

    Techtouch, Inc. · Institute of Science Tokyo

  27. Verify Claims, Not Scores: Evidence-Based Verification of Modular Agents

    Oct 1, 2026Ali Atiah AlzahraniAI Agent AuditingAgent Evaluation

  28. Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

    Sep 30, 2026Michael Hardy, Ruhana Azam, Anka Reuel +2AI Agent EvaluationAgent Evaluation

    Stanford University · University of Illinois Urbana-Champaign · Senior Author

  29. Hard-Gate Candidacy in a Deployed Validator Suite

    Sep 30, 2026Xin XuAgent ReliabilityAgent Evaluation

    Carnegie Mellon University

  30. Risk-Aware Adaptive Evaluation: Finding High-Impact Failures Under Limited Budgets

    Sep 30, 2026Priyanath Maji, Spandan Ghose ChowdhuryThompson SamplingAI Agent Evaluation

    Georgia Institute of Technology Atlanta, GA 30332