Overview

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

CardsList

563 papers47 papers in the last four weeks, down 48% on the four weeks before. 0.4% of all new papers.

  1. A Capability Framework for Evaluating Agentic AI Platforms in Regulated Enterprises

    Nov 18, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidAI Agent EvaluationAI Agent Benchmarks

  2. Verifying Agents in Rubric-Graded Environments

    Nov 17, 2026Markus Dücker, Vaibhav Kumar, Yi Liu +4LLM Agent VerificationAI Agent Evaluation

  3. Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment

    Nov 17, 2026Dongnuan Tian, Yan Yu, Yangfan Liu +6AI Agent EvaluationAgent Evaluation

  4. Behavioral Assurance for Socially Deployed AI Agents

    Nov 14, 2026Nicole H Brennan, Jiaee CheongAI Agent EvaluationBehavioral Consistency of LLM Agents

  5. Agent-as-a-Judge

    Nov 13, 2026Runyang You, Hongru Cai, Caiqi Zhang +5LLM-as-a-JudgeLLM Agent Evaluation

  6. Incentive-Responsive Agents: A Principal-Agent Framework, Metric Suite, and Runtime Architecture for Long-Lived Personal AI Agents

    Nov 11, 2026Jipu SuLong-Horizon LLM AgentsLLM Agent Efficiency

  7. When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

    Nov 10, 2026Sungbin ImAI Agent EvaluationLLM-as-a-Judge

  8. How Efficient Is Your Agent? Online Predictive Bounds under Budget Constraints

    Nov 8, 2026Roy Maor Lotan, Shai Feldman, Yaniv RomanoAI Agent EvaluationProcess-Based Agent Evaluation

  9. Knowing When to Intervene: Causal Evaluation of AI Agents

    Nov 6, 2026Ayush Sawarni, Wenqian Xing, Tijana Zrnic +2AI Agent EvaluationAI Agent Auditing

  10. DelegateEval: Evaluating Platforms for Personal External Service Agents

    Nov 5, 2026Jichao Li, Jinrong Lu, Manjiang Yu +1AI Agent EvaluationPersonal AI Agents

  11. FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

    Nov 5, 2026Yupeng Cao, Xueqing Peng, Lingfei Qian +14LLM Agent EvaluationAgent Evaluation

  12. Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration

    Nov 4, 2026Ruifeng Ren, Gun AhnAI Agent EvaluationAutomated Agent Evaluation

  13. PersonaCodeBench: Evaluating Coding Agents in Persona-driven Human-Interactive Sessions

    Nov 3, 2026Haixin Wang, Jianheng Hou, Hongyu Shen +27Coding AgentsAI Agent Evaluation

  14. Inter-Agent Influence: Evaluating Persuasion, Deception, and Coercion in Multi-Agent Systems

    Nov 3, 2026Chandler Smith, Qi Guo, Cecilia Elena Tilli +6AI Agent SafetyMulti-Agent Systems

  15. Signals: Continuous Diagnostics of AI Agents

    Nov 3, 2026Amey Rambatla, Vatsal Goel, Sagar Gaur +3AI Agent MonitoringAI Agent Evaluation