Overview

Agent evaluation studies whether AI systems can use tools, navigate changing environments, and complete multi-step tasks reliably—not just produce plausible answers. Benchmarks span software engineering, web and computer use, voice, and domain workflows, while methods include executable verifiers, human or model judges, and analysis of action traces and intermediate states. Researchers increasingly measure process quality, evidence use, safety, cost, latency, and recovery alongside task success. A central tension is making evaluations realistic and broad while keeping scores reproducible and trustworthy, especially when results depend on the agent’s harness, scoring rules, or evaluation channel.

Latest papers

CardsList

160 papers15 papers in the last four weeks, down 56% on the four weeks before. 0.1% of all new papers.

  1. Reliable Evaluation of AI Agents for Investment Research: A Framework for Performance and Validity Assessment

    Nov 17, 2026Dongnuan Tian, Yan Yu, Yangfan Liu +6AI Agent EvaluationAgent Evaluation

  2. When the World Is the Test Suite: Agentic Intelligence for the Physical World

    Nov 17, 2026Kaixin Li, Meng Luo, Yanlin Li +9Agentic AI for RoboticsAgent Evaluation

  3. DetourBench: Trajectory-Level Evaluation of Interactive Agents with Exact Minimum-Action Bounds

    Nov 16, 2026Hanlin Tian, Sihan Zhu, Zhao Yang +5Agent EvaluationProcess-Based Agent Evaluation

  4. EVIDENT: Evaluating Evidence Grounding in GUI Agents

    Nov 16, 2026Yerin Nam, Minkyung Cho, MinKyu Kim +4Agent EvaluationComputer-Use Agent Evaluation

  5. WordInferBench: Can Agents Infer How to Edit from Word Document Context?

    Nov 11, 2026Jiayu Ding, Yupan Huang, Jinyoung Park +14Agent EvaluationAutomated Agent Evaluation

  6. ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces

    Nov 10, 2026Jiahang Lin, Wanying Ren, Wanghan Xu +24Agent EvaluationScientific Agent Benchmarks

  7. Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

    Nov 5, 2026E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman +1Coding Agent BenchmarksBenchmark Validity

  8. FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance

    Nov 5, 2026Yupeng Cao, Xueqing Peng, Lingfei Qian +14LLM Agent EvaluationAgent Evaluation

  9. What Structurally Breaks an Agent? Auditing Difficulty in Stateful Tool-Use Workflows

    Nov 2, 2026Shereen Anand, Ju Wang, Yali DuAgent Failure AnalysisAI Agent Auditing

  10. Delivered, Not Used: Evaluating Agent Memory at Its Point of Use

    Nov 1, 2026Changran Xu, Zeju Li, Qiang XuAgent MemoryLLM Agent Evaluation

  11. RubricAtlas: Mining Type-Level Rubrics from Expert Reports to Evaluate Deep Research Agents

    Oct 27, 2026Delin Wang, Sinuo Wang, Rui Sun +5Agent EvaluationRubric-Based Evaluation

  12. Benchmarking Task Takeover Across Agent Handoffs

    Oct 27, 2026Wenyuan Li, Guang Li, Dongyuan Li +3LLM Agent EvaluationAgent Evaluation

  13. PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers

    Oct 23, 2026Siyang Wu, Weizhi Zhang, Chen Yang +34Scientific Agent BenchmarksLLM Agent Evaluation

  14. PersonaCUABench: Benchmarking Computer-Use Agents Serving Diverse Personas

    Oct 16, 2026Haoming Li, Defu Cao, Xu Hu +23Computer-Use Agent EvaluationComputer-Use Agent Benchmarks

  15. Harness-Delta Attribution: Dissecting Gains From Harness Evolution

    Oct 15, 2026Wenxuan Ding, Qin Lu, Changlong Yu +5Agent Harness OptimizationAgent Harness Self-Evolution

  16. Beyond Downstream QA: Rethinking How We Evaluate Agent Memory

    Oct 15, 2026Sebastian-Antonio Toma, Jean de Kernier, Mike Ply +1Agent MemoryLLM Agent Memory

  17. Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

    Oct 15, 2026Jingyu Tang, Xu Yang, Zhehao Zhang +7Agent EvaluationAI Agent Benchmarks

  18. OmicsBench: a long-horizon multi-turn benchmark of AI agents on research-study-scale bioinformatics

    Oct 15, 2026Gideon Shaked, Siddharth NagisettyBioinformaticsAI Agent Benchmarks

  19. Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach

    Oct 14, 2026Jiarui Qin, Junjie Huang, Yunjia Xi +5LLM Agent EvaluationAgent Evaluation

  20. Smart Alone, Blind Together: Why Accuracy Fails to CertifyTheory of Mind in Multi-Agent Teams

    Oct 13, 2026Will Beaumaster, Paul SchraterAI Agent ReliabilityAgent Evaluation

  21. Sensitivity or Criterion? Measuring and Moving When Language Models Act

    Oct 11, 2026Sixun Dong, Yebowen Hu, Ming Yin +3Agent EvaluationLLM Tool Use