AI Agent Evaluation

Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 341

All topics
CardsList

  1. Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

    Sep 12, 2026Mia Lassiter, Brinnae BentAI Agent EvaluationAI Agent Benchmarks

    Duke University

  2. Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents

    Sep 11, 2026Toshiaki Koike-Akino, Vladislav Blaykhman, Ye Wang +2LLM Self-RefinementLLM-as-a-Judge

    Mitsubishi Electric Research Laboratories (MERL) 202 Broadway, Cambrdige, MA 02139

  3. Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers

    Sep 11, 2026Hanhua Hong, Yizhi Li, Luu Gia Huy +3AI Agent EvaluationAgent Evaluation

    The University of Manchester · IQuest Research · Vietnam National University +2

  4. The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation

    Sep 10, 2026Makoto Fukushima, Hua-Dong Xiong, Ehsan Moradi PariGame-Playing AgentsAI Agent Evaluation

    Honda Research Institute Japan · Honda Research Institute USA

  5. OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

    Sep 10, 2026Aayam Bansal, Keertan BalajiAI for ScienceAI Agent Auditing

  6. AgentAudit: An Open, Extensible Framework for Full-Lifecycle Trust Evaluation of AI Agents

    Sep 9, 2026Shrey Nag, Sachita, Abhishek Kumar Singh +2AI Agent ReliabilityAI Agent Auditing

    TIET · IGDTUW · GTBIT +1

  7. Safe to Stop? Risk-Constrained Stopping for Sequential Clinical Diagnosis Agents

    Sep 9, 2026Yuexin Wu, Vasile RusSelective PredictionAI Agent Evaluation

    Department of Computer Science University of Memphis

  8. AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

    Sep 7, 2026Yunxiang Mo, Tianshi Zheng, Yisen Gao +7AI for ScienceAI Agent Evaluation

    Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China · NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA

  9. Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

    Sep 7, 2026Jingjie Ning, Shanshan Zhong, Xiaochuan Li +1AI Agents for Scientific DiscoveryAlgorithmic Auditing

    School of Computer Science, Carnegie Mellon University

  10. PatchBench: Evaluating AI Agents for Vulnerability Patching

    Sep 3, 2026Chihao Shen, Jiacheng Li, Aastha Mahajan +3Automated Program RepairNeural Network Memorization

    University of Maryland

  11. Interface-Induced Trajectory Censoring

    Sep 3, 2026Wenbo WangTool-Use EvaluationAI Agent Evaluation

    City University of Hong Kong

  12. READY or Not: Reliable Enterprise Agent Deployment

    Sep 2, 2026Veronica Chatrath, Bryan Zhu, Jingxuan Fan +15AI Agent Evaluation

    Christy · Scale AI · University of California, Santa Cruz +1

  13. Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

    Sep 1, 2026Kefeng Duan, Dewu Zheng, Yanlin Wang +7Software Engineering BenchmarksLLM Agent Evaluation

    School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China

  14. Disclosure-Gated User Simulation for Companion-Agent Evaluation

    Sep 1, 2026Yao Liu, Yu HeHuman Behavior SimulationAI Agent Evaluation

  15. Agentic Empirical Asset Pricing: Methodological Foundations

    Sep 1, 2026Yingjian Pan, Xiaowei Ding, Kay GieseckeQuantitative FinanceAI Agent Evaluation

    Advanced Financial Technologies Laboratory · Stanford University · School of Information Management +1

  16. Learning to Evaluate Before Improving: Automatic Rubric Induction for Automatic Research Agents

    Aug 31, 2026Xuehai Wang, Haowei Qin, Tongxin Liu +6AI Agent EvaluationAutonomous Scientific Discovery

    1Zhejiang University · University of Electronic Science and Technology of China · 3Beijing University of Posts and Telecommunications +1

  17. MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents

    Aug 31, 2026Vernon Toh, Navonil Majumder, Zhengyuan Liu +2Active PerceptionAI Agent Evaluation

    DeCLaRe Lab, Nanyang Technological University, Singapore · Agency for Science, Technology, and Research (A*STAR), Singapore

  18. Selection-Aware Stress Testing for Interactive Agents

    Aug 31, 2026Yang Xu, Chenang Li, Jiefu Zhang +3AI Agent EvaluationMultiple Hypothesis Testing

    Purdue University · University of California, Irvine

  19. BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks

    Aug 31, 2026Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs +3Reward HackingAI Agent Evaluation

    National University of Singapore · MIT · Vrije Universiteit Amsterdam +3

  20. Science sandboxes measure the scientific capability of AI agents

    Aug 31, 2026Arya S. Rao, Rodrigo I. Castro, Sager J. Gosai +8Protein Fitness PredictionAI Agent Evaluation

    The Broad Institute of MIT and Harvard; Cambridge, MA 02142, USA. · Department of Biomedical Informatics, Harvard Medical School; Boston, MA 02115, USA. · The Jackson Laboratory; Bar Harbor, ME 04609, USA. +10

  21. EDGE: Engine for Deterministic Graph Evaluation through Conversation Simulation from Graph Structured DSL Configuration

    Aug 30, 2026Ram Kulathumani, Pushkar Nagar, Regunathan Radhakrishnan +6AI Agent ReliabilityLLM Agent Evaluation

    Salesforce, Palo Alto, USA.

  22. Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment

    Aug 30, 2026Zhiyu Chen, Keyu Zhao, Jigao Fu +8AI Agents for Scientific DiscoveryLLM-as-a-Judge

    1Fudan University · 3Tsinghua University · 4Zhongguancun Institute of Artificial Intelligence +1

  23. FrontierChallenge: Evaluating Scientific Workflow Completion

    Aug 25, 2026Liangcai Su, Zhaopeng Feng, Zhuo Chen +14AI Agent EvaluationScientific Workflow Automation

    Apodex Team

  24. CatchBench: When Can an Agent Failure Be Caught?

    Aug 24, 2026Yue Zhao, Mengyuan Li, Ruolin Li +5Agent Failure AnalysisBenchmark Auditing

    University of Southern California · Carnegie Mellon University · AWS Agentic AI +3

  25. What is Missing from AI Post-Training AI: An Empirical Analysis

    Aug 19, 2026Joy Jia Yin Lim, Xin Huang, Hao Peng +5AI Agent EvaluationLLM Agent Self-Improvement

    Tsinghua University · University of Electronic Science and Technology of China · Renmin University of China

  26. Science Done on a Machine by a Machine: AI Agents in Computational Chemistry

    Aug 19, 2026Pavlo O. Dral, Hassan Nawaz, Arif UllahAI Agents for Scientific DiscoveryAI Agent Evaluation

    State Key Laboratory of Physical Chemistry of Solid Surfaces, College of Chemistry and Chemical Engineering, and Fujian Provincial Key Laboratory of Theoretical and Computational Chemistry, Xiamen University, Xiamen, Fujian 361005, China · Institute of Physics, Faculty of Physics, Astronomy, and Informatics, Nicolaus Copernicus University in Toruń, ul. Grudziadzka 5, 87-100 Toruń, Poland · Institute of Advanced Studies, Nicolaus Copernicus University in Toruń, ul. Wileńska 4, 87-100 Toruń, Poland +2

  27. DiG-bench: Discovery in Games

    Aug 12, 2026Ruairidh M. Battleday, Kai Sandbrink, Jimi Cullen-Drohan +13Benchmark DesignAI Agent Evaluation

    1Thinking About Thinking · 2Independent · University of Oxford +5

  28. ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

    Aug 12, 2026Yutao Mou, Pengfei Yang, Zhe Yin +6AI Agent EvaluationAdversarial Scenario Generation

    National Engineering Research Center for Software Engineering, Peking University · Harbin Institute of Technology · Beijing University of Posts and Telecommunications +1

  29. FrontierFinance: A Challenging Benchmark for Measuring Frontier Intelligence of Finance Agents

    Aug 12, 2026Yuhao Zhang, O. Ozan Koyluoglu, Thejas Venkatesh +4AI Agent EvaluationAI Agent Benchmarks

    Samaya AI

  30. Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

    Aug 11, 2026Brian Wang, Bin Feng, Xiaoman Pan +26AI for ScienceAI Agent Evaluation