AI Agent Evaluation

Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 341

All topics
CardsList

  1. Plausible but Wrong: A case study on Agentic Failures in Astrophysical Workflows

    Apr 28, 2026Shivam Rawat, Lucie FlekAgent Failure AnalysisAI for Science

    Bonn-Aachen International Center for Information Technology, University of Bonn, Germany; Lamarr Institute for Machine Learning and Artificial Intelligence, Germany.

  2. ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable

    Apr 28, 2026Sidi Chang, Peiying Zhu, Yuxiao ChenLLM-as-a-JudgeAI Agent Evaluation

    Blossom AI Labs, Tokyo, Japan · Blossom AI, San Francisco, USA

  3. Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents

    Apr 28, 2026Jianming Chen, Yawen Wang, Junjie Wang +4Agent Failure AnalysisAI Agent Evaluation

    Institute of Software, Chinese Academy of Sciences, Beijing, China · Science & Technology on Integrated Information System Laboratory, Beijing, China · State Key Laboratory of Complex System Modeling and Simulation Technology, Beijing, China +2

  4. Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver

    Apr 27, 2026Joshua Sherwood, Ben Aybar, Benjamin KaplanAI Coding AgentsAI Agent Evaluation

    Department of Computer Science University of Chicago Chicago, IL 60637 · Independent Researcher

  5. Toward a Science of Intent: Closure Gaps and Delegation Envelopes for Open-World AI Agents

    Apr 27, 2026Maximiliano Armesto, Christophe KolbAI Agent EvaluationAI Agent Safety

    Taller Technologies

  6. GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation

    Apr 27, 2026Yunsu Kim, Kaden Uhlig, Joern WuebkerAI Agent EvaluationAI Agent Benchmarks

    LILT

  7. OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents

    Apr 27, 2026Zheng Wu, Yi Hua, Zhaoyuan Huang +8Computer-Use AgentsAI Agent Evaluation

    School of Computer Science, Shanghai Jiao Tong University

  8. AgentPulse: A Continuous Multi-Signal Framework for Evaluating AI Agents in Deployment

    Apr 27, 2026Yuxuan Gao, Megan Wang, Yi Ling YuAI Agent EvaluationAI Agent Benchmarks

    University of Pennsylvania · 2Columbia University · 3OpenMesh AI

  9. Failure-Centered Runtime Evaluation for Deployed Trilingual Public-Space Agents

    Apr 27, 2026M. MengAgent Failure AnalysisAutomated Software Testing

    Shenzhen The Nine’s Light Technology Co., Ltd., Shenzhen, Guangdong, China

  10. AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking

    Apr 26, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuAgent Failure AnalysisAI Agent Evaluation

    The University of Hong Kong · Stellaris AI Limited

  11. Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents

    Apr 24, 2026Xirui Li, Ming Li, Yunze Xiao +4Multi-Agent CoordinationAI Agent Evaluation

    University of Maryland · 2Mohamed bin Zayed University of Artificial Intelligence · 3Carnegie Mellon University

  12. AgentSearchBench: A Benchmark for AI Agent Search in the Wild

    Apr 24, 2026Bin Wu, Arastun Mammadli, Xiaoyu Zhang +1Agentic SearchAI Agent Evaluation

    Centre for Artificial Intelligence, University College London

  13. MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills

    Apr 22, 2026Yingyong Hou, Xinyuan Lao, Huimei Wang +10HealthcareAI-Assisted Scientific Research

    Department of Pathology, Zhongshan Hospital, Fudan University, Shanghai, China · 1AIPOCH PTE. LTD., Singapore

  14. Information Aggregation with AI Agents

    Apr 21, 2026Spyros GalanisAI Agent EvaluationMulti-Agent Reasoning

    Department of Economics, University of Durham

  15. How Adversarial Environments Mislead Agentic AI?

    Apr 20, 2026Zhonghao Zhan, Huichi Zhou, Zhenhao Li +3Adversarial RobustnessAI Agent Evaluation

    Imperial College London

  16. AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

    Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-JudgeLLM Agent Evaluation

    University of Science and Technology of China · National University of Singapore · Meituan

  17. CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks

    Apr 20, 2026Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3AI Agent EvaluationHuman-in-the-Loop Evaluation

    Stanford University · Carnegie Mellon University · University of Rochester +1

  18. HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

    Apr 20, 2026Hanhua Hong, Yizhi LI, Jiaoyan Chen +4Multi-Agent LLM SystemsCode Generation

    The University of Manchester · Institute for Infocomm Research (I²R), A*STAR · ELLIS Manchester +1

  19. Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

    Apr 18, 2026Christopher Koch, Joshua Andreas WellbrockLLM Agent OrchestrationAI Agent Evaluation

    Independent Researcher

  20. Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents

    Apr 18, 2026Sukai Huang, Chenyuan Zhang, Fucai Ke +4AI Agent EvaluationAI Agent Benchmarks

    Faculty of Information Technology, Monash University

  21. PersonalHomeBench: Evaluating Agents in Personalized Smart Homes

    Apr 18, 2026Manasa Bharadwaj, Yolanda Liu, InJung Yang +5AI Agent EvaluationMultimodal Agents

    LG Toronto AI Lab · LG Electronics, Korea.

  22. Agentic Frameworks for Reasoning Tasks: An Empirical Study

    Apr 17, 2026Zeeshan Rasheed, Abdul Malik Sami, Muhammad Waseem +3LLM Agent OrchestrationAI Agent Evaluation

    *Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland.

  23. MOSAIC: A Universal Agent-Level Interface for Cross-Paradigm Agent Mixing and Human-AI Collaboration

    Mar 1, 2026Abdulhamid M. Mousa, Jinhui Pang, Rakhmonberdi Khajiev +5AI Agent EvaluationAgent Evaluation

    School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Mathematics Department, Faculty of Science, Ain Shams University, Cairo, Egypt +1

  24. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

    Feb 26, 2026Abhay Sheshadri, Aidan Ewart, Elias Kempf +8Benchmark AuditingAI Agent Evaluation

    Anthropic Fellows Program · Anthropic

  25. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

    Feb 15, 2026Lingxiang Hu, Yiding Sun, Tianle Xia +6Online AdvertisingLLM Agent Evaluation

    Tencent

  26. JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

    Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-JudgeAI Agent Evaluation

    Alibaba International Digital Commerce Group · Zhejiang University · Peking University

  27. Persuasion Propagation: Does Persuasion Change AI Agent Behavior?

    Jan 31, 2026Hyejun Jeong, Amir Houmansadr, Shlomo Zilberstein +1Persuasion in Language ModelsAI Agent Evaluation

    University of Massachusetts Amherst

  28. LogicEnvGen: Task-Logic Driven Generation of Diverse Simulated Environments for Embodied AI

    Jan 20, 2026Jianan Wang, Siyang Zhang, Bin Li +4Robotics SimulationAI Agent Evaluation

    College of Computer Science and Technology, National University of Defense Technology · Intelligent Game and Decision Lab (IGDL), Beijing · School of Artifical Intelligence, Shanghai Jiao Tong University

  29. DR-Arena: an Automated Evaluation Framework for Deep Research Agents

    Jan 15, 2026Yiwen Gao, Ruochen Zhao, Yang Deng +1Deep Research AgentsBenchmark Design

    National University of Singapore · Nanyang Technological University · Singapore Management University +1

  30. Monte Carlo Query Search: Active Capability Assessment of AI Agents

    Dec 18, 2025Daniel Bramblett, Rushang Karia, Adrian Ciotinga +3AI Agent EvaluationActive Learning

    Arizona State University, AZ, USA · Indian Institute of Technology Madras, India

  31. Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents

    Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent EvaluationMultimodal Large Language Models

    Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2

  32. TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward

    Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Reward ModelingLLM Planning

    Trip.com Group · Chongqing University · Stanford University

  33. AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance

    Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent OrchestrationAI Agent Evaluation

    IBM · University of South Carolina · Steel Authority of India +1

  34. CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

    Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent EvaluationAI Agent Benchmarks

    Princeton University

  35. When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments

    Jul 15, 2024Chong Zhang, Xinyi Liu, Zhongmou Zhang +10Multi-Agent LLM SystemsHuman Behavior Simulation

    University of Liverpool, UK · Peking University, China · Shanghai University of Finance and Economics, China +6

  36. The Agent Incident Registry: Toward Preventing Repeated AI Agent Failures

    Date pendingDivyanshu Kumar, Rohith HN, Nitin Aravind Birur +2Agent Failure AnalysisAI Agent Evaluation

    Anaconda

  37. SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?

    Date pendingYuqiao Tan, Shizhu He, Jun Zhao +1AI Agent EvaluationSparse Autoencoders

    The Key Laboratory of Cognitive Intelligence, Institute of Automation, CAS · School of Artificial Intelligence, University of Chinese Academy of Sciences

  38. An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc

    Date pendingHong Zhang, Barry Smith, Satish Balay +4High-Performance ComputingCode Generation

    Argonne National Laboratory, Lemont, IL, USA

  39. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    Date pendingWilliam CabanInter-Rater ReliabilityAI Agent Evaluation

    Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria

  40. EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

    Date pendingTara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz +10Voice Agent EvaluationSpoken Dialogue Systems

    ServiceNow AI Research

  41. Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation

    Date pendingLin Shi (Audrey), Haowei Lin (Audrey), Zixuan Zhu (Audrey) +123LLM Agent EvaluationAI Agent Evaluation