AI Agent Evaluation

Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 341

All topics
CardsList

  1. Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance

    Sep 28, 2026Dipankar SarkarLLM GuardrailsAI Agent Evaluation

    Skelf Research

  2. BIABench: Evaluating AI agents on real-world bioimage analysis tasks

    Sep 28, 2026Zixuan Pan, Davide Panzeri, Lukas Johanns +5AI Agent ReliabilityAI Agent Evaluation

    Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA · Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany · Institute of Computer Science, University of Tartu, Tartu, Estonia

  3. GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

    Sep 28, 2026Shaoqing Zhang, Kehai Chen, Xuefeng Bai +4Agent Failure AnalysisGUI Agents

    Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China

  4. SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

    Sep 27, 2026Xiaonan Luo, Yue Huang, Kehan Guo +9Software Vulnerability DetectionAI Agent Evaluation

    University of Notre Dame · Bake AI · Vanderbilt University +5

  5. What Happens During Autonomous Deep Research After the User Steps Away?

    Sep 27, 2026Yimin Liu, Yijia Zhang, Yanmin Li +4Counterfactual EvaluationDeep Research Agents

    University of Southern California · University of Michigan, Ann Arbor · Institute of Automation, Chinese Academy of Sciences +3

  6. Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

    Sep 27, 2026Yixuan LiuProbability CalibrationSelective Prediction

    TraceStone and Nanyang Technological University, Singapore

  7. ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

    Sep 24, 2026Ming Zhang, Zhenghao Xiang, Peizhong Gao +17AI Agent EvaluationAI Agent Benchmarks

    Fudan University

  8. Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

    Sep 24, 2026Tapan ParikhLLM SycophancyAI Agent Evaluation

    Cornell Tech

  9. PUBG Ally: A Conversational Embodied Agent as an AI Teammate

    Sep 24, 2026Beomsoo Kim, Byeongju Kim, Dohyun Kim +21Game-Playing AgentsAI Agent Evaluation

  10. Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations

    Sep 24, 2026Leon Goldberg, Gal Engelberg, Eden Yavin +3AI Coding AgentsAI Agent Evaluation

    Sola Security, Tel Aviv, Israel

  11. Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

    Sep 24, 2026Shuzhi Gong, Fengze Sun, Yuansan LiuTemporal Video GroundingVideo-Language Model Evaluation

    The University of Melbourne

  12. LabFactory: Building and Evaluating Executable AI Labs

    Sep 23, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +6AI-Assisted Scientific ResearchAI Agent Evaluation

    University of Oxford

  13. WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

    Sep 23, 2026Jingjie Ning, Xueqi Li, Yibo Kong +1AI Agent EvaluationActive Learning

    Carnegie Mellon University · Tsinghua University

  14. Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents

    Sep 23, 2026Chengguang Gan, Yunhao Liang, QingHao Zhang +1Web Agent BenchmarksAI Agent Evaluation

    Independent Researcher

  15. Behavior is Not Enough: A Mechanism-Based Evaluation of Social Norm Emergence in LLM Societies

    Sep 22, 2026Rasika Muralidharan, Haewoon Kwak, Jisun AnMulti-Agent CoordinationAI Agent Evaluation

    Indiana Univeristy Bloomington

  16. ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations

    Sep 21, 2026David Garg, Ritobrata Sarkar, Ehsan Azarnasab +1AI Agent EvaluationAI Agent Benchmarks

    Brackett Labs

  17. XYEval: Agents say yes to bad advice

    Sep 20, 2026Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord +1LLM SycophancyAI Agent Evaluation

    Google DeepMind

  18. DENSE: Distilling Agent Trajectories into Evidence-Grounded Shortcut Trees for Self-Refinement

    Sep 18, 2026Siyuan Liu, Fan Yu, Dongyu Ru +5LLM Self-RefinementAI Agent Evaluation

    Fudan University · Meituan Longcat Team

  19. Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

    Sep 17, 2026Sho Kawano, Zehang Richard Li, Paul A. ParkerAI Agent EvaluationPrediction-Powered Inference

    Department of Statistics, University of California, Santa Cruz, CA, USA

  20. DeltaSelect: Affordable A/B Testing for Coding Agents

    Sep 17, 2026Nicholas J. ConnAI Coding AgentsBenchmark Design

    Conn Castle Studios

  21. A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

    Sep 17, 2026Shaina Raza, Ahmed Y. Radwan, Imran Liaquat +1LLM EvaluationAI Agent Evaluation

    Vector Institute for Artificial Intelligence, Toronto, ON M5G 0C6, Canada

  22. Do AI Agents Understand Computer Architecture?

    Sep 16, 2026Ambika Sharan, Grigory Chirkov, Soheil AbbaslooElectronic Design AutomationAI Agent Evaluation

    Microsoft Research

  23. Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

    Sep 16, 2026Xinshuai Guo, Junjie Wu, Dolly Deng +4AI Agent EvaluationAI Agent Benchmarks

    Hunyuan Team, Tencent · Tsinghua University

  24. The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information

    Sep 14, 2026Ivy ZhangAgent Failure AnalysisSoftware Engineering Agents

    Apart Research

  25. Empirical Evaluation of Task-Based Permission Scoping Architecture for AI Agents

    Sep 14, 2026Halil Burak NoyanAI Agent EvaluationAI Agent Security

    Independent Researcher

  26. When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

    Sep 14, 2026Stefan G. Creadore, Peyton WoakzAlgorithmic AuditingBenchmark Auditing

    Tampa, FL, USA

  27. WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

    Sep 14, 2026Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy +1Synthetic Data GenerationAI Agent Evaluation

    Microsoft Research India

  28. GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents

    Sep 14, 2026Umesh Bodhwani, Thanh Tran, Kai WeiLLM-as-a-JudgeLLM Agent Evaluation

    Amazon

  29. DynSTEER: Dynamic Stage-wise Trajectory Evaluation and Execution-time Review for Agents

    Sep 13, 2026Zhichao Shi, Xuhui Jiang, Wenjie Zhang +5LLM Agent EvaluationAI Agent Evaluation

    School of Advanced Interdisciplinary Sciences, UCAS · State Key Lab of AI Safety, Institute of Computing Technology, CAS · DataArc Tech Ltd. +3

  30. Finishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating Challenge

    Sep 12, 2026Yuanchen Bai, Zijian Ding, Angelique TaylorAI Agent ReliabilityAgent Reliability

    Cornell Tech / Cornell University · University of Maryland, College Park · Korea Advanced Institute of Science and Technology