AI Agent Evaluation

Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 344

All topics
CardsList

  1. Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses

    Sep 29, 2026Ziluowen Luo, Senzhang Wang, Chaozhuo Li +9AI Agent EvaluationKnowledge Distillation

    Central South University, ChangSha, Hunan, China · Beijing Academy of Artificial Intelligence, Beijing, China · Hong Kong Polytechnic University, Hongkong, China +2

  2. PADMÉ: Preference Alignment Data Synthesis for Meta-Evaluation of LM Agent Evaluators

    Sep 28, 2026Cheng Chang, Yining Mao, Peng QiHuman Preference EvaluationLLM Evaluation

    Uniphore

  3. From Migration to Calibration: Preserving Agent Capabilities across Models, Jurisdictions, and Scale

    Sep 28, 2026Yaxiao Liu, Pengbo Liu, Yiwen Liu +2Domain AdaptationAI Agent Evaluation

    PwC China AI Center · Tsinghua University

  4. Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance

    Sep 28, 2026Dipankar SarkarLLM GuardrailsAI Agent Evaluation

    Skelf Research

  5. BIABench: Evaluating AI agents on real-world bioimage analysis tasks

    Sep 28, 2026Zixuan Pan, Davide Panzeri, Lukas Johanns +5AI Agent ReliabilityAI Agent Evaluation

    Department of Computer Science and Engineering, University of Notre Dame, Notre Dame, USA · Leibniz-Institut für Analytische Wissenschaften – ISAS – e.V., Dortmund, Germany · Institute of Computer Science, University of Tartu, Tartu, Estonia

  6. GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions

    Sep 28, 2026Shaoqing Zhang, Kehai Chen, Xuefeng Bai +4Agent Failure AnalysisGUI Agents

    Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Shanghai Jiao Tong University, China

  7. SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

    Sep 27, 2026Xiaonan Luo, Yue Huang, Kehan Guo +9Software Vulnerability DetectionAI Agent Evaluation

    University of Notre Dame · Bake AI · Vanderbilt University +5

  8. What Happens During Autonomous Deep Research After the User Steps Away?

    Sep 27, 2026Yimin Liu, Yijia Zhang, Yanmin Li +4Counterfactual EvaluationDeep Research Agents

    University of Southern California · University of Michigan, Ann Arbor · Institute of Automation, Chinese Academy of Sciences +3

  9. Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

    Sep 27, 2026Yixuan LiuProbability CalibrationSelective Prediction

    TraceStone and Nanyang Technological University, Singapore

  10. ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

    Sep 24, 2026Ming Zhang, Zhenghao Xiang, Peizhong Gao +17AI Agent EvaluationAI Agent Benchmarks

    Fudan University · Hunyuan Team, Tencent

  11. Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

    Sep 24, 2026Tapan ParikhLLM SycophancyAI Agent Evaluation

    Cornell Tech

  12. PUBG Ally: A Conversational Embodied Agent as an AI Teammate

    Sep 24, 2026Beomsoo Kim, Byeongju Kim, Dohyun Kim +21Game-Playing AgentsAI Agent Evaluation

  13. Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations

    Sep 24, 2026Leon Goldberg, Gal Engelberg, Eden Yavin +3AI Coding AgentsAI Agent Evaluation

    Sola Security, Tel Aviv, Israel

  14. Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

    Sep 24, 2026Shuzhi Gong, Fengze Sun, Yuansan LiuTemporal Video GroundingVideo-Language Model Evaluation

    The University of Melbourne

  15. LabFactory: Building and Evaluating Executable AI Labs

    Sep 23, 2026Jinge Wu, Hongjian Zhou, Mingde Zeng +6AI-Assisted Scientific ResearchAI Agent Evaluation

    University of Oxford

  16. WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

    Sep 23, 2026Jingjie Ning, Xueqi Li, Yibo Kong +1AI Agent EvaluationActive Learning

    Carnegie Mellon University · Tsinghua University

  17. Guides That Cause Actions: An Offline Study of Guide-Action Mutual Reinforcement in Multimodal Web Agents

    Sep 23, 2026Chengguang Gan, Yunhao Liang, QingHao Zhang +1Web Agent BenchmarksAI Agent Evaluation

    Independent Researcher

  18. Behavior is Not Enough: A Mechanism-Based Evaluation of Social Norm Emergence in LLM Societies

    Sep 22, 2026Rasika Muralidharan, Haewoon Kwak, Jisun AnMulti-Agent CoordinationAI Agent Evaluation

    Indiana Univeristy Bloomington

  19. ShowTellArena: Evaluating Business Workflow Understanding from Demonstrations

    Sep 21, 2026David Garg, Ritobrata Sarkar, Ehsan Azarnasab +1AI Agent EvaluationAI Agent Benchmarks

    Brackett Labs

  20. XYEval: Agents say yes to bad advice

    Sep 20, 2026Zhengxuan Wu, Yuxuan Li, Oyvind Tafjord +1LLM SycophancyAI Agent Evaluation

    Google DeepMind

  21. DENSE: Distilling Agent Trajectories into Evidence-Grounded Shortcut Trees for Self-Refinement

    Sep 18, 2026Siyuan Liu, Fan Yu, Dongyu Ru +5LLM Self-RefinementAI Agent Evaluation

    Fudan University · Meituan Longcat Team

  22. Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

    Sep 17, 2026Sho Kawano, Zehang Richard Li, Paul A. ParkerAI Agent EvaluationPrediction-Powered Inference

    Department of Statistics, University of California, Santa Cruz, CA, USA

  23. DeltaSelect: Affordable A/B Testing for Coding Agents

    Sep 17, 2026Nicholas J. ConnAI Coding AgentsBenchmark Design

    Conn Castle Studios

  24. A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

    Sep 17, 2026Shaina Raza, Ahmed Y. Radwan, Imran Liaquat +1LLM EvaluationAI Agent Evaluation

    Vector Institute for Artificial Intelligence, Toronto, ON M5G 0C6, Canada

  25. Do AI Agents Understand Computer Architecture?

    Sep 16, 2026Ambika Sharan, Grigory Chirkov, Soheil AbbaslooElectronic Design AutomationAI Agent Evaluation

    Microsoft Research

  26. Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

    Sep 16, 2026Xinshuai Guo, Junjie Wu, Dolly Deng +4AI Agent EvaluationAI Agent Benchmarks

    Hunyuan Team, Tencent · Tsinghua University

  27. The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information

    Sep 14, 2026Ivy ZhangAgent Failure AnalysisSoftware Engineering Agents

    Apart Research

  28. Empirical Evaluation of Task-Based Permission Scoping Architecture for AI Agents

    Sep 14, 2026Halil Burak NoyanAI Agent EvaluationAI Agent Security

    Independent Researcher

  29. When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

    Sep 14, 2026Stefan G. Creadore, Peyton WoakzAlgorithmic AuditingBenchmark Auditing

    Tampa, FL, USA

  30. WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

    Sep 14, 2026Amey Varhade, Ananya Sutradhar, Ravishankar Krishnaswamy +1Synthetic Data GenerationAI Agent Evaluation

    Microsoft Research India