AI Agent Evaluation

Follow this topic Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 341

All topics

Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations
Aug 11, 2026Vasundra SrinivasanAI Agent Reliability AI Agent Evaluation
Independent Researcher, Stanford School of Engineering
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?
Aug 11, 2026Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub +3Computer-Use Agent Benchmarks AI Agent Evaluation
York University · Nanyang Technological University · Salesforce AI Research
Neuroevolution Arena: Nested Ecological Evaluation of Update-and-Inheritance Regimes across Neural Architectures
Aug 10, 2026Yuxu Ge, Yifei ChengReinforcement Learning Artificial Life
University of York, United Kingdom
The CASE Framework: A Multi-Disciplinary Control Architecture for Governing Enterprise Agentic AI
Aug 10, 2026Srinivas Telukunta, Georgios Nektarios Lilis, Lucio BaronAI Agent Evaluation AI Agent Governance
Cornell University · Johns Hopkins University · AI71
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
Aug 10, 2026Shulin Tian, Ziqi Huang, Fan Zhang +3Tool-Augmented Language Model Agents AI Agent Evaluation
Shanghai Artificial Intelligence Laboratory
Software Engineering for and with GUI Agent
Aug 10, 2026Shengcheng Yu, Yuchen Ling, Junyang Xing +3AI Agent Reliability GUI Agents
State Key Laboratory for Novel Software Technology, Nanjing University, China · Technical University of Munich, Germany
Evo-Bench: Can Language Models Improve Agent Harness?
Aug 10, 2026Lisheng Huang, Chen Yang, Hao Zhou +6AI Agent Evaluation Long-Horizon Agent Evaluation
1Gaoling School of Artificial Intelligence, Renmin University of China · 2BOSS Zhipin, Beijing, China
Causal Behavioral Evaluation of AI Agents at Scale via Automated Behavioral Science
Aug 10, 2026Soo Yong Lee, Jongha Lee, Jaewan Chun +8AI-Assisted Scientific Research AI Agent Evaluation
KAIST, Kim Jaechul Graduate School of AI · Yale, Department of Psychiatry · University of Haifa, School of Public Health +3
Findings of the First Teaching Monster Challenge: A Benchmark of Pedagogical Content Knowledge in AI Agents
Aug 9, 2026Yi-Cheng Lin, Yu-Kai Guo, Szu-Chi Chen +15AI in Education AI Agent Evaluation
National Taiwan University · NTU AI Center of Research Excellence
OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents
Aug 9, 2026Andrea Caciolai, Pere-Lluís Huguet Cabot, Chierh Cheng +11AI Agent Evaluation AI Agent Benchmarks
Meta Superintelligence Labs
ForestBench: A Unified Graph Framework for Evaluating Multi-Agent Collaboration
Aug 9, 2026Guo Chen, Ziwen Li, Reed Li +4Multi-Agent LLM Systems AI Agent Evaluation
Southwest University Chongqing, China · Tencent Beijing, China · Institute of Computing Technology, Chinese Academy of Sciences Beijing, China
Not Worth Another Token: Marginal Value Estimation for Efficient Deep Research Agents
Aug 9, 2026Harshitha Kolukuluru, Reshma Ashok, Kirat Arora +7Deep Research Agents AI Agent Evaluation
University of Massachusetts Amherst · The University of Texas at Austin · Adobe Research
AndroidReality: How Far Are Mobile Agents from the Real World?
Aug 7, 2026Xiaoou Liu, Longchao Da, Hanyang Chen +2Agent Reliability GUI Agents
Arizona State University · Independent Researcher
Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability
Aug 6, 2026Guanghui Min, Liang Wu, Mayank Darbari +2AI Agent Evaluation Long-Horizon LLM Agents
Department of Computer Science, University of Virginia, Charlottesville, VA, USA · Nokia, Sunnyvale, CA, USA
When Agentic AI Meets Integrated Sensing and Communication
Aug 6, 2026Kai Li, Conggai Li, Sarah Ali Siddiqui +4Integrated Sensing and Communication AI Agent Evaluation
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg · Technology, CSIRO, Sydney, NSW, Australia · College of Computer, Qassim University, Buraydah, Saudi Arabia +1
ASTELD: A Six-Axis Classification Framework for Autonomous AI Agents - Design, Evaluation, and an OpenClaw Case Study
Aug 5, 2026Siyuan Li, Peng Shu, Churan Yu +19AI Agent Evaluation Agentic AI
School of Computing, University of Georgia, Athens, GA, USA · Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA · College of Engineering, University of Georgia, Athens, GA, USA +6
MatrAIx: Simulating the World with 8.3 Billion Persona Agents
Aug 4, 2026Xiaomin Li, Yuexing Hao, Jianheng Hou +90Human Behavior Simulation AI Agent Evaluation
FinProBench: Evaluating Financial AI Agents with Role-Grounded Rubrics Derived from Professional Deliverables
Aug 4, 2026Ben Wang, Kang Zhou, Lifan Guo +2AI Agent Evaluation AI Agent Benchmarks
Qwen DianJin Team, Alibaba Cloud Computing
Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details
Aug 4, 2026Maksymilian Wolski, Nicholas Hoernle, Johannes Forkel +1Multi-Agent Coordination AI Agent Evaluation
ProrokLab University of Cambridge United Kingdom · QuantCo · BOLD† University of Oxford United Kingdom
When Truth Is Distributed: Misinformation Derails Collective Fact Recovery in LLM-Based Multi-Agent Systems
Aug 4, 2026Chenfei Yan, Zeyang Yue, Feifei Zhao +6AI Agent Evaluation Multi-Agent Systems
Institute of Automation, Chinese Academy of Sciences · Beijing Institute of AI Safety and Governance · School of Artificial Intelligence, Beihang University +3
Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation
Aug 4, 2026Saqib Shouqi, Abdullah Nazly, Januki Wanniarachchi +1Adversarial Attacks on LLMs LLM Agent Evaluation
School of Computing, Informatics Institute of Technology, Colombo, Sri Lanka
Agentic Commerce World: An Auditable and Verifiable Environment for Vibe Commerce
Aug 3, 2026Shicheng Fan, Mingdai Yang, Duohao Wang +9AI Agent Evaluation Agent Evaluation
University of Illinois at Chicago · 2Springbrand · 3Northwestern University +6
Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation
Aug 3, 2026Stefan Hut, Lorenzo MasoeroCausal Effect Estimation AI Agent Evaluation
Amazon
What Could the Agent See at 19:05? Generating Temporal Enterprise Scenarios from Real Research and Replaying Them to Evaluate Agents
Aug 2, 2026Tezan Sahu, Himani AroraAI Agent Evaluation AI Agent Benchmarks
Microsoft Hyderabad, India
Auditing Discovery Claims: A Two-Sided Criterion for Agentic Science, with the Negative Side Decidable
Aug 2, 2026Wenhui Chen, Jianlin Chen, Ziyao Lin +1Reward Hacking AI for Science
University of Macau · 2South China University of Technology
AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
Aug 1, 2026Meher Bhaskar Madiraju, Meher Sai Preetam MadirajuAI Agent Evaluation Agent Evaluation
FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction
Aug 1, 2026Chaoqun Yang, Fengbin Zhu, Xinyu Lin +5Quantitative Finance Deep Research Agents
School of Computing, National University of Singapore · Institute of Computing Technology, Chinese Academy of Sciences · China Economics and Management Academy, Central University of Finance and Economics +1
Beyond Component Testing: Validating Agentic AI Systems
Jul 31, 2026Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi +4AI Agent Evaluation AI Agent Safety
Department of Biomedical, Dental, and Morphological and Functional Imaging Sciences, University of Messina, A.O.U. Policlinico “G.Martino” - Via Consolare Valeria, Messina, 98125, Italy. · Department of Engineering, University of Messina, Contrada di Dio, Sant’Agata, Messina, 98158, Italy. · Institute for High Performance Computing and Networking of National Research Council of Italy, ICAR-CNR, Via Pietro Castellino 111, Naples, 80131, Italy. +2
Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds
Jul 31, 2026Yuxuan Liu, Zhaochen Su, Yuhao Zhang +9AI Agent Evaluation Agent Skill Learning
The Hong Kong University of Science and Technology · Harbin Institute of Technology, Shenzhen · Harbin Institute of Technology +1
InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk
Jul 31, 2026Yuan Gao, Zeren Yang, Junnan Li +6AI Agent Reliability AI Agent Evaluation
Wanxiang · University of Wisconsin–Madison · 2Iowa State University

Aug 11, 2026, cs.AIJ/K move · Enter open · S save

Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations

Vasundra Srinivasan

Independent Researcher, Stanford School of Engineering

AI Agent Reliability AI Agent Evaluation Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany,

τ^2

-bench, and AppWorld), that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7-23%. Leaderboards rank specialization, not capability. We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. Four further findings sharpen what the leaderboard is hiding. First, aggregate reliability collapses on the hardest task quartile:

Eρ^2

τ^2

action_checks falls from 0.752 to 0.000. Second, training-cell reliability negatively correlates with held-out reliability (

r = -0.90

τ^2

), meaning the designs that look most reliable replicate worst. Third, population-level diagnostics transfer across enterprise benchmarks (capability-gap ratio stable at 0.35-0.40) but per-family agent rankings invert. Fourth, on the MAST failure taxonomy, trace-level mode profiles are idiosyncratic (MAE = 0.261) while cell-level profiles generalise (MAE = 0.056,

r = 0.83

). We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend. All code, data loaders, and fit artifacts are released under an open-source license.

Momentum

Related topics

Latest papers 341