AI Agent Evaluation

Follow this topic Ask about this topic

Momentum

58 papers in the last four weeks, up 100% on the four weeks before. 0.6% of all new papers.

Jul 13Week of Sep 28

Latest papers 344

All topics

Advancing Mathematics Research with AI-Driven Formal Proof Search
May 21, 2026George Tsoukalas, Anton Kovsharov, Sergey Shirobokov +18Automated Theorem Proving AI Agent Evaluation
1Google DeepMind1 · 2Aarhus University · 3Google
Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
May 21, 2026Sahar Abdelnabi, Chris Hicks, Konrad Rieck +1AI Agent Evaluation AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center, Germany · The Alan Turing Institute, London, UK · BIFOLD & Technische Universität Berlin, Germany +1
Towards Direct Evaluation of Harness Optimizers via Priority Ranking
May 21, 2026Kai Tzu-iunn Ong, Minseok Kang, Dongwook Choi +11AI Agent Evaluation Agent Harness Optimization
Yonsei University · Microsoft Research Asia · Texas A&M University
Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work
May 20, 2026Haiyang Shen, Jiuzheng Wang, Taian Guo +9AI Accountability Benchmark Construction
1Peking University · 2Advanced Institute of Big Data
Open-World Evaluations for Measuring Frontier AI Capabilities
May 19, 2026Sayash Kapoor, Peter Kirgis, Andrew Schwartz +15AI Agent Evaluation Long-Horizon Agent Evaluation
Princeton University · Cornflower Labs · Meridian Labs +10
What Do Evolutionary Coding Agents Evolve?
May 19, 2026Nico Pelleriti, Sree Harsha Nelaturu, Zhanke Zhou +4Evolutionary Optimization AI Coding Agents
1Zuse Institute Berlin · 2Technical University of Berlin · 3Hong Kong Baptist University +1
Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study
May 19, 2026Priyansh Trivedi, Olivier SchmittCoding Agents AI Agent Evaluation
SonarSource
Distribution-Free Uncertainty Quantification for Continuous AI Agent Evaluation
May 19, 2026Yuxuan Gao, Megan Wang, Yi Ling YuUncertainty Quantification AI Agent Evaluation
OpenMesh · University of Pennsylvania · Columbia Univer
OpenComputer: Verifiable Software Worlds for Computer-Use Agents
May 19, 2026Jinbiao Wei, Qianran Ma, Yilun Zhao +4Computer-Use Agent Benchmarks Computer-Use Agents
Yale NLP Lab · University of Pennsylvania · University of North Carolina at Chapel Hill
DexHoldem: An Agentic Robotics Benchmark for Dexterous Manipulation in Texas Hold'em
May 18, 2026Feng Chen, Tianzhe Chu, Li Sun +6Dexterous Manipulation AI Agent Evaluation
Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks
May 18, 2026Yubin Qu, Ying Zhang, Yanjun Zhang +4Computer-Use Agent Benchmarks Coding Agents
Griffith University · Wake Forest University · Nanyang Technological University +2
Interactive Evaluation Requires a Design Science
May 18, 2026Keyang Xuan, Peiyang Song, Pan Lu +10LLM Agent Evaluation AI Agent Evaluation
University of Texas Austin · California Institute of Technology · Carnegie Mellon University +5
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
May 17, 2026Qiran Zou, Hou Hei Lam, Wenhao Zhao +11AI Agents for Scientific Discovery Inference-Time Search
National University of Singapore · Tsinghua University · University of Minnesota +2
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
May 17, 2026Jinhu Qi, Muzhi Li, Jiahong Liu +9AI Agent Reliability AI Agent Evaluation
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?
May 16, 2026Robin-Nico Kampa, Fabian Deuser, Anna Bößendörfer +2AI Coding Agents AI Agent Evaluation
University of the Bundeswehr Munich, Munich, Germany
MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
May 14, 2026Minghao Guo, Qingyue Jiao, Zeru Shi +14Multimodal Memory AI Agent Evaluation
Rutgers · Notre Dame · Princeton +1
Holistic Evaluation and Failure Diagnosis of AI Agents
May 14, 2026Netta Madvil, Gilad Dym, Alon Mecilati +12Agent Failure Analysis AI Agent Evaluation
Deepchecks, Ramat Gan, Israel
Beyond Binary: Reframing GUI Critique as Continuous Semantic Alignment
May 14, 2026Yuchen Sun, Pei Fu, Shaojie Zhang +6GUI Agents AI Agent Evaluation
1Shanghai Jiao Tong University · 2Xiaomi Inc.
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
May 13, 2026Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang +2AI Agents for Scientific Discovery AI Agent Evaluation
New High Energy Theory Center Department of Physics & Astronomy Rutgers University · Faculty of Computing & Data Sciences Boston University
Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
May 12, 2026Miaosen Zhang, Xiaohan Zhao, Zhihong Tan +14Computer-Use Agent Benchmarks Computer-Use Agents
1Southeast University · 2Mohamed bin Zayed University of Artificial Intelligence · 3Wuhan University +2
No More, No Less: Task Alignment in Terminal Agents
May 12, 2026Sina Mavali, David Pape, Jonathan Evertz +5Computer-Use Agent Benchmarks AI Alignment
CISPA Helmholtz Center for Information Security · ELLIS Institute Tübingen and MPI for Intelligent Systems · Tübingen AI Center
Rollout Cards: A Reproducibility Standard for Agent Research
May 12, 2026Charlie Masters, Ziyuan Liu, Stefano V. AlbrechtAI Agent Evaluation Agent Evaluation
Deepflow · Nanyang Technological University
Toward Modeling Player-Specific Chess Behaviors
May 12, 2026Loris Sogliuzzo, Aloïs Rautureau, Eric PietteGame-Playing Agents AI Agent Evaluation
ICTEAM, UCLouvain · Louvain-la-Neuve, Belgium
ABRA: Agent Benchmark for Radiology Applications
May 11, 2026Bulat Maksudov, Vladislav Kurenkov, Kathleen M. Curran +1Radiology Report Generation Healthcare
School of Computing Dublin City University Dublin, Ireland · AXXX · School of Medicine University College Dublin Dublin, Ireland
Generalizing the Turing Test to Interactive Agents
May 11, 2026Daniel Mitropolsky, Riccardo Neumarker, Emanuele Rimoldi +2LLM Evaluation AI Agent Evaluation
MIT · ETH Zurich · EPFL
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
May 11, 2026Pedro Conde, Henrique Branquinho, Valerio Mazzone +3Software Vulnerability Detection AI Agent Evaluation
Ethiack Coimbra, Portugal · Ethiack Porto, Portugal · University of Notre Dame Notre Dame, Indiana, USA
Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability
May 11, 2026Harsh Raj, Niranjan Orkat, Suvrorup Mukherjee +3AI Agent Reliability Agent Reliability
Northeastern University · University of P´ecs · University of Michigan +2
Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation
May 11, 2026Shanshan Gao, Liyi ZhouComputer-Use Agent Benchmarks Benchmark Design
The University of Sydney
Agentic Performance at the Edge: Insights from Benchmarking
May 11, 2026Shiqiang Wang, Herbert WoisetschlägerLLM Agent Evaluation AI Agent Evaluation
University of Exeter · Exeter, United Kingdom · Technical University of Munich +1
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
May 11, 2026Haonan Dong, Qiguan Feng, Kehan Jiang +3AI Alignment AI Agent Evaluation
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Key Laboratory of Machine Perception (Ministry of Education), Peking University · School of Software and Microelectronics, Peking University +1

May 21, 2026, cs.AIJ/K move · Enter open · S save

Advancing Mathematics Research with AI-Driven Formal Proof Search

George Tsoukalas, Anton Kovsharov, Sergey Shirobokov, Anja Surina+17

1Google DeepMind1 · 2Aarhus University · 3Google

Automated Theorem Proving AI Agent Evaluation Lean Theorem Proving LLM Mathematical Reasoning Autonomous Research Agents

Open paper AskPDF arXiv

Large language models (LLMs) increasingly excel at mathematical reasoning, but their unreliability limits their utility in mathematics research. A mitigation is using LLMs to generate formal proofs in languages like Lean. We perform the first large-scale evaluation of this method's ability to solve open problems. Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures, and is being deployed in combinatorics, optimization, graph theory, algebraic geometry, and quantum optics research. A basic agent alternating LLM-based generation with Lean-based verification replicated the Erdős successes but proved costlier on the hardest problems. These findings demonstrate the power of AI-aided formal proof search and shed light on the agent designs that enable it.

Momentum

Related topics

Latest papers 344