Automated Agent Evaluation

Latest papers

76 papers24 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement
Nov 5, 2026Wenbo Huang, Jianxin Bi, Shunyu Yang +10Self-Improving Agents Benchmark Contamination
Peking University · National University of Singapore
How Do AI Agents Waste Your Money?
Nov 4, 2026Huimin Zeng, Yuxuan Lu, Jesse Cai +8LLM Agent Efficiency Automated Agent Evaluation
University of Illinois Urbana-Champaign · Northeastern University · Independent Researcher +2
Closed-Loop Agent Evaluation: Layered Measurement and Self-Calibration
Nov 4, 2026Ruifeng Ren, Gun AhnAI Agent Evaluation Automated Agent Evaluation
Massachusetts Institute of Technology, Cambridge, MA, USA
DARE: Difficulty Analysis for Repair Enhancement
Nov 4, 2026Jikwang Lee, Seungwook Jung, Jaechang NamAutomated Program Repair Automated Agent Evaluation
Handong Global University, Pohang, South Korea
Silent Instrument Errors Reorder Agent Leaderboards
Nov 3, 2026Yangze LiuAI Agent Evaluation Automated Agent Evaluation
Shandong University
An Agentic Framework for Automated Power System Studies and Grid Connection Assessment
Nov 2, 2026MaoXun Huang, Kshitij Kapoor, Eli LouisePower Systems AI Agent Benchmarks
Grid Agents
Harder Than the Task: From Technical to Organizational Difficulty in AI Benchmarking
Nov 2, 2026Victor Barres, Aden Barton, Nicolai J. Foss +1AI Agent Benchmarks Automated Agent Evaluation
Mercor · Copenhagen Business School
Parametric CAD Bench v3: Evaluating Agentic Parametric CAD Authoring from Text and Engineering Drawings
Nov 2, 2026Mei Chen, Yu Xia, Haipeng MaoParametric CAD Modeling Text-to-CAD Generation
gNucleus AI, Inc.
CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents
Nov 2, 2026Kristen Pereira, Neelabh Sinha, Rajat Ghosh +1Automated Code Review Automated Agent Evaluation
Nutanix, Inc.
Harvesting SWE Benchmarks from Wild Vibe Coding Sessions
Nov 2, 2026Ao Lu, Yuanhao Zeng, Shaocheng Lan +5Coding Agent Benchmarks Benchmark Construction
ShanghaiTech University · ByteDance, China · Shanghai AI Laboratory
When 200 OK Is Not an Answer: Silent Fetch Failures in Agentic Web Content Acquisition
Oct 31, 2026Shuyang ZhangWeb Agents Automated Agent Evaluation
The Hong Kong Polytechnic University
VeriCUA: Verification as Agentic Exploration for Computer-Using Agents
Oct 29, 2026Qiushi Sun, Da Yu, Alexander Knop +5Computer-Use Agent Evaluation Automated Agent Evaluation
Google · The University of Hong Kong
rTaste: Do Research Agents Have Taste? Evaluating Early Stage Research Judgment
Oct 29, 2026Art KankeAutonomous Research Agents Automated Agent Evaluation
University of Minnesota
DynaRev: Benchmarking Web Agents under Dynamic Goal Revision
Oct 29, 2026Shiqi Zhang, Guochen Zhou, Zesheng Wei +2Web Agent Benchmarks Web Agents
City University of Hong Kong · Shanghai Jiao Tong University
M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Oct 27, 2026Hannes Ill, Rafi Al Attrach, Rajna Fani +5Coding Agent Benchmarks AI Coding Agents
Laboratory for Computational Physiology, Massachusetts Institute of Technology, USA · Department of Surgery, Seoul National University Hospital, Korea · Charité, University Medicine Berlin, Germany +2
LongSearchEval: Evaluating Long-Horizon Web Search Capabilities of Large Language Models via a Trajectory-Aware Agent Framework with Failure-Node Localization
Oct 26, 2026Jialiang Liu, Shusheng Li, Wenjun TanLong-Horizon Agent Evaluation Automated Agent Evaluation
College of Information Science and Engineering, Northeastern University, China · Key Laboratory of Intelligent Computing of Medical Images, Ministry of Education, Northeastern University, China. · School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China +1
NumTrace: Auditing the Number Provenance of Financial Analyst Agents
Oct 26, 2026Tianlei Zhu, Matt White, Xue Liu +1Financial Reasoning Benchmarks Data Provenance
Columbia University · University of California, Berkeley · Linux Foundation +3
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
Oct 25, 2026Jingjie Ning, Xueqi Li, Yi Lu +1LLM Agent Verification Web Agent Benchmarks
Rethinking Verification in Agent Harnesses
Oct 25, 2026Zhengxin Zhang, Erkang Zhu, Yin Lin +3LLM Agent Verification LLM Agent Harnesses
Cornell University · Alibaba Group
SWE-Shift: Evaluating Coding Agents in Evolving Workspaces
Oct 23, 2026Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7Coding Agents Software Engineering Benchmarks
MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University +2
Conservative Meta-Agent Search: Architecture Adaptation under Non-Stationarity
Oct 22, 2026Ali Atiah AlzahraniMulti-Agent System Optimization Self-Evolving Agents
Lifetime-valid decisions by continually evolving AI analysts
Oct 22, 2026Xiaojing Sun, Yuhan Zeng, Xiao WangAutomated Agent Evaluation Sequential Hypothesis Testing
Department of Statistics, Purdue University
RubricCraft: Where Solutions Diverge, Criteria Emerge
Oct 21, 2026Hosein Hasanbeig, Priyanshu Gupta, Ashish Tiwari +2Automatic Rubric Generation Rubric-Based Evaluation
Microsoft
Preference Is Not Permission: Personalizing the Auto-Purchase Line in Agentic Commerce
Oct 21, 2026Ryotaro Shimizu, Sai HtaungKham, Mo FanAgentic Commerce Automated Agent Evaluation
ZOZO Research, Tokyo, Japan
VeeReview: Measuring Build-Verify Pairs in Agent-Authored Pull Requests from a V-Model Perspective
Oct 21, 2026Gabriel Dautovi, Jie JW WuAutomated Code Review Automated Agent Evaluation
Department of Computer Science, Michigan Technological University, Houghton, Michigan, United States
Fig: Automated Multiagent Behavior Elicitor
Oct 20, 2026Anrui Gu, Zhichao Wang, Nicolai OuporovAutomated Agent Evaluation Multi-Agent Systems
Fleet AI, San Francisco
An Explicit Feedback Contract for Iterative ML-Agent Harnesses
Oct 20, 2026Kehui Yao, Chuanbo Hu, Jiao Chen +8Agent Harness Optimization Automated Agent Evaluation
Walmart Global Tech · Lehigh University · Rutgers University
Reusing Multimodal Agents: Efficient Evaluation of Sound Designs in Games
Oct 20, 2026Thai Nguyen Van, Ibrahim Khan, Mury Fajar Dewantoro +3Automated Agent Evaluation Multimodal Agents
Graduate School of Information Science and Engineering, Ritsumeikan University, Japan · College of Information Science and Engineering, Ritsumeikan University, Japan · Independent Researcher +1
VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents
Oct 19, 2026Jianing Yin, Ziyi Chen, Hao Wang +4LLM Agent Verification AI Agent Benchmarks
University of Pennsylvania · University of Illinois at Urbana-Champaign · University of California, Berkeley +2
Self-Consistency of Entity Sets and Citations in a Deep Research System under Repeated Prompting
Oct 19, 2026Lauri Pykälä, Kimmo K. KaskiDeep Research Agents Behavioral Consistency of LLM Agents
Department of Computer Science, Aalto University School of Science, P.O. Box 11000, 00076 Aalto, Finland

Nov 5, 2026J/K move · Enter open · S save

Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement

Wenbo Huang, Jianxin Bi, Shunyu Yang, Ruiyang Si+9

Peking University · National University of Singapore

Self-Improving Agents Benchmark Contamination Automated Agent Evaluation Coding Agent Benchmarks Embodied Agent Benchmarks

Open paper AskPDF arXiv

Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.