AI Agent Evaluation

Follow this topic Ask about this topic

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

525 papers77 papers in the last four weeks, up 4% on the four weeks before. 0.4% of all new papers.

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks
Apr 20, 2026Yijia Shao, Zora Zhiruo Wang, Neel Ahuja +3AI Agent Evaluation Human-in-the-Loop Evaluation
Stanford University · Carnegie Mellon University · University of Rochester +1
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
Apr 20, 2026Hanhua Hong, Yizhi LI, Jiaoyan Chen +4Multi-Agent LLM Systems Code Generation
The University of Manchester · Institute for Infocomm Research (I²R), A*STAR · ELLIS Manchester +1
Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI
Apr 18, 2026Christopher Koch, Joshua Andreas WellbrockLLM Agent Orchestration AI Agent Evaluation
Independent Researcher
Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents
Apr 18, 2026Sukai Huang, Chenyuan Zhang, Fucai Ke +4AI Agent Evaluation AI Agent Benchmarks
Faculty of Information Technology, Monash University
PersonalHomeBench: Evaluating Agents in Personalized Smart Homes
Apr 18, 2026Manasa Bharadwaj, Yolanda Liu, InJung Yang +5AI Agent Evaluation Multimodal Agents
LG Toronto AI Lab · LG Electronics, Korea.
Agentic Frameworks for Reasoning Tasks: An Empirical Study
Apr 17, 2026Zeeshan Rasheed, Abdul Malik Sami, Muhammad Waseem +3LLM Agent Orchestration AI Agent Evaluation
*Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland.
AsymmetryZero: A Framework for Operationalizing Human Expert Preferences as Semantic Evals
Apr 15, 2026Tadhg Looram, Lucas Nuzzi, Kyle Waters +1Automated Evaluation Human Preference Evaluation
PortexAI · Stanford University
Evaluation of Agents under Simulated AI Marketplace Dynamics
Apr 15, 2026To Eun Kim, Alireza Salemi, Hamed Zamani +1AI Agent Evaluation IR Evaluation
Carnegie Mellon University, Pittsburgh, PA, United States · University of Massachusetts Amherst, Amherst, MA, United States
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
Apr 14, 2026Yizhe Chi, Deyao Hong, Dapeng Jiang +18AI Agent Evaluation AI Agent Benchmarks
Navers Lab, Einsia.AI
AlphaEval: Evaluating Agents in Production
Apr 14, 2026Pengrui Lu, Bingyu Xu, Wenjun Zhang +24AI Agent Evaluation AI Agent Benchmarks
SII · SJTU · GAIR +9
SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents
Apr 13, 2026Daniel Begimher, Cristian Leo, Jack Huang +2Cybersecurity AI Agent Security Benchmarks
Amazon Web Services
ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
Apr 13, 2026Fei Tang, Zhiqiong Lu, Boxuan Zhang +4GUI Agents Mobile GUI Automation
Zhejiang University
Retrieval Is Not Enough: Why Organizational AI Needs Epistemic Infrastructure
Apr 13, 2026Federico Bottino, Carlo Ferrero, Nicholas Dosio +1Knowledge Representation Epistemic Uncertainty
Kakashi Ventures Accelerator (KVA) · Massachusetts Institute of Technology
BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
Apr 13, 2026Elaine Lau, Markus Dücker, Ronak Chaudhary +24AI Agent Evaluation AI Agent Benchmarks
Handshake AI · McGill University
WebForge: Breaking the Realism-Reproducibility-Scalability Trilemma in Browser Agent Benchmark
Apr 13, 2026Peng Yuan, Yuyang Yin, Yuxuan Cai +1Web Agent Benchmarks Benchmark Design
Tencent BAC · Tsinghua University
Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
Apr 11, 2026Miles Q. Li, Benjamin C. M. Fung, Boyang Li +2AI Agent Safety Benchmarks AI Agent Evaluation
School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE
COMPOSITE-Stem
Apr 10, 2026Kyle Waters, Lucas Nuzzi, Tadhg Looram +20Scientific Agent Benchmarks AI Agent Benchmarks
PortexAI · University of Milano-Bicocca · University of Calgary +17
ClawBench: Can AI Agents Complete Everyday Online Tasks?
Apr 9, 2026Yuxuan Zhang, Yubo Wang, Yipeng Zhu +25Web Agent Benchmarks AI Agent Evaluation
University of British Columbia · Vector Institute · Etude AI +9
Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems
Apr 9, 2026Maria Movin, Claudia Hauff, Aron Henriksson +1GUI Agents AI Agent Evaluation
Spotify, Sweden · Stockholm University, Sweden · Spotify, Netherlands
An Agentic Evaluation Architecture for Historical Bias Detection in Educational Textbooks
Apr 9, 2026Gabriel Stefan, Adrian-Marius DumitranAutomated Evaluation AI Agent Evaluation
University of Bucharest, Bucharest, Romania
GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
Apr 8, 2026Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin +2AI Agent Evaluation AI Agent Benchmarks
National University of Singapore · University of Oxford
Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Tracking
Apr 8, 2026Jiwan Chung, JiHyuk Byun, Vibhav Vineet +1Process-Based Agent Evaluation Agent Evaluation
Yonsei University · Microsoft Research
ATANT: An Evaluation Framework for AI Continuity
Apr 8, 2026Samuel Sameer TanguturiLLM Memory AI Agent Evaluation
Kenotic Labs
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Apr 7, 2026Bowen Ye, Rang Li, Qibin Yang +10AI Agent Evaluation AI Agent Benchmarks
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong
Context-Value-Action Architecture for Value-Driven Large Language Model Agents
Apr 7, 2026TianZe Zhang, Sirui Sun, Yuhang Xie +3Human Behavior Modeling LLM Agents
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Yuanpei College, Peking University · School of Psychological and Cognitive Sciences, Peking University +2
Designing Ethical Learning for Agentic AI: Toegye Yi Hwang's Ethical Emotion Regulation Framework
Apr 7, 2026Ji Yeon KimAgentic AI AI Agent Governance
Seoul National University, Seoul, South Korea
Dynamic Agentic AI Expert Profiler System Architecture for Multidomain Intelligence Modeling
Apr 7, 2026Aisvarya Adeseye, Jouni Isoaho, Seppo Virtanen +1User Modeling Agentic AI
Department of Computing, University of Turku, Turku, Finland
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Apr 6, 2026Rongqian Chen, Yu Li, Zeyu Fang +3Computer-Use Agent Evaluation Reward Modeling
George Washington University, Washington D.C., USA
GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
Apr 6, 2026Yuwen Zhai, Runze Li, Liang Wang +6AI Agent Evaluation GUI Agents
Alibaba Group · School of Computer Science and Technology, East China Normal University · Shanghai Innovation Institute

Apr 20, 2026J/K move · Enter open · S save

AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation

Wentao Shi, Yu Wang, Yuyang Zhao, Yuxin Chen+7

University of Science and Technology of China · National University of Singapore · Meituan

LLM-as-a-Judge LLM Agent Evaluation AI Agent Evaluation LLM Agent Verification AI Agent Benchmarks

Open paper AskPDF arXiv

As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.