AI Agent Evaluation

Follow this topic Ask about this topic

AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.

Latest papers

525 papers77 papers in the last four weeks, up 4% on the four weeks before. 0.4% of all new papers.

DREAM: Deep Research Evaluation with Agentic Metrics
Feb 21, 2026Elad Ben Avraham, Changhao Li, Ron Dorfman +8AI Agent Evaluation Deep Research Agents
AWS Agentic AI · Georgia Institute of Technology
Towards More Standardized AI Evaluation: From Models to Agents
Feb 20, 2026Ali El Filali, Inès BedarAI Agent Evaluation LLM Evaluation
G42 · Independent
Towards a Science of AI Agent Reliability
Feb 18, 2026Stephan Rabanser, Sayash Kapoor, Peter Kirgis +3AI Agent Reliability AI Agent Evaluation
Princeton University.
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
Feb 15, 2026Lingxiang Hu, Yiding Sun, Tianle Xia +6Online Advertising LLM Agent Evaluation
Tencent
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Feb 6, 2026Lanbo Lin, Jiayao Liu, Tianyuan Yang +5LLM-as-a-Judge AI Agent Evaluation
Alibaba International Digital Commerce Group · Zhejiang University · Peking University
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM Evaluation AI Agent Evaluation
Computer Science and Information, Ben-Gurion University of the Negev
Persuasion Propagation: Does Persuasion Change AI Agent Behavior?
Jan 31, 2026Hyejun Jeong, Amir Houmansadr, Shlomo Zilberstein +1Persuasion in Language Models AI Agent Evaluation
University of Massachusetts Amherst
LogicEnvGen: Task-Logic Driven Generation of Diverse Simulated Environments for Embodied AI
Jan 20, 2026Jianan Wang, Siyang Zhang, Bin Li +4Robotics Simulation AI Agent Evaluation
College of Computer Science and Technology, National University of Defense Technology · Intelligent Game and Decision Lab (IGDL), Beijing · School of Artifical Intelligence, Shanghai Jiao Tong University
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Jan 15, 2026Yiwen Gao, Ruochen Zhao, Yang Deng +1Deep Research Agents Benchmark Design
National University of Singapore · Nanyang Technological University · Singapore Management University +1
Monte Carlo Query Search: Active Capability Assessment of AI Agents
Dec 18, 2025Daniel Bramblett, Rushang Karia, Adrian Ciotinga +3AI Agent Evaluation Active Learning
Arizona State University, AZ, USA · Indian Institute of Technology Madras, India
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
Nov 24, 2025Dayong Liu, Chao Xu, Weihong Chen +5AI Agent Evaluation Multimodal Large Language Models
Zhejiang University · IROOTECH TECHNOLOGY · Wolf 1069 b Lab, Sany Group +2
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
Oct 10, 2025Yincen Qu, Huan Xiao, Feng Li +5Reward Modeling LLM Planning
Trip.com Group · Chongqing University · Stanford University
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
Jun 4, 2025Dhaval Patel, Shuxin Lin, James Rayfield +7LLM Agent Orchestration AI Agent Evaluation
IBM · University of South Carolina · Steel Authority of India +1
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Sep 17, 2024Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2AI Agent Evaluation AI Agent Benchmarks
Princeton University
When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments
Jul 15, 2024Chong Zhang, Xinyi Liu, Zhongmou Zhang +10Multi-Agent LLM Systems Human Behavior Simulation
University of Liverpool, UK · Peking University, China · Shanghai University of Finance and Economics, China +6

Feb 21, 2026J/K move · Enter open · S save

DREAM: Deep Research Evaluation with Agentic Metrics

Elad Ben Avraham, Changhao Li, Ron Dorfman, Roy Ganz+7

AWS Agentic AI · Georgia Institute of Technology

AI Agent Evaluation Deep Research Agents Reference-Free Evaluation Tool-Use Evaluation

Open paper AskPDF arXiv

Deep Research Agents generate analyst-grade reports, yet evaluating them remains challenging due to the absence of a single ground truth and the multidimensional nature of research quality. Recent benchmarks propose distinct methodologies, yet they suffer from the Mirage of Synthesis, where strong surface-level fluency and citation alignment can obscure underlying factual and reasoning defects. We characterize this gap by introducing a taxonomy across four verticals that exposes a critical capability mismatch: static evaluators inherently lack the tool-use capabilities required to assess temporal validity and factual correctness. To address this, we propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that instantiates the principle of capability parity by making evaluation itself agentic. DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent, enabling temporally aware coverage, grounded verification, and systematic reasoning probes. Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks, offering a scalable, reference-free evaluation paradigm.