AI Agent Evaluation
Overview
AI agent evaluation studies how well systems use tools and interact with changing environments to complete tasks, from coding and web navigation to scientific and professional workflows. Researchers build interactive benchmarks and assess outcomes alongside execution traces, using deterministic state checks, expert rubrics, and language-model judges. Repeated trials and process-level measures help reveal failures in tool use, safety, consistency, and recovery that a final pass rate can hide. A central challenge is ensuring that benchmark scores reflect real capability rather than leakage, reward hacking, evaluator bias, or quirks of a particular model harness.
Latest papers
563 papers47 papers in the last four weeks, down 48% on the four weeks before. 0.4% of all new papers.
A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines
Aletheia tackles FirstProof autonomously
Implicit Intelligence -- Evaluating Agents on What Users Don't Say
Quantifying the Expectation-Realisation Gap for Agentic AI Systems
MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems
Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations
DREAM: Deep Research Evaluation with Agentic Metrics
Towards More Standardized AI Evaluation: From Models to Agents
Towards a Science of AI Agent Reliability
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents
JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
Persuasion Propagation: Does Persuasion Change AI Agent Behavior?
LogicEnvGen: Task-Logic Driven Generation of Diverse Simulated Environments for Embodied AI
DR-Arena: an Automated Evaluation Framework for Deep Research Agents
Monte Carlo Query Search: Active Capability Assessment of AI Agents
Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents
TripScore: Aligning LLMs for Real-World Travel Planning via Expert-Calibrated Reward
AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenance
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
When AI Meets Finance (StockAgent): Large Language Model-based Stock Trading in Simulated Real-world Environments
Towards Shutdownable Agents via Stochastic Choice
Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.