LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI
May 15, 2026Keshava Chaitanya, Jahnavi GundakaramLLM Prompting LLM Agent Evaluation
Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
May 14, 2026Sahil Sen, Akhil Kasturi, Elias Lumer +2Agentic Search LLM Agent Evaluation
PricewaterhouseCoopers, U.S.
From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents
May 14, 2026Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo +3Audio-Language Model Evaluation Voice Agent Evaluation
Dialpad Inc.
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
May 14, 2026Pei Yang, Wanyi Chen, Tongyun Yang +15Benchmark Design LLM Agent Evaluation
Gradient · Soochow University · Independent Researcher +7
Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining
May 14, 2026Robert Müller, Clemens MüllerMulti-Agent Negotiation Game-Playing Agents
GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
May 14, 2026Jingbo Yang, Kwei-Herng Lai, Xiaowen Wang +3Conversational Memory LLM Agent Evaluation
UC Santa Barbara · Microsoft
Herculean: An Agentic Benchmark for Financial Intelligence
May 14, 2026Xueqing Peng, Zhuohan Xie, Yupeng Cao +60AI Agent Benchmarks LLM Agent Evaluation
Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows
May 14, 2026Zixin Chen, Peng Liu, Rui Sheng +6Intelligent Tutoring Systems Educational Assessment
Hong Kong University of Science and Technology · Qwen Team, Alibaba Group · Qwen Team
ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents
May 13, 2026Yuxiang Lai, Peng Xia, Haonian Ji +8Computer-Use Agent Benchmarks Terminal-Agent Benchmarks
University of North Carolina at Chapel Hill · Stanford University · University of Southern California +2
ChromaFlow: A Negative Ablation Study of Orchestration Overhead in Tool-Augmented Agent Evaluation
May 13, 2026Tarun MittalAI Agent Reliability LLM Agent Orchestration
Octave-X
PBT-Bench: Benchmarking AI Agents on Property-Based Testing
May 13, 2026Lucas Jing, Xinqi Wang, Liao Zhang +1Automated Software Testing Software Engineering Agents
Tsinghua University · University of Washington · Liaoyuan Technology
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
May 13, 2026Qingyun Zou, Feng Yu, Hongshi Tan +2Electronic Design Automation LLM Agent Evaluation
National University of Singapore
TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints
May 13, 2026Zabir Al Nazi, Shubhashis Roy DiptaLLM Agent Evaluation AI Agent Benchmarks
TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate
May 13, 2026Erica Zhang, Fangzhao Zhang, Aneesh Pappu +5Multi-Agent Negotiation Automated Negotiation
♠Stanford School of Engineering · ♣Stanford Department of Economics · ♥Stanford Graduate School of Business +1
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
May 13, 2026Priyam Sahoo, Gaurav Mittal, Xiaomin Li +4Agent Failure Analysis Process-Based Agent Evaluation
University of Illinois, Urbana-Champaign, IL, USA · Microsoft, Redmond, WA, USA
Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents
May 13, 2026Harshita Chopra, Kshitish Ghate, Aylin Caliskan +3Human Behavior Simulation LLM Agent Evaluation
University of Washington, Seattle · Georgetown University, Washington DC
Mechanism Plausibility in Generative Agent-Based Modeling
May 12, 2026Patrick Zhao, David Huu Pham, Nicholas VincentLLM Agent Evaluation Agent-Based Modeling
Simon Fraser University, Burnaby, BC, Canada
Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
May 12, 2026Muhammad Bilal, Jon Crowcroft, Ruizhi Wang +2LLM Agent Evaluation AI Agent Security
School of Computing and Communications Lancaster University, United Kingdom · Department of Computer Science and Technology, University of Cambridge, United Kingdom · School of Software, Nanjing University of Information Science and Technology, China +1
MEME: Multi-entity & Evolving Memory Evaluation
May 12, 2026Seokwon Jung, Alexander Rubinstein, Arnas Uselis +2Agent Memory Persistent Memory for Language Models
1KAIST AI · 2Tübingen AI Center, University of Tübingen · 3NAVER AI Lab
CAX-Agent: A Lightweight Agent Harness for Reliable APDL Automation
May 12, 2026Chenying Lin, Yichen Hai, Yi He +3LLM Agent Orchestration LLM Agent Evaluation
Shanghai Ultradimension Technology Co., Ltd., Shanghai, China · College of Logistics Engineering, Shanghai Maritime University, Shanghai, 201306, China · School of Civil Aviation, Northwestern Polytechnical University, Xi’an 710072, China +3
Counterfactual Trace Auditing of LLM Agent Skills
May 12, 2026Xiaolin Zhou, Jinbo Liu, Li Li +2Software Engineering Agents LLM Agent Skill Evaluation
Arizona State University · University of Southern California · Adobe Research
Cochise: A Reference Harness for Autonomous Penetration Testing
May 12, 2026Andreas Happe, Jürgen CitoLLM Agent Orchestration LLM Agent Evaluation
TU Wien · Vienna, Austria
Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
May 12, 2026Junjue Wang, Weihao Xuan, Heli Qi +7LLM Agent Evaluation Geospatial Reasoning
The University of Tokyo · RIKEN AIP · Waseda University +1
CTFusion: A CTF-based Benchmark for LLM Agent Evaluation
May 12, 2026Dongjun Lee, Ga-eun Bae, Insu YunBenchmark Contamination LLM Agent Evaluation
School of Electrical Engineering, KAIST, Daejeon, Republic of Korea
RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
May 11, 2026Imad Aouali, Flavian Vasile, Otmane Sakhi +2Tool-Use Evaluation LLM Agent Evaluation
Criteo AI Lab
WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation
May 11, 2026Shuangrui Ding, Xuanlang Dai, Long Xing +14LLM Agent Harnesses AI Agent Benchmarks
1Shanghai AI Laboratory, 2The Chinese University of Hong Kong · 1Shanghai AI Laboratory, 3Fudan University · 1Shanghai AI Laboratory, 4University of Science and Technology of China +8
Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?
May 11, 2026Tz-Huan Hsu, Jheng-Hong Yang, Jimmy LinTool-Augmented Language Model Agents Agentic Search
University of Waterloo · Stencilzeit
ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox
May 11, 2026Yuanyang Li, Xue Yang, Longyue Wang +2LLM Agent Evaluation AI Agent Benchmarks
1Zhejiang University · 2Zhejiang Lab · 3Alibaba Group
CrackMeBench: Binary Reverse Engineering for Agents
May 11, 2026Isaac David, Arthur GervaisSoftware Reverse Engineering LLM Agent Evaluation
University College London
Argumentation-Based Person-Tailored Storytelling for Older Adults: A Reflective Hybrid AI Architecture
May 11, 2026Jayalakshmi Baskar, Vera C. Kaelin, Kaan Kilic +1Argument Mining LLM Agent Evaluation
Umeå University, Department of Computing Science, Sweden

May 15, 2026J/K move · Enter open · S save

PRISM: Prompt Reliability via Iterative Simulation and Monitoring for Enterprise Conversational AI

Keshava Chaitanya, Jahnavi Gundakaram

LLM Prompting LLM Agent Evaluation Prompt Optimization Language Model Robustness LLM Agent Reliability

Open paper AskPDF arXiv

Deploying large language model (LLM)-driven conversational agents in enterprise settings requires prompts that are simultaneously correct at launch and resilient to the non-deterministic behavioral drift that characterizes production LLM deployments. Existing prompt optimization frameworks address prompt quality as a one-time compile-time problem, leaving open the equally critical question of how to detect and repair prompt regressions caused by silent LLM behavior changes over time. We present PRISM (Prompt Reliability via Iterative Simulation and Monitoring), a closed-loop framework that treats prompt engineering as a continuous reliability engineering problem rather than a one-time authorship task. PRISM takes as input plain-language agent requirements, a set of configured tools and memory variables, and an initial draft prompt. It automatically generates test cases from requirements, simulates full multi-turn conversations against a platform-faithful LLM environment, evaluates pass/fail using an LLM-as-judge, diagnoses root causes of failures, and surgically repairs the prompt -- iterating until all tests pass. Critically, PRISM is designed to run on a scheduled basis (daily), treating LLM behavioral drift as a first-class reliability concern. We evaluate PRISM across 35 enterprise conversational agents over a three-week deployment period on the Yellow.ai V3 platform. PRISM reduces median prompt authoring time from 2 days to under 30 minutes, achieves 99% production reliability across all evaluated agents, and successfully identifies and repairs production regressions caused by LLM behavioral drift within a 24-hour detection window. Our results suggest that continuous, simulation-driven prompt optimization is both tractable and necessary for reliable enterprise conversational AI at scale.