LLM Agent Evaluation

LLM: Large Language Model

Ask about this topic

Overview

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

CardsList

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

  1. LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks

    Mar 20, 2026Xiang Long, Li Du, Yilong Xu +11Computer-Use Agent BenchmarksTool-Augmented Language Model Agents

    Samsung Research, Beijing, China · HKUST (Guangzhou) · City University of Hong Kong +1

  2. When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

    Mar 20, 2026Artem Maryanskyy, Dmitry Budnikov, Alibek T. KaliyevMulti-Agent LLM SystemsMulti-LLM Collaboration

    Uber Technologies, Sunnyvale, CA 94085, USA · Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia · Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA

  3. The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents

    Mar 19, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent SafetyLLM Safety Alignment

    Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, UK · School of Professoinal Studies, Clark University, Worcester, USA · Harvard Medical School, Harvard University, Boston, USA

  4. Multi-User Large Language Model Agents

    Mar 19, 2026Shu Yang, Shenzhe Zhu, Hao Zhu +5LLM AgentsLLM Agent Evaluation

    KAUST · UT Austin · Stanford University +1

  5. Security awareness in LLM agents: the NDAI zone case

    Mar 19, 2026Enrico Bottazzi, Pia ParkAI Agent SecurityLLM Agent Security

    Leku

  6. I Can't Believe It's Corrupt: Evaluating Corruption in Multi-Agent Governance Systems

    Mar 19, 2026Vedanta S P, Ponnurangam KumaraguruAI Agent GovernanceLLM Agent Safety

    IIIT Kottayam · IIIT Hyderabad

  7. CN-Buzz2Portfolio: A Chinese-Market Dataset and Benchmark for LLM-Based Macro and Sector Asset Allocation from Daily Trending Financial News

    Mar 18, 2026Liyuan Chen, Shilong Li, Jiangpeng Yan +3Financial Decision-MakingLLM Agent Evaluation

    Tsinghua Shenzhen International Graduate School · E Fund Management Co., Ltd. · Hong Kong Polytechnic University

  8. How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment

    Mar 17, 2026Rebecca Ansell, Autumn Toney-WailsFine-Tuning Language ModelsLLM Agent Evaluation

    Department of Computer Science, Georgetown University, Washington, DC, USA · Syntheos, Corp; UNU-Merit, Maastricht, NL

  9. Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure

    Mar 17, 2026Caglar YildirimLLM Agent SafetyLLM Agent Evaluation

    Khoury College of Computer Sciences Northeastern University Boston, MA 02115, USA

  10. Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis

    Mar 16, 2026Penny Chong, Harshavardhan Abichandani, Jiyuan Shen +4LLM Agent EvaluationAgent Evaluation

    SAP · Stanford University

  11. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?

    Mar 16, 2026Tingxu Han, Yi Zhang, Wei Song +4Software Engineering AgentsSoftware Engineering Benchmarks

    Nanjing University · Mohamed bin Zayed University of Artificial Intelligence · South China University of Technology +1

  12. SKILLS: Structured Knowledge Injection for LLM-Driven Telecommunications Operations

    Mar 16, 2026Ivo BrettTool-Augmented Language Model AgentsLLM Agent Evaluation

    independent.academia.edu/ivobrett

  13. AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

    Mar 15, 2026Shengda Fan, Xuyan Ye, Yupeng Huo +9LLM Agent EvaluationProcess-Based Agent Evaluation

    Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China +1

  14. Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspective

    Mar 15, 2026Mohamed Aghzal, Gregory J. Stein, Ziyu YaoWeb AgentsLLM Agent Reliability

    Department of Computer Science, George Mason University

  15. Human-Centered Evaluation of an LLM-Based Process Modeling Copilot: A Mixed-Methods Study with Domain Experts

    Mar 13, 2026Chantale Lauer, Peter Pfeiffer, Nijat MehdiyevBusiness Process ModelingLLM Agent Evaluation

    Saarland University, Saarbrücken, Saarland, Germany · German Research Institute for Artificial Intelligence, Saarbrücken, Saarland, Germany · 5Plus GmbH, Würzburg, Bavaria, Germany

  16. AI Planning Framework for LLM-Based Web Agents

    Mar 13, 2026Orit Shahnovsky, Rotem DrorLLM Agent PlanningWeb Agents

    Faculty of Computer and Information Science, University of Haifa, Haifa, Israel

  17. Sell Me This Stock: Unsafe Recommendation Drift in LLM Agents

    Mar 13, 2026Zekun Wu, Adriano Koshiyama, Sahan Bulathwela +1LLM Agent SafetyLanguage Model Safety Evaluation

    Centre for Artificial Intelligence, University College London · Holistic AI

  18. Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

    Mar 12, 2026Aadyaa Maddi, Prakhar Naval, Deepti Mande +3LLM Agent EvaluationData Analysis Agents

    Rockfish Data and Carnegie Mellon University

  19. Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

    Mar 12, 2026Łukasz Borchmann, Jordy Van Landeghem, Michał Turski +12Multi-Document QALLM Agent Evaluation

  20. Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance

    Mar 12, 2026Akshey Sigdel, Rista BaralLLM Tool UseLLM Agent Reliability

    Independent Researcher

  21. RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents

    Mar 11, 2026Yonas Atinafu, Robin CohenLLM Agent EvaluationAI Agent Security Benchmarks

    David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

  22. Mind the Sim2Real Gap in User Simulation for Agentic Tasks

    Mar 11, 2026Xuhui Zhou, Weiwei Sun, Qianou Ma +8User SimulationSim-to-Real Transfer

    Carnegie Mellon University, Language Technologies Institute

  23. SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

    Mar 11, 2026Yadi Cao, Sicheng Lai, Jiahe Huang +12AI Agent BenchmarksLarge Language Model-Guided Optimization

    University of California San Diego · The Chinese University of Hong Kong, Shenzhen · Peking University +3

  24. Interview-Informed Generative Agents for Product Discovery: A Validation Study

    Mar 10, 2026Zichao Wang, Alexa SiuUser SimulationLarge Language Model-Based Social Simulation

    Adobe Research San Jose, CA, USA

  25. Time, Identity and Consciousness in Language Model Agents

    Mar 10, 2026Elija Perrier, Michael Timothy BennettLLM Agent EvaluationLLM Agents

    Centre for Quantum Software and Information, UTS, Sydney · Australian National University, Canberra

  26. OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning

    Mar 9, 2026Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins +10Multi-Document QALLM Agent Evaluation

    Databricks

  27. PostTrainBench: Can LLM Agents Automate LLM Post-Training?

    Mar 9, 2026Ben Rank, Hardik Bhatnagar, Ameya Prabhu +4LLM Post-TrainingAutonomous Research Agents

    ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +2

  28. FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

    Mar 9, 2026Jiaxuan Lu, Kong Wang, Yemin Wang +9LLM Agent EvaluationTool-Using Language Agents

    Shanghai AI Laboratory · Hunan University · Xiamen University +3

  29. $OneMillion-Bench: How Far are Language Agents from Human Experts?

    Mar 9, 2026Yang Liu, Jiaqi Li, Jun Bai +20LLM Agent EvaluationAI Agent Benchmarks

    BIGAI · Humanlaya · xbench +1

  30. Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety

    Mar 8, 2026David GringrasLLM Safety BenchmarksLLM Agent Evaluation

    Harvard University