LLM Agent Evaluation

LLM: Large Language Model

Overview

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

CardsList

1,313 papers159 papers in the last four weeks, down 28% on the four weeks before. 1.2% of all new papers.

  1. Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads

    Nov 19, 2026Shai Gretz, Alon Halfon, Yoav Katz +10LLM ServingLLM Agent Evaluation

  2. CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks

    Nov 19, 2026Jiayi Chen, Lizhe Chen, Yujia Chen +8Context Compaction for LLM AgentsAI Agent Benchmarks

  3. AppSci-Bench: Can LLMs Build AI Agents and Make Them Better?

    Nov 19, 2026Sanjari Srivastava, Manpreet Kaur, Cheng Chang +7AI Agent BenchmarksLLM Agents

  4. Moving Generality from Model to Modeler: LLM Agents for PHM on a Deployment-Style Benchmark

    Nov 18, 2026Zhenling Chen, Zhiguo Zeng, Anne BarrosLLM Agent EvaluationAI Agent Benchmarks

  5. DVBench: End-to-End Benchmark for Real-World UVM Design Verification Agents

    Nov 18, 2026Jiaru Qian, Jiawei Li, Xiaolong Hu +16Electronic Design AutomationAgent Benchmarks

  6. Local AI Agents in Power Electronics Simulation: Deployment, Measurements and Costs

    Nov 18, 2026Yuzhuo Li, Pasan Gunawardena, Yunwei LiLLM Agent EfficiencyLLM Agent Evaluation

  7. The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

    Nov 17, 2026Yuanhao Zeng, Shaocheng Lan, Ao Lu +8Self-Improving AgentsLLM Agent Evaluation

  8. How Web Research Agents Interpret User Priorities

    Nov 17, 2026Robin Sharma, Parisa SuchdevWeb AgentsAutonomous Research Agents

  9. OfficePrism: Benchmarking Agents on Single-Domain Operations Across Enterprise Business Functions

    Nov 17, 2026Hanfeng Liu, Zhenyu Luo, Dongchao Zhang +5Enterprise AI AgentsLLM Agent Evaluation

  10. Multi-Agent Memory Inconsistency: Where Factual Conflicts Disappear in LLM Agents

    Nov 17, 2026Hejia Zhang, Zhongming Yu, Yimin Ding +9Multi-Agent SystemsKnowledge Conflicts in Language Models

  11. Where Does Hindsight Enter Clinical Agent Benchmarks? An Audit of Simulators and Case Construction

    Nov 17, 2026Meilu Zhu, Xiaohan Li, Axiu MaoLLM Agent EvaluationBenchmark Validity

  12. DetourBench: Trajectory-Level Evaluation of Interactive Agents with Exact Minimum-Action Bounds

    Nov 16, 2026Hanlin Tian, Sihan Zhu, Zhao Yang +5Agent EvaluationProcess-Based Agent Evaluation

  13. GainCtrl: Predicting Remaining Gains for Efficient LLM Agent Execution

    Nov 16, 2026Kangwen Zhao, Jianfeng Cai, Ruopei Sun +4LLM Agent EfficiencyLanguage Model-Based Control

  14. Instrumenting Failure: Scoring Language Model Agents at Every Decision Node in Imperfect-Information Games

    Nov 16, 2026Michael Kingston, Sonit Singh, Alan BlairLLM Agent EvaluationBehavioral Consistency of LLM Agents

  15. HydroMind: A Checkable Agent for Autonomous Hydrological Research

    Nov 15, 2026Zheng Fang, Yiqun Sun, Simin Qu +1LLM Agent HarnessesAutonomous Research Agents

  16. Macro Control and Capability Evaluation of Black-Box World Models: A Two-Regime Framework for LLM-Agent Markets

    Nov 14, 2026Zhiyang Niu, Chuanjie Li, Huayuan Shen +2LLM Agent EvaluationMulti-Agent LLM Systems

  17. Agent-as-a-Judge

    Nov 13, 2026Runyang You, Hongru Cai, Caiqi Zhang +5LLM-as-a-JudgeLLM Agent Evaluation

  18. MineralExp: Benchmarking LLM Agents for Experimental Design in Real-World Mineral Exploration

    Nov 13, 2026Bing Wang, Mingyang Wang, Yuanhang Mi +8Active Experimental DesignScientific Agent Benchmarks