LLM Agent Evaluation

LLM: Large Language Model

Overview

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

CardsList

1,313 papers159 papers in the last four weeks, down 28% on the four weeks before. 1.2% of all new papers.

  1. Evaluating Test-Time Scaling of General LLM Agents

    Feb 22, 2026Xiaochuan Li, Ryan Ming, Pranav Setlur +6LLM Agent EvaluationTest-Time Scaling

  2. WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics

    Feb 20, 2026Madhav Kanda, Sharad Agarwal, Rodrigo Fonseca +2LLM Agent EvaluationAI Agent Benchmarks

  3. Evaluating Chain-of-Thought Reasoning through Reusability and Verifiability

    Feb 19, 2026Shashank Aggarwal, Ram Vikas Mishra, Amit AwekarCoT EvaluationLLM Reasoning

  4. AI Agents Alone Are Not (Yet) Sufficient for Social Simulation

    Feb 19, 2026Yiming Li, Dacheng TaoLarge Language Model-Based Social SimulationHuman Behavior Simulation

  5. Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History

    Feb 19, 2026Serin Kim, Sangam Lee, Dongha LeeWeb AgentsWeb Agent Benchmarks

  6. Evaluating Collective Behaviour of Hundreds of LLM Agents

    Feb 18, 2026Richard Willis, Jianing Zhao, Yali Du +1Social DilemmasMulti-Agent LLM Systems

  7. AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems

    Feb 18, 2026Hadar Mulian, Sergey Zeltyn, Ido Levy +3LLM Agent ReliabilityLLM Agents

  8. Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents

    Feb 18, 2026Nivya Talokar, Ayush K Tarun, Murari Mandal +2LLM Red TeamingLLM Agent Safety

  9. Empirical Cumulative Distribution Function Clustering for LLM-based Agent System Analysis

    Feb 18, 2026Chihiro Watanabe, Jingyu SunLLM Agent EvaluationLanguage Model Generation Evaluation

  10. Decision Quality Evaluation Framework at Pinterest

    Feb 17, 2026Yuqi Tian, Robert Paine, Attila Dobi +3LLM Safety EvaluationContent Moderation

  11. BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions

    Feb 14, 2026Minyuan Ruan, Ziyue Wang, Kaiming Liu +3LLM Agent EvaluationAI Agent Benchmarks

  12. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

    Feb 12, 2026Thibaud Gloaguen, Niels Mündler-Sasahara, Mark Niklas Müller +2AI Coding AgentsCoding Agents

  13. ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

    Feb 11, 2026Bang Nguyen, Dominik Soós, Qian Ma +8LLM Agent EvaluationAI Agent Benchmarks

  14. Evaluating Memory Structure in LLM Agents

    Feb 11, 2026Alina Shutova, Alexandra Olenina, Ivan Vinogradov +1LLM Agent MemoryLLM Agent Evaluation

  15. Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents

    Feb 8, 2026Zhi Chen, Zhensu Sun, Yuling Shi +4Software Engineering AgentsAutomated Test Generation

  16. Do Web Agents Investigate Before They Decide?

    Feb 5, 2026Syed Nazmus Sakib, Nafiul Haque, Tapodhir Karmakar Taton +2Web Agent BenchmarksHallucination in Language Models

  17. PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios

    Feb 5, 2026Chris Zhu, Sasha Cui, Will Sanok Dufallo +4LLM Agent EvaluationMulti-Agent Negotiation

  18. MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems

    Feb 3, 2026Vishal Venkataramani, Haizhou Shi, Zixuan Ke +6Multi-Agent LLM SystemsLLM-as-a-Judge

  19. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

    Feb 2, 2026Shraddha Barke, Arnav Goyal, Alind Khare +3Fault DiagnosisAI Agent Reliability

  20. Why Your Deep Research Agent Fails? On Hallucination Evaluation in Full Research Trajectory

    Jan 30, 2026Yuhao Zhan, Tianyu Fan, Linxuan Huang +2Deep Research AgentsHallucination Detection

  21. Automated structural testing of LLM-based agents: methods, framework, and case studies

    Jan 25, 2026Jens Kohl, Otto Kruse, Youssef Mostafa +9Regression TestingAutomated Software Testing

  22. DSAEval: Evaluating Data Science Agents on a Wide Range of Real-World Data Science Problems

    Jan 20, 2026Maojun Sun, Yifei Xie, Yue Wu +5Multimodal Agent EvaluationLLM Agent Evaluation

  23. Real-Time Deadlines Reveal Fragile Temporal Adaptation in LLM Strategic Dialogues

    Jan 19, 2026Neil K. R. Sehgal, Sharath Chandra Guntuku, Lyle UngarLLM Agent EvaluationLLM Decision-Making

  24. AstroAgentBench: Evaluating Agentic Planning on Space Mission Planning Tasks

    Jan 16, 2026Weiyi Wang, Xinchi Chen, Jingjing Gong +2LLM Agent EvaluationAI Agent Benchmarks