LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,221 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baseline
Jun 3, 2026Zhikai Chen, Jialiang Gu, Junyu Yin +6Agent Memory LLM Agent Memory
Michigan State University · George Mason University · Purdue University
SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
Jun 2, 2026Joel Sol, Homayoun NajjaranMulti-Agent Coordination LLM Agent Evaluation
Faculty of Engineering and Computer Science University of Victoria Victoria, BC, Canada
MedCUA-Bench: A Screenshot-Only Benchmark for Clinical Computer-Use Agents
Jun 2, 2026Jia Yu, Zilong Wang, Xinyang Jiang +2Computer-Use Agent Benchmarks LLM Agent Evaluation
Microsoft Research Asia, Shanghai, China · Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China · Shanghai Key Laboratory of MICCAI, Shanghai, China
The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation
Jun 2, 2026Herun Wan, Jiaying Wu, Minnan Luo +4Multi-Agent LLM Systems LLM Agent Evaluation
Xi’an Jiaotong University · National University of Singapore · Yunnan University +1
Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks
Jun 1, 2026Dipesh KC, Anjila BudathokiCoding Agents Software Engineering Benchmarks
Independent Researcher · Georgia State University
The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models
Jun 1, 2026Petra Ferencz, Ava Keeling, Tobias O'Keefe +4Human Behavior Simulation LLM Agent Evaluation
Big Data Institute, Li Ka Shing Center for Health Information and Discovery, University of Oxford, Oxford, United Kingdom · Leverhulme Centre for Demographic Science, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom · Pandemic Sciences Institute, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom +3
$Ψ$ -Bench: Evaluating Persona-Sensitive Influencing in Persuasive Dialogues
Jun 1, 2026Peixuan Han, Hongyi Du, Jiayu Liu +3LLM Personalization LLM Agent Evaluation
University of Illinois Urbana-Champaign
HERO'S JOURNEY: Testing Complex Rule Induction with Text Games
Jun 1, 2026Anshun Asher Zheng, Kanishka Misra, David I. Beaver +1LLM Agent Evaluation AI Agent Benchmarks
Department of Linguistics The University of Texas at Austin
MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation
Jun 1, 2026Wenhao Wang, Peizhi Niu, Gongyi Zou +9LLM Agent Evaluation AI Agent Benchmarks
Multi-Agent Governance & Intelligence Crew (MAGIC), Shanghai Jiao Tong University, Shanghai, China · Department of Computer Science and Technology, Zhejiang University, Hangzhou, China · University of Illinois at Urbana-Champaign, Illinois, USA +2
K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
Jun 1, 2026Nahyun Lee, Dongkeun Yoon, Guijin Son +12Computer-Use Agent Benchmarks LLM Evaluation
Chung-Ang University1 · KAIST2 · Seoul National University3 +3
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
Jun 1, 2026Yuyan Bu, Haowei Li, Qirui Zheng +7LLM Agent Evaluation AI Agent Safety
Beijing Academy of Artificial Intelligence · University of Science and Technology of China · Peking University +2
Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains
Jun 1, 2026Garvin Guo, Donglei Yu, Yu Chen +6Tool-Use Evaluation LLM Agent Evaluation
University of Chinese Academy of Sciences · Tongyi Lab, Alibaba Group · Amap, Alibaba Group +1
POIROT: Interrogating Agents for Failure Detection in Multi-Agent Systems
Jun 1, 2026Iñaki Dellibarda Varela, R. Sendra-Arranz, Pablo Romero-Sorozabal +5Agent Failure Analysis AI Agent Reliability
Center for Automation and Robotics, Spanish National Research Council (CSIC-UPM), Madrid, Spain. · ETSI Telecomunicación, Universidad Politécnica de Madrid (UPM), Madrid, Spain.
BADGER: Bridging Agentic and Deterministic Evaluation for Generative Enterprise Reasoning
Jun 1, 2026Shannon Serrao, Soumitra Chatterjee, Dorina Strori +2LLM Agent Evaluation AI Agent Evaluation
Merkle Analytics
SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
Jun 1, 2026Kuan Li, Shuo Zhang, Huacan Wang +12LLM Agent Evaluation Internet of Things
Midea Group · Beijing University of Posts and Telecommunications · Donghua University +2
CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
Jun 1, 2026Danqing Wang, Akshay Sivaraman, Lei LiHuman Behavior Simulation LLM Agent Evaluation
Carnegie Mellon University
Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation
Jun 1, 2026Donghwan Kim, Prakhar Singh, Younghoon Min +3LLM Agent Evaluation AI Agent Benchmarks
The Pennsylvania State University · SK Hynix · KAIST
From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents
Jun 1, 2026Laksh AdvaniAgent Failure Analysis AI Agent Reliability
University of Colorado
TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning
May 31, 2026Yaxuan Kong, Qingren Yao, Yuqi Nie +7Time Series Reasoning LLM Agent Evaluation
University of Oxford · VulpiVox Intelligence · Eindhoven University of Technology +3
Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
May 31, 2026Xianyou Li, Weiran Yan, Yichao Wu +4LLM Inference Efficiency Multi-Agent LLM Systems
New York University · New York, NY 10012, USA · Independent Researcher +7
A New Framework for Cybersecurity Refusals in AI Agents
May 31, 2026Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson +1LLM Agent Evaluation LLM Refusal Behavior
Work done while at Gray Swan · Gray Swan AI · Carnegie Mellon University
TravelEval: A Comprehensive Benchmarking Framework for Evaluating LLM-Powered Travel Planning Agents
May 31, 2026Weiyi Chen, Shuaixiong Wang, Ziyun Gao +5Spatiotemporal Reasoning LLM Agent Evaluation
Zhejiang University · Hangzhou, China · Hong Kong Polytechnic University +5
Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
May 30, 2026Rana Muhammad UsmanAdversarial Attacks on LLMs LLM Agent Evaluation
Independent Researcher
Not All Flips Are Conformity: Decomposing Stance Convergence in Multi-Agent LLM Debate
May 30, 2026Xiqi Hao, Zengqing Wu, Yu-Xuan Qiu +4LLM Agent Evaluation Multi-Agent Debate
Beijing Institute of Technology, Zhuhai · University of Osaka · Shenzhen University
ForeSci: Evaluating LLM Agents for Forward-Looking AI Research Judgment
May 30, 2026Qiuyu Tian, Haojie Yin, Yingce Xia +2LLM Agent Evaluation AI Agent Benchmarks
1Southeast University, Nanjing, China · 3Duke Kunshan University, Kunshan, China · 2Beijing Zhongguancun Academy, Beijing, China
Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents
May 30, 2026Yufeng WangAI Agent Reliability LLM Agent Evaluation
BAGEN: Are LLM Agents Budget-Aware?
May 29, 2026Yuxiang Lin, Zihan Wang, Mengyang Liu +9LLM Agent Evaluation AI Agent Benchmarks
1Northwestern University · 2O2 Lab · 3Independent +5
Skill Availability and Presentation Granularity in Large-Language-Model Agents: A Controlled SkillsBench Study
May 29, 2026Xiaonan Xu, Wenjing WuLLM Agent Evaluation LLM Agent Skill Learning
Computer Information Technology, Northern Arizona University, Flagstaff, AZ, USA · Computer Science, University of Colorado Boulder, Boulder, CO, USA
Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation
May 29, 2026Grégoire Martinon, Ibrahim Merad, Mohammed RakiLLM-Assisted Annotation LLM Agent Evaluation
Emerton Data, 16 Avenue Hoche, 75008, Paris, France.
TUX: Measuring Human--AI Tacit Understanding
May 29, 2026Yueshen Li, Hanyi Min, Vedant Das Swain +1LLM Alignment LLM Agent Evaluation
University of Illinois Urbana-Champaign · New York University

Jun 3, 2026J/K move · Enter open · S save

Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baseline

Zhikai Chen, Jialiang Gu, Junyu Yin, Xianxuan Long+5

Michigan State University · George Mason University · Purdue University

Agent Memory LLM Agent Memory LLM Agent Evaluation AI Agent Benchmarks LLM Agent Harnesses Agent Memory Management

Open paper AskPDF arXiv

LLM agents accumulate histories that outgrow their context windows, motivating a growing literature on memory systems. Yet most existing designs are tuned to a single scenario (multi-session chat or a single trajectory format), and there is little evidence that they generalize across the heterogeneous trajectories agents encounter in deployment. We revisit eight memory systems plus an agentic harness for search problems, on five scenarios: single-turn QA, multi-session chat, agentic-trajectory QA, memory stress tests, and long-horizon agentic tasks. The harness, which self-manages flat text-file storage via tool calls, achieves the best cross-task ranking, suggesting that memory performance hinges on giving the agent active control over storage and retrieval rather than on a passive store behind a fixed pipeline. We instantiate this insight in AutoMEM, an agentic memory harness with a self-managed tool interface that achieves the best cross-scenario generality among the systems we evaluate.