LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

Toward User Comprehension Supports for LLM Agent Skill Specifications
May 19, 2026Zikai Alex WenLLM Agent Evaluation LLM Agent Skill Evaluation
University of Washington, Tacoma School of Engineering & Technology
PAVE: A Cognitive Architecture for Legitimate Violation in Generative Agent Societies
May 19, 2026Ahmad Yehia, Abduallah Mohamed, Kun Qian +4Cognitive Architectures for AI Agents LLM Agent Evaluation
The University of Texas at Austin · Meta Reality Labs · University of Calgary
Agentic Trading: When LLM Agents Meet Financial Markets
May 19, 2026Yihan Xia, Panpan You, Taotao Wang +4LLM Agent Evaluation Finance
College of Electronic and Information Engineering, Shenzhen University, Shenzhen, China
PrefBench: Evaluating Zero-Shot LLM Agents in Hidden-Preference Personalized Pricing Negotiations
May 19, 2026Yingjie LeiAutomated Negotiation LLM Agent Evaluation
University of Aberdeen
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
May 18, 2026Leyao Wang, Yanan He, Peng Chen +5AI Agent Reliability LLM-as-a-Judge
Yale University · IBM Research
How Far Are We From True Auto-Research?
May 18, 2026Zhengxin Zhang, Ning Wang, Sainyam Galhotra +1LLM Agent Evaluation AI Agent Benchmarks
Cornell University
Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents
May 18, 2026Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya +1Agent Failure Analysis AI Agent Reliability
Department of Computer Science Cornell University
POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
May 18, 2026Qiaoyuan Zheng, Yiqu Yang, Qi Gao +1LLM Agent Evaluation LLM Agent Security
ETH Zurich Zurich, Switzerland · ETH AI Center Zurich, Switzerland
SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents
May 18, 2026Yifan Zhou, Zhentao Zhang, Ziming Cheng +8LLM Agent Evaluation LLM Agent Skill Learning
1SJTU · 2XJTU · 3NUS +6
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
May 18, 2026Yuyao Wang, Zhongjian Zhang, Mo Chi +7LLM Agent Memory LLM Agent Evaluation
Hong Kong University of Science and Technology (Guangzhou) · Beijing University of Posts and Telecommunications · Beijing Institute of Technology +1
Same Signal, Different Semantics: A Cross-Framework Behavioral Analysis of Software Engineering Agents
May 18, 2026Wei Ma, Zhi Chen, Jingxu Gu +3Software Engineering Agents LLM Agent Evaluation
Blekinge Tekniska Högskola Karlskrona, Sweden · Singapore Management University Singapore, Singapore · Nanjing University Nanjing, China +1
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
May 18, 2026Jiawei He, Jie Jia, Chenbo Liu +4AI Coding Agents LLM Agent Evaluation
Amap, Alibaba Group
PROTEA: Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows
May 18, 2026Kazuki Kawamura, Satoshi Waki, Kei TatenoMulti-Agent LLM Systems LLM Agent Evaluation
Sony Group Corporation
Evaluating Cognitive Age Alignment in Interactive AI Agents
May 18, 2026Yifan Shen, Jiawen Zhang, Jian Xu +4LLM Agent Evaluation AI Agent Benchmarks
1PediaMed AI · University of Illinois Urbana-Champaign · 4Peking University +2
Interactive Evaluation Requires a Design Science
May 18, 2026Keyang Xuan, Peiyang Song, Pan Lu +10LLM Agent Evaluation AI Agent Evaluation
University of Texas Austin · California Institute of Technology · Carnegie Mellon University +5
SocialMemBench: Are AI Memory Systems Ready for Social Group Settings?
May 18, 2026Olukunle OwolabiAgent Memory Conversational Memory
Independent Researcher
Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps
May 17, 2026Tanmay Asthana, Aman Saksena, Divyansh SahuAI Agent Reliability Deep Research Agents
Deccan AI Research
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
May 17, 2026Lecheng Yan, Ruizhe Li, Xicheng Han +5AI Agent Reliability LLM Agent Evaluation
Southern University of Science and Technology · University of Science and Technology of China · Zhejiang University +2
DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents
May 17, 2026Sirui Hong, Zhijie Liu, Tengfei Li +3Agent Failure Analysis GUI Agents
1DeepWisdom · 2Independent Researcher · 3The Hong Kong University of Science and Technology (Guangzhou)
MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation
May 17, 2026Chenyu Wang, Yang ShuLLM Agent Orchestration LLM Agent Evaluation
School of Computer and Artificial Intelligence, Zhengzhou University, Zhengzhou, China · Zhejiang University, Hangzhou, China
ContractBench: Can LLM Agents Preserve Observation Contracts?
May 17, 2026Jicheng Wang, Yifeng He, Zili Wang +3AI Agent Reliability LLM Agent Evaluation
University of California, Davis · University of Southern California · University of Hong Kong
Representation Signatures and Risk-Feedback Alignment in LLM Trading Agents
May 16, 2026Weicheng XueLLM Alignment LLM Auditing
Virginia Tech
PersonaArena: Dynamic Simulation for Evaluating and Enhancing Persona-Level Role-Playing in Large Language Models
May 16, 2026Wenlong Shi, Jianxun Lian, Mingqi Wu +5Large Language Model-Based Role-Play Simulation Personality Modeling in Language Models
College of Computer Science and Software Engineering, Shenzhen University, China · Microsoft Research Asia · Microsoft Gaming +2
The Alpha Illusion: Reported Alpha from LLM Trading Agents Should Not Be Treated as Deployment Evidence
May 16, 2026Yuxuan Ye, Jun Han, Ao Hu +7LLM Agent Evaluation AI Agent Benchmarks
Fudan University · Shanghai University of Finance and Economics · Southwest University of Finance and Economics +3
Multi-Paradigm Agent Interaction in Practice:A Systematic Analysis of Generator-Evaluator, ReAct Loop,and Adversarial Evaluation in the buddyMe Framework
May 16, 2026Xiaohua Wang, Chao Han, Kai Yu +2LLM Agent Orchestration LLM Agent Evaluation
BuddyMe Research Team, Nanjing 210000, China · CHARMMIRAEL Biotech Co., Ltd, Nanjing 210000, China
PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures
May 15, 2026Yunan Lu, Luigi Liu, Omar Yahia +2AI Agent Reliability Adversarial Prompt Generation
Columbia University · University of California San Diego · Walmart
Confirming Correct, Missing the Rest: LLM Tutoring Agents Struggle Where Feedback Matters Most
May 15, 2026Tahreem Yasir, Wenbo Li, Sam Gilson +3Educational Feedback Generation Intelligent Tutoring Systems
North Carolina State University
Context, Reasoning, and Hierarchy: A Cost-Performance Study of Compound LLM Agent Design in an Adversarial POMDP
May 15, 2026Igor Bogdanov, Chung-Horng Lung, Thomas Kunz +3Multi-Agent LLM Systems LLM Agent Orchestration
Carleton University Ottawa, Ontario, Canada · Defence R&D Canada Ottawa, Ontario, Canada · Cistel Technology Ottawa, Ontario, Canada
SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?
May 15, 2026Kean Shi, Zihang Li, Tianyi Ma +13Computer-Use Agent Benchmarks Computer-Use Agents
1UniPat AI · 2PKU · 3HKU +2
Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation
May 15, 2026Jinuk Kim, Junsoo Byun, Donghwi Hwang +2Electronic Design Automation LLM Agent Evaluation
Department of Computer Science and Engineering, Seoul National University · Neural Processing Research Center · Samsung Electronics Co., Ltd.

May 19, 2026J/K move · Enter open · S save

Toward User Comprehension Supports for LLM Agent Skill Specifications

Zikai Alex Wen

University of Washington, Tacoma School of Engineering & Technology

LLM Agent Evaluation LLM Agent Skill Evaluation

Open paper AskPDF arXiv

Users often interpret and select agent skills through their SKILL markdown specifications. To protect users, existing audits mainly focus on malicious or unsafe skills. We study the complementary question of whether specifications help users form bounded expectations about what a skill consumes, produces, and covers. Across 878 cybersecurity skills, we used rule-based coding to measure textual cues for four comprehension anchors, namely operational basis, output contract, boundary disclosure, and example capability demonstration. Cues for operational basis were common, but only 19.0% of specifications exhibited cues for an example task, sample, or expected outcome, and only 2.3% exhibited cues for all four anchors. We further examined a small DNS/C2 telemetry subset (n

=

6) to illustrate why missing examples may matter. Examples appeared to make first local checks easier to construct, while no-example skills typically required helper code inspection to recover command arguments or output fields. We argue that agent-skill evaluation should treat specifications as user-facing capability disclosures, not merely as containers for executable instructions.