LLM Agent Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

115 papers in the last four weeks, up 140% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 793

All topics

WorkBench Revisited: Workplace Agents Two Years On
Jun 10, 2026Olly Styles, Sam MillerComputer-Use Agent Benchmarks AI Agent Reliability
Independent researcher
Skill-Augmented AI Agents for Medical Research Analysis: An Exploratory Multi-Model Human Evaluation in an NSCLC Transcriptomic Biomarker Task
Jun 10, 2026Qianyu Yao, Fei Sun, Bocheng Huang +10AI Agents for Scientific Discovery LLM Agent Evaluation
AIPOCH PTE. LTD., Singapore
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
Jun 10, 2026Yizhou Chi, Eric Chamoun, Zifeng Ding +1Event Time Prediction LLM Agent Evaluation
Department of Computer Science and Technology University of Cambridge
Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness
Jun 10, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLLM Agent Evaluation
Lumivate (Lumi)
Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment
Jun 10, 2026Derek Yohn, Luke Flancher, Mirajul Islam +1Software Vulnerability Detection LLM Agent Evaluation
College of Engineering and Science, Florida Institute of Technology, Melbourne, Florida, USA
SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior
Jun 10, 2026Zhiyu Chen, Zihan Guo, Bo Huang +4LLM Agent Evaluation
Tongji University · Shanghai Innovation Institute · Sun Yat-sen University +1
AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerable
Jun 9, 2026Meysam Alizadeh, Fabrizio Gilardi, Mohsen Mosleh +1AI Agent Reliability AI Coding Agents
University of Oxford · University of Zurich · Technical University of Munich
From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents
Jun 9, 2026Yifan Li, Shengbin Yue, Boyu Feng +6LLM Agent Evaluation AI Agent Benchmarks
The Chinese University of Hong Kong · Fudan University · Tencent +2
T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains
Jun 9, 2026Genta Indra Winata, Amartya Chakraborty, Yuzhen Lin +12Customer Support Automation LLM Agent Evaluation
AI Foundations, Capital One
Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?
Jun 9, 2026Tengchao Lv, Dongdong Zhang, Jiayu Ding +10LLM Agent Evaluation AI Agent Benchmarks
Microsoft Research
Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
Jun 9, 2026Aman Sharma, Sushrut Thorat, Paras ChopraCoding Agents LLM Agent Evaluation
Lossfunk
Skill Coverage: A Test Adequacy Metric for Agent Skills
Jun 9, 2026Boyin Tan, Xiaowei Huang, Youcheng SunLLM Agent Evaluation AI Agent Benchmarks
1Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates · University of Liverpool Liverpool, United Kingdom
STAGE-Claw: Automated State-based Agent Benchmarking for Realistic Scenarios
Jun 9, 2026Sirui Liang, Bohan Yu, Peiyu Wang +8Computer-Use Agent Benchmarks LLM Agent Evaluation
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation · Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences · Zhongguancun Academy +3
Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents
Jun 9, 2026Sawyer Zhang, Alexander Wang, Sophie LeiLLM-as-a-Judge LLM Agent Evaluation
Lumivate (Lumi)
Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate
Jun 9, 2026Ali Keramati, Justin Cheok, Jacob Horne +1LLM Agent Evaluation Multi-Agent Debate
University of California, Irvine
The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge
Jun 9, 2026Ali Keramati, Justin Cheok, Jacob Horne +1LLM Agent Evaluation Multi-Agent Debate
University of California, Irvine
H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions
Jun 8, 2026Shiping Zhu, Yibo Yang, Zhengyang Wang +3Multimodal Memory LLM Agent Memory
Jilin University · Shanghai Jiao Tong University · University of California at Merced
REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces
Jun 8, 2026Xiaofeng Lin, Yingxu Wang, Tung Sum Thomas Kwok +4LLM Agent Evaluation LLM Agents
University of California, Los Angeles, USA · Cisco, USA · University of Mississippi, USA
Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human
Jun 8, 2026Emre TuranLearning to Defer LLM Guardrails
Independent Researcher
PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting
Jun 7, 2026Youran Sun, Xingyu Ren, Kejia Zhang +2Multi-Agent Orchestration LLM Agent Evaluation
University of Maryland · The Chinese University of Hong Kong · Stanford University
Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework
Jun 7, 2026Aman Gupta, Kevin Rossell, Edesio Alcobaça +8Customer Support Automation LLM Agent Evaluation
Nubank · Palo Alto, USA · Mexico City, Mexico +1
Data Agents Under Attack: Vulnerabilities in LLM-Driven Analytical Systems
Jun 7, 2026Kuncan Wang, Ziting Wang, Peizhuo Lv +4LLM Agent Evaluation Cybersecurity
Nanyang Technological University, Singapore · The Hong Kong Polytechnic University · Tsinghua University
Scaffold Effects on GAIA: A Controlled Comparison
Jun 7, 2026Jason StaraceLLM Agent Evaluation AI Agent Benchmarks
Independent Researcher
The Consistency Illusion: How Multi-Agent Debate Hides Reasoning Misalignment
Jun 7, 2026Xiaoyang Wang, Christopher C. YangLLM Agent Evaluation Multi-Agent Debate
1Drexel University
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Jun 6, 2026Kale-ab Abebe Tessera, Andras Szecsenyi, Cameron Barker +7Multi-Agent Coordination LLM Agent Evaluation
University of Edinburgh · University of Oxford · University College London
To Nuke or Not to Nuke: LLMs' (Missing) Ethical Reasoning and Actions in a High-Stakes Decision-Making Simulation
Jun 6, 2026John Chen, Sihan Cheng, Can Gurkan +1Moral Reasoning in Language Models LLM Agent Evaluation
University of Arizona, Tucson, AZ, USA · Northwestern University, Evanston, IL, USA
Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents
Jun 6, 2026Hyogon Ryu, Jeonghwan Kim, Yewon Lim +3LLM-as-a-Judge LLM Agent Evaluation
KRAFTON
Aligned but Not Partner-Specific: How Multimodal LLM Agents Succeed in Reference Games Without Forming Conceptual Pacts
Jun 6, 2026Po-Ya Angela Wang, Chinmaya Mishra, Aslı Özyürek +2Repeated Games LLM Agent Evaluation
Co-Intelligence Humanities AI Future Lab and Graduate Institute of Linguistics, National Taiwan University · Multimodal Language Department, Max Planck Institute for Psycholinguistics · Donders Institute for Brain, Cognition and Behaviour, Radboud University +1
Does Persona Make LLMs K-pop Fans? A Pilot Study of LLM-Based Online Concert Audience Agents
Jun 5, 2026Kirak Kim, Hyojin Kim, Yejin Son +2Multi-Agent LLM Systems LLM Agent Evaluation
Graduate School of Culture Technology, KAIST, Daejeon, South Korea · Department of Artificial Intelligence, Yonsei University, Seoul, South Korea
MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights
Jun 5, 2026Yilun Liu, Miao Zhang, Shimin Tao +9Multilingual Language Model Evaluation LLM Evaluation
Huawei, China

Jun 10, 2026, cs.AIJ/K move · Enter open · S save

WorkBench Revisited: Workplace Agents Two Years On

Olly Styles, Sam Miller

Independent researcher

Computer-Use Agent Benchmarks AI Agent Reliability LLM Agent Evaluation AI Agent Benchmarks AI Agent Safety Benchmarks LLM Agent Safety+1

Open paper AskPDF arXiv

The best agent on WorkBench in March 2024, GPT-4, completed just 43% of tasks. We revisit the benchmark in June 2026 and find that the best agent to date, Claude Fable 5, now completes 98%. Beyond this considerable progress in frontier agent performance, three things stand out. First, unintended harmful actions, such as emailing the wrong person, fell from 26% of tasks for GPT-4 to 1.9% for Claude Fable 5; capability and safety go together on WorkBench rather than trade off, so the models that finish the most tasks also do the least unintended damage. Second, the rise of open-weight models has drastically lowered costs for a performance level that was only accessible to proprietary models, while frontier costs have stayed stable. Third, while several classes of error have been eliminated, frontier models still make some basic mistakes that occasionally result in irreversible harm. We release an updated version of the benchmark with data and code quality improvements, new model scores, and analysis of agent progress on WorkBench since 2024.

Momentum

Related topics

Latest papers 793