LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

Finding Is Not Returning, Returning Is Not Reconstructing: A Multi-Dimensional Evaluation of Agent Social Re-entry
Oct 29, 2026Sheng Wan, Cong Ma, Sichen Dong +4LLM Agent Evaluation LLM Agents
The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · City University of Hong Kong, Hong Kong, China +4
Practical Assessment of Agentic Systems in Acoustics
Oct 29, 2026Arne Hildenbrand, Philipp Diel, Zhuo Cao +3LLM Agent Harnesses LLM Agent Evaluation
Chair of Vibroacoustics of Vehicles and Machines, Department of Engineering Physics and Computation, TUM School of Engineering and Design, Technical University of Munich, Boltzmannstrasse 15, 85748 Garching, Germany · Chair of Thermodynamics of Mobile Energy Conversion Systems, Faculty of Mechanical Engineering, RWTH Aachen University, Forckenbeckstrasse 4, 52074 Aachen, Germany · IAS-8, Forschungszentrum Jülich, Wilhelm-Johnen-Straße, 52425 Jülich, Germany
LLM Agent-Based Design of RISC-V CPUs with FPGA-in-the-Loop Evaluation
Oct 29, 2026Leo Itoh, Junichiro KadomotoElectronic Design Automation LLM Agent Evaluation
The University of Tokyo Tokyo, Japan
How Specific Do I Need to Be? Effects of User Query Underspecification on Deep Research Agents
Oct 29, 2026Yi Wen, Xiangjue Dong, Bingxuan Li +4Deep Research Agents Agentic Search
University of Maryland, College Park · Texas A&M University · University of Illinois, Urbana-Champaign
AgentDecept: Benchmarking Deceptive Behavior in Tool-Using LLM Agents under Pressure
Oct 29, 2026Yehan Zhang, Qianqi Yan, Hang Yin +5Deception in Language Models LLM Agent Evaluation
University of California, Santa Barbara · eBay
HERA: Assessing an Autonomous Agent for Quantum Device Measurement
Oct 29, 2026Zachary W. Gomez, Nathan Bittner, Jordan T. McCourt +7Autonomous Experimentation Autonomous Research Agents
Stanford Institute for Materials and Energy Sciences, SLAC National Accelerator Laboratory, Menlo Park, CA 94025 · Department of Physics, Stanford University, Stanford, CA 94305 · Research Center for Electronic and Optical Materials, National Institute for Materials Science, 1-1 Namiki, Tsukuba 305-0044, Japan +1
TreasuryQA Bench: Can AI Agents Work an Archive of Scanned Financial Records?
Oct 28, 2026Ritiz Tambi, Sahar Shayegan, Joshua Meyer +5AI Agent Benchmarks Document QA
Veris AI
Beyond fabrication: Misattributed numbers in papers written by AI research agents
Oct 28, 2026Mosab Alfaqeeh, Omar Hourani, Muath Hourani +1Claim Verification AI Agent Auditing
Al-Ahliyya Amman University, Amman, Jordan · Binghamton University, State University of New York, Binghamton, NY, USA
Think Before You Act: Decision Gate for LLM Agents
Oct 28, 2026Iftach Shoham, Yuval David, Offer Akrabi +5LLM Agent Verification Runtime Enforcement for AI Agents
IBM SIL
Repairs for Whom? Evaluator-Executor Mismatch in Selecting Repairs for Failed LLM Agents
Oct 28, 2026Yijun Liao, Fanwei LiangLLM Agent Evaluation Process-Based Agent Evaluation
Independent Researcher · South China University of Technology
EPSBench: Benchmarking LLM Agents for Earnings Forecasting
Oct 28, 2026Yanbin Yin, Chuyu Xu, Liqin Ye +2Financial Forecasting LLM Agent Evaluation
Georgia Institute of Technology
Vendster: An Evaluation Arena to Investigate How Forecasting Tool Quality Affects LLM Agent Behaviors in Buyer-Vendor Negotiation
Oct 28, 2026Charin PolpanumasLLM Agent Evaluation Forecasting Benchmarks
Amazon, Japan
FilingLoop: Evaluating and Evolving LLM Agents for Corporate Disclosure Review and Revision
Oct 27, 2026Wentao Zhang, Yilei Zhao, Guangyan Gan +2LLM Agent Evaluation Continual Learning for LLM Agents
Nanyang Technological University, Singapore · The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
What Does an Ontology Catch? Curated, LLM-Generated and Data-Derived Ontologies as Checkers of LLM Agents over Knowledge Graphs
Oct 27, 2026Michal Orzechowski, Rund Tawfiq, Tomasz KosciolekLLM Answer Verification LLM Agent Evaluation
ACK Cyfronet AGH, Krakow, Poland · AGH University of Krakow, Krakow, Poland · Sano Centre for Computational Medicine, Krakow, Poland
Where Should the Evaluation Budget Go in LLM-Driven Agent Scaffold Evolution?
Oct 27, 2026Sichun Luo, Yi Huang, Mukai Li +7LLM Agent Evaluation LLM Agent Efficiency
The University of Hong Kong · JIUTIAN Research, China Mobile · City University of Hong Kong +2
Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution
Oct 27, 2026Jiawei Zhou, Bo Peng, Yuxuan Huang +4Agent Skill Learning LLM Agent Skill Evaluation
Shanghai AI Laboratory · Fudan University · Shanghai Jiao Tong University +2
OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
Oct 27, 2026Humza Nusrat, Kundan ThindMedical AI Agents LLM Agent Evaluation
Department of Radiation Oncology, Henry Ford Health, Detroit, Michigan, USA · Department of Radiology, College of Human Medicine, Michigan State University, East Lansing, Michigan, USA
AutoVend: Benchmarking LLM Agents on Long-Horizon Autonomous Vending Operations
Oct 27, 2026Qiming Shi, Yikun Li, Beidi Luan +5Long-Horizon LLM Agents LLM Agent Evaluation
StepFun · Zhejiang University · Xi’an Jiaotong-Liverpool University +1
Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue
Oct 27, 2026Baiqiao Zhang, Shuai Ma, Zhifeng Liao +9Human-AI Decision Making Human Behavior Prediction
The Hong Kong University of Science and Technology, Hong Kong SAR, China · Institute of Software, Chinese Academy of Sciences, Beijing, China · Shandong University, Weihai, China +1
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
Oct 27, 2026Binhang Qi, Yiming Liu, Yun Lin +3Automated Test Generation LLM Agent Evaluation
National University of Singapore, Singapore · Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China +1
Benchmarking Task Takeover Across Agent Handoffs
Oct 27, 2026Wenyuan Li, Guang Li, Dongyuan Li +3LLM Agent Evaluation Agent Evaluation
Hokkaido University Sapporo, Japan · The University of Tokyo Tokyo, Japan
When to Intervene? An Empirical Study of Errors and Self-Recovery in LLM Agents
Oct 27, 2026Chiming Duan, Tong Jia, Yingxi Chen +3LLM Agents LLM Agent Reliability
Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China
Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
Oct 27, 2026Nardine Basta, Venus Haghighi, Dali Kaafar +1LLM Agent Evaluation LLM Agent Safety
Macquarie University, Sydney, Australia
Your Agent Acts Differently When You're Around: How User Presence Changes Agent Behavior in Negotiation
Oct 27, 2026Jane Pan, You Lyu, Abe Bohan Hou +2Human-AI Interaction LLM Agent Evaluation
New York University · Stanford University
Context Is a Pipeline, Not a Window: Information Attrition in LLM Agent Harnesses
Oct 26, 2026Yuvraj Sehgal, Mahsa Panahandeh, Naser Ezzati-JivanLLM Context Management LLM Agent Harnesses
Department of Computer Science Brock University St. Catharines, Ontario, Canada · School of Electrical Engineering and Computer Science University of Ottawa Ottawa, Ontario, Canada
Do Coding Agents Still Need a Debugger? Frontier-Model-Based Fault Localization at the Accuracy Ceiling
Oct 26, 2026Christoph Daxerer, Johann Glock, Christian Macho +3AI Coding Agents Software Engineering
Software Competence Center Hagenberg GmbH (SCCH), Hagenberg, Austria · Department of Informatics Systems, University of Klagenfurt, Klagenfurt, Austria · School of Informatics, Communications and Media, University of Applied Sciences Upper Austria, Hagenberg, Austria
A Unified Arena for LLM Agents across Diverse Eurogame
Oct 26, 2026Fengrui Zhang, Xiang Li, Ruomeng Ding +6LLM Agent Evaluation Game-Playing Agents
Alibaba Group · Nanjing University · National University of Singapore +3
Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents
Oct 26, 2026Tingzhu Bi, Xun Zhang, Jinglin Li +2Root Cause Analysis LLM Agent Harnesses
Peking University, Beijing, China · Beijing University of Posts and Telecommunications, Beijing, China
ShutdownMod: Turning Any Trajectory into a Shutdown Evaluation
Oct 26, 2026Param Biyani, Sebastien Luyasu, Cedric Potvliege +3AI Agent Safety AI Agent Evaluation
MATS · SEAR · University of Washington +2
Intent-Preserving Adversarial User Simulation for Multi-Turn Agent Evaluation
Oct 26, 2026Atri Vivek Sharma, Brian Formento, Alessio LomuscioLLM Agent Evaluation User Simulation
Imperial College London · Safe Intelligence

Oct 29, 2026J/K move · Enter open · S save

Finding Is Not Returning, Returning Is Not Reconstructing: A Multi-Dimensional Evaluation of Agent Social Re-entry

Sheng Wan, Cong Ma, Sichen Dong, Qian'Ang Mao+3

The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China · City University of Hong Kong, Hong Kong, China+4

LLM Agent Evaluation LLM Agents Conversational Memory Long-Term Conversational Memory

Open paper AskPDF arXiv

As Large Language Model (LLM) agents increasingly participate in persistent social environments, understanding their long-term social interaction capabilities becomes increasingly important. While existing research has advanced long-term memory and conversational context retrieval, less attention has been paid to whether agents can autonomously re-engage with previously inactive conversations. In particular, remembering a past interaction does not necessarily mean that an agent can identify when to revisit it, decide to return, or reconstruct the context needed for a coherent continuation. To investigate this gap, we introduce agent social re-entry as a multi- dimensional evaluation problem, distinguishing three complementary capabilities: Finding, Returning, and Reconstructing. We develop a shared-snapshot evaluation framework that measures these dimensions independently, allowing us to examine how conversational resurfacing and persistent memory relate to different aspects of social re-entry. Evaluations across six LLM configurations show that resurfacing substantially improves target conversation identification, while persistent memory is associated with higher rates of autonomous target-directed replying. However, neither improvement is accompanied by consistent gains in contextual reconstruction. These results highlight the distinction between accessing previous conversations, choosing to re-engage, and maintaining conversational continuity. Our framework provides a way to evaluate these capabilities separately rather than relying on memory retrieval or conversational response quality alone.