LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors
Apr 23, 2026Chenghao Yang, Yuning Zhang, Zhoufutu Wen +4Tool-Augmented Language Model Agents LLM Agent Evaluation
School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P +1
Chasing the Public Score: User Pressure and Evaluation Exploitation in Coding Agent Workflows
Apr 22, 2026Hardy Chen, Nancy Lau, Haoqin Tu +8Coding Agents LLM Agent Evaluation
UC Santa Cruz · UT Dallas · NUS
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Apr 22, 2026Shanshan Zhong, Yi Lu, Jingjie Ning +7Continual Learning for LLM Agents LLM Agent Evaluation
Carnegie Mellon University · Amazon AGI
Time Series Augmented Generation for Financial Applications
Apr 21, 2026Anton Kolonin, Alexey Glushchenko, Evgeny Bochkov +1LLM Agent Evaluation AI Agent Benchmarks
SingularityNET Foundation
Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture the Flag Challenges
Apr 21, 2026Ali Al-Kaswan, Maksim Plotnikov, Maxim Hájek +3LLM Agent Evaluation AI Agent Security Benchmarks
Delft University of Technology Delft, The Netherlands
Forage V2: Knowledge Evolution and Transfer in Autonomous Agent Organizations
Apr 21, 2026Huaqing XieLLM Agent Evaluation Memory-Augmented Agents
Independent Researcher
AutomationBench
Apr 21, 2026Daniel Shepard, Robin SalimansLLM Agent Evaluation Business Process Automation
AI scientists produce results without reasoning scientifically
Apr 20, 2026Martiño Ríos-García, Nawaf Alampara, Chandan Gupta +5AI Agent Reliability Scientific Reasoning
Laboratory of Organic and Macromolecular Chemistry (IOMC), Friedrich Schiller University Jena, Humboldstr. 10, 07743 Jena, Germany · Department of Civil Engineering, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India · School of Interdisciplinary Research, Indian Institute of Technology Delhi, Hauz Khas, New Delhi 110016, India +3
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
Apr 20, 2026Wentao Shi, Yu Wang, Yuyang Zhao +8LLM-as-a-Judge LLM Agent Evaluation
University of Science and Technology of China · National University of Singapore · Meituan
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Apr 20, 2026Xinping Lei, Xinyu Che, Junqi Xiong +16LLM Agent Evaluation Code Language Models
Nanjing University · Kuaishou Technology
Do LLMs Need to See Everything? A Benchmark and Study of Failures in LLM-driven Smartphone Automation using Screentext vs. Screenshots
Apr 20, 2026Shiquan Zhang, Tianyi Zhang, Le Fang +3Mobile GUI Automation LLM Agent Evaluation
School of Computing and Information Systems, University of Melbourne, Melbourne, Australia · School of Computer Science, University of Auckland, Auckland, New Zealand
Agents Explore but Agents Ignore: LLMs Lack Environmental Curiosity
Apr 19, 2026Leon Engländer, Sophia Althammer, Ahmet Üstün +2LLM Agent Evaluation Tool-Using Agents
Cohere · Poolside
HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
Apr 19, 2026Chao Jin, Wenkui Yang, Hao Sun +6GUI Agents Hallucination in Language Models
MAIS&NLPR, Institute of Automation, Chinese Academy of Sciences
Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
Apr 18, 2026Tyler H. Merves, Michael H. Conaway, Joseph M. Escobar +2LLM Agent Evaluation Cybersecurity
Department of Cybersecurity · Department of Information Sciences and Technology
If Only My CGM Could Speak: A Privacy-Preserving Agent for Question Answering over Continuous Glucose Data
Apr 18, 2026Yanjun Cui, Ali Emami, Temiloluwa Prioleau +1Privacy-Preserving Language Models Healthcare
Dartmouth College · Emory University
Don't Start What You Can't Finish: A Counterfactual Audit of Support-State Triage in LLM Agents
Apr 17, 2026Eren UnluLLM Agent Evaluation LLM Refusal Behavior
Globeholder Paris, France
Evaluating Tool-Using Language Agents: Judge Reliability, Propagation Cascades, and Runtime Mitigation in AgentProp-Bench
Apr 17, 2026Bhaskar GurramAI Agent Reliability Tool-Augmented Language Model Agents
Zasti Inc., Ashburn, VA, USA
SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems
Apr 17, 2026Hikaru Shindo, Hanzhao Lin, Lukas Helff +2LLM Agent Evaluation AI Agent Benchmarks
Technical University of Darmstadt · German Research Center for Artificial Intelligence · CERTAIN +2
Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis
Apr 16, 2026Ke Zhang, Patricio Gallardo, Maziar Raissi +1Code Translation LLM Agent Evaluation
University of California, Riverside.
CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas
Apr 16, 2026Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita +2Social Dilemmas Cooperative Game Theory
Carnegie Mellon University · Foundations of Cooperative AI Lab (FOCAL) · Jinesis Lab, University of Toronto & Vector Institute +3
From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench
Apr 16, 2026Ke Xu, Yuhao Wang, Yu WangLLM Agent Evaluation Agent Evaluation
Shanghai Jiao Tong University
Task-Level AI Readiness Assessment for Business Process Management:The T-IPO Model and LARA Matrix in Financial-Services IT Operations
Apr 16, 2026Mingjun Li, Xiaojun YeLLM Agent Evaluation Multi-Criteria Decision Analysis
Tsinghua University, Beijing, China
HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks
Apr 16, 2026Fan Cui, Hongyuan Hou, Zizhang Luo +2Automated Program Repair Electronic Design Automation
Peking University · Beijing, China
DR $^{3}$ -Eval: Towards Realistic and Reproducible Deep Research Evaluation
Apr 16, 2026Qianqian Xie, Qingheng Xiong, He Zhu +16LLM Agent Evaluation Long-Horizon Agent Evaluation
Nanjing University · M-A-P · Nanjing University of Science and Technology +2
GeoAgentBench: A Dynamic Execution Benchmark for Tool-Augmented Agents in Spatial Analysis
Apr 15, 2026Bo Yu, Cheng Yang, Dongyang Hou +6LLM Agent Evaluation AI Agent Benchmarks
School of Geosciences and Info-Physics, Central South University, Changsha, China. · School of Earth Sciences and Spatial Information Engineering, Hunan University of Science and Technology, Xiangtan, China.
Exploration and Exploitation Errors Are Measurable for Language Model Agents
Apr 14, 2026Jaden Park, Jungtaek Kim, Jongwon Jeong +3Exploration-Exploitation Tradeoff LLM Agent Evaluation
University of Wisconsin–Madison · KRAFTON · Ludo Robotics
Grounded autonomous scrutiny at scale: emergent critique from reproduction of published computational physics papers
Apr 14, 2026Haonan HuangLLM Agent Evaluation Scientific Reproducibility
Department of Physics, Princeton University
From Plan to Action: How Well Do Agents Follow the Plan?
Apr 13, 2026Shuyang Liu, Saman Dehghan, Jatin Ganhotra +2Software Engineering Agents LLM Agent Planning
University of Illinois Urbana-Champaign Urbana, USA · IBM New York, USA
The A-R Behavioral Space: Execution-Level Profiling of Tool-Using Language Model Agents in Organizational Deployment
Apr 13, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent Evaluation Tool-Using Language Agents
Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, Wales CF5 2YB, UK · School of Professional Studies, Clark University, Worcester, MA 01610 USA · Harvard Medical School, Harvard University, Boston, MA 02115 USA
RPA-Check: A Multi-Stage Automated Framework for Evaluating Dynamic LLM-based Role-Playing Agents
Apr 13, 2026Riccardo Rosati, Edoardo Colucci, Massimiliano Bolognini +2Role-Playing Agent Evaluation LLM Agent Evaluation
Department of Political Sciences, Communication and International Relations, University of Macerata, Via Don Minzoni 22/A, Macerata, 62100, Italy · Deparment of Information Engineering, Università Politecnica delle Marche, Via Brecce Bianche 12, Ancona, 60131, Italy · Department of Law, University of Macerata, Piaggia dell’Università 2, Macerata, 62100, Italy

Apr 23, 2026J/K move · Enter open · S save

When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

Chenghao Yang, Yuning Zhang, Zhoufutu Wen, Tao Gong+3

School of Cyber Science and Technology, USTC · Anhui Province Key Laboratory of Digital Security · 3M-A-P+1

Tool-Augmented Language Model Agents LLM Agent Evaluation AI Agent Benchmarks Tool-Using Agents Knowledge Distillation Behavioral Consistency of LLM Agents

Open paper AskPDF arXiv

Model distillation is a primary driver behind the rapid progress of LLM agents, yet it often leads to behavioral homogenization. Many emerging agents share nearly identical reasoning steps and failure modes, suggesting they may be distilled echoes of a few dominant teachers. Existing metrics, however, fail to distinguish mandatory behaviors required for task success from non-mandatory patterns that reflect a model's autonomous preferences. We propose two complementary metrics to isolate non-mandatory behavioral patterns: Response Pattern Similarity (RPS) for verbal alignment and Action Graph Similarity (AGS) for tool-use habits modeled as directed graphs. Evaluating 18 models from 8 providers on

τ

-Bench and

τ^2

-Bench against Claude Sonnet 4.5 (thinking), we find that within-family model pairs score 5.9 pp higher in AGS than cross-family pairs, and that Kimi-K2 (thinking) reaches 82.6%

S_{\text{node}}

and 94.7%

S_{\text{dep}}

, exceeding Anthropic's own Opus 4.1. A controlled distillation experiment further confirms that AGS distinguishes teacher-specific convergence from general improvement. RPS and AGS capture distinct behavioral dimensions (Pearson

r

= 0.491), providing complementary diagnostic signals for behavioral convergence in the agent ecosystem. Our code is available at https://github.com/Syuchin/AgentEcho.