LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.org
Dec 12, 2025Jaehyung Lee, Justin Ely, Kent Zhang +3AI Agents for Scientific Discovery LLM Agent Evaluation
Department of Materials Science and Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Physics and Astronomy, West Virginia University, Morgantown, WV 26506, USA
PPTArena: A Benchmark for PowerPoint Editing
Dec 2, 2025Michael Ofengenden, Yunze Man, Ziqi Pang +2LLM Agent Evaluation AI Agent Benchmarks
University of Illinois Urbana-Champaign · University of California, Berkeley
Multi-Agent LLM Orchestration Achieves Deterministic, High-Quality Decision Support for Incident Response
Nov 19, 2025Philip DrammehMulti-Agent LLM Systems LLM Agent Orchestration
Independent Researcher
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents
Nov 4, 2025Jiayu Liu, Cheng Qian, Zhaochen Su +4LLM Planning LLM Agent Evaluation
The Hong Kong University of Science and Technology · University of Illinois Urbana-Champaign · Tsinghua University
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
Oct 22, 2025Gil Pasternak, Dheeraj Rajagopal, Julia White +4LLM Agent Evaluation AI Agent Benchmarks
Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
Oct 18, 2025Vamshi Krishna Bonagiri, Ponnurangam Kumaragurum, Khanh Nguyen +1LLM Agent Evaluation Tool-Using Agents
University of California, Berkeley · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · International Institute of Information Technology, Hyderabad (IIIT Hyderabad)
BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
Sep 27, 2025Zehua Zhang, Ati Priya Bajaj, Divij Handa +14Software Engineering Agents Software Engineering Benchmarks
School of Computing and Augmented Intelligence, Arizona State University
HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
Sep 10, 2025Benjamin Sturgeon, Daniel Samuelson, Jacob Haimes +1LLM Agent Evaluation Human-AI Interaction
Apart Research · AI Safety Cape Town · University of Chicago +2
Are LLM Agents Behaviorally Coherent? A Latent Profiling Approach for Social Simulation
Sep 3, 2025James Mooney, Josef Woldense, Zheng Robert Jia +4Behavioral Consistency of LLM Agents LLM Agent Evaluation
Department of Computer Science and Engineering, University of Minnesota · Department of African American & African Studies, University of Minnesota · Department of Sociology, University of Chicago +1
Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents
Aug 6, 2025Thassilo M. Schiepanski, Nicholas PiëlLLM Agent Evaluation Web Agents
Surfly BV
Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
Jul 7, 2025Yuanzhe Hu, Yu Wang, Julian McAuleyLLM Agent Memory LLM Agent Evaluation
University of California, San Diego
SheetMind: Actions Set Accuracy, Agents Set the Failure Mode
Jun 14, 2025Lyuhao Chen, Xi Cheng, Yanming Kang +11Multi-Agent LLM Systems LLM Agent Evaluation
Carnegie Mellon University · Cornell University · University of California, Berkeley +2
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge
LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?
May 17, 2025Idriss Malek, Omar Choukrani, Daniil Orel +5LLM Planning LLM Agent Evaluation
Tandon School of Engineering New York University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) Abu Dhabi, UAE
Towards LLM Agents for Earth Observation
Apr 16, 2025Chia Hsiang Kao, Wenting Zhao, Cheryl Lam +10Code Generation LLM Agent Evaluation
Cornell University · Columbia University
Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
Mar 28, 2025Shengyue Guan, Jindong Wang, Jiang Bian +3LLM Agent Evaluation Multi-Turn Dialogue Evaluation
Microsoft, China
How Well Can Modern LLMs Act as Agent Cores in Radiology Environments?
Dec 12, 2024Qiaoyu Zheng, Chaoyi Wu, Weike Zhao +5Healthcare LLM Agent Evaluation
Shanghai Jiao Tong University, Shanghai, China · Department of Radiology, Renmin Hospital of Wuhan University, Wuhan, China

Dec 12, 2025J/K move · Enter open · S save

AGAPI-Agents: An Open-Access Agentic AI Platform for Accelerated Materials Design on AtomGPT.org

Jaehyung Lee, Justin Ely, Kent Zhang, Akshaya Ajith+2

Department of Materials Science and Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Physics and Astronomy, West Virginia University, Morgantown, WV 26506, USA

AI Agents for Scientific Discovery LLM Agent Evaluation Materials Property Prediction Tool-Using Agents Computational Materials Science

Open paper AskPDF arXiv

Agentic AI systems increasingly connect large language models (LLMs) to external scientific tools, yet whether and when tool access improves prediction accuracy remains uncharacterized. We present AGAPI (AtomGPT.org API), an open access platform integrating eight open-source LLMs with 18 REST endpoints (28 agent tools, 50 web apps) spanning materials databases, force fields, tight-binding band structures, X-ray diffraction, and protein structure. A three-evaluation residual decomposition on JARVIS-Leaderboard electronic-structure test sets separates agent pipeline fidelity from inherited density functional theory (DFT) functional bias. For bulk modulus and bandgap the agent reproduces JARVIS-DFT entries to numerical precision, so the experimental-reference degradation is functional bias, not agentic malfunction. On memorization-resistant test sets (57 defective supercells, 60 hypothetical compositions), tool-augmented mean absolute error (MAE) is below 0.005 eV versus 1.25 to 1.86 eV tool-free, confirming tools are indispensable where parametric knowledge is unavailable. We further demonstrate autonomous multi-step workflows including 10-operation defect-engineering pipelines. AGAPI is available at https://github.com/atomgptlab/agapi.