LLM Agent Evaluation

LLM: Large Language Model

The field studies whether language-model agents can use tools, manage state, follow instructions, and achieve users’ goals across multi-step interactions. Benchmarks place agents in executable settings such as coding repositories, websites, terminals, business workflows, and medical or scientific tasks, while methods use hidden tests, state checks, trajectory analysis, and simulated users to assess performance. Evaluation increasingly includes cost, latency, privacy, safety, consistency, and process quality—not just whether the final answer is correct. Results remain difficult to compare because harnesses and scoring choices can change rankings, and strong performance on short or isolated tasks often fails to transfer to long-horizon real-world work.

Latest papers

1,217 papers231 papers in the last four weeks, up 62% on the four weeks before. 1.3% of all new papers.

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents
Apr 13, 2026Yijuan Liang, Xinghao Chen, Yifan Ge +5LLM Agent Evaluation AI Agent Benchmarks
University of Science and Technology of China · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University
SLALOM: Simulation Lifecycle Analysis via Longitudinal Observation Metrics for Social Simulation
Apr 13, 2026Juhoon Lee, Joseph SeeringLarge Language Model-Based Social Simulation LLM Agent Evaluation
School of Computing, KAIST, Daejeon, Republic of Korea
YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents
Apr 13, 2026Victor De Lima, Grace Hui YangOutput Elicitation LLM Agent Evaluation
InfoSense Lab, Georgetown University
AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web
Apr 13, 2026Shanshan Zhong, Kate Shen, Chenyan XiongWeb Agent Benchmarks Web Agents
Language Technologies Institute, School of Computer Science, Carnegie Mellon University · Anaxi Labs
OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation
Apr 13, 2026Xiaomeng Hu, Yinger Zhang, Fei Huang +7LLM Agent Evaluation AI Agent Benchmarks
Qwen Team, Alibaba Group · The Chinese University of Hong Kong
Diversity Without Fidelity: A Solver-Sampler Mismatch in Multi-Agent LLM Negotiation Simulation
Apr 12, 2026Sandro AndricMulti-Agent LLM Systems Multi-Agent Negotiation
Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Apr 12, 2026Wanyi Chen, Xiao Yang, Xu Yang +7Agentic RL AI Agent Benchmarks
Soochow University · Microsoft Research Asia · Stony Brook University +2
The Amazing Agent Race: Strong Tool Users, Weak Navigators
Apr 11, 2026Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2LLM Agent Evaluation AI Agent Benchmarks
University of Minnesota Twin Cities · Yonsei University · Grammarly
FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks
Apr 11, 2026Yupeng Cao, Haohang Li, Weijin Liu +11Agent Trajectory Analysis LLM Tool Use
Stevens Institute of Technology · Independent Researcher · The FinAI +2
Can Coding Agents Be General Agents?
Apr 10, 2026Maksim Ivanov, Abhijay Rana, Gokul PrabhakaranCoding Agents Business Process Automation
Agentic Labs
HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
Apr 10, 2026Suhana Bedi, Ryan Welch, Ethan Steinberg +12Computer-Use Agent Benchmarks LLM Agent Evaluation
Stanford University · Kinetic Systems · Stanford Health Care
Agentic Jackal: Live Execution and Semantic Value Grounding for Text-to-JQL
Apr 10, 2026Vishnu Murali, Anmol Gulati, Elias Lumer +3Tool-Augmented Language Model Agents LLM Agent Evaluation
Commercial Technology and Innovation Office, PricewaterhouseCoopers, U.S.
Many-Tier Instruction Hierarchy in LLM Agents
Apr 10, 2026Jingyu Zhang, Tianjian Li, William Jurayj +3LLM Agents AI Agent Safety Benchmarks
Johns Hopkins University
SAGE: A Service Agent Graph-guided Evaluation Benchmark
Apr 10, 2026Ling Shi, Yuqin Dai, Ziyin Wang +7LLM Agent Evaluation Agent Evaluation
Tianjin University, Tianjin, China · Tsinghua University, Beijing, China · Beihang University, Beijing, China +3
DRBENCHER: Can Your Agent Identify the Entity, Retrieve Its Properties and Do the Math?
Apr 10, 2026Young-Suk Lee, Ramon Fernandez Astudillo, Radu FlorianAI Agent Benchmarks Scientific Agent Benchmarks
IBM Research
ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents
Apr 9, 2026Kenan Li, Qirui Jin, Liao Zhu +13Software Engineering Agents LLM Agent Evaluation
Microsoft · Georgia Institute of Technology
CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V
Apr 9, 2026John Chen, Sihan Cheng, Can Gurkan +1LLM Agent Evaluation AI Agent Benchmarks
Northwestern University
Sell More, Play Less: Benchmarking LLM Realistic Selling Skill
Apr 8, 2026Xuanbo Su, Wenhao Hu, Le Zhan +8LLM Agent Evaluation Persuasion in Language Models
Bairong Inc. · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy +4
Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents
Apr 8, 2026Heng Zhou, Zelin Tan, Zhemeng Zhang +15LLM Agent Workflow Optimization LLM Agents
On Emotion-Sensitive Decision Making of Small Language Model Agents
Apr 8, 2026Jiaju Lin, Xingjian Du, Qingyun Wu +2LLM Decision-Making LLM Agent Evaluation
Pennsylvania State University · University of Rochester · William & Mary
RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation
Apr 7, 2026Ming Zhu, Juntao Tan, Rithesh Murthy +6User Simulation Human Behavior Simulation
Salesforce AI Research
WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks
Apr 7, 2026Guruprasad Viswanathan Ramesh, Asmit Nayak, Basieem Siddique +1Web Agents LLM Agent Evaluation
University of Wisconsin-Madison
CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments
Apr 7, 2026Gustav Keppler, Moritz Gstür, Veit HagenmeyerIndustrial Control System Security LLM Agent Evaluation
Karlsruhe Institute of Technology (KIT)
Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution
Apr 7, 2026Kai Yu, Zhenhao Zhou, Junhao Zeng +8LLM Agent Evaluation Software Engineering Benchmarks
Fudan University, China · Nanyang Technological University, Singapore
LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo
Apr 7, 2026Ojas Jain, Dhruv KumarLLM Agent Evaluation Strategic Reasoning Risks in Language Models
Department of Computer Science and Information Systems, BITS Pilani
Market-Bench: Benchmarking Large Language Models on Economic and Trade Competition
Apr 7, 2026Yushuo Zheng, Huiyu Duan, Zicheng Zhang +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Apr 6, 2026Xiangyi Li, Kyoung Whan Choe, Yimin Liu +12LLM Agent Evaluation LLM Agents
BenchFlow · RLWRLD · Ohio State University +9
How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings
Apr 6, 2026Yujian Liu, Jiabao Ji, Li An +3LLM Agent Skill Retrieval LLM Agents
UC Santa Barbara · MIT CSAIL · MIT-IBM Watson AI Lab
ESL-Bench: An Event-Driven Synthetic Longitudinal Benchmark for Health Agents
Apr 3, 2026Chao Li, Cailiang Liu, Ang Gao +7Synthetic Clinical Data Generation AI Agent Benchmarks
Shanda Group
NetAgentBench: A State-Centric Benchmark for Evaluating Agentic Network Configuration
Apr 3, 2026Ahmed Twabi, Yepeng Ding, Tohru KondoFinite-State Machine LLM Agent Evaluation
Informatics & Data Science, Hiroshima University, Hiroshima, Japan

Apr 13, 2026J/K move · Enter open · S save

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

Yijuan Liang, Xinghao Chen, Yifan Ge, Ziyi Wu+4

University of Science and Technology of China · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University

LLM Agent Evaluation AI Agent Benchmarks LLM Tool Use LLM Agent Training Tool Use

Open paper AskPDF arXiv

Tool-use capability is a fundamental component of LLM agents, enabling them to interact with external systems through structured function calls. However, existing research exhibits inconsistent interaction representations, largely overlooks the structural distribution of tool-use trajectories, and relies on incompatible evaluation benchmarks. We present UniToolCall, a unified framework for tool learning that standardizes the entire pipeline from toolset construction and dataset generation to evaluation. The framework curates a large tool pool of 22k+ tools and constructs a hybrid training corpus of 390k+ instances by combining 10 standardized public datasets with structurally controlled synthetic trajectories. It explicitly models diverse interaction patterns, including single-hop vs. multi-hop and single-turn vs. multi-turn, while capturing both serial and parallel execution structures. To support coherent multi-turn reasoning, we further introduce an Anchor Linkage mechanism that enforces cross-turn dependencies. Furthermore, we convert 7 public benchmarks into a unified Query--Action--Observation--Answer (QAOA) representation with fine-grained evaluation at the function-call, turn, and conversation levels. Experiments show that fine-tuning Qwen3-8B on our dataset substantially improves tool-use performance. Under the distractor-heavy Hybrid-20 setting, achieves 93.0% single-turn Strict Precision, outperforming commercial models including GPT, Gemini, and Claude.