Tool-Use Evaluation

Follow this topic Ask about this topic

Tool-use evaluation studies whether language-model agents choose appropriate tools, provide valid arguments, interpret results, and complete tasks reliably. Benchmarks use executable APIs, simulators, and application environments, with deterministic state checks, trace analysis, and tests of long workflows, constraints, multimodal inputs, and tool failures. Researchers also measure efficiency, privacy, recovery, and whether tools actually improve performance using controls such as tool-free runs and deliberately misleading results. A central challenge is that final-answer scores and model-judge ratings can hide faulty processes, while harnesses, interfaces, and scoring rules can substantially change measured performance.

Latest papers

97 papers17 papers in the last four weeks, up 21% on the four weeks before. 0.1% of all new papers.

LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents
Oct 27, 2026Sinuo Wang, Wentao Zhao, Jiacheng Lu +6Financial Language Model Evaluation Benchmark Design
StepFun · Adelaide University · Tsinghua University +3
Separate Field Scoring Protects Unused Tools When Agents Pool Tool-Use Experience
Oct 22, 2026Abhijit ChakrabortyTool Selection for Language Model Agents Tool-Augmented Language Model Agents
Arizona State University
What the Trace Shows and What the Label Says: Auditing Behavioral Detectors in Agent Evaluation
Oct 21, 2026Idunnuoluwa A. Adeniji, Shilpika, Michael E. PapkaLLM Agent Evaluation Process-Based Agent Evaluation
Department of Computer Science, University of Illinois Chicago, Chicago, IL, USA · Argonne Leadership Computing Facility, Argonne National Laboratory, Lemont, IL, USA
VeriEvolve: Evaluator-First Evolution for Tool-Using Agents
Oct 21, 2026Yueshen Li, Kamer Ali Yuksel, Gokhan Tur +2Self-Evolving Agents Tool-Use Evaluation
University of Illinois Urbana-Champaign · aiXplain
Evolving Self-tuning Benchmarks for Long-Horizon Tool Use
Oct 19, 2026Hyunji Alex Nam, Michael Dennis, Natasha Jaques +1AI Agent Benchmarks Tool-Use Evaluation
Stanford University · Google Deepmind · University of Washington
EGI-BENCH: An Enterprise General Intelligence Benchmark Suite
Oct 19, 2026Zhiwei Liu, Zhujun Lan, Jielin Qiu +12LLM Agent Evaluation AI Agent Benchmarks
Salesforce AI Research
AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
Oct 15, 2026Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi +4LLM Evaluation LLM Agent Evaluation
Lexsi Labs
ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets
Oct 15, 2026Zichuan Xu, Yuyang Dong, Makoto Onizuka +1Tool-Use Evaluation LLM Agent Evaluation
University of Osaka · Initial S · Nagoya University
Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments
Oct 13, 2026Linhai Ma, Yan Wang, Lingfei Qian +2Tool-Use Evaluation LLM-as-a-Judge
The Fin AI, USA · Yale University, USA · Cardiff University, UK
What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents
Oct 10, 2026Shaikh Akib Shahriyar, MD. Akil Raihan Iftee, Armon Barton +1LLM Agent Evaluation Tool-Use Evaluation
Rochester Institute of Technology, Rochester, New York, USA · Naval Postgraduate School, Monterey, California, USA
GeoNatureAgent (GNA): A Framework and Benchmark for Pre-Production Evaluation of Tool-Using Agents on Geospatial and Environmental Tasks
Oct 6, 2026Gabriel Diaz-Ireland, Diego Prieto-Herráez, Mario García Peces +3Tool-Use Evaluation LLM Agent Evaluation
Universidad Católica de Ávila (UCAV) Ávila, Spain · Independent Researcher Madrid, Spain · Department of Earth and Planetary Sciences, Johns Hopkins University Baltimore, MD, USA +1
Asking Earns Nothing: Scoring the Decision to Act in BFCL Multi-Turn
Oct 3, 2026Yangze Liu, Zhongyi HanAgent Evaluation AI Agent Benchmarks
Shandong University
Knowledge or Calculator? Decomposing the Skill Premium in Verifiable Financial Agent Workflows
Oct 2, 2026Jermyn Zhen Yong Bek, Zhuang Qiang Bok, Zhongtian SunLLM Agents AI Agent Benchmarks
Deep Insight Labs Singapore, Singapore · University of Kent Canterbury, United Kingdom · University of Cambridge Cambridge, United Kingdom
Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models
Oct 1, 2026Juan S. SantillanaSupervised Fine-Tuning Tool-Use Evaluation
Independent Researcher
Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills
Oct 1, 2026Ngoc Phuoc An Vo, Aarya Doshi, Vadim SheininProcess-Based Agent Evaluation Tool-Use Evaluation
IBM · Georgia Institute of Technology · IBM Research
EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
Sep 30, 2026Shulin Tian, Junsu Kim, Shuai Liu +17Tool-Use Evaluation Tool Use in VLMs
S-Lab, Nanyang Technological University · A*STAR · KAIST +4
Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments
Sep 29, 2026Rohith Reddy Bellibatlu, Zichong Wang, Wenbin ZhangTool-Use Evaluation Automated Software Testing
Florida International University, Miami, FL, USA
When Tools Silently Lie: Evaluating and Mitigating Blind Compliance in Tool-Augmented Data Agents
Sep 29, 2026Zifu Tao, Changqing YinAI Agent Reliability Tool-Use Evaluation
Tongji University
What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review
Sep 27, 2026Shuyang ZhangTool-Use Evaluation LLM Agent Skill Evaluation
The Hong Kong Polytechnic University, Hong Kong, China
EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks
Sep 25, 2026Mukul Singh, Mansi Uniyal, Devin Devlin +5LLM Agent Evaluation AI Agent Benchmarks
Microsoft
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
Sep 22, 2026Lijuan Tang, Yuemeng ZhengLLM Evaluation Tool-Use Evaluation
Northeastern University, Seattle
Counterfactual Tool Ranking under Utility, Cost, and Privilege Constraints
Sep 19, 2026Jiapeng LiCounterfactual Evaluation Tool-Use Evaluation
Microsoft
RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
Sep 16, 2026Qingnuan Han, Boli Fang, Mingzhi Hou +1Tool-Use Evaluation AI Agent Benchmarks
Didi Global
Beyond AI Literacy: A Structured Review and Exploratory Meta-Analysis of Measures for Competent Generative-AI Use
Sep 14, 2026Daniele Veri'Tool-Use Evaluation Generative AI Evaluation
Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification
Sep 14, 2026Nimit Shah, Haitz Sáez de Ocáriz BordeTool-Use Evaluation LLM Agent Evaluation
AION
What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
Sep 10, 2026Haseeb Mohammed AfsarTool-Use Evaluation Model Context Protocol
Independent researcher
The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
Sep 9, 2026Benjamin Gruenbaum, Doron Porat, Assaf Natanzon +3Tool-Use Evaluation LLM Agent Evaluation
Eon
Interface-Induced Trajectory Censoring
Sep 3, 2026Wenbo Wang, Di SangTool-Use Evaluation AI Agent Evaluation
City University of Hong Kong
ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction
Sep 2, 2026Ke Zhang, Yankang Liu, Roya Zandi +1Tool-Use Evaluation Benchmark Construction
University of California, Riverside
Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
Sep 1, 2026Kangjia Zhao, Jiajun Li, Haozhan Shen +8Tool-Use Evaluation LLM Agent Evaluation
Zhejiang University · Shanghai Jiao Tong University · National University of Singapore +3

Oct 27, 2026J/K move · Enter open · S save

LiveSearch: Questions as Executable Verifier Code for a Live Benchmark of Professional-Source Financial Search Agents

Sinuo Wang, Wentao Zhao, Jiacheng Lu, Rui Sun+5

StepFun · Adelaide University · Tsinghua University+3

Financial Language Model Evaluation Benchmark Design Tool-Use Evaluation IR Evaluation

Open paper AskPDF arXiv

In finance, the answer that matters is usually the current one: prices, rates, and disclosures hold only as of a given day, so retrieving up-to-date facts is a core capability of a financial agent. Yet most financial benchmarks overlook timeliness. They fix each answer at annotation time, so the gold goes stale as markets move and, once leaked into training data, rewards recall rather than search and analysis. We introduce LiveSearch, a live benchmark that stores each question as executable verifier code: a typed program of calls to a professional financial data service, followed by a declarative extractor. The question is frozen, while its gold answer is determined by executing the program against professional financial data at evaluation time, and a daily job re-executes it with no human or language model in the loop. A memorized answer is thus likely outdated, and a model must reach the current one through tools and analysis. Construction is equally automatic: a language model only verbalizes programs enumerated from real tool responses and never chooses an answer. Under general web search, six contemporary tool-calling models reach 70.9% to 76.7% answer accuracy. Over three consecutive days, 70% of served questions change their gold value, and even under tolerance-aware matching, a literally reused earlier snapshot fails on a majority of questions within a single day. Per-question tracking uncovers dynamic shifts in correctness masked by aggregate accuracy.