OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction

Skyler Hallinan; Thejas Venkatesh; Xiang Ren; Sai Praneeth Karimireddy; Ashwin Paranjape; Yuhao Zhang; Jack Hessel

cs.CLFeb 16, 2026

OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction

Authors: Skyler Hallinan, Thejas Venkatesh, Xiang Ren, Sai Praneeth Karimireddy, Ashwin Paranjape, Yuhao Zhang, Jack Hessel

Organizations: University of Southern California · Samaya AI

Abstract

Tool-calling is essential for Large Language Model (LLM) agents to complete real-world tasks. While most existing benchmarks assume simple, perfectly documented tools, real-world tools (e.g., general "search" APIs) are often opaque, lacking clear best practices or failure modes. Can LLM agents improve their performance in environments with opaque tools by interacting and subsequently improving documentation? To study this, we create OpaqueToolsBench, a benchmark consisting of three distinct task-oriented environments: general function calling, interactive chess playing, and long-trajectory agentic search. Each environment provides underspecified tools that models must learn to use effectively to complete the task. Results on OpaqueToolsBench suggest existing methods for automatically documenting tools are expensive and unreliable when tools are opaque. To address this, we propose a simple framework, ToolObserver, that iteratively refines tool documentation by observing execution feedback from tool-calling trajectories. Our approach outperforms existing methods on OpaqueToolsBench across datasets, even in relatively hard settings. Furthermore, for test-time tool exploration settings, our method is also efficient, consuming 3.5-7.5x fewer total tokens than the best baseline.

Figures & tables

Figure 1: LLM agents may struggle when presented with opaque tools – tools lacking clear description of their usage best practices or their failure modes. To succeed in these settings, we posit that LLM agents must explore tool usage to learn their true behaviors.

Figure 2: Examples from each of the environments in OpaqueToolsBench . For Chess and BrowseComp Domains, each setting has a fixed set of tools across all instances (chess and search engines respectively). In BFCL-Opaque, each instance has custom query-dependent tools. The dataset defines the input and output (green). For each instance, the agent makes tool calls iteratively (blue). For BFCL-Opaque and BrowseComp Domains we check for a match with the gold answer. For Chess, we match engine choices in green with optimal engine choices.

Tool Setting: Individual Tools per Problem
Documentation	Model	ReAct
		Gold			Base			+ TO			+ P2P			+ ET
		E	P	A	E	P	A	E	P	A	E	P	A	E	P	A
Anon. Fn. Names Only	GPT-5	0.95	0.96	0.96	0.00	0.00	0.60	0.80	0.78	0.89	0.44	0.45	0.76	0.00	0.00	0.60
	GPT-5-mini	0.93	0.94	0.96	0.00	0.00	0.60	0.66	0.63	0.85	0.43	0.43	0.77	0.00	0.00	0.60
Anon. Fn. + Real Desc	GPT-5	0.95	0.96	0.96	0.00	0.00	0.60	0.86	0.82	0.91	0.53	0.54	0.82	0.00	0.00	0.60

Table 2: Performance on BFCL across models and baselines. We use ReAct. Gold is the ground truth documentation and base is the opacified set. TO denotes ToolObserver , P2P denotes Play2Prompt, and ET denotes EasyTool. Columns denote E (Execution-based overall accuracy), P (Parameter accuracy), and A (AST accuracy). The highest E value in each row is bolded .

Model	ReAct
	Gold		Base		+ TO		+ P2P		+ ET
	Acc	ELO	Acc	ELO	Acc	ELO	Acc	ELO	Acc	ELO
Tool Setting: Opening, midgame, endgame, late-endgame specialists
GPT-5	64.4	1411	23.5	728	40.1	1020	35.8	966	23.2	761
GPT-5-mini	52.8	1243	24.9	772	32.1	949	19.5	754	25.8	739
Tool Setting: Beginner, intermediate, advanced skill

Table 3: Average percentage of best tool calls (Acc) and ELO rating at a given state for Chess. TO denotes ToolObserver , P2P denotes Play2Prompt, and ET denotes EasyTool. Gold is the ground truth documentation. The highest value in each row is bolded .

Model	ReAct
	Gold		Base		+ TO		+ P2P		+ ET
	Acc.	#TC	Acc.	#TC	Acc.	#TC	Acc.	#TC	Acc.	#TC
Tool Setting: Domain-specific (9) Search
GPT-5	22.5	7.0	18.1	10.0	21.9	6.3	17.9	7.8	18.9	8.2
GPT-5-mini	9.7	2.8	3.1	4.1	3.2	3.5	2.8	3.5	7.1	3.8
Tool Setting: Domain-specific (9) + Full Search

Table 4: Performance comparison table on BrowseComp Domains using Qwen-0.6B. Acc. refers to accuracy and #TC refers to number of tool calls. TO denotes ToolObserver , P2P denotes Play2Prompt, and ET denotes EasyTool. The highest method Acc. value in each row is bolded .

Panel of Figure 3 — Figure 3: Performance of ToolObserver on BFCL-Opaque with increased iterations on the tool settings: A): Anon. Function Names, B): Anon. Function Names + Descriptions, C): Anon. Function Names + Param. Names. Iterations stop after full convergence. These are expanded versions of the Table 2 results.

Documentation	TO (GPT-5)		TO (GPT-5-Mini)		P2P		ET
Documentation	Sem.	Lex.	Sem.	Lex.	Sem.	Lex.	Sem.	Lex.
Anon. Fn. Names	0.44	0.21	0.58	0.28	0.51	0.18	0	0
Anon. Fn. Names + Desc.	0.78	0.43	0.78	0.44	0.70	0.31	0.71	0.43
Anon. Fn. Names + Param. Names	0.71	0.40	0.71	0.40	0.69	0.28	0.69	0.39

Table 6: Average similarity metrics (semantic and lexical) of final documentation generated vs. gold documentation on BFCL-Opaque. The highest sem. is bolded, while the highest lex. is italicized. TO denotes ToolObserver , P2P denotes Play2Prompt, and ET denotes EasyTool.

LLM Agent	GPT5	GPT5-mini	o3
GPT5	0.64	0.62	0.60
GPT5-mini	0.62	0.54	0.48

Table 7: Performance on BFCL-Opaque with different editors on Anon. Fn. Names setting for the executable multiple function subset

LLM Tool Use

Large language model tool use studies how models select, call, and coordinate external tools to extend their reasoning and take actions.

See all 289 papers

DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents
Aug 10, 2026You Lu, Kun Zhang, Bihuan Chen +1Tool-Augmented Language Model Agents Tool-Using Agents
College of Computer Science and Artificial Intelligence, Fudan University, China
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
LLM Agents Already Know When to Call Tools -- Even Without Reasoning
May 10, 2026Chung-En Sun, Linbo Liu, Ge Yan +2Tool-Augmented Language Model Agents AI Agent Benchmarks
University of California, San Diego

Aug 10, 2026J/K move · Enter open · S save

DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

You Lu, Kun Zhang, Bihuan Chen, Xin Peng

College of Computer Science and Artificial Intelligence, Fudan University, China

Tool-Augmented Language Model Agents Tool-Using Agents LLM Tool Use

Open paper AskPDF arXiv

Large language models (LLMs) increasingly rely on external tools to accomplish complex real-world tasks, making tool documentation a critical grounding resource for LLM agents. Existing studies mainly focus on improving the tool-use capabilities of LLM agents, while largely treating tool documentation as a fixed input. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings. To bridge this gap, we conduct a large-scale empirical study on tool documentation for LLM agents. Our study reveals substantial heterogeneity in the information fields provided by existing tool documentation. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings. Motivated by these findings, we propose DocsChisel, an adaptive tool documentation optimization framework for LLM agents. DocsChisel analyzes failed execution traces of a target LLM agent to identify documentation-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool. We evaluate DocsChisel against two state-of-the-art baselines, i.e., EasyTool and DRAFT. Experimental results show that DocsChisel improves the task success rate of LLM agents by 95.89% over the original tool documentation and by 75.15%, on average, over existing baselines, while incurring limited optimization time and token overhead

Abstract

LLM Tool Use

Explore similar work

DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

LLM Agents Already Know When to Call Tools -- Even Without Reasoning