Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents

Hy Dang; Quang Dao; Meng Jiang

cs.AIMar 31, 2026

Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents

Authors: Hy Dang, Quang Dao, Meng Jiang

Organizations: University of Notre Dame · Rose-Hulman Institute of Technology

Abstract

Tool-integrated LLMs retrieve information, perform computations, and take real-world actions, but their reliability depends on both tool-use accuracy and intrinsic tool accuracy, including tool correctness, stability, and safety. While prior work primarily emphasizes tool use, intrinsic tool accuracy remains underexamined. We introduce OpenTools, a community-driven and maintainable toolbox for discovering, using, evaluating, and contributing open-source tools. OpenTools standardizes tool interfaces, converts documented Python functions into reviewable bundles, supports maintainer-triggered evaluation, and combines non-executing risk inspection with optional advisory LLM review. A public web demo allows users to run tools and agents, inspect evidence, contribute tests, and submit tools for maintainer review, while MCP enables controlled access from external applications. Experiments show that community-contributed, task-specific tools yield relative gains of 6% to 22% over an existing toolbox across multiple agent architectures, highlighting the importance of intrinsic tool accuracy.

Figures & tables

Figure 1: OpenTools supports two complementary workflows. Top, tool maintenance: community members submit tools, tests, prior evaluations, and feedback. Non-executing scanners and an optional LLM advisor produce a review report, while human verifiers make the final acceptance decision and manually update the shared tools, tests, and reliability evidence. Accepted tools are assessed using tool-specific metrics and availability tests. Bottom, agentic use: users provide a query, optional files, and tool cards from the OpenTools -Box. Agent modules select actions and request tool execution, execution modules return observations, and the system produces a final answer with tool and reasoning logs. Users may provide optional feedback through the WebUI.

Task	-	OctoTools-T	OpenTools-T	OctoTools-T	OpenTools-T	OctoTools-T	OpenTools-T
	Prompting	ReAct		OctoTools		MultiAgent
VQA/Puzzle	45.38 / 74.36	55.52 / 73.13	68.95 / 79.92	52.13 / 69.75	67.01 / 80.48	72.40	75.28
Math/Reasoning	43.77 / 83.58	48.71 / 83.67	59.03 / 85.33	42.93 / 78.23	55.54 / 81.28	81.25	83.15
Scientific	46.54 / 80.20	50.67 / 75.67	53.38 / 79.89	47.87 / 76.11	52.97 / 77.11	69.48	73.78
Medical	43.56 / 69.56	52.00 / 69.11	57.59 / 71.52	53.00 / 66.78	55.48 / 68.50	66.05	68.67
Agent	5.51 / 21.41	14.17 / 51.97	28.50 / 55.91	4.72/ 46.03	24.41 / 53.54	53.92	66.14

Table 1: Task-group performance for Prompting and three agent approaches using OctoTools-T vs. OpenTools-T. Scores are reported as gpt-4o-mini / gpt-5-mini for Prompting, ReAct, and OctoTools; MultiAgent reports a single gpt-5-mini score. OpenTools-T improves the overall average across all frameworks, demonstrating that higher intrinsic tool quality can translate into better end-to-end reliability. Higher means better.

	OctoTools-T	OpenTools-T
Total tools	13	42
Program-based	1	16
API-based	8	19
Prompting-based	4	7

Table 2: Composition of the evaluated toolboxes by primary execution mechanism.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: OpenTools tool-contribution interface. Contributors upload a Python implementation and README, provide metadata, and may request advisory LLM review. The interface performs deterministic conversion and non-executing inspection, then reports conversion status, static risk, scanner coverage and findings, and the LLM recommendation. Uploaded code remains unexecuted and unpublished, and the resulting pending-review bundle is returned for contributor revision and maintainer review.

Figure 3: Test-case contribution UI: submission instructions and supported formats for adding new tool tests.

Figure 4: Test-case authoring/execution UI: run tools on submitted cases and store outputs in a bulk-results table with selectable evaluation metrics.

Figure 5: Agent Playground UI: select an agent and base LLM, optionally filter tools via retrieval, and inspect execution logs, responses

	ZeroShot	CoT
VQA/Puzzle	45.38 / 74.36	46.95 / 74.20
Math/Reasoning	43.77 / 83.58	41.38 / 76.16
Scientific	46.54 / 80.20	48.89 / 80.74
Medical	43.56 / 69.56	45.33 / 69.33
Agent	5.51 / 21.41	4.46 / 20.01
Average	42.16 / 73.49	42.70 / 71.45

Appendix

Table 4: Task-group performance for Prompting without tools between ZeroShot and Chain of Thought (CoT). Scores are reported as gpt-4o-mini / gpt-5-mini.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 286 papers

OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction
Feb 16, 2026Skyler Hallinan, Thejas Venkatesh, Xiang Ren +4LLM Tool Use LLM Agents
University of Southern California · Samaya AI
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley
Agents' Overreliance on Unreliable Tools
Sep 4, 2026Hoyeol Yang, Woojung Song, Taewon Kim +3LLM Tool Use LLM Agent Reliability
Seoul National University, Seoul, Republic of Korea

Feb 16, 2026J/K move · Enter open · S save

OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction

Skyler Hallinan, Thejas Venkatesh, Xiang Ren, Sai Praneeth Karimireddy+3

University of Southern California · Samaya AI

LLM Tool Use LLM Agents AI Agent Benchmarks

Open paper AskPDF arXiv

Tool-calling is essential for Large Language Model (LLM) agents to complete real-world tasks. While most existing benchmarks assume simple, perfectly documented tools, real-world tools (e.g., general "search" APIs) are often opaque, lacking clear best practices or failure modes. Can LLM agents improve their performance in environments with opaque tools by interacting and subsequently improving documentation? To study this, we create OpaqueToolsBench, a benchmark consisting of three distinct task-oriented environments: general function calling, interactive chess playing, and long-trajectory agentic search. Each environment provides underspecified tools that models must learn to use effectively to complete the task. Results on OpaqueToolsBench suggest existing methods for automatically documenting tools are expensive and unreliable when tools are opaque. To address this, we propose a simple framework, ToolObserver, that iteratively refines tool documentation by observing execution feedback from tool-calling trajectories. Our approach outperforms existing methods on OpaqueToolsBench across datasets, even in relatively hard settings. Furthermore, for test-time tool exploration settings, our method is also efficient, consuming 3.5-7.5x fewer total tokens than the best baseline.

Abstract

Tool-Using Agents

Explore similar work

OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Agents' Overreliance on Unreliable Tools