Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent

Bowei Xia; Mengkang Hu; Shijian Wang; Jiarui Jin; Wenxiang Jiao; Yuan Lu; Kexin Li; Ping Luo

cs.SEMar 5, 2026

Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent

Authors: Bowei Xia, Mengkang Hu, Shijian Wang, Jiarui Jin, Wenxiang Jiao, Yuan Lu, Kexin Li, Ping Luo

Organizations: UESTC · The University of Hong Kong · Xiaohongshu Inc.

Abstract

Research on self-evolving language agents has accelerated, drawing increasing attention to their ability to create, adapt, and maintain tools from task requirements. However, existing benchmarks predominantly rely on predefined specifications, which limits scalability and hinders truly autonomous evolution. While recent studies attempt to dynamically generate tools, they primarily emphasize downstream performance, resulting in a "black-box" evaluation that makes it difficult to attribute failures to specific causes. To address this, we propose Tool-Genesis, a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions, including interface compliance, functional correctness, and downstream utility. Tool-Genesis evaluates whether agents can construct task-relevant tools solely from abstract requirements (without preset specifications) and use them to solve realistic problems. Crucially, we find that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting. These minor initial flaws are amplified through the pipeline, leading to a sharp degradation in downstream metrics. We hope Tool-Genesis will guide future research toward training and steering models to synthesize persistent, general-purpose tools that better address real-world challenges.

Figures & tables

Benchmark	Scale (Reported)			I. Artifacts Required				II. Verification Signals
Benchmark	Tool Sets	Avg Tools	# Domains	No Doc	Schema Gen	Reuse Tool	Toolset ( $\geq 2$ )	Held-out UnitTests	Neg Tests	GT Tools
CREATOR ( Qian et al., 2023b )	2K	1	9	✗	✗	✓	✓	✗	✗	✗
LATM ( Cai et al., 2024 )	6	1	6	✓	✗	✗	✗	✗	✗	✗
CRAFT ( Yuan et al., 2024 )	150	3	3	✗	✗	✓	✓	✗	✗	✗
TM-Bench ( Wölflein et al., 2025 )	15	1	4	✓	✗	✓	✗	✓	✗	✗
SciEvo ( Zhang et al., 2025a )	925	6	25	✓	✗	✓	✗	✓	✗	✗

Table 1: Feature-wise comparison of representative tool-creation benchmarks under a strict binary rubric. Held-out UnitTests indicates tests/invocations are not available during tool creation and used strictly for evaluation (e.g., TM-Bench).

Figure 1: Comparison of tool creation paradigms: (a) Outcome-Driven: Ad-hoc solving with disposable scripts; (b) Code-Centric: Spec-based translation with limited safety; (c) Tool-Genesis(Ours): Inductive design for verified, reusable assets.

Figure 2: Dataset construction pipeline of Tool Genesis .

Figure 3: Comparison of benchmarks in terms of task reasoning depth and tool compositionality.

Figure 4: Overview statistics of Tool Genesis . Left: functional domain coverage of MCP servers across 24 domain classes. Right: dataset scale and task/trajectory structure statistics.

Model Family	Version	L1		L2	L3		L4
Model Family	Version	Compliance $\uparrow$	Exec. $\uparrow$	Schema-F1 $\uparrow$	UT ${}_{\text{soft}}\uparrow$	UT ${}_{\text{hard}}\uparrow$	SR $\uparrow$
Direct
OpenAI ( OpenAI, 2025 )	gpt-4o	0.779	0.209	0.175	0.089	0.049	0.153
	gpt-4.1-mini	0.686	0.318	0.293	0.127	0.088	0.199
	gpt-4.1	0.860	0.738	0.675	0.261	0.129	0.330
	gpt-5.1	0.826	0.759	0.688	0.281	0.161	0.372

Table 2: Main results under two evaluation paradigms ( Direct and Code-Agent ) updated with the latest metrics. Metrics follow our four-level evaluation: L1 surface compliance (Compliance, Server Execution), L2 semantic interface fidelity (Schema-F1), L3 functional correctness via unit tests (UT

{}_{\text{soft}}

, UT

{}_{\text{hard}}

), and L4 downstream task utility (Task Success Rate, SR).

Figure 5: Direct prompting vs. code-agent repair on Qwen3 scales: (a) Signal–Validation Alignment (SVA) of Schema-F1 and UT (soft/hard); (b) stage-wise cumulative pass-through across verification stages (Start, L1–L4) for selected scales, with solid/dashed lines denoting the two paradigms.

Figure 6: Base vs. fine-tuned performance under Direct and Code-Agent evaluation, summarized across our four-layer metric suite (L1–L4).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 286 papers

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks
May 27, 2026Tomer Keren, Nitay Calderon, Asaf Yehudai +3Computer-Use Agent Benchmarks Tool-Use Evaluation
Faculty of Data and Decision Science Technion · IBM Research
Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents
Apr 1, 2026Alibek Kaliyev, Artem MaryanskyyAI Agent Reliability Tool-Use Evaluation
The University of Texas at Austin · Austin, Texas, USA · Uber +1
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
Jul 6, 2026Zhiheng Xi, Dingwen Yang, Jiaqi Liu +22Tool Discovery for LLM Agents LLM Agent Evaluation
Fudan University · Zhejiang University · Shanghaijiaotong University +1

May 27, 2026J/K move · Enter open · S save

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Tomer Keren, Nitay Calderon, Asaf Yehudai, Yotam Perlitz+2

Faculty of Data and Decision Science Technion · IBM Research

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents Benchmark Construction LLM Agent Evaluation Synthetic Benchmark Generation+1

Open paper AskPDF arXiv

As agent capabilities advance, existing benchmarks, such as

τ^2

-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive

n

-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct

τ^c

-Bench, a challenging extension of the three domains of

τ^2

-Bench. We evaluate

11

agent/user LLM pairs and find that models nearly saturating

τ^2

-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from

0.82\!-\!0.94

0.28\!-\!0.61

). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.

Abstract

Tool-Using Agents

Explore similar work

A Matter of TASTE: Improving Coverage and Difficulty of Agent Benchmarks

Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments