Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

Emre Can Acikgoz; Cheng Qian; Jonas Hübotter; Heng Ji; Dilek Hakkani-Tür; Gokhan Tur

cs.LGFeb 24, 2026

Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

Authors: Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji, Dilek Hakkani-Tür, Gokhan Tur

Organizations: UIUC · ETH Zurich

Abstract

Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.

Figures & tables

Figure 2: Tool-R0 Self-Evolution Framework. A base LLM is initialized into two roles: a Generator and a Solver . The Generator synthesizes challenging tool-calling tasks (question, tool menu, and gold tool-call), targeting the frozen Solver ’s competence frontier with designed rewards ( $r_{\text{fmt}}+r_{\text{valid}}+r_{\text{curr}}$ ). Generated tasks are filtered and ranked easy-to-hard into a curriculum pool. The Solver trains on this curated data to predict tool calls ( $r_{\text{fmt}}+r_{\text{acc}}$ ), completing a self-evolving cycle that requires no pre-existing human datasets.

Figure 3: Difficulty reward ( $r_{\text{diff}}$ ).

Qwen Models
Name	ToolAlpaca	SealTool	NexusRaven	API-Bank	SNIPS	Avg
Qwen2.5-0.5B-Instruct	20.17	37.07	4.71	13.85	1.57	15.47
w/ Tool-R0	31.58	63.95	17.61	28.00	14.29	30.57
$\Delta$	+11.41	+26.88	+12.90	+14.15	+12.72	+15.62
	$\uparrow$ 56.57%	$\uparrow$ 72.51%	$\uparrow$ 273.89%	$\uparrow$ 102.17%	$\uparrow$ 810.19%	$\uparrow$ 101.03%
Qwen2.5-1.5B-Instruct	35.96	47.27	17.61	19.13	4.29	24.85

Table 1: Main results of Tool-R0 . We evaluate Tool-R0 on five different tool-calling benchmarks, each targeting different aspects of tool use. Highlighted rows indicate models trained with Tool-R0 . +

\Delta

reports absolute accuracy improvements over the corresponding instruction-tuned baseline, while percentages denote relative improvement.

Agents Trained on Existing Curated Tool-Calling Data
Model	#data	ToolAlpaca	SealTool	Nexus-Raven	API-Bank	SNIPS	Avg
Qwen2.5-1.5B-Instruct	–	35.96	47.27	17.61	10.13	4.29	24.85
$\vdash$ w/ xLAM Dataset ( Zhang et al., 2025c )	60k	51.75	69.05	38.68	34.65	23.85	43.60
$\vdash$ w/ Hammer Dataset ( Lin et al., 2025 )	210k	45.61	68.70	51.88	33.10	19.42	43.74
$\vdash$ w/ ToolAce Dataset ( Liu et al., 2025c )	12k	45.61	67.01	43.08	53.71	14.14	44.71
$\vdash$ w/ ToolRL Dataset ( Qian et al., 2025 )	4k	46.49	72.78	34.14	62.04	14.86	46.06

Table 2: Performance Comparison with Supervised Baselines. Performance of various models is evaluated on five agentic benchmarks (ToolAlpaca, SealTool, NexusRaven, API-Bank, SNIPS). We use

+

for absolute accuracy increase from base model.

Method	Average Accuracy	$\Delta$ Accuracy Drop
Method	Average Accuracy	pp	Relative (%)
Tool-R0	47.84	–	–
Ablations
$\vdash$ shared weights	30.42	-17.42	$\downarrow$ 36.41%
$\vdash$ frozen Generator ( )	41.65	-6.19	$\downarrow$ 12.94%
$\vdash$ w/o Difficulty Reward ( Equation 4 )	43.54	-4.30	$\downarrow$ 8.99%

Table 3: Ablation analysis relative to Tool-R0 . Absolute changes are reported in percentage points (pp), and relative changes as percentage drop (

\downarrow

%).

Figure 5: Self-play convergence for extended iterations across model scales.

Figure 7: RLVR training dynamics of Generator – Solver . Top Row: total reward trajectories for Generator and Solver (left), Generator total reward with format ( $r_{\text{fmt}}$ ), validity ( $r_{\text{valid}}$ ), and curriculum ( $r_{\text{curr}}$ ) components (middle), and Solver total reward with format and accuracy components (right). Bottom Row: detailed Generator reward components, showing format reward (left; $r_{\text{fmt}}$ ), validity reward (middle; $r_{\text{valid}}$ ), and curriculum reward decomposed into difficulty and semantic coherence (right; $r_{\text{curr}}$ ), over self-play iterations.

Figure 8: Categorical analysis of tool failures for base model and Tool-R0 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Generator	Solver
Dataset Size	2000	2000
Global Batch Size	24	32
Optimizer	AdamW	AdamW
Learning Rate	$1\times 10^{-6}$	$1\times 10^{-6}$
Weight Decay	$1\times 10^{-2}$	$1\times 10^{-2}$
KL Penalty Coefficient ( $\lambda_{\mathrm{KL}}$ )	$1\times 10^{-2}$	$1\times 10^{-2}$

Appendix

Table 4: Training hyperparameters. Details of the main training configurations for the Generator and Solver during Self-Play iterations.

Validity Reward ( Equation 2 )
Hyperparameter	Value
$\lambda_{\text{menu}}$ (tool existence)	$0.4$
$\lambda_{\text{gold}}$ (required arguments)	$0.4$
$\lambda_{\text{value}}$ (value grounding)	$0.2$
Curriculum Reward ( Equation 4 )
Solver samples ( $K$ )	$8$

Appendix

Table 5: Generator reward hyperparameters. Hyperparameter values for the validity and curriculum reward components used during Generator training ( Section 3.2 ). The format reward ( Equation 1 ) uses unweighted binary indicators and has no tunable parameters.

Hyperparameter	Value
Accuracy weights $(\lambda_{\text{name}},\lambda_{\text{key}},\lambda_{\text{val}})$	$(0.2,\,0.3,\,0.5)$
Extra-call penalty coefficient $\alpha$	$0.25$
Format reward weights $(\mathbb{I}_{\text{tag}},\mathbb{I}_{\text{parse}},\mathbb{I}_{\text{norm}})$	$(0.3,\,0.3,\,0.4)$

Appendix

Table 6: Reward hyperparameters for Solver training.

Panel of Figure 10 — Figure 10: Generator prompt used for task synthesis in Tool-R0 . The system template enforces domain/context constraints, tool-menu cardinality, and primitive-only arguments, while requiring the final output to follow a strict four-block schema ( <think> , <question> , <available_tools> , <tool_call_answer> ).

Panel of Figure 11 — Figure 11: Semantic alignment reward prompt template. Prompt used to compute the semantic alignment reward $r_{\text{sem}}$ in Equation 5 , evaluating whether the synthesized user question is realistic and well-formed, and whether the corresponding tool call semantically and functionally satisfies the user’s request.

Panel of Figure 12 — Figure 12: Solver prompt template. User_Query indicates the user’s natural-language input, while Tool_Menu denotes the set of available tools; both placeholders are instantiated dynamically at each training step.

Panel of Figure 13 — Figure 13: Early-stage Generator behavior in Tool-R0 . In the first training iteration, the Generator produces surface-level, minimally structured tool-use tasks. The user request is simple and underspecified, the tool menu contains a single available function, and the solution consists of a single, straightforward tool call with canonical arguments.

Panel of Figure 14 — Figure 14: Late-stage Generator behavior in Tool-R0 . By Iteration 3, the Generator synthesizes complex, multi-constraint user requests that require multi-step tool execution. The task jointly involves booking a round-trip flight and reserving a centrally located hotel, incorporating temporal constraints, passenger count, and cabin class preferences. The generated solution correctly decomposes the request into multiple ordered tool calls, demonstrating improved task abstraction, constraint integration, and compositional planning.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning
Nov 20, 2025Peng Xia, Kaide Zeng, Jiaqi Liu +5LLM Agent Self-Improvement Multi-Agent LLMs
UNC-Chapel Hill · Salesforce Research · Stanford University
EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection
Mar 5, 2026Shuo Yang, Soyeon Caren Han, Xueqi Ma +3Policy Optimization LLM Agent Self-Improvement
School of Computing and Information Systems, The University of Melbourne
Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model
Apr 20, 2026Chenming Tang, Hsiu-Yuan Huang, Weijie Liu +3LLM Agent Training RL for Tool Use
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent

Nov 20, 2025J/K move · Enter open · S save

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning

Peng Xia, Kaide Zeng, Jiaqi Liu, Can Qin+4

UNC-Chapel Hill · Salesforce Research · Stanford University

LLM Agent Self-Improvement Multi-Agent LLMs Self-Evolving Agents

Open paper AskPDF arXiv

Large Language Model (LLM) Agents, often trained with Reinforcement Learning (RL), are constrained by a dependency on human-curated data, limiting scalability and tethering AI to human knowledge. Existing self-evolution frameworks offer an alternative but are typically restricted by the model's inherent capabilities and single-round interactions, hindering the development of complex curricula involving tool use or dynamic reasoning. We introduce Agent0, a fully autonomous framework that evolves high-performing agents without external data through multi-step co-evolution and seamless tool integration. Agent0 establishes a symbiotic competition between two agents initialized from the same base LLM: a curriculum agent that proposes increasingly challenging frontier tasks, and an executor agent that learns to solve them. We integrate external tools to enhance the executor's problem-solving capacity; this improvement, in turn, pressures the curriculum agent to construct more complex, tool-aware tasks. Through this iterative process, Agent0 establishes a self-reinforcing cycle that continuously produces high-quality curricula. Empirically, Agent0 substantially boosts reasoning capabilities, improving the Qwen3-8B-Base model by 18% on mathematical reasoning and 24% on general reasoning benchmarks. Code is available at https://github.com/aiming-lab/Agent0.

Abstract

LLM Agent Training

Explore similar work

Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning

EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection

Democratizing Tool Learning with Environments Fully Simulated by a Free 8B Language Model