Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.
Figures & tables
Figure 1: Tool-R0 self-evolution loop.
Figure 2: Tool-R0 Self-Evolution Framework. A base LLM is initialized into two roles: a Generator and a Solver . The Generator synthesizes challenging tool-calling tasks (question, tool menu, and gold tool-call), targeting the frozen Solver ’s competence frontier with designed rewards ( rfmt+rvalid+rcurr ). Generated tasks are filtered and ranked easy-to-hard into a curriculum pool. The Solver trains on this curated data to predict tool calls ( rfmt+racc ), completing a self-evolving cycle that requires no pre-existing human datasets.
Figure 3: Difficulty reward ( rdiff ).
Name
ToolAlpaca
SealTool
NexusRaven
API-Bank
SNIPS
Avg
Qwen Models
Qwen2.5-0.5B-Instruct
20.17
37.07
4.71
13.85
1.57
15.47
w/ Tool-R0
31.58
63.95
17.61
28.00
14.29
30.57
Δ
+11.41
+26.88
+12.90
+14.15
+12.72
+15.62
↑ 56.57%
↑ 72.51%
↑ 273.89%
↑ 102.17%
↑ 810.19%
↑ 101.03%
Qwen2.5-1.5B-Instruct
35.96
47.27
17.61
19.13
4.29
24.85
Table 1: Main results of Tool-R0 . We evaluate Tool-R0 on five different tool-calling benchmarks, each targeting different aspects of tool use. Highlighted rows indicate models trained with Tool-R0 . + Δ reports absolute accuracy improvements over the corresponding instruction-tuned baseline, while percentages denote relative improvement.
Model
#data
ToolAlpaca
SealTool
Nexus-Raven
API-Bank
SNIPS
Avg
Agents Trained on Existing Curated Tool-Calling Data
Qwen2.5-1.5B-Instruct
–
35.96
47.27
17.61
10.13
4.29
24.85
⊢ w/ xLAM Dataset ( Zhang et al., 2025c )
60k
51.75
69.05
38.68
34.65
23.85
43.60
⊢ w/ Hammer Dataset ( Lin et al., 2025 )
210k
45.61
68.70
51.88
33.10
19.42
43.74
⊢ w/ ToolAce Dataset ( Liu et al., 2025c )
12k
45.61
67.01
43.08
53.71
14.14
44.71
⊢ w/ ToolRL Dataset ( Qian et al., 2025 )
4k
46.49
72.78
34.14
62.04
14.86
46.06
Table 2: Performance Comparison with Supervised Baselines. Performance of various models is evaluated on five agentic benchmarks (ToolAlpaca, SealTool, NexusRaven, API-Bank, SNIPS). We use + for absolute accuracy increase from base model.
Figure 4: Self-play coverage analysis.
Method
Average Accuracy
Δ Accuracy Drop
pp
Relative (%)
Tool-R0
47.84
–
–
Ablations
⊢ shared weights
30.42
-17.42
↓ 36.41%
⊢ frozen Generator ( )
41.65
-6.19
↓ 12.94%
⊢ w/o Difficulty Reward ( Equation 4 )
43.54
-4.30
↓ 8.99%
Table 3: Ablation analysis relative to Tool-R0 . Absolute changes are reported in percentage points (pp), and relative changes as percentage drop ( ↓ %).
Figure 5: Self-play convergence for extended iterations across model scales.
Figure 6: Tool-R0 as Mid-training.
Figure 7: RLVR training dynamics of Generator – Solver . Top Row: total reward trajectories for Generator and Solver (left), Generator total reward with format ( rfmt ), validity ( rvalid ), and curriculum ( rcurr ) components (middle), and Solver total reward with format and accuracy components (right). Bottom Row: detailed Generator reward components, showing format reward (left; rfmt ), validity reward (middle; rvalid ), and curriculum reward decomposed into difficulty and semantic coherence (right; rcurr ), over self-play iterations.
Figure 8: Categorical analysis of tool failures for base model and Tool-R0 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Generator
Solver
Dataset Size
2000
2000
Global Batch Size
24
32
Optimizer
AdamW
AdamW
Learning Rate
1×10−6
1×10−6
Weight Decay
1×10−2
1×10−2
KL Penalty Coefficient ( λKL )
1×10−2
1×10−2
Appendix
Table 4: Training hyperparameters. Details of the main training configurations for the Generator and Solver during Self-Play iterations.
Hyperparameter
Value
Validity Reward ( Equation 2 )
λmenu (tool existence)
0.4
λgold (required arguments)
0.4
λvalue (value grounding)
0.2
Curriculum Reward ( Equation 4 )
Solver samples ( K )
8
Appendix
Table 5: Generator reward hyperparameters. Hyperparameter values for the validity and curriculum reward components used during Generator training ( Section 3.2 ). The format reward ( Equation 1 ) uses unweighted binary indicators and has no tunable parameters.
Hyperparameter
Value
Accuracy weights (λname,λkey,λval)
(0.2,0.3,0.5)
Extra-call penalty coefficient α
0.25
Format reward weights (Itag,Iparse,Inorm)
(0.3,0.3,0.4)
Appendix
Table 6: Reward hyperparameters for Solver training.
Figure 10: Generator prompt used for task synthesis in Tool-R0 . The system template enforces domain/context constraints, tool-menu cardinality, and primitive-only arguments, while requiring the final output to follow a strict four-block schema ( <think> , <question> , <available_tools> , <tool_call_answer> ).
Figure 11: Semantic alignment reward prompt template. Prompt used to compute the semantic alignment reward rsem in Equation 5 , evaluating whether the synthesized user question is realistic and well-formed, and whether the corresponding tool call semantically and functionally satisfies the user’s request.
Figure 12: Solver prompt template. User_Query indicates the user’s natural-language input, while Tool_Menu denotes the set of available tools; both placeholders are instantiated dynamically at each training step.
Figure 13: Early-stage Generator behavior in Tool-R0 . In the first training iteration, the Generator produces surface-level, minimally structured tool-use tasks. The user request is simple and underspecified, the tool menu contains a single available function, and the solution consists of a single, straightforward tool call with canonical arguments.
Figure 14: Late-stage Generator behavior in Tool-R0 . By Iteration 3, the Generator synthesizes complex, multi-constraint user requests that require multi-step tool execution. The task jointly involves booking a round-trip flight and reserving a centrally located hotel, incorporating temporal constraints, passenger count, and cabin class preferences. The generated solution correctly decomposes the request into multiple ordered tool calls, demonstrating improved task abstraction, constraint integration, and compositional planning.
Large Language Model (LLM) Agents, often trained with Reinforcement Learning (RL), are constrained by a dependency on human-curated data, limiting scalability and tethering AI to human knowledge. Existing self-evolution frameworks offer an alternative but are typically restricted by the model's inherent capabilities and single-round interactions, hindering the development of complex curricula involving tool use or dynamic reasoning. We introduce Agent0, a fully autonomous framework that evolves high-performing agents without external data through multi-step co-evolution and seamless tool integration. Agent0 establishes a symbiotic competition between two agents initialized from the same base LLM: a curriculum agent that proposes increasingly challenging frontier tasks, and an executor agent that learns to solve them. We integrate external tools to enhance the executor's problem-solving capacity; this improvement, in turn, pressures the curriculum agent to construct more complex, tool-aware tasks. Through this iterative process, Agent0 establishes a self-reinforcing cycle that continuously produces high-quality curricula. Empirically, Agent0 substantially boosts reasoning capabilities, improving the Qwen3-8B-Base model by 18% on mathematical reasoning and 24% on general reasoning benchmarks. Code is available at https://github.com/aiming-lab/Agent0.
Peng Xia, Kaide Zeng, Jiaqi Liu +5
UNC-Chapel Hill · Salesforce Research · Stanford University
LLM-based agents depend on effective tool-use policies to solve complex tasks, yet optimizing these policies remains challenging due to delayed supervision and the difficulty of credit assignment in long-horizon trajectories. Existing optimization approaches tend to be either monolithic, which are prone to entangling behaviors, or single-aspect, which ignore cross-module error propagation. To address these limitations, we propose EvoTool, a self-evolving framework that optimizes a modular tool-use policy via a gradient-free evolutionary paradigm. EvoTool decomposes agent's tool-use policy into four modules, including Planner, Selector, Caller, and Synthesizer, and iteratively improves them in a self-improving loop through three novel mechanisms. Trajectory-Grounded Blame Attribution uses diagnostic traces to localize failures to a specific module. Feedback-Guided Targeted Mutation then edits only that module via natural-language critique. Diversity-Aware Population Selection preserves complementary candidates to ensure solution diversity. Across four benchmarks, EvoTool outperforms strong baselines by over 5 points on both GPT-4.1 and Qwen3-8B, while achieving superior efficiency and transferability. The code will be released once paper is accepted.
Shuo Yang, Soyeon Caren Han, Xueqi Ma +3
School of Computing and Information Systems, The University of Melbourne
Reinforcement learning has become a prevalent paradigm for training tool calling agents, which typically requires online interactive environments. Existing approaches either rely on training data with ground truth annotations or require advanced proprietary language models (LMs) to synthesize environments that remain fixed once created. In this work, we propose TRUSTEE, a cost-friendly method for training tool calling agents with dynamic environments fully simulated by open-source LMs that can be as small as 8B, including task generation, user simulation, tool simulation, and trajectory evaluation, paired with an adaptive curriculum learning mechanism that controls task difficulty dynamically. Our empirical results show that TRUSTEE outperforms baselines which require extra external resources in most cases. These confirm that, with a sufficiently sophisticated design, even simulated environments with a local 8B LM as the backbone could set a strong baseline for tool learning. We hope our proposed paradigm could democratize tool learning and inspire future research on environment scaling with limited resources.
Chenming Tang, Hsiu-Yuan Huang, Weijie Liu +3
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · LLM Department, Tencent