Equipping Large Language Models (LLMs) with multi-turn tool-calling capabilities is essential for building autonomous agents. However, progress is fundamentally limited by the reliance on full-length trajectory imitation. For tasks involving multiple order-independent sub-goals, the optimal solution space forms a vast combinatorial diamond lattice. Forcing this rich topology into monolithic trajectories causes a severe topological collapse, indiscriminately penalizing valid alternative explorations and severely degrading policy diversity. To address this, we propose DART-SD (Diamond-topology Aware Retrieval and Tuning for Self-Distillation), a novel framework that shifts the paradigm from global forcing to topology-guided localized correction. DART-SD first models the execution process as a converging Interaction-State Transition Graph (ISTG), faithfully capturing the inherent diamond topology of successful and failed exploratory paths. During autonomous rollouts, the framework identifies the Critical Topological Breakpoint (CTB) and retrieves success-supported recovery references. Finally, we introduce a progressive self-distillation paradigm through CTB-guided localized supervision, ensuring that the training loss is calculated exclusively on the generated recovery steps while strictly protecting the valid reasoning prefix from destructive gradient updates. Experiments on complex multi-turn tool-calling benchmarks demonstrate that DART-SD significantly outperforms traditional full-trajectory baselines.
Figures & tables
Figure 1: Comparison of training paradigms. (a) SFT Teacher Boosting applies an indiscriminate global loss, which overwrites valid exploration. (b) Standard-RL (GRPO) misassigns credit through a uniformly distributed reward spread. (c) DART-SD dynamically identifies the Critical Topological Breakpoints (CTB) and applies localized correction while preserving valid exploration.
Figure 2: Overview of DART-SD. (1) DART-SD constructs an ISTG from teacher rollouts, where main and auxiliary nodes model information acquisition and useless exploration. (2–3) Failed student rollouts are collected and replayed in the same interaction-state space, then projected onto the budget-filtered success-reachable region Rx+ , where the first projectable-to-non-projectable transition defines the CTB. (4–5) Conditioned on the retained student prefix and privileged teacher references, DART-SD generates a recovery continuation after the CTB and applies localized supervision only to the generated assistant tokens. (6) This CTB-guided SFT loop is progressively repeated across self-distillation rounds.
Methods
FTRL
BFCL
ToolHop
τ -bench
RoTBench
Avg.
Solve-P
Solve-R
Solve-F1
Multi-Turn
AC
Pass^1
TS
PI
CF
Qwen3-4B
Base
21.00
26.54
21.81
10.14
20.20
15.15
69.52
26.31
16.07
25.19
♠ SFT
34.26
49.02
37.96
14.57
40.50
21.82
71.67
44.40
24.40
37.62
♠ SCoRe-SFT
23.79
28.20
24.61
12.75
25.83
11.52
60.36
31.19
18.33
26.29
♠ OPSD
22.37
29.21
23.50
11.00
20.20
23.03
68.93
35.24
21.67
28.35
Table 1: Performance comparison of different training methods on five tool-use benchmarks using Qwen3-4B and Qwen3-8B backbones. Training-based baselines are organized into distillation-based ♠ and reinforcement learning ♢ paradigms. All trainable methods are trained on FTRL and evaluated on both the in-domain FTRL test set and four out-of-domain benchmarks. The best and second-best results within each backbone are highlighted.
Figure 3: Performance comparison of Qwen3-8B, DART-SD, and the teacher across five tool-use benchmarks. DART-SD improves upon Qwen3-8B on all benchmarks and surpasses the teacher on FTRL, ToolHop, and τ -bench.
Task
DART-SD
Golden
Iter1
Iter2
Iter3
Iter4
Iter5
Single
1.23
1.15
1.10
1.09
1.07
1.00
Multi
3.83
4.63
4.54
4.49
4.41
4.71
Para-Single
3.29
2.65
2.27
2.26
2.22
2.11
Para-Multi
7.25
6.48
5.97
5.82
5.62
6.97
Overall
4.23
3.99
3.71
3.65
3.55
4.02
Table 2: Average tool-call length of successful trajectories across progressive SFT iterations on the FTRL test set. DART-SD progressively improves Solve-F1 while shortening its tool traces, eventually producing more efficient traces than the golden solutions given during data construction.
Task
CTB Position
Iter1
Iter2
Iter3
Iter4
Iter5
Single
0.034
0.186
0.137
0.249
0.077
Multi
0.500
1.624
1.729
1.876
1.953
Para-Single
0.095
0.219
0.299
0.328
0.313
Para-Multi
0.395
1.524
1.692
1.791
1.816
Overall
0.348
1.185
1.310
1.421
1.452
Table 3: Average CTB positions of failed training trajectories across progressive SFT. Larger values indicate that the first departure from empirically recoverable behavior occurs later, meaning that the model correctly executes a longer trajectory prefix before localized recovery is required.
Method
FTRL
BFCL
ToolHop
Qwen3-8B
29.74
40.00
42.21
FTRL-GRPO
32.85
41.50
36.72
ToolRL
26.72
34.25
32.93
MatchTIR (KM)
37.33
47.13
46.16
DART-SD
41.03
49.75
46.43
Table 4: Performance comparison under the thinking setting. Results are reported using FTRL Solve-F1, the average BFCL Multi-Turn score, and ToolHop AC.
Method
IFEval
AIME24
AIME25
MMLU
Avg.
Qwen3-8B
34.75
46.67
23.33
70.94
43.92
SFT
35.30
43.33
26.67
71.43
44.18
DART-SD
45.29
50.00
30.00
74.27
49.89
Table 5: General capability evaluation on representative benchmarks. The best results are highlighted.
Method
Solve-P
Solve-R
Solve-F1
Qwen3-8B
21.18
30.71
23.48
+SD
36.23
44.46
38.10
+CTB
36.62
46.32
39.51
+Progressive SFT
41.32
49.65
43.93
+ISTG ( Ours )
42.00
54.13
45.66
Table 6: Component ablation of DART-SD on the FTRL test set using Solve-P, Solve-R, and Solve-F1.
Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8
Zhejiang University · Meituan · Tsinghua University
Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/
Hao Wang, Guozhi Wang, Han Xiao +8
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2