DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents

Hangrui Xu; Jiarui Wang; Yang Yang; Chuanbo Zhu; Fangda Chen; Ziqi Wu; Jingming Cai; Yan Song

cs.CLAug 19, 2026

DART-SD: Diamond-topology Aware Retrieval and Tuning for Self-Distillation of Multi-Turn Tool-Calling Agents

Authors: Hangrui Xu, Jiarui Wang, Yang Yang, Chuanbo Zhu, Fangda Chen, Ziqi Wu, Jingming Cai, Yan Song

Organizations: ByteDance · University of Science and Technology of China

Abstract

Equipping Large Language Models (LLMs) with multi-turn tool-calling capabilities is essential for building autonomous agents. However, progress is fundamentally limited by the reliance on full-length trajectory imitation. For tasks involving multiple order-independent sub-goals, the optimal solution space forms a vast combinatorial diamond lattice. Forcing this rich topology into monolithic trajectories causes a severe topological collapse, indiscriminately penalizing valid alternative explorations and severely degrading policy diversity. To address this, we propose DART-SD (Diamond-topology Aware Retrieval and Tuning for Self-Distillation), a novel framework that shifts the paradigm from global forcing to topology-guided localized correction. DART-SD first models the execution process as a converging Interaction-State Transition Graph (ISTG), faithfully capturing the inherent diamond topology of successful and failed exploratory paths. During autonomous rollouts, the framework identifies the Critical Topological Breakpoint (CTB) and retrieves success-supported recovery references. Finally, we introduce a progressive self-distillation paradigm through CTB-guided localized supervision, ensuring that the training loss is calculated exclusively on the generated recovery steps while strictly protecting the valid reasoning prefix from destructive gradient updates. Experiments on complex multi-turn tool-calling benchmarks demonstrate that DART-SD significantly outperforms traditional full-trajectory baselines.

Figures & tables

Figure 1: Comparison of training paradigms. (a) SFT Teacher Boosting applies an indiscriminate global loss, which overwrites valid exploration. (b) Standard-RL (GRPO) misassigns credit through a uniformly distributed reward spread. (c) DART-SD dynamically identifies the Critical Topological Breakpoints (CTB) and applies localized correction while preserving valid exploration.

Figure 2: Overview of DART-SD. (1) DART-SD constructs an ISTG from teacher rollouts, where main and auxiliary nodes model information acquisition and useless exploration. (2–3) Failed student rollouts are collected and replayed in the same interaction-state space, then projected onto the budget-filtered success-reachable region $\mathcal{R}_{x}^{+}$ , where the first projectable-to-non-projectable transition defines the CTB. (4–5) Conditioned on the retained student prefix and privileged teacher references, DART-SD generates a recovery continuation after the CTB and applies localized supervision only to the generated assistant tokens. (6) This CTB-guided SFT loop is progressively repeated across self-distillation rounds.

Methods	FTRL			BFCL	ToolHop	$\tau$ -bench	RoTBench			Avg.
Methods	Solve-P	Solve-R	Solve-F1	Multi-Turn	AC	Pass^1	TS	PI	CF	Avg.
Qwen3-4B
Base	21.00	26.54	21.81	10.14	20.20	15.15	69.52	26.31	16.07	25.19
$\spadesuit$ SFT	34.26	49.02	37.96	14.57	40.50	21.82	71.67	44.40	24.40	37.62
$\spadesuit$ SCoRe-SFT	23.79	28.20	24.61	12.75	25.83	11.52	60.36	31.19	18.33	26.29
$\spadesuit$ OPSD	22.37	29.21	23.50	11.00	20.20	23.03	68.93	35.24	21.67	28.35

Table 1: Performance comparison of different training methods on five tool-use benchmarks using Qwen3-4B and Qwen3-8B backbones. Training-based baselines are organized into distillation-based

\spadesuit

and reinforcement learning

\diamondsuit

paradigms. All trainable methods are trained on FTRL and evaluated on both the in-domain FTRL test set and four out-of-domain benchmarks. The best and second-best results within each backbone are highlighted.

Figure 3: Performance comparison of Qwen3-8B, DART-SD, and the teacher across five tool-use benchmarks. DART-SD improves upon Qwen3-8B on all benchmarks and surpasses the teacher on FTRL, ToolHop, and

\tau

-bench.

Task	DART-SD					Golden
Task	Iter1	Iter2	Iter3	Iter4	Iter5	Golden
Single	1.23	1.15	1.10	1.09	1.07	1.00
Multi	3.83	4.63	4.54	4.49	4.41	4.71
Para-Single	3.29	2.65	2.27	2.26	2.22	2.11
Para-Multi	7.25	6.48	5.97	5.82	5.62	6.97
Overall	4.23	3.99	3.71	3.65	3.55	4.02

Table 2: Average tool-call length of successful trajectories across progressive SFT iterations on the FTRL test set. DART-SD progressively improves Solve-F1 while shortening its tool traces, eventually producing more efficient traces than the golden solutions given during data construction.

Task	CTB Position
Task	Iter1	Iter2	Iter3	Iter4	Iter5
Single	0.034	0.186	0.137	0.249	0.077
Multi	0.500	1.624	1.729	1.876	1.953
Para-Single	0.095	0.219	0.299	0.328	0.313
Para-Multi	0.395	1.524	1.692	1.791	1.816
Overall	0.348	1.185	1.310	1.421	1.452

Table 3: Average CTB positions of failed training trajectories across progressive SFT. Larger values indicate that the first departure from empirically recoverable behavior occurs later, meaning that the model correctly executes a longer trajectory prefix before localized recovery is required.

Method	FTRL	BFCL	ToolHop
Qwen3-8B	29.74	40.00	42.21
FTRL-GRPO	32.85	41.50	36.72
ToolRL	26.72	34.25	32.93
MatchTIR (KM)	37.33	47.13	46.16
DART-SD	41.03	49.75	46.43

Table 4: Performance comparison under the thinking setting. Results are reported using FTRL Solve-F1, the average BFCL Multi-Turn score, and ToolHop AC.

Method	IFEval	AIME24	AIME25	MMLU	Avg.
Qwen3-8B	34.75	46.67	23.33	70.94	43.92
SFT	35.30	43.33	26.67	71.43	44.18
DART-SD	45.29	50.00	30.00	74.27	49.89

Table 5: General capability evaluation on representative benchmarks. The best results are highlighted.

Method	Solve-P	Solve-R	Solve-F1
Qwen3-8B	21.18	30.71	23.48
+SD	36.23	44.46	38.10
+CTB	36.62	46.32	39.51
+Progressive SFT	41.32	49.65	43.93
+ISTG ( Ours )	42.00	54.13	45.66

Table 6: Component ablation of DART-SD on the FTRL test set using Solve-P, Solve-R, and Solve-F1.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 585 papers

Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data
Feb 24, 2026Emre Can Acikgoz, Cheng Qian, Jonas Hübotter +3LLM Agent Training Self-Evolving Agents
UIUC · ETH Zurich
Self-Distilled Agentic Reinforcement Learning
May 14, 2026Zhengxi Lu, Zhiyuan Yao, Zhuowen Han +8Agentic RL Self-Distillation
Zhejiang University · Meituan · Tsinghua University
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
Apr 12, 2026Hao Wang, Guozhi Wang, Han Xiao +8RL for LLM Agents LLM Agent Skill Learning
Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences · vivo AI Lab · The Chinese University of Hong Kong +2

Feb 24, 2026J/K move · Enter open · S save

Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji+2

UIUC · ETH Zurich

LLM Agent Training Self-Evolving Agents RL for Tool Use Self-Play RL

Open paper AskPDF arXiv

Large language models (LLMs) are becoming the foundation for autonomous agents that can use tools to solve complex tasks. Reinforcement learning (RL) has emerged as a common approach for injecting such agentic capabilities, but typically under tightly controlled training setups. It often depends on carefully constructed task-solution pairs and substantial human supervision, which creates a fundamental obstacle to open-ended self-evolution toward superintelligent systems. In this paper, we propose Tool-R0 framework for training general purpose tool-calling agents from scratch with self-play RL, under a zero-data assumption. Initialized from the same base LLM, Tool-R0 co-evolves a Generator and a Solver with complementary rewards: one proposes targeted challenging tasks at the other's competence frontier and the other learns to solve them with real-world tool calls. This creates a self-evolving cycle that requires no pre-existing tasks or datasets. Evaluation on different tool-use benchmarks show that Tool-R0 yields 92.5 relative improvement over the base model and surpasses fully supervised tool-calling baselines under the same setting. Our work further provides empirical insights into self-play LLM agents by analyzing co-evolution, curriculum dynamics, and scaling behavior.

Abstract

LLM Agent Training

Explore similar work

Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Data

Self-Distilled Agentic Reinforcement Learning

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents