Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints

Haoxuan Gao; Qin-Wen Luo; Xiang Chen; Yu Tian; Zequn Sun; Xiaofeng Cao; Jiangchao Yao; Sheng-Jun Huang

cs.LGOct 13, 2026

Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints

Authors: Haoxuan Gao, Qin-Wen Luo, Xiang Chen, Yu Tian, Zequn Sun, Xiaofeng Cao, Jiangchao Yao, Sheng-Jun Huang

Organizations: Nanjing University of Aeronautics and Astronautics · Security Capability Center · Nanjing University · Tongji University ·

Abstract

External tools can extend the capabilities of small language models (SLMs), enabling their deployment in resource constrained environments. However, their limited capabilities make it difficult for SLMs to discover effective tool use behaviors through reinforcement learning alone, while direct teacher supervision suffers from teacher student behavioral distribution mismatch and may propagate erroneous preferences. To address these challenges, we propose Selective Teacher guided Policy Optimization (SeTPO), which coordinates student exploration with selective teacher guidance and persistent demonstration supervision. Specifically, SeTPO incorporates reverse KL derived token level teacher feedback into policy advantages on low reward trajectories, while auxiliary forward KL supervision preserves teacher demonstrated behaviors throughout training. Under an idealized bidirectional KL objective, we theoretically establish that the optimal policy achieves expected reward no lower than the teacher's while retaining a guaranteed fraction of the teacher's probability mass. Extensive experiments on BFCL v4 and API Bank demonstrate that SeTPO achieves superior overall performance over supervised fine tuning, reinforcement learning, and distillation baselines across all three student scales.

Figures & tables

Figure 1: Motivation for coordinating supervision in tool learning for small language models. Top: a schematic weather-tool example comparing SFT, GRPO-based RL, the pure OPD baseline used here, and SeTPO . Bottom: BFCL-v4 scores for Qwen2.5-0.5B.

Figure 2: Overview of SeTPO . Selective online teacher guidance and auxiliary supervision from complete teacher trajectories jointly support policy optimization. A composite checker selects low-reward student trajectories for teacher scoring; token-wise log-probability discrepancies refine their GRPO advantages. Both probabilities are evaluated for the same sampled token under the student’s prefix.

Method	BFCL-v4	API-Bank
Always-on guidance	16.51	39.53
Random intervention	13.29	42.21
SeTPO	18.99	46.06

Table 2: Comparison of online teacher-intervention strategies for Qwen2.5-0.5B on BFCL-v4 and API-Bank. The variants retain the same formulation of teacher feedback and other training components, while applying guidance to all trajectories, a random subset, or low-reward trajectories. The best result on each benchmark is shown in bold.

Figure 3: Teacher-intervention rate during SeTPO training. The inset shows the validation reward.

Figure 4: Performance of a Qwen2.5-0.5B student guided by teachers of different sizes across four benchmarks.

Figure 5: Counts of tool-selection and argument-construction errors for Qwen2.5-0.5B on BFCL-v4. Blue and coral denote the comparison model without online teacher guidance and SeTPO , respectively.

Method	BFCL-v4	API-Bank
SeTPO	18.99	46.06
Trajectory-level signal	7.79 (-11.20)	32.50 (-13.56)
w/o SFT loss	16.05 (-2.94)	37.35 (-8.71)
w/o Online Teacher	10.78 (-8.21)	39.36 (-6.70)
GRPO+OPD	15.85 (-3.14)	45.23 (-0.83)

Table 4: Ablation results on BFCL-v4 and API-Bank. Values in parentheses indicate the percentage-point change relative to SeTPO .

Figure 6: Training dynamics of SeTPO with and without the auxiliary SFT loss: (a) correctness reward and (b) format reward.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Learning Rate	$2\times 10^{-5}$
LR Scheduler	Cosine
Total Epochs	3
Global Batch Size	32
Micro Batch Size / GPU	4
Max Sequence Length	4096

Appendix

Table 5: Cold-start initialization settings.

Hyperparameter	Value
Precision	Bfloat16
LoRA Rank ( $r$ )	32
LoRA Alpha ( $\alpha$ )	64
Actor Learning Rate	$2\times 10^{-5}$
GRPO Group Size ( $G$ )	4
RL Total Epochs	15

Appendix

Table 6: SeTPO policy-optimization settings.

Parameter	Value	BFCL-v4	API-Bank
$\gamma$	3.0	6.19	35.18
	3.5	17.34	43.55
	3.9	17.66	43.07
	3.99	18.99	46.06
$\beta$	0.5	17.69	45.39
	1.0	18.99	46.06

Appendix

Table 7: Hyperparameter sensitivity for the Qwen2.5-0.5B student. The best result in each column within each parameter group is bolded.

Teacher Model	BFCL-v4	API-Bank	GSM8K	HotpotQA
Qwen2.5-0.5B	16.73	38.53	30.55	23.0
Qwen2.5-1.5B	18.99	46.06	38.51	35.0
Qwen2.5-3B	17.52	42.04	31.77	35.0
Qwen2.5-7B	12.93	31.16	35.10	29.0

Appendix

Table 8: Performance of the Qwen2.5-0.5B student under teachers of different model scales.

RL for Tool Use

Reinforcement learning for tool use trains agents to choose, execute, and learn from external tools through interaction and feedback.

See all 191 papers

DemoPSD: Disagreement-Modulated Policy Self-Distillation
Jul 2, 2026Yunhe Li, Hao Shi, Wenhao Liu +5On-Policy Self-Distillation RL for Language Model Reasoning
1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology +1
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
May 13, 2026Xinyu Liu, Kechen Jiao, Chunyang Xiao +10RL for Language Model Reasoning Policy Optimization
School of Computer Science and Engineering, Northeastern University, China · Tsinghua University · Meituan +2
A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone
May 19, 2025Jitai Hao, Qiang Huang, Hao Liu +3Language Model Pretraining LLM Pruning
School of Intelligence Science and Engineering, Harbin Institute of Technology, Shenzhen · Baidu Inc. · Leiden University +1

Jul 2, 2026J/K move · Enter open · S save

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan+4

1City University of Hong Kong · 2Tsinghua University · 3Shenzhen University of Advanced Technology+1

On-Policy Self-Distillation RL for Language Model Reasoning Language Model Distillation

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

Abstract

RL for Tool Use

Explore similar work

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence

A Token is Worth over 1,000 Tokens: Efficient Knowledge Distillation through Low-Rank Clone