OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

Qinglin Ye; Zhiyuan Gu; Jingjie Xia; Yiheng Zhang; Kaiyan Zhao; Shunchao Zheng; Yuhang Mu; Wenchao Du

cs.AIAug 25, 2026

OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning

Authors: Qinglin Ye, Zhiyuan Gu, Jingjie Xia, Yiheng Zhang, Kaiyan Zhao, Shunchao Zheng, Yuhang Mu, Wenchao Du,

Organizations: University of Chinese Academy of Sciences · Institute of Computing Technology, Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · University of Macau ·

Abstract

Search-augmented reasoning remains difficult for small language models. On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality multi-turn search trajectories depend on dynamic retriever responses, making SFT data prohibitively expensive to collect at scale; (2) task-specifically trained teachers incur substantial training cost, while directly applying OPD with an off-the-shelf teacher without task-specific fine-tuning constrains the student to the teacher's performance ceiling and suffers from severe training instability. We propose OPDSearch+, the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning. We investigate the role of a frozen off-the-shelf instruct model as the teacher in on-policy distillation, and reveal a key insight: the teacher reshapes the student's policy distribution so that subsequent RL converges to a superior solution that RL alone cannot reach. In stage one, the student interacts with a live search engine and is distilled via a per-position forward KL objective, transferring reasoning decomposition and evidence integration skills without any task-specific teacher training. In stage two, RL refines the distilled student from a richer behavioral foundation, achieving performance that RL alone cannot reach from scratch. Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA.

Figures & tables

Figure 1: Left: Prior OPD requires costly task-specific teacher training; pure RL shows limited multi-hop gains. Right: OPDSearch+ eliminates teacher training by using a frozen off-the-shelf teacher. The teacher reshapes the student’s distribution via on-policy distillation, enabling subsequent RL to escape local optima and achieve strong single-hop, multi-hop, and in-domain performance.

Figure 2: Comparison between RL-based search agent training (Search-R1) and our On-Policy Distillation (OPDSearch+) framework. (a) Search-R1 uses outcome reward (EM/F1) to train the model via GRPO, which conflates retrieval quality with answer correctness and suffers from reward hacking. (b) OPDSearch+ generates on-policy trajectories from the student interacting with a live search engine, then distills the teacher’s token-level distribution onto these trajectories. The teacher’s search behavior serves as implicit supervision for both reasoning and retrieval, providing a strong initialization for subsequent RL refinement.

3B Methods (same model size as OPDSearch+)
Method	Size	Single-Hop QA			SH-Avg	Multi-Hop QA				Avg
Method	Size	NQ †	TriviaQA ⋆	PopQA ⋆	SH-Avg	HotpotQA †	2Wiki ⋆	Musique ⋆	Bamboogle ⋆	Avg
Direct Generation ⋄	3B	0.106	0.288	0.108	0.167	0.149	0.244	0.020	0.024	0.134
SFT ⋄	3B	0.249	0.292	0.104	0.215	0.186	0.248	0.044	0.112	0.176
Naive RAG ⋄	3B	0.348	0.544	0.387	0.426	0.255	0.226	0.047	0.080	0.270
Search-o1 ⋄	3B	0.238	0.472	0.262	0.324	0.221	0.218	0.054	0.320	0.255

Table 1: Performance comparison on QA benchmarks (Exact Match). OPDSearch+ uses a 3B student model. † indicates in-distribution training data. ⋆ indicates out-of-distribution test sets. ⋄ indicates numbers cited from prior work. Best in bold , second best underlined .

Figure 3: OPD training dynamics (14B

\to

3B, forward-KL). (a) Distillation loss decreases rapidly in the first

\sim

50 steps and stabilizes around

0.2

. (b) Validation EM improves progressively across all benchmarks.

Objective	Stability	Best Val EM	Grad Norm
Forward-KL (Ours)	Stable (400+ steps)	0.493	0.73–1.19
Rev-KL Adaptive	Stable (160+ steps)	0.470	0.23–0.32
Rev-KL Clipped	Degrades by step $\sim$ 60	0.136	2.2–4.5
Rev-KL JSD	Collapses at step $\sim$ 20	—	75.0 $\to$ NaN
Rev-KL Log-Ratio	Collapses at step $\sim$ 20	—	$\to$ NaN

Table 2: Comparison of distillation objectives under matched hyperparameters (all 3B student, 14B teacher, same lr/

\alpha_{\text{kl}}

/rollout config). Under these shared settings, only the entropy-adaptive variant avoids collapse among reverse-KL alternatives, but it still underperforms our forward-KL objective. Collapses may be mitigable with objective-specific tuning (see text).

Figure 4: Distillation objective comparison under matched hyperparameters (all 14B

\to

3B). (a) Validation EM: Forward-KL (ours) achieves the highest performance and continues improving through 400 steps. Rev-KL Adaptive avoids collapse but plateaus below forward-KL. Three other reverse variants collapse under default settings. (b) Gradient norm: Forward-KL maintains stable norms (

\sim

0.8–1.2); Adaptive has lower norms (

\sim

0.3) but this does not translate to better EM.

Teacher model evaluated directly (no training)
Method	Single-Hop QA			SH-Avg	Multi-Hop QA				Avg
Method	NQ	TriviaQA	PopQA	SH-Avg	HotpotQA	2Wiki	Musique	Bamboogle	Avg
Qwen2.5-7B-Instruct (teacher)	0.3019	0.5633	0.3404	0.4019	0.2811	0.2624	0.1059	0.3040	0.3084
Qwen2.5-14B-Instruct (teacher)	0.3630	0.6479	0.3919	0.4676	0.3916	0.3513	0.1592	0.4844	0.3985
Pure RL baseline (GRPO, no distillation)
Search-R1-Base (3B) ⋄	0.421	0.583	0.413	0.472	0.297	0.274	0.066	0.128	0.312

Table 3: Ablation study on teacher scale and two-stage training (full test set evaluation, Exact Match). All methods use Qwen2.5-3B as the student/policy model. “OPD only” denotes pure on-policy distillation without RL. “OPD

\to

RL” denotes the two-stage approach (OPD warmup followed by GRPO fine-tuning). “Offline SFT” trains on teacher-generated correct trajectories (EM-filtered). ⋄ indicates numbers cited from prior work. ‡ indicates our reproduced RL baseline using

R=0.9\cdot R_{\text{EM}}+0.1\cdot R_{\text{format}}

(same reward as our OPD

\to

RL stage). Best in bold .

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 511 papers

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning
May 21, 2026Zihan Liang, Yufei Ma, Ben Chen +4Tool-Augmented Language Model Agents On-Policy Self-Distillation
SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning
May 18, 2026Yufei Ma, Zihan Liang, Ben Chen +6On-Policy Self-Distillation RL for Language Model Reasoning
Kuaishou Technology
How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation
Oct 4, 2026Xiaoyu Ma, Haoyue Liu, Zhichao Wang +2On-Policy Distillation Teacher-Student Learning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNiI-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK(SZ)

May 21, 2026J/K move · Enter open · S save

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning

Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian+3

Tool-Augmented Language Model Agents On-Policy Self-Distillation RL for Language Model Reasoning LLM Agent Self-Improvement LLM Reasoning

Open paper AskPDF arXiv

Post-training has become the dominant recipe for turning a language model into a competent search-augmented reasoning agent. A line of recent work pushes its performance further by adding elaborate machinery on top of this standard pipeline. These augmentations import external supervision from stronger external systems, attach auxiliary modules such as process reward models or retrospective critics, restructure the rollout itself with tree search or multi-stage curricula, or shape the reward with hand-crafted bonuses and penalties. Each addition delivers a measurable gain, but each also inflates the training pipeline and ties the recipe to resources or designs that may not always be available. We take a step back and ask whether any of this machinery is actually necessary, and propose Search-E1, a self-evolution method that lets a search-augmented agent improve through only vanilla GRPO interleaved with on-policy self-distillation (OPSD). After each GRPO round, the policy rolls out on its own training questions. A token-level forward KL objective then aligns the policy's inference-time distribution to its own distribution under a privileged context that exposes a more efficient sibling trajectory. Despite this simplicity, the procedure naturally provides dense per-step supervision. On seven QA benchmarks, Search-E1 reaches 0.440 average EM with Qwen2.5-3B, surpassing all open-source baselines at both scales. Code and complete version will be made public soon.

Abstract

On-Policy Distillation

Explore similar work

Search-E1: Self-Distillation Drives Self-Evolution in Search-Augmented Reasoning

SD-Search: On-Policy Hindsight Self-Distillation for Search-Augmented Reasoning

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation