Learning from Think-Mode Advantage via On-Policy Distillation

Wanqi Ren; Jianxiang Wang; Danxuan Liu; Linyi Ding; Huaixiao Tou

cs.CLSep 29, 2026

Learning from Think-Mode Advantage via On-Policy Distillation

Authors: Wanqi Ren, Jianxiang Wang, Danxuan Liu, Linyi Ding, Huaixiao Tou

Organizations: ByteDance, China

Abstract

Explicit intermediate reasoning gives large language models (LLMs) a stronger problem-solving mode. We study learning from this think-mode advantage via on-policy distillation (OPD). OPD preserves student-generated trajectories and provides dense token-level teacher targets at student-visited prefixes. Privileged reasoning is used during distillation rather than student inference. Uniform ThinkOPD, a natural think-enabled OPD baseline, conditions a fixed teacher on one shared think trace and uniformly distills every sibling student response. Although its prefixes are on-policy, the trace need not follow a route compatible with every complete response: the same privileged trace can induce different teacher-student discrepancies even when responses reach the same outcome. We summarize this interaction with trace-response divergence (TRD) and introduce ThinkOPD, which routes supervision at the response level by combining group-relative reward gain with a TRD-based compatibility proxy. Final response weights are normalized within each rollout group. Across mathematical reasoning and code generation, ThinkOPD outperforms Uniform ThinkOPD in both same-model settings and both cross-model teacher-student pairs, and it exceeds representative rationale and self-distillation baselines in a controlled comparison. Controlled interventions show that outcome benefit and the TRD-based proxy provide complementary routing signals in this setting. Think-enabled OPD provides a controlled setting for studying how teacher advantage becomes transferable along student responses.

Figures & tables

Figure 1: ThinkOPD response routing under a shared privileged trace. Uniform ThinkOPD weights sibling responses equally; ThinkOPD routes them using outcome benefit and a TRD-based compatibility proxy with group-normalized weights.

Method	On-policy	Compatibility	Benefit	Multi-Trace
SDFT	$\checkmark$	$\times$	$\times$	$\times$
OPSD/SDPO	$\checkmark$	$\times$	$\times$	$\times$
BRTS	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
U-ThinkOPD	$\checkmark$	$\times$	$\times$	$\times$
ThinkOPD	$\checkmark$	$\checkmark$	$\checkmark$	$\times$

Table 1: Supervision interfaces compared by on-policy scoring, compatibility modeling, outcome-aware selection or weighting, and use of multiple teacher traces. Multi-Trace means that a method generates or compares multiple teacher reasoning trajectories per training instance; it does not denote sibling student responses. All methods use privileged teacher context, and U-ThinkOPD denotes Uniform ThinkOPD.

Figure 2: A controlled trace–response discrepancy case. One closed-form think trace conditions two correct responses with the same answer and reward; the route-aligned response has lower TRD than the alternative recurrence route.

Figure 3: Sample mass (left) and mean response weight (right) for 768 valid responses in 192 four-response prompt groups from Qwen3-1.7B, binned by reward advantage and TRD. Each coarse region spans two fine bins per axis.

Method	Math			Coding		Overall
Method	AIME24	AIME25	AMC23	Skywork	LCBv5	Overall
SDFT	28.4	25.2	56.9	18.8	12.0	28.2
OPSD	28.8	24.7	57.7	19.4	13.0	28.7
SDPO	30.0	25.1	58.4	20.2	13.5	29.4
BRTS	30.1	26.6	58.2	20.4	13.4	29.7
U-ThinkOPD	29.2	25.0	57.6	20.1	13.3	29.0

Table 2: Qwen3-1.7B baseline comparison under eval16 (%, avg@16; higher is better). Overall is the unweighted mean over the five benchmarks. Bold and underlined entries indicate first and second place.

Model	Method	Math						Coding				Overall
		AIME24		AIME25		AMC23		Skywork held.		LCBv5		Overall
		Avg.	Best	Avg.	Best	Avg.	Best	Avg.	Best	Avg.	Best	Avg.	Best
Qwen3 0.6B	Think off	2.5	16.6	2.5	6.6	23.5	62.6	10.1	24.8	6.1	15.0	8.9	25.1
	Think on	8.7	36.6	17.0	40.0	46.3	78.3	19.5	41.7	12.8	24.7	20.9	44.3
Qwen3 1.7B	Think off	13.5	46.6	10.0	30.0	40.6	73.4	18.1	33.3	11.2	21.8	18.6	41.0
	Think on	43.3	76.6	36.0	70.0	73.7	92.7	40.1	58.0	31.1	46.2	44.8	68.7

Table 3: Same-model performance under eval16 across mathematical reasoning and code generation (%, higher is better). Overall Avg. and Best are the unweighted means of their corresponding five benchmark columns. Think-off and think-on rows are undistilled base-policy references; within each distilled model group, boldface marks the higher result.

Model	Method	AIME24	AIME25	AMC23	Avg.	$\Delta$
4B $\downarrow$ 1.7B	OPD	33.9	29.7	66.2	51.8	0.0
	U-ThinkOPD	35.2	30.6	65.5	51.8	0.0
	ThinkOPD	37.5	34.3	66.5	53.7	+1.9
1.7B $\downarrow$ 0.6B	OPD	2.5	1.4	24.5	15.0	-6.0
	U-ThinkOPD	5.2	6.6	31.9	21.0	0.0
	ThinkOPD	5.0	7.0	34.5	22.6	+1.6

Table 4: Cross-model Qwen3 math performance under eval16 (%, avg@16, higher is better); the first column gives teacher and student sizes. Avg. pools all AIME24, AIME25, and AMC23 problems, and

\Delta

is its gain over Uniform ThinkOPD within each pair.

Routing component ablations
Configuration	AIME24	AIME25	AMC23	Avg.	$\Delta$
U-ThinkOPD	29.2	25.0	57.6	44.8	+0.0
w/o TRD	31.0	26.3	59.4	46.5	+1.7
w/o Reward	27.5	25.4	55.6	43.4	-1.4
w/ Reversed TRD	26.3	25.1	54.8	42.5	-2.3
Alternative compatibility proxies

Table 5: Math routing ablations under eval16 (%, avg@16; higher is better). Avg. pools all AIME24, AIME25, and AMC23 problems;

\Delta

is relative to Uniform ThinkOPD.

Figure 4: Sensitivity of pooled Math avg@16 to future horizon $H$ and inverse-TRD strength $\eta$ . Each sweep varies one parameter while fixing the other at its default.

Figure 5: Recurring cost under matched asynchronous OPD, normalized to standard OPD ( $1.00$ ). Shared offline-bank construction is excluded; BRTS-4 selects among four online teacher trajectories.

Explore similar work

What Does Privileged Information Add to On-Policy Self-Distillation?
Sep 17, 2026XiuYu Zhang, Wei Chow, Junfeng Fang +2Unsupervised On-Policy Self-Distillation Tutors
National University of Singapore
Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance
May 29, 2026Yuxuan Jiang, Francis FerraroLLM Reasoning Strategies
University of Maryland, Baltimore County
Rethinking On-Policy Self-Distillation for Thinking Models
Jul 6, 2026Simran Kaur, Narutatsu Ri, Yinghui He +2Unsupervised On-Policy Self-Distillation Large Reasoning Models
Princeton Language and Intelligence, Princeton University

Sep 17, 2026, cs.CLJ/K move · Enter open · S save

What Does Privileged Information Add to On-Policy Self-Distillation?

XiuYu Zhang, Wei Chow, Junfeng Fang, Zhenkai Liang+1

National University of Singapore

Unsupervised On-Policy Self-Distillation Tutors

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. These benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision can leave student behavior largely unchanged. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.

Abstract

Explore similar work

What Does Privileged Information Add to On-Policy Self-Distillation?

Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance

Rethinking On-Policy Self-Distillation for Thinking Models