What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

Zhinan Hou; Jiaqi Zhang; Xunliang Cai; Keyou You

cs.AISep 4, 2026

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

Authors: Zhinan Hou, Jiaqi Zhang, Xunliang Cai, Keyou You

Organizations: Tsinghua University · Meituan LongCat Team

Abstract

On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents an empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problem naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., "Alternatively"). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even "unsolvable" examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on five models ranging from 1.5B to 8B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline. Code and models will be publicly available.

Figures & tables

Figure 1: Learning Dynamics of 1-shot OPD. We visualize the training trajectories of six representative examples across three difficulty levels: two easy problems (

\{\pi_{105}\}

and

\{\pi_{178}\}

, left), two medium problems (

\{\pi_{316}\}

and

\{\pi_{794}\}

, middle), and two hard problems (

\{\pi_{954}\}

and

\{\pi_{973}\}

, right). The solid blue curves denote the Average Validation Accuracy (computed across AMC 2023, AIME 2024, and AIME 2025). The solid orange curves with markers track the Overlap Ratio, indicating progressive token-level policy alignment with the teacher model. The horizontal dashed lines represent the full-set baseline (Full-Set OPD) for comparison.

Figure 2: Per-Token KL Divergence Reduction post 1-Shot OPD. We report the top 16 tokens with the highest expected KL reduction after distillation for models trained respectively on

\{\pi_{178}\}

(left),

\{\pi_{794}\}

(middle), and

\{\pi_{954}\}

(right).

Easy Problems
Model	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	Avg
Base	31.7	23.0	60.7	82.6	34.1	22.4	42.4
Teacher	55.8	35.8	83.4	87.4	43.4	29.8	55.9
Full-Set	51.9	34.0	78.9	86.7	44.3	26.6	53.7
$\{\pi_{5}\}$	39.2	28.8	69.9	85.3	39.3	25.3	48.0
$\{\pi_{20}\}$	40.3	30.6	69.1	86.2	40.8	25.7	48.8

Table 1: 1-Shot OPD Performance Across Mathematical Reasoning Benchmarks. We compare the student model DeepSeek-R1-Distill-Qwen-1.5B (Base), the teacher model JustRL-DeepSeek-1.5B (Teacher), and the full-dataset distillation baseline (Full-Set) against our 1-shot OPD models. Notably, 1-shot models trained on hard problems consistently outperform those trained on easy problems, with the best-performing example (

\{\pi_{973}\}

) achieving

51.7\%

average accuracy, nearly matching the Full-Set baseline (

53.7\%

Figure 3: Learning Trajectories across Difficulty Levels. Left: Average Validation Accuracy over training steps. Right: Average Training Accuracy on the selected training example over training steps. Solid lines indicate the mean and shaded areas represent the standard deviation across the examples in each category.

Figure 4: Training Dynamics in Length-Constrained Setting Training dynamics are presented for problem pairs grouped by difficulty: Easy vs. Hard (1K token limit, left), Medium vs. Hard (2K token limit, middle), and Hard vs. Hard (4K token limit, right). The top plots monitor token entropy, while the bottom plots track validation accuracy.

Figure 5: Validation performance across different maximum response length. We compare distillation validation accuracy across training steps under

1\text{K}

2\text{K}

4\text{K}

, and

7\text{K}

maximum token limits.

Dataset	Size	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	Avg
DAPO-Math-17K	17K	51.9	34.0	78.9	86.7	44.3	26.6	53.7
DAPO-Subset	1K	50.2	34.8	79.3	87.1	42.8	27.3	53.6
$\{\pi_{5},\pi_{20},\dots,\pi_{192}\}$	8	45.8	30.6	76.1	87.1	41.6	26.2	51.2
$\{\pi_{316},\pi_{487},\dots,\pi_{820}\}$	8	49.0	32.7	76.8	86.8	42.1	26.0	52.2
Random	8	48.1	33.8	76.2	86.3	41.9	26.4	52.1
$\{\pi_{973}\}$	1	47.5	31.7	75.1	87.7	41.5	26.9	51.7

Table 2: Few-Shot OPD Performance across Different Dataset Sizes. The baselines include the full-dataset distillation (DAPO-Math-17K) and a randomly sampled 1K subset (DAPO-Subset). Distilling on just 8 hard examples achieves

53.6\%

average accuracy, nearly matching the Full-Set 17K baseline (

53.7\%

Skywork-OR1-Math-7B $\rightarrow$ DeepSeek-R1-Distill-Qwen-7B
Model	Size	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	Avg
DeepSeek-R1-Distill-Qwen-7B	–	51.9	38.8	81.0	89.6	43.6	32.6	56.3
Skywork-OR1-Math-7B	–	62.1	45.6	84.0	91.9	47.7	34.1	60.9
Full-Set	17K	58.3	44.6	83.4	91.1	46.0	34.2	59.6
Random	8	57.3	42.1	81.1	91.0	45.2	33.2	58.3
$\{\pi_{874},\dots,\pi_{997}\}$	8	58.3	43.9	83.1	91.3	46.3	34.0	59.5

Table 3: Selecting few hard examples for OPD is effective across different models. We report benchmark accuracies across diverse model pairings and training sizes.

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Value
Training temperature	1.0
Global batch size	64
Mini batch size	64
Rollout number	8
Temperature	1.0
LogProb top- $K$	16

Appendix

Table 4: Default hyperparameters for OPD.

Item	Value
Training temperature	1.0
Global batch size	64
Mini batch size	64
Rollout number	8
Temperature	1.0
Max prompt length	1024

Appendix

Table 5: Default hyperparameters for GRPO.

Easy Problems			Medium Problems			Hard Problems
Model	LCB	GPQA	Model	LCB	GPQA	Model	LCB	GPQA
$\{\pi_{5}\}$	37.3	21.4	$\{\pi_{316}\}$	39.3	22.2	$\{\pi_{874}\}$	40.9	24.6
$\{\pi_{20}\}$	38.3	21.1	$\{\pi_{487}\}$	39.6	23.7	$\{\pi_{890}\}$	40.1	23.4
$\{\pi_{21}\}$	37.1	23.5	$\{\pi_{530}\}$	39.4	23.0	$\{\pi_{948}\}$	41.8	25.9
$\{\pi_{63}\}$	38.8	21.6	$\{\pi_{543}\}$	38.9	22.3	$\{\pi_{954}\}$	40.8	23.7
$\{\pi_{84}\}$	36.9	22.7	$\{\pi_{661}\}$	39.1	23.6	$\{\pi_{961}\}$	39.7	23.8

Appendix

Table 6: Out of Distribution Generalization of 1-Shot OPD Models from Table 1 . We report the accuracy on LiveCodeBench (LCB) and GPQA across individual models. The Avg rows denote the average accuracy across the 8 models trained on Easy, Medium, and Hard samples, respectively.

Figure 6: Training dynamics and validation trajectories of 1-shot OPD in the coding domain. We report the in-distribution validation accuracy on LiveCodeBench, the out of distribution Math Accuracy (computed as the average accuracy across AMC23, AIME24, and AIME25), and the distillation loss over training steps.

Model	Size	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	LiveCodeBench	GPQA	Avg
Student	N/A	31.7	23.0	60.7	82.6	34.1	22.4	32.4	19.4	38.3
Teacher	N/A	55.8	35.8	83.4	87.4	43.4	29.8	42.8	28.0	50.8
Full-Set	2641	44.4	32.1	78.7	86.3	40.8	27.0	42.7	26.4	47.3
Random	8	41.2	29.0	72.0	86.1	39.3	26.1	41.6	25.6	45.1
Hard	8	43.7	31.8	77.2	87.7	39.9	27.0	42.8	26.3	47.1

Appendix

Table 8: Data Selection Efficacy in the Coding Domain. We report benchmark performance across six mathematical reasoning benchmarks alongside LiveCodeBench and GPQA. We evaluate the student model (DeepSeek-R1-Distill-Qwen-1.5B) distilled on LeetCodeDataset training prompts under different selection strategies.

Figure 7: Validation Accuracy over Extended Training in 1-Shot OPD. We plot the validation trajectories over 2,000 steps for 1-shot OPD models trained on three representative examples of varying difficulty levels:

\{\pi_{105}\}

(Easy, left),

\{\pi_{794}\}

(Medium, middle), and

\{\pi_{973}\}

(Hard, right). Despite extended training, the validation performance across all models remains exceptionally stable without policy collapse.

Figure 8: Token-level KL Loss over Generation Position. For visual clarity, we partition the token positions into non-overlapping intervals of 100 tokens and plot the binned average KL loss over positions.

Figure 9: Study on student rollout temperature

T

for the easy example

\pi_{178}

during 1-shot OPD. The left subfigure monitors the average per-token entropy of the student’s rollouts across training steps under different sampling temperatures

T\in\{0.6,0.8,1.0,1.1\}

. The right subfigure tracks the corresponding Average Validation Accuracy (computed over AMC 2023, AIME 2024, and AIME 2025).

Random Data Selection
Model	AIME24	AIME25	AMC23	MATH500	Olympiad	Minerva	LiveCodeBench	GPQA	Avg
Seed 1	48.1	33.8	76.2	86.3	41.9	26.4	40.3	24.3	47.2
Seed 2	46.7	35.0	76.1	87.0	41.6	26.1	39.9	21.9	46.8
Seed 3	46.2	33.8	77.0	86.4	42.4	26.2	40.1	24.9	47.1
Our Data Selection
Seed 1	51.7	34.2	79.2	87.2	42.7	26.4	42.0	27.2	48.8

Appendix

Table 9: Robustness of Data Selection Across Different Random Seeds. We report benchmark accuracies across eight evaluation benchmarks alongside their average (Avg) for different data selection under three independent random sampling seeds. Group 1 represents the default seed used in the main text, while Group 2 and Group 3 represent additional independent random seeds.

Easy Problems			Medium Problems			Hard Problems
ID	Student	Teacher	ID	Student	Teacher	ID	Student	Teacher
$\pi_{5}$	100.0	100.0	$\pi_{316}$	68.8	100.0	$\pi_{874}$	0.0	6.3
$\pi_{20}$	100.0	100.0	$\pi_{487}$	37.5	100.0	$\pi_{890}$	0.0	0.0
$\pi_{21}$	100.0	100.0	$\pi_{530}$	25.0	100.0	$\pi_{948}$	0.0	0.0
$\pi_{63}$	93.8	100.0	$\pi_{543}$	50.0	75.0	$\pi_{954}$	0.0	0.0
$\pi_{84}$	93.8	100.0	$\pi_{661}$	25.0	68.8	$\pi_{961}$	0.0	0.0

Appendix

Table 10: Accuracies of Sampled Problems. We report the rollout accuracies of both the student and teacher models across 16 rollouts per problem during our data preparation phase.

Prompt:
⬇ Let $a$ and $b$ be positive real numbers . Find the minimum value of $a ^2 + b ^2 + \ frac {1}{( a + b )^2}.$ The answer is in the form k \ sqrt { m }+ n ,. Please provide the value of k + m + n . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $3$ .

Appendix

Table 11: Details of example

\pi_{5}

Prompt:

⬇ In trapezoid

ABCD

the lengths of the bases

AB

and

CD

are 8 and 17 respectively . The legs of the trapezoid are extended beyond

A

and

B

to meet at point

E

. What is the ratio of the area of triangle

EAB

to the area of trapezoid

ABCD

? Express your answer as a common fraction . The answer is in the form rac { m }{ n }, where gcd ( m , n ) = 1. Please provide the value of m + n . Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

289

Appendix

Table 12: Details of example

\pi_{20}

Prompt:

⬇ Lines

l_1 ^{}

and

l_2 ^{}

both pass through the origin and make first - quadrant angles of

\ frac {\ pi }{70}

and

\ frac {\ pi }{54}

radians , respectively , with the positive

x

- axis . For any line

l

, the transformation

R ( l )

produces another line as follows :

l

is reflected in

l_1

, and the resulting line is reflected in

l_2

. Let

R ^{(1)}( l )= R ( l )

and

R ^{( n )}( l )= R \ left ( R ^{( n -1)}( l )\ right )

. Given that

l

is the line

y =\ frac {19}{92} x

, find the smallest positive integer

m

for which

R ^{( m )}( l )= l

. Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

945

Appendix

Table 13: Details of example

\pi_{21}

Prompt:
⬇ On the party , every boy gave $1$ candy to every girl , and every girl gave $1$ candy to every boy . Then every boy ate $2$ candies , and every girl ate $3$ candies . It is known that $\ frac {1}{4}$ of all candies were eaten . Find the greatest possible number of children at the party . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $35$ .

Appendix

Table 14: Details of example

\pi_{63}

Prompt:
⬇ The prime numbers $a$ , $b$ , and $c$ satisfy the equation $a + b ^2 = 4 c ^2$ . Determine the sum of all possible values of $a + b + c$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $31$ .

Appendix

Table 15: Details of example

\pi_{84}

Prompt:
⬇ Compute the smallest positive integer $N$ for which $N \ cdot 2^{2024}$ is a multiple of $2024$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $253$ .

Appendix

Table 16: Details of example

\pi_{105}

Prompt:
⬇ Determine the number of pairs $( a , b )$ of real numbers such that $10 , a , b , ab$ is an arithmetic progression . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $2$ .

Appendix

Table 17: Details of example

\pi_{178}

Prompt:
⬇ Compute the remainder when $2 ^{3^5}+ 3^{5^2}+ 5^{2^3}$ is divided by $30$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $6$ .

Appendix

Table 18: Details of example

\pi_{192}

Prompt:

⬇ A dartboard consists of three concentric circles with radii 4, 6, and 8. Three darts are thrown at the board , sticking at random locations . Determine the probability that each dart lands in a different region of the dartboard . The probability can be expressed as

\ frac { m }{ n }

, where

m

and

n

are relatively prime positive integers . Calculate

m + n

. Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

617

Appendix

Table 19: Details of example

\pi_{316}

Prompt:
⬇ In how many ways can the integers from 1 to n be ordered subject to the condition that , except for the first integer on the left , every integer differs by 1 from some integer to the left of it ? Please provide the number of ways for $n = 6$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $32$ .

Appendix

Table 20: Details of example

\pi_{487}

Prompt:

⬇ Determine the largest integer

N

for which there exists a

6 \ times N

table

T

that has the following properties :\ n \ n - Every column contains the numbers

1 , 2, \ ldots , 6

in some ordering .\ n - For any two columns

i \ ne j

, there exists a row

r

such that

T ( r , i ) = T ( r , j )

.\ n - For any two columns

i \ ne j

, there exists a row

s

such that

T ( s , i ) \ ne T ( s , j )

. Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

120

Appendix

Table 21: Details of example

\pi_{530}

Prompt:
⬇ Let f be a polynomial of degree $3$ with integer coefficients such that $f (0) = 3$ and $f (1) = 11$ . If f has exactly $2$ integer roots , how many such polynomials $f$ exist ? Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $0$ .

Appendix

Table 22: Details of example

\pi_{543}

Prompt:
⬇ Find the area of the figure on the coordinate plane bounded by the straight lines $x = 0$ , $x = 2$ , and the graphs of the functions $y = \\ sqrt { x ^3 + 1}$ and $y = -\\ sqrt [3]{ x ^2 + 2 x }$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $6$ .

Appendix

Table 23: Details of example

\pi_{661}

Prompt:

⬇ An organization has

30

employees ,

20

of whom have a brand A computer while the other

10

have a brand B computer . For security , the computers can only be connected to each other and only by cables . The cables can only connect a brand A computer to a brand B computer . Employees can communicate with each other if their computers are directly connected by a cable or by relaying messages through a series of connected computers . Initially , no computer is connected to any other . A technician arbitrarily selects one computer of each brand and installs a cable between them , provided there is not already a cable between that pair . The technician stops once every employee can communicate with each other . What is the maximum possible number of cables used ? Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

191

Appendix

Table 24: Details of example

\pi_{763}

Prompt:
⬇ Tanya wrote numbers in the form $n ^7 - 1$ for $n = 2, 3, \ ldots$ and noticed that for $n = 8$ , she obtained a number divisible by $337$ . For what minimal $n$ did she get a number divisible by $2022$ ? Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $79$ .

Appendix

Table 25: Details of example

\pi_{794}

Prompt:
⬇ Vasya has $n$ candies of several types , where $n > 145$ . It is known that for any group of at least 145 candies , there is a type of candy which appears exactly 10 times . Find the largest possible value of $n$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $160$ .

Appendix

Table 26: Details of example

\pi_{820}

Prompt:
⬇ Find the number of subsets of $\{1,3,5,7,9,11,13,15,17,19\}$ where the elements in the subset add to $49$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $22$ .

Appendix

Table 27: Details of example

\pi_{874}

Prompt:
⬇ Let $x_1 , x_2 , \ dots , x_ {100}$ be real numbers such that $\| x_1 \| = 63$ and $\| x_ { n +1}\| = \| x_n + 1\|$ for $n = 1, 2, \ dots , 99$ . Find the largest possible value of $(- x_1 - x_2 - \ cdots - x_ {100})$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $2034$ .

Appendix

Table 28: Details of example

\pi_{890}

Prompt:
⬇ Two real numbers $x$ and $y$ are chosen at random in the interval (0,1) with respect to the uniform distribution . What is the probability that the closest integer to $x / y$ is even ? The original answer is in the form $r + s \ pi$ , please give the value of $r + s$ . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $4$ .

Appendix

Table 29: Details of example

\pi_{948}

Prompt:

⬇ Let

f :\{1,2,\ dots ,2019\}\ to \{-1,1\}

be a function , such that for every

k \ in \{1,2,\ dots ,2019\}

, there exists an

\ ell \ in \{1,2,\ dots ,2019\}

such that

\ sum_ { i \ in \ mathbb { Z }:(\ ell - i )( i - k )\ geqslant 0} f ( i )\ leqslant 0.

Determine the maximum possible value of

\ sum_ { i \ in \ mathbb { Z }:1\ leqslant i \ leqslant 2019} f ( i ).

Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

673

Appendix

Table 30: Details of example

\pi_{954}

Prompt:

⬇ A herder has forgotten the number of cows she has and does not want to count all of them . She remembers these four facts about the number of cows :\ n \ n - It has

3

digits .\ n - It is a palindrome .\ n - The middle digit is a multiple of

4

.\ n - It is divisible by

11

.\ n \ nWhat is the sum of all possible numbers of cows that the herder has ?\ n \ n

\\ textbf {( A ) }343 \\ \\ textbf {( B ) }494 \\ \\ textbf {( C ) }615 \\ \\ textbf {( D ) }635 \\ \\ textbf {( E ) }726

Please reason step by step , and put your final answer within \ boxed {{}}.

Ground truth:

726

Appendix

Table 31: Details of example

\pi_{961}

Prompt:
⬇ Find all ordered pairs $( a , b )$ of positive integers for which \ n $\ n \ frac {1}{ a }+\ frac {1}{ b }=\ frac {3}{2018} .\ n$ \ nPlease provide the sum of all integers in the ordered pairs . Please reason step by step , and put your final answer within \ boxed {{}}.
Ground truth: $1438383$ .

Appendix

Table 32: Details of example

\pi_{973}

Explore similar work

Sep 3, 2026

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches

71.5\%

, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach

98.9\%

and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Zixuan Fu, Bingxiang He, Yuxin Zuo +10

Tsinghua University · University of Chinese Academy of Sciences · Northeastern University +2

Language Model Distillation On-Policy Distillation

Ask PDF

May 10, 2026

On-Policy Distillation with Best-of-N Teacher Rollout Selection

On-policy distillation (OPD), which supervises a student on its own sampled trajectories, has emerged as a data-efficient post-training method for improving reasoning while avoiding the reward dependence of reinforcement learning and the catastrophic forgetting often observed in standard supervised fine-tuning. However, standard OPD typically computes teacher supervision under noisy student-generated contexts and often relies on a single stochastic teacher rollout per prompt. As a result, the supervision signal can be high-variance: the sampled teacher trajectory can be incorrect, uninformative, or poorly matched to the student's current reasoning behavior. To address this limitation, we propose BRTS, a Best-of-N Rollout Teacher Selection framework for on-policy distillation. BRTS augments standard student-context OPD with a teacher-context supervision branch constructed from the curated teacher trajectory. Rather than distilling from the first sampled teacher rollout, BRTS samples a small pool of teacher trajectories and selects the auxiliary trajectory using a simple priority rule: correctness first, student alignment second. When multiple correct teacher trajectories are available, BRTS chooses the one most aligned with the student's current behavior; when unconditioned teacher samples fail on harder prompts, it invokes a ground-truth-conditioned recovery step to elicit a natural derivation. The selected trajectory is then used to provide reliable teacher-context supervision inside the OPD loop, augmented with an auxiliary loss on the teacher trajectory. Experiments on AIME 2024, AIME 2025, and AMC 2023 show that BRTS improves over standard OPD on challenging reasoning benchmarks, with the largest gains on harder datasets. Our code is available at https://github.com/BWGZK-keke/BRTS.

Ke Zhang, Yunjie Tian, Dongdi Zhao +4

Johns Hopkins University · TikTok · University of California, San Diego +1

CoT Distillation Best-of-N Sampling Mathematical Reasoning Language Model Distillation On-Policy Distillation Teacher-Student Learning

Ask PDF

Aug 19, 2026

Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.

Chen Yang, Haiyuan Wan, Rengrong Xiong +2

The Hong Kong University of Science and Technology (Guangzhou) · Tsinghua University · Zhejiang University +1

Language Model Distillation On-Policy Distillation RL for Language Model Reasoning

Ask PDF