CD-OPD: Contextual Digest for On-Policy Distillation

Mingyuan Li; Yucheng Li; Xiaxia Wang; Gong Cheng

cs.LGOct 18, 2026

CD-OPD: Contextual Digest for On-Policy Distillation

Authors: Mingyuan Li, Yucheng Li, Xiaxia Wang, Gong Cheng

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford

Abstract

Teacher takeover in on-policy distillation (OPD) can produce incorrect responses even when the student and teacher independently answer the same problem correctly, a failure we quantify using Collaboration Failure Rate (CFR). To address it, we propose Contextual Digest for On-Policy Distillation (CD-OPD). CD-OPD structures the earlier context of teacher guidance through a teacher-generated dependency graph, curating the required problem conditions and preceding reasoning as node summaries alongside the unchanged guidance in a Contextual Digest before student continuation. We further develop online meta-reweighting, which compares Digest segment gradients with a task gradient from independent student responses to improve student-only performance, favoring updates predicted to reduce task loss. Across eight benchmarks, CD-OPD improves average accuracy by 1.32--7.05 percentage points over representative existing OPD methods. It also reduces CFR by 27.4%--56.4% relative to existing takeover-based OPD methods.

Figures & tables

Figure 1: Accuracy and CFR under teacher takeover (1,024 mathematical problems randomly sampled from our local copy of DAPO-Math-17k-dedup; student: Qwen3-1.7B; teacher: Qwen3-4B-Instruct-2507; accuracy reported as mean@4).

Figure 2: Comparison of independent student and teacher responses, takeover-based OPD, and CD-OPD.

Figure 3: Overview of CD-OPD: (a) Trajectory Construction with Contextual Digest (Section 3.3 ); (b) Policy Update with Online Meta-Reweighting (Section 3.4 ).

Student: Qwen3-0.6B-Non-Thinking
Method	AIME2024	AIME2025	AIME2026	MATH500	AMC2023	OlympiadBench	HMMT-Feb26	HMMT-Nov25	Avg.
Teacher	60.42	46.04	52.19	94.20	93.83	70.62	31.25	41.88	61.30
Student	1.77	2.40	0.73	44.10	24.45	16.36	0.76	3.85	11.80
SFT @110	4.90	7.60	4.06	59.45	34.92	26.74	1.89	3.02	17.82
KD @110	4.17	7.19	4.79	57.75	35.23	27.15	2.94	2.71	17.74
GRPO @110	8.23	15.21	10.00	68.60	46.56	34.42	7.86	6.04	24.62

Table 1: Performance across eight mathematical reasoning benchmarks. The best and second-best results within each student size are shown in bold and underlined, respectively. Blue-shaded rows denote our method, and light-blue-shaded rows denote improvements in percentage points relative to the indicated baselines.

Method	AIME2024	AIME2025	AIME2026	MATH500	AMC2023	OlympiadBench	HMMT-Feb26	HMMT-Nov25	Avg.	$\boldsymbol{\Delta}$
Student: Qwen3-0.6B-Non-Thinking
CD-OPD	16.77	20.94	15.63	77.50	56.33	49.07	12.97	10.10	32.41	—
w/o Dependency Graph	14.58	18.33	14.58	76.35	55.47	48.74	11.65	7.60	30.91	$-1.50$
w/o Online Meta-Reweighting	16.35	17.50	15.21	76.65	56.02	48.70	11.46	8.65	31.32	$-1.09$
Student: Qwen3-1.7B-Non-Thinking
CD-OPD	43.65	31.56	33.23	89.80	79.53	63.46	24.81	20.21	48.28	—

Table 2: Ablation results across eight benchmarks.

\Delta

denotes the change in average accuracy relative to CD-OPD, in percentage points.

Figure 4: CFR for Qwen3-0.6B and Qwen3-1.7B, computed by pooling numerator and denominator counts across eight benchmarks.

Figure 5: Effects of

B

and

R

on CD-OPD’s average accuracy across eight benchmarks and additional training time relative to training without online meta-reweighting. Left : Qwen3-0.6B; Right : Qwen3-1.7B.

Figure 6: Contextual Digest length distributions during training for Qwen3-0.6B and Qwen3-1.7B.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	$\boldsymbol{N}$	$\boldsymbol{K}$	License
AIME2024	30	32	Apache-2.0
AIME2025	30	32	CC BY-NC-SA 4.0
AIME2026	30	32	CC BY-NC-SA 4.0
MATH500	500	4	MIT
AMC2023	40	32	Apache-2.0
OlympiadBench	674	4	Apache-2.0

Appendix

Table 3: Evaluation dataset statistics and declared licenses.

$\boldsymbol{(B,R)}$	AIME2024	AIME2025	AIME2026	MATH500	AMC2023	OlympiadBench	HMMT-Feb26	HMMT-Nov25	Avg.
Student: Qwen3-0.6B-Non-Thinking
$(2,2)$	16.42	18.07	15.28	76.80	56.07	48.76	11.71	8.89	31.50
$(4,2)$	16.55	19.17	15.41	77.06	56.17	48.88	12.19	9.35	31.85
$\boldsymbol{(8,2)}$	16.77	20.94	15.63	77.50	56.33	49.07	12.97	10.10	32.41
$(2,4)$	16.50	18.70	15.36	76.95	56.13	48.83	11.99	9.16	31.70
$(4,4)$	16.64	19.89	15.50	77.24	56.24	48.96	12.51	9.66	32.08

Appendix

Table 7: Per-benchmark accuracy (%) of CD-OPD for different

(B,R)

configurations. Bold rows indicate the default setting

(B,R)=(8,2)

Method	Avg. Accuracy (%) $\uparrow$	CFR (%) $\downarrow$	Training Time (min/step) $\downarrow$	Response Length (tokens) $\downarrow$	Inference Time (min/response) $\downarrow$
Student: Qwen3-0.6B-Non-Thinking
Relay-OPD	31.04	13.20	4.67	14,473.80	0.22
CD-OPD	32.41	9.58	5.76	14,699.02	0.23
Student: Qwen3-1.7B-Non-Thinking
Relay-OPD	46.96	10.00	4.93	11,738.32	0.28
CD-OPD	48.28	6.49	6.04	11,921.79	0.29

Appendix

Table 8: Training and inference efficiency of Relay-OPD and CD-OPD across Qwen3-0.6B and Qwen3-1.7B.


Problem (AIME2025). Let $m,n$ be relatively prime positive integers such that $\prod_{k=4}^{63}\frac{\log_{k}\!\left(5^{k^{2}-1}\right)}{\log_{k+1}\!\left(5^{k^{2}-4}\right)}=\frac{m}{n}.$ Find $m+n$ . Reference answer: 106. Both responses are generated by the respective students independently, without teacher takeover or a Contextual Digest at inference.
Relay-OPD	CD-OPD
1. Rewrite and separate the product “ We can cancel $\log 5$ from numerator and denominator: ” $\frac{(k^{2}-1)\log(k+1)}{(k^{2}-4)\log k}.$ “ Let’s now split the product into two parts: ” $\underbrace{\prod_{k=4}^{63}\frac{k^{2}-1}{k^{2}-4}}_{\text{rational product}}\;\cdot\;\underbrace{\prod_{k=4}^{63}\frac{\log(k+1)}{\log k}}_{\text{log product}}.$	1. Rewrite and separate the product “ Now, we can write everything in terms of base $5$ . ” $\frac{(k^{2}-1)\log_{5}(k+1)}{(k^{2}-4)\log_{5}k}.$ “ Let’s separate the product into two parts: factorial-like terms and logarithmic terms. ” $\begin{gathered}\prod_{k=4}^{63}\left(\frac{k-1}{k-2}\cdot\frac{k+1}{k+2}\right)\\[-1.0pt] {}\cdot\prod_{k=4}^{63}\frac{\log_{5}(k+1)}{\log_{5}k}.\end{gathered}$
2. Evaluate the two rational factors “ This is a telescoping product: ” $\prod_{k=4}^{63}\frac{k-1}{k-2}=\frac{3}{2}\cdot\frac{4}{3}\cdots\frac{62}{61}=31.$ “ Similarly, look at the second part: ” $\prod_{k=4}^{63}\frac{k+1}{k+2}=\frac{5}{6}\cdot\frac{6}{7}\cdots\frac{64}{65}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}.$ “ So now, the total product of the terms is: ” $(31)\cdot\left(\frac{1}{13}\right)={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.$	2. Evaluate the two rational factors “ Wait — no. Actually: ” (after revisiting the factorial simplification) $\prod_{k=4}^{63}\frac{k-1}{k-2}=\frac{62}{2}=31.$ “ Again, telescoping! ” $\prod_{k=4}^{63}\frac{k+1}{k+2}=\frac{5}{6}\cdot\frac{6}{7}\cdots\frac{64}{65}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}.$ “ The product of the ratio parts is ” $31\cdot\frac{1}{13}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.$
3. Reassemble the original expression “ Wait — but do we have only the first two products? ” “ We have now: ” 1. First product: $\prod_{k=4}^{63}\frac{k^{2}-1}{k^{2}-4}={\color[rgb]{0.6445,0.168,0.1484}\boldsymbol{31}}.$ Reading note. The earlier factor $1/13$ is no longer included in the reported rational product.	3. Recheck the two rational factors “ Wait — let me double-check: ” $\displaystyle\prod_{k=4}^{63}\frac{k-1}{k-2}$ $\displaystyle=31,$ $\displaystyle\prod_{k=4}^{63}\frac{k+1}{k+2}$ $\displaystyle={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}.$ “ The product of ratios is ” $31\cdot\frac{1}{13}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.$
4. Evaluate the logarithmic product “ This is a telescoping product of ratios. ” $\prod_{k=4}^{63}\frac{\log(k+1)}{\log k}=\frac{\log 64}{\log 4}=3.$	4. Evaluate the logarithmic product “ So the product telescopes: ” $\prod_{k=4}^{63}\frac{\log_{5}(k+1)}{\log_{5}k}=\frac{\log_{5}64}{\log_{5}4}.$ “ Wait — this simplifies exactly! ” $\frac{\log_{5}64}{\log_{5}4}=\frac{6\log_{5}2}{2\log_{5}2}=3.$

Appendix

Table 12

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 507 papers

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
Sep 2, 2026Zhiwei Zhang, Zechen Sun, Fei Zhao +6On-Policy Distillation RL Post-Training
AllSpark Team
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
Oct 13, 2026Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1Language Model Distillation On-Policy Distillation
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
Oct 8, 2026Siyan Zhao, Yonggan Fu, Jindong Jiang +8Language Model Distillation On-Policy Distillation
NVIDIA

Sep 2, 2026J/K move · Enter open · S save

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng+5

AllSpark Team

On-Policy Distillation RL Post-Training

Open paper AskPDF arXiv

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.

Abstract

On-Policy Distillation

Explore similar work

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Correcting or Continuing? What On-Policy Distillation Learns from Student Errors

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better