CD-OPD: Contextual Digest for On-Policy Distillation
Authors: Mingyuan Li, Yucheng Li, Xiaxia Wang, Gong Cheng
Organizations: State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford
Teacher takeover in on-policy distillation (OPD) can produce incorrect responses even when the student and teacher independently answer the same problem correctly, a failure we quantify using Collaboration Failure Rate (CFR). To address it, we propose Contextual Digest for On-Policy Distillation (CD-OPD). CD-OPD structures the earlier context of teacher guidance through a teacher-generated dependency graph, curating the required problem conditions and preceding reasoning as node summaries alongside the unchanged guidance in a Contextual Digest before student continuation. We further develop online meta-reweighting, which compares Digest segment gradients with a task gradient from independent student responses to improve student-only performance, favoring updates predicted to reduce task loss. Across eight benchmarks, CD-OPD improves average accuracy by 1.32--7.05 percentage points over representative existing OPD methods. It also reduces CFR by 27.4%--56.4% relative to existing takeover-based OPD methods.
Figures & tables
Figure 1: Accuracy and CFR under teacher takeover (1,024 mathematical problems randomly sampled from our local copy of DAPO-Math-17k-dedup; student: Qwen3-1.7B; teacher: Qwen3-4B-Instruct-2507; accuracy reported as mean@4).
Figure 2: Comparison of independent student and teacher responses, takeover-based OPD, and CD-OPD.
Figure 3: Overview of CD-OPD: (a) Trajectory Construction with Contextual Digest (Section 3.3 ); (b) Policy Update with Online Meta-Reweighting (Section 3.4 ).
Method
AIME2024
AIME2025
AIME2026
MATH500
AMC2023
OlympiadBench
HMMT-Feb26
HMMT-Nov25
Avg.
Teacher
60.42
46.04
52.19
94.20
93.83
70.62
31.25
41.88
61.30
Student: Qwen3-0.6B-Non-Thinking
Student
1.77
2.40
0.73
44.10
24.45
16.36
0.76
3.85
11.80
SFT @110
4.90
7.60
4.06
59.45
34.92
26.74
1.89
3.02
17.82
KD @110
4.17
7.19
4.79
57.75
35.23
27.15
2.94
2.71
17.74
GRPO @110
8.23
15.21
10.00
68.60
46.56
34.42
7.86
6.04
24.62
Table 1: Performance across eight mathematical reasoning benchmarks. The best and second-best results within each student size are shown in bold and underlined, respectively. Blue-shaded rows denote our method, and light-blue-shaded rows denote improvements in percentage points relative to the indicated baselines.
Method
AIME2024
AIME2025
AIME2026
MATH500
AMC2023
OlympiadBench
HMMT-Feb26
HMMT-Nov25
Avg.
Δ
Student: Qwen3-0.6B-Non-Thinking
CD-OPD
16.77
20.94
15.63
77.50
56.33
49.07
12.97
10.10
32.41
—
w/o Dependency Graph
14.58
18.33
14.58
76.35
55.47
48.74
11.65
7.60
30.91
−1.50
w/o Online Meta-Reweighting
16.35
17.50
15.21
76.65
56.02
48.70
11.46
8.65
31.32
−1.09
Student: Qwen3-1.7B-Non-Thinking
CD-OPD
43.65
31.56
33.23
89.80
79.53
63.46
24.81
20.21
48.28
—
Table 2: Ablation results across eight benchmarks. Δ denotes the change in average accuracy relative to CD-OPD, in percentage points.
Figure 4: CFR for Qwen3-0.6B and Qwen3-1.7B, computed by pooling numerator and denominator counts across eight benchmarks.
Figure 5: Effects of B and R on CD-OPD’s average accuracy across eight benchmarks and additional training time relative to training without online meta-reweighting. Left : Qwen3-0.6B; Right : Qwen3-1.7B.
Figure 6: Contextual Digest length distributions during training for Qwen3-0.6B and Qwen3-1.7B.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
N
K
License
AIME2024
30
32
Apache-2.0
AIME2025
30
32
CC BY-NC-SA 4.0
AIME2026
30
32
CC BY-NC-SA 4.0
MATH500
500
4
MIT
AMC2023
40
32
Apache-2.0
OlympiadBench
674
4
Apache-2.0
Appendix
Table 3: Evaluation dataset statistics and declared licenses.
(B,R)
AIME2024
AIME2025
AIME2026
MATH500
AMC2023
OlympiadBench
HMMT-Feb26
HMMT-Nov25
Avg.
Student: Qwen3-0.6B-Non-Thinking
(2,2)
16.42
18.07
15.28
76.80
56.07
48.76
11.71
8.89
31.50
(4,2)
16.55
19.17
15.41
77.06
56.17
48.88
12.19
9.35
31.85
(8,2)
16.77
20.94
15.63
77.50
56.33
49.07
12.97
10.10
32.41
(2,4)
16.50
18.70
15.36
76.95
56.13
48.83
11.99
9.16
31.70
(4,4)
16.64
19.89
15.50
77.24
56.24
48.96
12.51
9.66
32.08
Appendix
Table 7: Per-benchmark accuracy (%) of CD-OPD for different (B,R) configurations. Bold rows indicate the default setting (B,R)=(8,2) .
Method
Avg. Accuracy (%) ↑
CFR (%) ↓
Training Time (min/step) ↓
Response Length (tokens) ↓
Inference Time (min/response) ↓
Student: Qwen3-0.6B-Non-Thinking
Relay-OPD
31.04
13.20
4.67
14,473.80
0.22
CD-OPD
32.41
9.58
5.76
14,699.02
0.23
Student: Qwen3-1.7B-Non-Thinking
Relay-OPD
46.96
10.00
4.93
11,738.32
0.28
CD-OPD
48.28
6.49
6.04
11,921.79
0.29
Appendix
Table 8: Training and inference efficiency of Relay-OPD and CD-OPD across Qwen3-0.6B and Qwen3-1.7B.
Problem (AIME2025). Let m,n be relatively prime positive integers such that ∏k=463logk+1(5k2−4)logk(5k2−1)=nm. Find m+n . Reference answer: 106. Both responses are generated by the respective students independently, without teacher takeover or a Contextual Digest at inference.
Relay-OPD
CD-OPD
1. Rewrite and separate the product “ We can cancel log5 from numerator and denominator: ” (k2−4)logk(k2−1)log(k+1). “ Let’s now split the product into two parts: ” rational productk=4∏63k2−4k2−1⋅log productk=4∏63logklog(k+1).
1. Rewrite and separate the product “ Now, we can write everything in terms of base 5 . ” (k2−4)log5k(k2−1)log5(k+1). “ Let’s separate the product into two parts: factorial-like terms and logarithmic terms. ” k=4∏63(k−2k−1⋅k+2k+1)⋅k=4∏63log5klog5(k+1).
2. Evaluate the two rational factors “ This is a telescoping product: ” ∏k=463k−2k−1=23⋅34⋯6162=31. “ Similarly, look at the second part: ” \prod_{k=4}^{63}\frac{k+1}{k+2}=\frac{5}{6}\cdot\frac{6}{7}\cdots\frac{64}{65}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}. “ So now, the total product of the terms is: ” (31)\cdot\left(\frac{1}{13}\right)={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.
2. Evaluate the two rational factors “ Wait — no. Actually: ” (after revisiting the factorial simplification) ∏k=463k−2k−1=262=31. “ Again, telescoping! ” \prod_{k=4}^{63}\frac{k+1}{k+2}=\frac{5}{6}\cdot\frac{6}{7}\cdots\frac{64}{65}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}. “ The product of the ratio parts is ” 31\cdot\frac{1}{13}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.
3. Reassemble the original expression “ Wait — but do we have only the first two products? ” “ We have now: ” 1. First product: \prod_{k=4}^{63}\frac{k^{2}-1}{k^{2}-4}={\color[rgb]{0.6445,0.168,0.1484}\boldsymbol{31}}. Reading note. The earlier factor 1/13 is no longer included in the reported rational product.
3. Recheck the two rational factors “ Wait — let me double-check: ” k=4∏63k−2k−1=31,k=4∏63k+2k+1\displaystyle={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{1}{13}}}. “ The product of ratios is ” 31\cdot\frac{1}{13}={\color[rgb]{0.1406,0.3594,0.5742}\boldsymbol{\frac{31}{13}}}.
4. Evaluate the logarithmic product “ This is a telescoping product of ratios. ” ∏k=463logklog(k+1)=log4log64=3.
4. Evaluate the logarithmic product “ So the product telescopes: ” ∏k=463log5klog5(k+1)=log54log564. “ Wait — this simplifies exactly! ” log54log564=2log526log52=3.
On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
Shuo Yang, Zheyu Zhang, Mahito Sugiyama +1
Technical University of Munich · National Institute of Informatics · Munich Center for Machine Learning
On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.