OC-OPD: Observation-coordinated Supervision for On-Policy Distillation
Organizations: Yale NLP Lab
Abstract
Reverse and forward Kullback--Leibler (KL) divergences provide complementary supervision in on-policy distillation: reverse KL refines the student's active token rankings, and forward KL allocates mass to teacher-supported candidates. At student-generated prefixes, we form a four-observation matrix by treating the student and teacher as both candidate sources and evaluators. The diagonal self-observations measure uncertainty, while source-centered cross-observations recover the two KL objectives. We establish an exact identity equating the Fisher inverse-metric coupling of their logit gradients with the sum of these contrasts, yielding a common local descent direction for distinct positive distributions. Observation-coordinated on-policy distillation (OC-OPD) uses observations to set preferences, closed-form bargaining to coordinate gradients, and distributional discrepancy to scale the update. Across three heterogeneous student settings (non-thinking generation, thinking generation, and base-model initialization), OC-OPD achieves strong gains in mathematical reasoning and transfers effectively to out-of-distribution scientific question answering and code generation. Our code is publicly available at: https://github.com/RuoranXu/OC-OPD.
Figures & tables
| Methods | Math (I.D.) | O.O.D. | ||||||||
| AIME 2024 | AIME 2025 | AIME 2026 | MATH500 | OlympiadBench | GPQA D | MBPP | ||||
| Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@8 | Pass@8 | Avg@32 | ||
| Teacher: Qwen3-30B-A3B-Instruct-2507 | ||||||||||
| Qwen3-1.7B | 14.16 | 43.33 | 9.90 | 30.00 | 7.71 | 23.33 | 71.88 | 52.67 | 29.59 | 56.03 |
| + GRPO | 34.17 | 70.00 | 25.52 | 46.67 | 26.04 | 50.00 | 82.21 | 62.57 | — | — |
| + OPD | 31.98 | 73.33 | 20.73 | 50.00 | 23.54 | 46.67 | 85.84 | 60.77 | — | — |
| Methods | AIME 2024 | AIME 2025 | OlympiadBench | MATH500 | |||
|---|---|---|---|---|---|---|---|
| Avg@8 | Pass@8 | Avg@8 | Pass@8 | Pass@8 | Avg@8 | Pass@8 | |
| Qwen3-4B-Base | 12.19 | 23.33 | 7.29 | 23.33 | 52.52 | 48.60 | 85.20 |
| + KD | 12.50 | 26.67 | 12.08 | 23.33 | 62.43 | 74.73 | 92.00 |
| + OPD | 18.91 | 33.33 | 12.50 | 30.00 | 58.80 | 78.40 | 90.40 |
| + EOPD | 17.92 | 33.33 | 17.50 | 33.33 | 62.61 | 80.05 | 92.60 |
| + OC-OPD | 21.67 | 40.00 | 18.75 | 40.00 | 63.20 | 81.75 | 93.20 |
| Methods | Math (I.D.) | O.O.D. | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| AIME24 | AIME25 | AIME26 | OlympiadBench | GPQA D | MBPP | |||||
| avg@32 | pass@32 | avg@32 | pass@32 | avg@32 | pass@32 | avg@8 | pass@8 | pass@32 | ||
| 1.7B | 50.94 | 83.33 | 37.19 | 73.33 | 38.65 | 70.00 | 59.33 | 79.38 | 77.78 | 84.66 |
| + OPD | 48.54 | 86.67 | 40.21 | 73.33 | 42.08 | 73.33 | 58.91 | 79.08 | 79.29 | 83.60 |
| [][] + OC-OPD | 52.81 | 86.67 | 46.46 | 76.66 | 48.85 | 76.67 | 59.70 | 80.12 | 81.82 | 85.45 |
| 4B | 72.71 | 86.67 | 65.52 | 83.33 | 61.04 | 83.33 | 69.08 | 84.91 | 84.34 | 87.04 |
| Method | AIME 2024 | AIME 2025 | AIME 2026 | |||
|---|---|---|---|---|---|---|
| Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@32 | Pass@32 | |
| OC-OPD | 60.94 | 93.33 | 56.67 | 83.33 | 58.54 | 86.67 |
| Forward KL (FKL) | 55.42 | 83.33 | 53.75 | 73.33 | 54.17 | 83.33 |
| Reverse KL (RKL) | 61.04 | 86.67 | 51.04 | 76.67 | 52.19 | 80.00 |
| Adaptive linear mixture | 60.21 | 86.67 | 53.02 | 80.00 | 53.96 | 80.00 |
| Equal-preference | 58.65 | 86.67 | 53.54 | 80.00 | 56.67 | 83.33 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| A prefix visited by the current student behavior policy | |
| Frozen teacher and current student distributions with positive support | |
| Student logits, with | |
| Self-/cross-entropy observation matrix in Eq. ( 3 ) | |
| Student-sourced and teacher-sourced observation contrasts | |
| Their Euclidean gradients with respect to current logits |
| Method | ||||
|---|---|---|---|---|
| REOPOLD | – | – | – | |
| TIP | – | – | – | |
| EOPD | – | – | – | |
| SEAD | – | – | ||
| OC - OPD (Ours) |
| Probe | Updates | Tokens | Sign R | Sign F | Both | ||
|---|---|---|---|---|---|---|---|
| On-policy rollout | 19 | 464 | 0.9977 | 99.57 | 0.9989 | 99.57 | 99.57 |
| Fixed in-domain | 25 | 300 | 0.9751 | 99.67 | 0.9992 | 98.00 | 99.00 |
| HMMT25 OOD | 25 | 400 | 0.9987 | 99.00 | 0.9993 | 100.00 | 99.25 |
| Hyperparameter | OC-OPD |
|---|---|
| General | |
| Max prompt length | 8192 |
| GPUs | NVIDIA A800 80GB |
| Training precision | BF16 |
| Optimization | |
| Global batch size | 8 |
| Method | AIME 2024 | AIME 2025 | AIME 2026 | |||
|---|---|---|---|---|---|---|
| Avg@32 | Pass@32 | Avg@32 | Pass@32 | Avg@32 | Pass@32 | |
| OC-OPD | 60.94 | 93.33 | 56.67 | 83.33 | 58.54 | 86.67 |
| Forward KL (FKL) | 55.42 | 83.33 | 53.75 | 73.33 | 54.17 | 83.33 |
| Reverse KL (RKL) | 61.04 | 86.67 | 51.04 | 76.67 | 52.19 | 80.00 |
| Adaptive linear mixture | 60.21 | 86.67 | 53.02 | 80.00 | 53.96 | 80.00 |
| Equal-preference | 58.65 | 86.67 | 53.54 | 80.00 | 56.67 | 83.33 |
Language Model Distillation
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
See all 556 papers