cs.LGNov 4, 2026

EvoOPD: Evolving Privileged Information Strategies for Black-Box On-Policy Distillation

Authors: Ximo Zhu, Ruiqi Liu, Yi Han, Zhengan Wu, Gengsheng Li, ZiAn Wang, Zhu Yuantian, Jin Jiang, 

Organizations: University of the Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Southwest University · Southeast University · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Proprietary language models exhibit strong reasoning and coding capabilities, but their output distributions are inaccessible, making fine-grained distillation methods that rely on teacher logits difficult to apply directly to black-box models. We observe that rich textual feedback from a black-box teacher can be treated as privileged information (PI) and converted, through a PI-conditioned privileged teacher, into dense supervision signals on the student's own trajectories. However, what PI should be acquired from the teacher to most effectively support student learning remains unclear. PI strategies that differ in where they focus, what information they request, and how much detail they provide can have substantially different effects on the current student, and their relative effectiveness can further change as training progresses. Based on this observation, we propose Evolving Privileged Information Strategies for Black-Box On-Policy Distillation (EvoOPD), which treats PI acquisition itself as an optimization problem and maintains a PI Strategy Bank that evolves with the student state. EvoOPD uses the measured repair utility of PI on the current student to adapt the focus, information type, and granularity of teacher feedback, allowing the acquired information to continually match the student's learning needs. We systematically evaluate EvoOPD on mathematical reasoning and code generation with Qwen3-1.7B, 4B, and 8B students. EvoOPD achieves the best overall performance in all six model-task settings, improving over the strongest baseline in each setting by an average of 2.14 percentage points in mathematics and 1.30 points in code. Further analysis shows that the relative effectiveness of PI acquisition strategies changes with student state, making a fixed strategy difficult to sustain as the best choice throughout distillation.

Figures & tables

Figure 1: Motivation for adapting PI acquisition. (a) White-box distillation uses accessible teacher logits for token-level supervision; black-box distillation uses teacher text for SeqKD or reward-based training. (b) Four fixed PI acquisition strategies evaluated on AIME 2025 and 2026 (16 samples per question). Left: unassisted accuracy along separate training trajectories. Right: unassisted accuracy after training from each GRPO parent checkpoint on the same data for two epochs (20 additional data steps and 80 optimizer updates); gray crosses mark parent-model accuracy before branch training. Strategy rankings change with student state.
Figure 2: Overview of EvoOPD. The upper loop acquires response-specific PI under the current strategy bank and uses the student’s local response to PI to strengthen negative advantage at supported error tokens. The lower loop adapts PI acquisition by proposing a bounded bank revision and comparing repair utility at the current student on independent selection data.
MethodAIME24AIME25AIME26HMMT25 FebHMMT25 NovHMMT26 FebOverall
Qwen3-1.7B
Base12.7110.217.505.633.756.827.75
SeqKD12.7110.008.335.214.587.398.03
OPSD10.427.716.675.213.967.206.86
GRPO24.7918.9617.0811.888.7513.4515.78
ROPD35.6324.7922.0816.2516.2521.0222.64
Table 1: Mathematical reasoning after training on DAPO-Math-17K. Scores are Avg@16 for all three student sizes (%).
MethodHumanEval+MBPP+BigCodeBench HardLiveCodeBench v5LiveCodeBench v6Overall
Qwen3-1.7B
Base63.9149.827.8511.6415.3934.02
SeqKD64.1049.707.3911.8315.2533.95
OPSD62.5050.028.6611.0815.1833.86
GRPO66.8851.328.9915.3418.1136.27
ROPD64.3348.519.8413.6217.6834.61
Table 2: Code generation on the OpenThoughts-114k code subset. Pass@1 is estimated from 16 samples per question for every benchmark and model size (%).
Table 5
Figure 3: PI acquisition strategy optimization during distillation. (a) Candidate-minus-current recovery utility, ΔU\Delta U , over eight reviews. Positive differences are accepted (blue); otherwise the current bank is retained (gray). (b) LiveCodeBench v6 pass@1 (%) for three banks, all starting at 15.39% from the same initial student checkpoint.
Figure 4: Student-bank compatibility across training stages. We compare early-, middle-, and late-stage PI Strategy Banks across three student checkpoints. (a) Recovery utility UU for each bank-student pairing. (b) Short-horizon unassisted gain GG after training from each checkpoint with the corresponding bank. (c) Association between UU and GG across all nine pairings. The stage-matched bank achieves the highest UU and GG at each checkpoint.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

FieldMeaning and example content
idUnique reusable rule identifier, e.g., constraint formalization.
conditionApplicability trigger, e.g., a missing mathematical condition.
knowledge_requestInformation to supply, e.g., the missing relation and its interpretation.
target_policyWhich unresolved error to localize; locations remain internal metadata.
granularityRule-specific detail: a condition cue, causal explanation, minimal action, representation change, or self-check.
combination_guidanceHow to combine this guidance with complementary rules.
Appendix
Table 5: PI Strategy Bank rule schema. Field names are preserved in the revised proposal. Examples describe strategy content; global output restrictions are enforced separately.
More in this topic

Black-Box Knowledge Distillation

See all 12 papers

Explore similar work

CardsList
Aug 18, 2026

Rethinking Privileged Information in On-Policy Self-Distillation

On-policy self-distillation (OPSD) trains a student on its own responses using token-level supervision from the same model conditioned on privileged reference information. We investigate whether performance gains from OPSD show that the student learned the information in the reference or instead reflect recovery of reasoning behavior already present in the base model. We perform OPSD experiments on science and mathematics datasets using Qwen3 models ranging from 1.7B to 8B. Our analysis framework separates the supervision induced by the reference from the supervision provided by the teacher without the reference and measures how each aligns with changes in the student's predictions. The correct reference does not provide a consistent performance benefit across teacher generation modes, model sizes, and training datasets. Students can improve without the correct reference, and a solution from another problem can outperform the correct solution on several mathematical reasoning benchmarks. The student's predictions align more strongly with the base model's thinking behavior than with the supervision induced by the reference, but controls constructed from other problems reproduce much of both alignments. Moreover, stronger alignment attributable to the correct reference does not reliably coincide with a greater performance benefit from the reference. Performance gains and distributional alignment alone therefore cannot determine how privileged reference information contributes to student learning in OPSD.

Samyak Shrestha, Alexander Tessier

FirstPrinciples

On-Policy Self-DistillationLanguage Model DistillationLearning Using Privileged Information
AskPDF
Oct 15, 2026

Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

On-policy self-distillation (OPSD) enables large language models (LLMs) to learn from privileged information (PI) without a separate teacher model. However, prior work largely treats PI as a fixed design choice, leaving the effects of its content and construction poorly understood. This challenge is particularly important in multi-turn agentic settings, where trajectories contain actions, observations, and reasoning that can be transformed into PI in many ways. We systematically evaluate 16 PI configurations across three multi-turn agentic benchmarks and three student models that vary in scale and reasoning mode. We find that PI effectiveness depends on its content, generation source, and the student model. Information-dense trajectories and PI produced by more capable generators generally yield larger gains, whereas thinking models benefit less consistently. To assess PI before training, we compare PI utility with two KL-based metrics: sampled-token reverse KL (DD) and error-step KL contrast (Δderr−Δd^{-}_{\mathrm{err}}). The KL-based metrics track post-training performance more consistently than PI utility, which can be distorted by information leakage. Empirically, selecting each task's PI by Δderr−Δd^{-}_{\mathrm{err}} also improves post-training performance over random selection, most when the PI candidates differ widely. Finally, effective PI remains beneficial when combined with GRPO, allowing the combined objective to outperform a strong GRPO baseline. Overall, our results provide practical guidance for constructing and evaluating PI in OPSD training.

Kaijian Zou, Emiliano Penaloza, MohammadHossein Rezaei +5

University of Michigan – Ann Arbor · Mila · Scale AI

On-Policy Self-DistillationLearning Using Privileged InformationLLM Agent Training
AskPDF
Nov 5, 2026

Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

On-policy distillation (OPD) improves language-model reasoning through dense teacher feedback on student-generated responses. We first observe an amplitude disparity in this feedback: on math reasoning responses, style-related positions exhibit 3.5 times the mean absolute teacher--student log-probability difference of mathematical positions. Scaling OPD updates by these discrepancies thus directs much of the update strength toward matching the teacher's expression. We instead take the student's own pretrained base as a reference: the student--base difference records the token preferences acquired during post-training and provides supervision that does not depend on the student's distance from the teacher. Building on this reference, we propose Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round. Across 5 teacher--student configurations with Qwen3 students of 1.7B to 8B parameters, SD-OPD improves six-task mathematics average pass@1 (avg@16) over OPD by 7.76 to 10.37 percentage points. Code is available at https://github.com/aim-uofa/sd-opd.

Bozhen Fang, Linhao Zhong, Xuehao Wang +3

Zhejiang University, State Lab of CAD & CG · Ant Group

Language Model DistillationOn-Policy DistillationLanguage Model Post-Training
AskPDF