Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.
Figures & tables
Figure 1: Correctness-dependent target construction in APSD. Directly matching the self-teacher can disrupt successful behaviors in correct rollouts while providing insufficient correction for incorrect ones. APSD addresses this asymmetry by adapting the distillation target according to rollout correctness.
Figure 2: Teacher–student discrepancy in SDPO on LiveCodeBench v6. The left panels show token-probability discrepancies for correct and incorrect rollouts, while the right panels provide representative examples of perturbing a correct choice and retaining an incorrect choice.
Figure 3: Overview of APSD. Step 1: Sample on-policy rollout groups and construct the self-teacher. Step 2: Interpolate student–self-teacher targets for correct rollouts ( 0<α<1 ) and extrapolate beyond the self-teacher for incorrect rollouts ( α>1 ). Step 3: Token-level RKL self-distillation with gradients through the targets. Analysis: The right panel shows target geometry and gradient directions.
Figure 4: Training dynamics and target gradient-flow ablation on LCBv6 across Qwen3 model scales under RLRF. (a) Accuracy of APSD, SDPO, and GRPO over training steps. Shaded regions indicate the standard deviation across three random seeds. (b) Accuracy of SDPO, APSD with a detached target (APSD Stop-Grad), and full APSD.
Sampling Distribution
Correct rollout
Incorrect rollout
Acc.
Δ
Acc.
Δ
Student
78.57
–
1.79
–
Inter. ( πθ,0.5APSD )
83.93
+5.36
5.36
+3.57
Teacher
78.57
+0.00
19.64
+17.85
Extra. ( πθ,1.5APSD )
75.00
− 3.57
23.21
+21.42
Table 3: Execution accuracy (%) under different sampling distributions. Δ denotes the percentage-point change relative to the student distribution.
Method
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
GRPO
30.34
37.98
40.27
SDPO
32.06
42.56
45.04
APSD-C
33.59
46.18
48.85
APSD-I
32.82
45.61
48.47
APSD
34.16
47.71
49.62
Table 4: Ablation of the correctness-specific components of APSD on LCBv6 (Avg@4).
Figure 5: Sensitivity of APSD to the asymmetry strength δ under RLVR and RLRF.
Figure 6: Additional analysis of teacher–student discrepancy on LiveCodeBench v6. (a) Qwen3-8B student versus a separate Qwen3-32B OPD teacher. The teacher–student perplexity gap is larger for incorrect rollouts than for correct rollouts. (b) Correctness-dependent APSD targets constructed from the SDPO self-teacher. Interpolation reduces the discrepancy for correct rollouts, whereas extrapolation increases it for incorrect rollouts. Each panel contains 10,000 sampled tokens. Δ PPL denotes the mean perplexity difference over the corresponding rollouts.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Token-level effect of APSD on representative correct and incorrect rollouts. For the correct rollout, interpolation shifts the target toward the student’s successful choice, while for the incorrect rollout, extrapolation increases the probability of the corrective alternative.
Category
Parameter
Value
Common (shared by all methods)
Data
Max. prompt length
2048
Max. response length
8192
Batching
Question batch size
32
Number of rollouts
8
Rollout
Inference engine
vLLM
Appendix
Table 5: Training and evaluation hyperparameters for the five-hour RLVR experiments with Olmo3-7B-Instruct and Qwen3-8B. Shared configurations are listed first, followed by method-specific settings.
Category
Parameter
Value
Common (shared by all methods)
Data
Max. prompt length
2048
Max. response length
8192
Batching
Question batch size
32
Training rollouts per question
8
Rollout
Inference engine
vLLM
Appendix
Table 6: Training and evaluation hyperparameters for the RLRF experiments on LiveCodeBench with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Shared configurations are listed first, followed by method-specific policy-optimization, distillation, and distribution-mixing settings.
On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.
On-policy self-distillation (OPSD) converts feedback into dense token-level supervision on student-generated trajectories, complementing reinforcement learning with sparse outcome rewards. Its self-teacher, derived from the student's current or exponentially averaged parameters and conditioned on additional context, evolves alongside the student and its rollout context distribution. The benefit of modifying this moving target depends on how target--student probability mismatches translate into updates. We propose Self-Referenced On-Policy Self-Distillation (SR-OPSD), which constructs a normalized geometric target from the self-teacher and a frozen initial policy, then minimizes the forward Rényi divergence from this target to the student. The interpolation coefficient controls the self-teacher's contribution, while the Rényi order controls the power weighting of target-to-student probability ratios in the gradient. For fixed contexts and target components, we establish a conditional variational characterization and derive the exact token-logit gradient, revealing how anchoring and projection jointly shape the effective update target. Experiments across scientific reasoning, tool use, mathematical reasoning, and code generation demonstrate strong performance across multiple model families and scales. Ablations further show that reference anchoring can improve or degrade performance depending on the projection objective, supporting the joint design of target construction and projection geometry.
Zhuo Sun, Entong Li, Yanlong Zhao +9
Shanghai University of Finance and Economics · Imperial College London · Independent Researcher +6
On-policy self-distillation (OPSD) combines student generated trajectories with dense supervision from a privileged teacher of the same model. Its pointwise clipping protects the student from locally incompatible teacher directions, but the fixed teacher receives no feedback about the supervision the student rejects, which limits effective learning from the teacher. We introduce DualOPSD, which alternates updates in a coupled teacher-student system. The student retains OPSD's clipped forward KL objective. The privileged teacher adapts to the updated student after each student step. A residual target based on the frozen initial model preserves the effect of privileged context during teacher adaptation. Across Qwen3-8B/4B/1.7B and three math benchmarks, DualOPSD improves avg@12 accuracy over OPSD by 9.83 percentage points. Further diagnostic analyses show lower bidirectional KL, less persistent mismatch, and retained sensitivity to privileged reference. Together, these results demonstrate that DualOPSD can be a more effective form of privileged self-distillation. Code is available at: https://github.com/on1262/DualOPSD.
Yutong Chen, Guangfu Guo, Zhichao Xu +1
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah