Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

Suwan Yoon; Seongryong Jung; Changhee Lee

cs.LGOct 19, 2026

Correctness-Dependent Asymmetric Target Distributions for Self-Distillation Policy Optimization

Authors: Suwan Yoon, Seongryong Jung, Changhee Lee

Organizations: Korea University, Seoul, Korea · Chung-Ang University, Seoul, Korea

Abstract

Self-distillation policy optimization provides dense token-level supervision for LLM post-training through feedback-conditioned self-teachers. However, directly matching the self-teacher can disrupt successful behaviors on correct rollouts while providing insufficient correction on incorrect ones. We propose Asymmetric Policy Optimization via Self-Distillation (APSD), which constructs correctness-dependent targets by mixing student and self-teacher distributions: interpolation preserves successful behaviors on correct rollouts, while extrapolation beyond the self-teacher strengthens correction on incorrect ones. We analyze APSD through divergence decomposition, reverse-KL barycentric geometry, and gradient analysis, highlighting the importance of gradient flow through student-dependent targets. Experiments under reinforcement learning with verifiable rewards and rich feedback show that APSD achieves strong overall performance against reinforcement-learning and self-distillation baselines and matches SDPO performance with 2.6x fewer training steps on average across Qwen3 model scales.

Figures & tables

Figure 1: Correctness-dependent target construction in APSD. Directly matching the self-teacher can disrupt successful behaviors in correct rollouts while providing insufficient correction for incorrect ones. APSD addresses this asymmetry by adapting the distillation target according to rollout correctness.

Figure 2: Teacher–student discrepancy in SDPO on LiveCodeBench v6. The left panels show token-probability discrepancies for correct and incorrect rollouts, while the right panels provide representative examples of perturbing a correct choice and retaining an incorrect choice.

Figure 3: Overview of APSD. Step 1: Sample on-policy rollout groups and construct the self-teacher. Step 2: Interpolate student–self-teacher targets for correct rollouts ( $0<\alpha<1$ ) and extrapolate beyond the self-teacher for incorrect rollouts ( $\alpha>1$ ). Step 3: Token-level RKL self-distillation with gradients through the targets. Analysis: The right panel shows target geometry and gradient directions.

Panel of Figure 4 — Figure 4: Training dynamics and target gradient-flow ablation on LCBv6 across Qwen3 model scales under RLRF. (a) Accuracy of APSD, SDPO, and GRPO over training steps. Shaded regions indicate the standard deviation across three random seeds. (b) Accuracy of SDPO, APSD with a detached target (APSD Stop-Grad), and full APSD.

Sampling Distribution	Correct rollout		Incorrect rollout
Sampling Distribution	Acc.	$\boldsymbol{\Delta}$	Acc.	$\boldsymbol{\Delta}$
Student	78.57	–	1.79	–
Inter. ( $\pi^{\text{APSD}}_{\theta,0.5}$ )	83.93	+5.36	5.36	+3.57
Teacher	78.57	+0.00	19.64	+17.85
Extra. ( $\pi^{\text{APSD}}_{\theta,1.5}$ )	75.00	$-$ 3.57	23.21	+21.42

Table 3: Execution accuracy (%) under different sampling distributions.

\Delta

denotes the percentage-point change relative to the student distribution.

Method	Qwen3-1.7B	Qwen3-4B	Qwen3-8B
GRPO	30.34	37.98	40.27
SDPO	32.06	42.56	45.04
APSD-C	33.59	46.18	48.85
APSD-I	32.82	45.61	48.47
APSD	34.16	47.71	49.62

Table 4: Ablation of the correctness-specific components of APSD on LCBv6 (Avg@4).

Figure 5: Sensitivity of APSD to the asymmetry strength $\delta$ under RLVR and RLRF.

Panel of Figure 6 — Figure 6: Additional analysis of teacher–student discrepancy on LiveCodeBench v6. (a) Qwen3-8B student versus a separate Qwen3-32B OPD teacher. The teacher–student perplexity gap is larger for incorrect rollouts than for correct rollouts. (b) Correctness-dependent APSD targets constructed from the SDPO self-teacher. Interpolation reduces the discrepancy for correct rollouts, whereas extrapolation increases it for incorrect rollouts. Each panel contains 10,000 sampled tokens. $\Delta$ PPL denotes the mean perplexity difference over the corresponding rollouts.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Token-level effect of APSD on representative correct and incorrect rollouts. For the correct rollout, interpolation shifts the target toward the student’s successful choice, while for the incorrect rollout, extrapolation increases the probability of the corrective alternative.

Category	Parameter	Value
Common (shared by all methods)
Data	Max. prompt length	2048
Data	Max. response length	8192
Batching	Question batch size	32
Batching	Number of rollouts	8
Rollout	Inference engine	vLLM

Appendix

Table 5: Training and evaluation hyperparameters for the five-hour RLVR experiments with Olmo3-7B-Instruct and Qwen3-8B. Shared configurations are listed first, followed by method-specific settings.

Category	Parameter	Value
Common (shared by all methods)
Data	Max. prompt length	2048
Data	Max. response length	8192
Batching	Question batch size	32
Batching	Training rollouts per question	8
Rollout	Inference engine	vLLM

Appendix

Table 6: Training and evaluation hyperparameters for the RLRF experiments on LiveCodeBench with Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Shared configurations are listed first, followed by method-specific policy-optimization, distillation, and distribution-mixing settings.

Language Model Post-Training

Language model post-training adapts pretrained models to follow instructions, reason, use tools, and meet task-specific or human-defined goals.

See all 197 papers

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation
Oct 19, 2026Harrison Zhao, Lei Huang, Fei Liu +2On-Policy Self-Distillation Language Model Distillation
Meta
SR-OPSD: Self-Referenced On-Policy Self-Distillation
Aug 10, 2026Zhuo Sun, Entong Li, Yanlong Zhao +9On-Policy Self-Distillation Language Model Distillation
Shanghai University of Finance and Economics · Imperial College London · Independent Researcher +6
DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation
Aug 26, 2026Yutong Chen, Guangfu Guo, Zhichao Xu +1Adaptive Knowledge Distillation On-Policy Self-Distillation
Department of Computer Science, Clemson University, 105 Sikes Hall, Clemson, SC 29634, USA · Department of Computer Science, University of Utah

Oct 19, 2026J/K move · Enter open · S save

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Harrison Zhao, Lei Huang, Fei Liu, Wen Shi+1

Meta

On-Policy Self-Distillation Language Model Distillation Selective Knowledge Distillation

Open paper AskPDF arXiv

On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.

Abstract

Language Model Post-Training

Explore similar work

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

SR-OPSD: Self-Referenced On-Policy Self-Distillation

DualOPSD: Closing the Feedback Loop in On-Policy Self-Distillation