SAPD: Step-Aligned Privileged Distillation

Tianle Wang; Jiayu Liu; Ruizhi Zhao; Ning Miao

cs.CLOct 7, 2026

SAPD: Step-Aligned Privileged Distillation

Authors: Tianle Wang, Jiayu Liu, Ruizhi Zhao, Ning Miao

Organizations: Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Shenzhen University

Abstract

On-policy post-training can improve large language models by learning from their own trajectories, but requires costly rollout generation. We ask whether fixed demonstrations can support competitive off-policy learning through better supervision. Our premise is that their usefulness depends not only on the training trajectories, but also on whether supervision provides informative preferences among continuations and connects this guidance to the reasoning decision being learned. We introduce Step-Aligned Privileged Distillation (SAPD), a rollout-free self-distillation method that turns demonstrations into step-aligned distributional supervision. Its key insight is to use the known progression of a reference solution to associate each reasoning transition with targeted privileged guidance, rather than treating the solution as undifferentiated context. On mathematical reasoning benchmarks, SAPD outperforms supervised fine-tuning and label smoothing on average while remaining competitive with on-policy reinforcement learning and self-distillation. Analyses support both the value of context-dependent distributional guidance and the benefit of aligning privileged information with the current step. SAPD also largely preserves out-of-domain coding performance and achieves approximately 2x training-loop speedups over the on-policy baselines. These findings suggest that carefully constructed supervision can make fully off-policy post-training a competitive and computationally efficient alternative. Our code is available at https://github.com/Miaow-Lab/SAPD.

Figures & tables

	In-domain: Mathematics					Out-of-domain: Code
Method	AIME24	AIME25	HMMT25	Avg.	Avg. length	MBPP+	HumanEval+	LCBv6	Avg.
Base	62.22	46.94	29.17	46.11	10,595	64.15	81.86	24.24	56.75
SFT	58.33	46.67	28.61	44.54	12,835	63.23	80.34	18.32	53.96
LS-SFT	59.72	49.17	31.11	46.67	10,102	61.97	81.71	22.14	55.27
GRPO	62.22	47.50	30.56	46.76	10,145	64.95	82.62	24.05	57.21
OPSD	61.94	47.78	32.78	47.50	10,870	63.82	83.23	23.66	56.90

Table 1: Overall Avg@12 and average output length . Mathematics benchmarks measure in-domain performance; coding benchmarks measure out-of-domain (OOD) capability retention after mathematics post-training. Average length is measured in tokens on mathematics benchmarks. Among post-trained methods, bold and underline indicate the best and second-best results, respectively. Base is shown separately as the initial-model reference.

Figure 1: Informative supervision is not simply target softening. Comparison of SFT, label-smoothed SFT (LS), and SAPD at two reference-token positions: (a) equivalent expressions and (b) a critical inequality decision. Background gives brief problem context; Response shows a local response, with highlighted tokens marking prediction sites. Cell shading uses a shared linear probability scale from 0 to 1.

H

denotes full-vocabulary entropy.

Figure 2: Training dynamics reveal preservation of non-reference distributional structure. (a) Full teacher–student KL,

D_{\mathrm{KL}}(p_{T}\|p_{S})

. (b) Conditional non-reference KL, computed after excluding the demonstrated token and renormalizing both distributions. (c) Mean token-level absolute entropy gap,

|H(p_{S})-H(p_{T})|

. The inset magnifies the range 0.19–0.32. (d) Student probability mass on the teacher’s top-five tokens.

Method	AIME24	AIME25	HMMT25	Avg.
SAPD (ours)	65.28	48.06	30.28	47.87
Misaligned PI	60.00	48.89	28.06	45.65
Full PI	60.00	44.72	30.27	45.00

Table 2: Ablation of privileged supervision. We report Avg@12 on three math benchmarks and their average. Best results are in bold.

Panel of Figure 3 — Figure 3: Training efficiency. (a) AIME24 Avg@12 accuracy versus training-loop GPU-hours. (b) Runtime decomposed by computational component; numbers at the bar ends indicate total runtime.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Setting	SFT	LS-SFT	GRPO	OPSD	SAPD
Learning rate	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$
Effective batch size	32	32	32	32	32
LoRA rank	64	64	64	64	64
LoRA alpha	128	128	128	128	128
Training steps	100	100	500	100	100
Max sequence length	16K	16K	–	–	16K

Appendix

Table 3: Training settings for baselines and SAPD. LS-SFT extends the SFT configuration with label smoothing. A dash denotes a setting that does not apply. Sequence limits refer to teacher-forced inputs; rollout lengths refer to generated responses.

Explore similar work

Aug 6, 2026cs.LG

On-Policy Self-Distillation without Any Supervision

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at https://github.com/williamium3000/u-opsd.

Yijiang Li, Bingyang Wang, Yijun Liang +3

UC San Diego · Georgia Institute of Technology · University of Maryland, College Park +1

Unsupervised On-Policy Self-Distillation Self-Distillation Framework On-Policy Self-Consistency

Ask PDF

Aug 3, 2026cs.AI

DAPD: Dual-Anchored Policy Distillation

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

Jianyu Wu, Yizhou Wang, Encheng Su +2

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +1

Privileged Context Missing Bridge

Ask PDF

Jul 5, 2026cs.CL

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

Diffusion large language models (dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative to autoregressive models, but eliciting strong reasoning through post-training remains difficult: supervised fine-tuning is off-policy and suffers from exposure bias, while reinforcement learning gives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods. On-policy self-distillation (OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacher privileged information (PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process; on Dream and LLaDA, dOPSD improves both in-domain math reasoning and out-of-domain code generation, outperforming supervised and on-policy baselines.

Phuong Tuan Dat, Qi Li, Xinchao Wang

National University of Singapore

Unsupervised On-Policy Self-Distillation Diffusion Language Models LLM Reasoning Strategies

Ask PDF