Black-box knowledge distillation for large language models presents a strict trade-off. Simple off-policy methods (e.g., sequence-level knowledge distillation) struggle to correct the student's inherent errors. Fully on-policy methods (e.g., Generative Adversarial Distillation) solve this via adversarial training but introduce well-known training instability and crippling computational overhead. To address this dilemma, we propose SODA (Semi On-policy Distillation with Alignment), a highly efficient alternative motivated by the inherent capability gap between frontier teachers and much smaller base models. Because a compact student model's natural, zero-shot responses are almost strictly inferior to the powerful teacher's targets, we can construct a highly effective contrastive signal simply by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This demonstrates that exposing the small student to its own static inferior behaviors is sufficient for high-quality distribution alignment, eliminating the need for costly dynamic rollouts and fragile adversarial balancing. Extensive evaluations across four compact Qwen2.5 and Llama-3 models validate this semi on-policy paradigm. SODA matches or outperforms the state-of-the-art methods on 15 out of 16 benchmark results. More importantly, it achieves this superior distillation quality while training 10 times faster, consuming 27% less peak GPU memory, and completely eliminating adversarial instability.
Figures & tables
Figure 1: SODA achieves competitive or better distillation quality than GAD: 10 × faster and 27% more memory-efficient, while being substantially easier and more stable to train. From left to right: GPT-4o Score averaged over four student models (higher is better; 50 denotes GPT-4o parity); training stability; wall-clock training time; and peak GPU memory.
Figure 2: Dual learning signal in SODA. (a) A brief warmup shifts the student toward the teacher via imitation, but residual modes from q0 persist. (b) Preference-based distillation additionally suppresses these student-specific inferior behaviors via mode pruning, yielding qSODA≈p .
Model
Method
In-Dist.
Out-of-Distribution
LMSYS
Dolly
SelfInst
Vicuna
GPT-5-Chat
Teacher
51.7
49.8
49.7
49.9
Qwen2.5-3B-Instruct
Base
45.8
45.1
45.6
47.3
SeqKD
47.5
44.8
45.7
48.0
GAD
48.9
46.7
47.7
49.4
SODA
49.2
46.1
48.2
49.8
Table 1: Automatic evaluation results (GPT-4o Score). Each student response is scored against a GPT-4o reference; the reported metric is S/(S+R)×100 averaged over all test prompts, where 50 indicates parity with GPT-4o. Base, SeqKD, and GAD numbers are from Ye et al. (2025) . Best result per model is in bold .
Table 4
Figure 3: Representation analysis on Llama-3.1-8B-Instruct (200 held-out LMSYS prompts). (a) Layer-wise CKA similarity to the base model: SODA diverges most, indicating deeper representational restructuring. (b, c) Last-layer activation entropy and kurtosis: SODA achieves the highest entropy and lowest kurtosis, correlating with its strongest distillation performance.
Model
SODA (Static q0 )
Resample @ Step 100
Resample @ Step 200
Qwen2.5-3B
49.2
48.5
47.8
Llama-3.2-3B
49.1
48.1
47.4
Table 4: Ablation on dynamic resampling (GPT-4o Score, LMSYS). Updating the negative samples mid-training strictly degrades performance compared to retaining the initial static snapshot.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Prompt wrapper for training and evaluation.
Figure 5: GPT-4o evaluation prompt, following Ye et al. (2025) and Gu et al. (2024b) .
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
Jianyu Wu, Yizhou Wang, Encheng Su +2
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +1
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On τ2-bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.
Autoregressive large language models (LLMs) have rapidly advanced in capability, but their increasing scale comes with substantial computational and memory costs at inference time. Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large teacher model to a smaller student model via alignment of discrete probability distributions. However, existing KD methods for LLMs primarily rely on divergences that evaluate discrepancies through probability values at each vocabulary index, without explicitly leveraging token-level semantic information. We propose Wasserstein-based knowledge distillation (WASD) for LLMs, which incorporates token-level semantic information via the Wasserstein-based distance with a cost matrix derived from token embeddings. To ensure computational tractability, we adopt the Sinkhorn divergence and derive a gradient-equivalent objective that can be efficiently optimized without introducing additional networks. Experiments across multiple LLM families and scales show that WASD consistently improves distillation performance on diverse tasks, including instruction following, mathematical reasoning, and code generation. Our results highlight the importance of semantic information encoded in the token space for effective distribution alignment in LLM distillation. The implementation is publicly available at https://github.com/aailab-kaist/WASD .