Black-box On-Policy Distillation (OPD) seeks to improve a student from its own generations when the teacher provides sampled responses but not token probabilities. Adversarial distillation offers one route: it learns a discriminator over prompt-matched teacher and student responses and uses its score as the policy reward. However, sampling discriminator negatives from the latest student at each step couples the learned reward to a negative distribution that changes after every policy update. We address this moving-target problem with persistent-negative adversarial distillation, a live-pool method that replaces a fraction of each discriminator batch with historical, prompt-matched teacher--student comparisons. Under matched discriminator compute, historical comparisons train the discriminator, while GRPO remains on-policy with fresh student responses. Our analysis identifies the Bayes-optimal reward as a teacher-to-negative log-density ratio and, under explicit assumptions, shows how persistent negatives anchor the discriminator and reduce reward-estimation MSE relative to fresh-negative training. Across two student families, three judges, and four judged-chat benchmarks, persistent-negative adversarial distillation consistently improves performance over current methods at matched discriminator compute. It also yields smoother fresh-policy discriminator trajectories, with fewer below-chance dips. These findings identify the discriminator's negative distribution as an important design axis in black-box on-policy distillation.
Figures & tables
Figure 1: Workflow of persistent-negative on-policy distillation. At iteration t , each prompt x is paired with a teacher response yT and a group of fresh student responses yt,1:G . Fresh comparisons Ft are combined with historical comparisons from Pt to form the fixed-size discriminator batch BtD , from which the updated reward rφt+1 is learned. The discriminator then scores the fresh student group, and the resulting within-group advantages drive the GRPO update θt→θt+1 for student.
Figure 2: Why persistent negatives help. GRPO updates continually change the student distribution, so a discriminator trained from a single policy snapshot can lose constraints supplied by earlier student responses. Persistent-negative training uses qtρ=(1−ρ)πθt+ρht , allowing fresh and historical comparisons to share a fixed-size discriminator batch. The historical term remains active in the discriminator objective, making the learned reward less dependent on any single rollout and providing steadier feedback for GRPO. Reward trajectories are schematic.
Dataset
Base
SeqKD
GAD
Persistent-negative (ours)
Δ vs. Base ↑ (pp)
Δ vs. GAD ↑ (pp)
Qwen2.5-72B-Instruct judge
LMSYS
48.5
48.9
49.1
50.7
+2.2
+1.6
Dolly
47.5
46.8
47.9
49.1
+1.6
+1.2
Vicuna
48.4
48.6
49.2
50.2
+1.8
+1.0
Self-Instruct
48.1
48.5
48.7
49.4
+1.3
+0.7
Average
48.1
48.2
48.7
49.9
+1.7
+1.1
Table 1: Judged-chat performance with Qwen2.5-7B-Instruct. Scores are percentages, with 50 denoting parity with the reference. Base is the undistilled student, and GAD uses current-only negatives. Improvements are reported in percentage points (pp). The average weights the four evaluation sets equally.
Dataset
Base
SeqKD
GAD
Persistent-negative (ours)
Δ vs. Base ↑ (pp)
Δ vs. GAD ↑ (pp)
Qwen2.5-72B-Instruct judge
LMSYS
46.7
47.2
48.5
49.4
+2.7
+0.9
Dolly
45.8
45.9
46.7
48.2
+2.4
+1.5
Vicuna
45.2
47.7
48.8
49.8
+4.6
+1.0
Self-Instruct
46.3
47.2
48.2
48.9
+2.6
+0.7
Average
46.0
47.0
48.1
49.1
+3.1
+1.0
Table 2: Judged-chat performance with Llama-3.1-8B-Instruct.
Table 3: Training configuration. Pool rows describe the only intervention.
Figure 3: Performance gains across paired training seeds. Each point shows the equal-set-weighted improvement of persistent-negative training over GAD for one paired seed, measured in percentage points. Large markers denote the mean across three seeds, and error bars show one standard deviation. Positive values favor persistent-negative training.
Method
Mean
Std.
Minimum
Dips <0.5
GAD
0.894
0.087
0.414
6
Persistent-negative (ours)
0.903
0.068
0.461
1
Appendix
Table 4: Fresh-policy discriminator accuracy in the initial primary run. Standard deviation is computed across adversarial updates, and “dips” counts pre-update evaluations below chance.
Black-box distillation is a practical route for transferring capabilities from API-accessible large language models that expose only text outputs into smaller student models. Recent on-policy adversarial methods such as GAD improve over SeqKD by forming an adversarial loop between a critic and a student, where the critic provides rewards for GRPO-based student policy optimization over the student's sampled responses. However, GRPO computes advantages from the within-group relative rewards of student samples for the same prompt, whereas the critic is trained primarily to distinguish teacher responses from student responses. This objective mismatch can produce reward groups with collapsed scale or fragile margins, leading to brittle grouped optimization signals. We propose Groupwise Reward Geometry Conditioning (GRGC), a two-stage framework that improves advantage construction by shaping student-side reward groups during both critic training and policy optimization. To improve critic-side conditioning, Gaussian groupwise Optimal Transport calibration regularizes the critic during training to produce reward groups with non-collapsed spread and smooth rank-wise gaps by matching sorted prompt-wise rewards to group-centered Gaussian quantiles. Building on this conditioned reward geometry, policy-side group power modulation reshapes the prompt-wise reward groups before they are converted into advantages, preserving the critic-induced ordering while increasing optimization-relevant margin separability. Extensive experiments across diverse teachers, student model families and scales, and training datasets demonstrate the effectiveness of GRGC on both in-distribution and out-of-distribution evaluations, while introducing negligible overhead over GAD. The code is available at https://github.com/2018cx/GRGC.
Xiao Cui, Mo Zhu, Yulei Qin +3
University of Science and Technology of China · Zhejiang University · Independent Researcher
On-policy distillation (OPD) has been widely studied as a post-training method in which a student model obtains token-level supervision from a stronger teacher on its own rollouts. Recent studies have improved OPD through alternative distillation reward formulations and teacher configurations, while the objective of distillation remains centered on mimicking the teacher. However, when a stronger teacher has limited distributional overlap with the student, such positive guidance can provide insufficient learning signals. In this work, we introduce Negative-Policy OPD (NP-OPD), which complements teacher supervision with rollouts from a lower-performing, lower-capability negative policy that serves as a negative reference for the student. Rather than modifying the distillation reward formulation, NP-OPD introduces the negative policy at the rollout stage, continuously supplying tokens preferred by the negative policy over the teacher so that they remain exposed to teacher supervision throughout training. This provides an explicit negative signal through negative-policy rollouts while preserving the positive teacher supervision used in OPD. Through extensive experiments, we show that NP-OPD improves OPD across model scales, generation modes, reasoning domains, and different OPD variants. Furthermore, our analyses show that NP-OPD effectively suppresses tokens preferred by the negative policy over the teacher and moves the student away from the negative policy. These results support our design of introducing negative signals through negative-policy rollouts and provide new insight into the role of the rollout policy in OPD. Code will be available at https://github.com/naver-ai/np-opd.
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA