Persistent Negatives for Adversarial Black-Box On-Policy Distillation

Haixu Ma; Saad Lahrichi; Weiwei Li; Kevin Han; Weiqiang Wu; Peggy Yang; Dongzhuo Li; Ruiyi Li

cs.CLSep 25, 2026

Persistent Negatives for Adversarial Black-Box On-Policy Distillation

Authors: Haixu Ma, Saad Lahrichi, Weiwei Li, Kevin Han, Weiqiang Wu, Peggy Yang, Dongzhuo Li, Ruiyi Li,

Organizations: Meta AI · University of Missouri

Abstract

Black-box On-Policy Distillation (OPD) seeks to improve a student from its own generations when the teacher provides sampled responses but not token probabilities. Adversarial distillation offers one route: it learns a discriminator over prompt-matched teacher and student responses and uses its score as the policy reward. However, sampling discriminator negatives from the latest student at each step couples the learned reward to a negative distribution that changes after every policy update. We address this moving-target problem with persistent-negative adversarial distillation, a live-pool method that replaces a fraction of each discriminator batch with historical, prompt-matched teacher--student comparisons. Under matched discriminator compute, historical comparisons train the discriminator, while GRPO remains on-policy with fresh student responses. Our analysis identifies the Bayes-optimal reward as a teacher-to-negative log-density ratio and, under explicit assumptions, shows how persistent negatives anchor the discriminator and reduce reward-estimation MSE relative to fresh-negative training. Across two student families, three judges, and four judged-chat benchmarks, persistent-negative adversarial distillation consistently improves performance over current methods at matched discriminator compute. It also yields smoother fresh-policy discriminator trajectories, with fewer below-chance dips. These findings identify the discriminator's negative distribution as an important design axis in black-box on-policy distillation.

Figures & tables

Figure 1: Workflow of persistent-negative on-policy distillation. At iteration $t$ , each prompt $x$ is paired with a teacher response $y_{T}$ and a group of fresh student responses $y_{t,1:G}$ . Fresh comparisons $F_{t}$ are combined with historical comparisons from $P_{t}$ to form the fixed-size discriminator batch $B_{t}^{D}$ , from which the updated reward $r_{\varphi_{t+1}}$ is learned. The discriminator then scores the fresh student group, and the resulting within-group advantages drive the GRPO update $\theta_{t}\rightarrow\theta_{t+1}$ for student.

Figure 2: Why persistent negatives help. GRPO updates continually change the student distribution, so a discriminator trained from a single policy snapshot can lose constraints supplied by earlier student responses. Persistent-negative training uses $q_{t}^{\rho}=(1-\rho)\pi_{\theta_{t}}+\rho h_{t}$ , allowing fresh and historical comparisons to share a fixed-size discriminator batch. The historical term remains active in the discriminator objective, making the learned reward less dependent on any single rollout and providing steadier feedback for GRPO. Reward trajectories are schematic.

Dataset	Base	SeqKD	GAD	Persistent-negative (ours)	$\Delta$ vs. Base $\uparrow$ (pp)	$\Delta$ vs. GAD $\uparrow$ (pp)
Qwen2.5-72B-Instruct judge
LMSYS	48.5	48.9	49.1	50.7	$+2.2$	$+1.6$
Dolly	47.5	46.8	47.9	49.1	$+1.6$	$+1.2$
Vicuna	48.4	48.6	49.2	50.2	$+1.8$	$+1.0$
Self-Instruct	48.1	48.5	48.7	49.4	$+1.3$	$+0.7$
Average	48.1	48.2	48.7	49.9	$+1.7$	$+1.1$

Table 1: Judged-chat performance with Qwen2.5-7B-Instruct. Scores are percentages, with

50

denoting parity with the reference. Base is the undistilled student, and GAD uses current-only negatives. Improvements are reported in percentage points (pp). The average weights the four evaluation sets equally.

Dataset	Base	SeqKD	GAD	Persistent-negative (ours)	$\Delta$ vs. Base $\uparrow$ (pp)	$\Delta$ vs. GAD $\uparrow$ (pp)
Qwen2.5-72B-Instruct judge
LMSYS	46.7	47.2	48.5	49.4	$+2.7$	$+0.9$
Dolly	45.8	45.9	46.7	48.2	$+2.4$	$+1.5$
Vicuna	45.2	47.7	48.8	49.8	$+4.6$	$+1.0$
Self-Instruct	46.3	47.2	48.2	48.9	$+2.6$	$+0.7$
Average	46.0	47.0	48.1	49.1	$+3.1$	$+1.0$

Table 2: Judged-chat performance with Llama-3.1-8B-Instruct.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Configuration
Student / teacher	Qwen2.5-7B-Instruct / sampled GPT-5-Chat responses
Warmup	one epoch of SFT on teacher responses (374 steps at $50\%$ ); discriminator jointly initialized; GAD arms resume, SeqKD does not
Adversarial phase	two epochs; 748 steps at $50\%$ , 492 at $33\%$
Optimization	learning rate $10^{-6}$ ; batch 256; rollout group 8; temperature 0.8
Lengths	maximum prompt 2048 tokens; maximum response 1536
Regularization	KL coefficient $10^{-3}$ ; gradient clip 0.2; discriminator fp32

Appendix

Table 3: Training configuration. Pool rows describe the only intervention.

Figure 3: Performance gains across paired training seeds. Each point shows the equal-set-weighted improvement of persistent-negative training over GAD for one paired seed, measured in percentage points. Large markers denote the mean across three seeds, and error bars show one standard deviation. Positive values favor persistent-negative training.

Method	Mean	Std.	Minimum	Dips $<0.5$
GAD	0.894	0.087	0.414	6
Persistent-negative (ours)	0.903	0.068	0.461	1

Appendix

Table 4: Fresh-policy discriminator accuracy in the initial primary run. Standard deviation is computed across adversarial updates, and “dips” counts pre-update evaluations below chance.

On-Policy Distillation

On-policy distillation trains a model on its own outputs using teacher guidance, aligning supervision with the states it encounters at deployment.

See all 512 papers

Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction
Sep 30, 2026Xiao Cui, Mo Zhu, Yulei Qin +3Group Relative Policy Optimization Language Model Distillation
University of Science and Technology of China · Zhejiang University · Independent Researcher
On-Policy Distillation with Negative-Policy Rollouts
Oct 6, 2026Jaehui Hwang, Dongyoon Han, Sangdoo Yun +1Language Model Distillation On-Policy Distillation
NAVER AI Lab
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
Aug 31, 2026Yi Ding, Ruqi ZhangRL for Language Models Language Model Distillation
Department of Computer Science, Purdue University, USA

Sep 30, 2026J/K move · Enter open · S save

Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction

Xiao Cui, Mo Zhu, Yulei Qin, Yuze Wu+2

University of Science and Technology of China · Zhejiang University · Independent Researcher

Group Relative Policy Optimization Language Model Distillation On-Policy Distillation Knowledge Distillation Adversarial Knowledge Distillation

Open paper AskPDF arXiv

Black-box distillation is a practical route for transferring capabilities from API-accessible large language models that expose only text outputs into smaller student models. Recent on-policy adversarial methods such as GAD improve over SeqKD by forming an adversarial loop between a critic and a student, where the critic provides rewards for GRPO-based student policy optimization over the student's sampled responses. However, GRPO computes advantages from the within-group relative rewards of student samples for the same prompt, whereas the critic is trained primarily to distinguish teacher responses from student responses. This objective mismatch can produce reward groups with collapsed scale or fragile margins, leading to brittle grouped optimization signals. We propose Groupwise Reward Geometry Conditioning (GRGC), a two-stage framework that improves advantage construction by shaping student-side reward groups during both critic training and policy optimization. To improve critic-side conditioning, Gaussian groupwise Optimal Transport calibration regularizes the critic during training to produce reward groups with non-collapsed spread and smooth rank-wise gaps by matching sorted prompt-wise rewards to group-centered Gaussian quantiles. Building on this conditioned reward geometry, policy-side group power modulation reshapes the prompt-wise reward groups before they are converted into advantages, preserving the critic-induced ordering while increasing optimization-relevant margin separability. Extensive experiments across diverse teachers, student model families and scales, and training datasets demonstrate the effectiveness of GRGC on both in-distribution and out-of-distribution evaluations, while introducing negligible overhead over GAD. The code is available at https://github.com/2018cx/GRGC.

Abstract

On-Policy Distillation

Explore similar work

Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Construction

On-Policy Distillation with Negative-Policy Rollouts

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement