On-policy distillation has been widely adopted for alleviating the distribution mismatch of off-policy distillation. Self-distillation further removes the need for an external teacher by using the student's own outputs, enhanced with extra context or demonstrations, as supervision. This teacher-free design makes on-policy self-distillation practical, yet its privacy risks remain largely unexplored. In particular, using two coupled views of the same training example may strengthen its memorization. Motivated by this, we present the first study of membership inference attacks against on-policy self-distillation. We investigate both gray-box and black-box threat models and propose a unified attack framework, named OP-Print. OP-Print builds on our finding that the supervision imprint in on-policy self-distillation is sparse and structured, concentrating on uncertainty-sensitive correction sites rather than across the entire rollout. Exploiting this property, we construct attack signals tailored to the two access levels: in the gray-box setting, OP-Print directly aggregates the privileged teacher-student residual correction along the same rollout; in the black-box setting, it measures restart-based output-basin stability as an observable surrogate for the hidden correction process. Across diverse models and distillation schemes, all baseline attacks remain essentially at random guessing even under gray-box access, whereas OP-Print is broadly validated to be effective in both gray-box and black-box settings.
Figures & tables
Figure 1: Gray-box MIA signals from rollout-level dynamics and attack behavior (AUC) of Gray-box score.
Figure 2: (a). Divergence varies only mildly across token positions, indicating that position alone is not the main source of the signal. (b). Divergence increases with student uncertainty, showing that correction concentrates on difficult reasoning tokens. (c). The reliability map reveals localized rather than uniform divergence, confirming that the supervision imprint is sparse and structured.
Figure 3: An illustration of the black-box attack process of OP-Print. Given multiple initial rollouts, OP-Print truncates each rollout at different prefix ratios, restarts generation from the partial prefix, and scores membership by measuring the similarity between restarted outputs. Members tend to produce more stable and similar restart trajectories, while non-members exhibit lower restart consistency.
Method
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
AUC
Acc.
TPR@5%FPR
AUC
Acc.
TPR@5%FPR
AUC
Acc.
TPR@5%FPR
Gray-box setting
Min-K% Prob
0.512
0.518
0.052
0.497
0.505
0.046
0.529
0.511
0.055
ModRényi ( α=0.5 )
0.484
0.506
0.044
0.508
0.508
0.051
0.513
0.514
0.056
ModRényi ( α=2 )
0.516
0.514
0.057
0.507
0.512
0.047
0.506
0.502
0.050
Rényi ( α=0.5 )
0.503
0.509
0.049
0.517
0.511
0.054
0.495
0.503
0.045
Table 1: Gray-box and black-box results with the JSD objective. Colors indicate the top-2 and worst-2 results in the gray-box setting, and the top-1 and worst-1 results in the black-box setting.
Figure 4: Representative gray-box and black-box OP-Print signals. (a)(b) In the gray-box setting, members show lower chunk-level KL gaps and higher scores than non-members. (c)(d) In the black-box setting, members exhibit more stable outputs under prefix restarts, whereas non-members are less stable.
Figure 5: Ablation studies of OP-Print on Qwen3-4B. (a) Gray-box performance across rollout numbers and chunk sizes. (b) Black-box performance with varying numbers of prefix ratios. (c) Restart cost-performance trade-off, where bubble size denotes TPR@5%FPR.
Method
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
AUC
Acc.
TPR@5%FPR
AUC
Acc.
TPR@5%FPR
AUC
Acc.
TPR@5%FPR
Gray-box setting
Min-K% Prob
0.509
0.510
0.051
0.491
0.502
0.045
0.522
0.503
0.054
ModRényi ( α=0.5 )
0.478
0.504
0.042
0.503
0.500
0.052
0.506
0.507
0.055
ModRényi ( α=2 )
0.507
0.505
0.055
0.496
0.508
0.046
0.500
0.501
0.049
Rényi ( α=0.5 )
0.495
0.502
0.050
0.510
0.504
0.056
0.484
0.500
0.043
Table 2: Gray-box and black-box results with tinker-style objective. Colors indicate the top-2 and worst-2 results in the gray-box setting, and the top-1 and worst-1 results in the black-box setting.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
Number of epochs
3
3
3
Learning rate
5×10−6
5×10−6
5×10−6
Precision
bfloat16
bfloat16
bfloat16
Attention implementation
SDPA
SDPA
SDPA
Gradient checkpointing
Yes
Yes
Yes
Max completion length
1024
1024
1024
Appendix
Table 3: Training configurations for target models. All models are trained under the same on-policy self-distillation pipeline; only model scale and a few runtime settings differ across runs.
On-policy distillation (OPD) trains a student to match a teacher's next-token distributions on student-generated trajectories. However, privileged information supplied to the teacher for OPD training may contain sensitive data. Whether the student leaks private information about the records supplied to the teacher during distillation remains poorly understood. To the best of our knowledge, we present the first systematic study of membership inference in this setting. We find that fresh student trajectories expose sparse membership signals that fixed reference-answer losses often miss. These signals are mixed with probability changes caused by training on other records. We introduce Leaky, which samples fresh trajectories from the target model and compares its token log-probabilities with the maximum across matched reference models trained without the candidate records. It applies Leaky ReLU to the resulting gaps, preserving positive gaps and downweighting negative gaps as an approximate correction for incidental positive gaps in non-members. Across fifteen targets spanning mathematics, medical question answering, and code generation, Leaky outperforms all evaluated baselines and achieves mean AUROC 0.875, compared with 0.614 for the strongest baseline on each target in the main evaluation. On the same sampled trajectories, the strongest baseline achieves mean AUROC 0.826. These results show that students trained through OPD can expose the membership of records used for teacher supervision, even when fixed reference-answer losses provide little evidence of membership.
On-policy distillation (OPD) trains a student model by aligning its policy with a teacher model on trajectories generated by the student model itself. Through this process, the student policy moves toward the teacher on the prompts used for distillation. However, these prompts are often private and costly, creating a need for prompt-level membership auditing. Existing methods mainly rely on likelihood-based confidence signals or student policy drift between checkpoints, but they do not capture the teacher-induced direction of the student update. In this paper, we propose Policy Alignment Membership Auditing (PAMA), a new auditing framework tailored for OPD. Our key observation is that a member prompt directly contributes to the teacher-guided policy update, while a non-member prompt only experiences indirect effects through cross-prompt generalization. Based on this directional trace, PAMA measures whether the student update moves toward reducing the teacher loss on a candidate prompt. Specifically, we introduce Teacher Alignment Gain (TAG) to estimate the teacher-aligned update direction from model outputs, and further combine it with student drift and uncertainty alignment signals for reliable membership auditing. We evaluate PAMA on six datasets and three teacher-student model families. On MATH, the primary evaluation benchmark, PAMA achieves AUC values of 0.791--0.941, improving AUC by 14.6--20.6% over state-of-the-art baselines.
Yilong Yang, Wenzhuo Shang, Yule Liu +2
Xidian University · Hong Kong University of Science and Technology (Guangzhou)
On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Yi Ding, Ruqi Zhang
Department of Computer Science, Purdue University, USA