Knowledge distillation is a widely adopted technique for transferring capabilities from LLMs to smaller, more efficient student models. However, unauthorized use of knowledge distillation takes unfair advantage of the considerable effort and cost put into developing frontier models. We investigate methods for modifying teacher-generated reasoning traces to achieve two objectives that deter unauthorized distillation: (1) anti-distillation, or degrading the training usefulness of query responses, and (2) API watermarking, which embeds verifiable signatures in student models. We introduce several approaches for dynamically rewriting a teacher's reasoning outputs while preserving answer correctness and semantic coherence. Two of these leverage the rewriting capabilities of LLMs, while others use gradient-based techniques. Our experiments show that a simple instruction-based rewriting approach achieves a strong anti-distillation effect while maintaining or even improving teacher performance. Furthermore, we show that our rewriting approach also enables embedding watermarks that can be reliably detected with essentially no false alarms. Our code is available at https://github.com/xhOwenMa/trace-rewriting.
Figures & tables
Figure 1: Overview of instruction-based rewriting.: (a) Clean trace generation : The teacher model T generates a reasoning trace r for given task (query) q using a standard generation instruction pg . (b) Rewriting : A rewrite model R with a rewrite instruction pr transforms r into r′ to achieve IP protection while maintaining utility.
Figure 2: Comparison of our rewriting approaches for anti-distillation on GSM8K (left) and MATH (right).
Figure 3: Anti-distillation comparisons on GSM8K (left) and MATH (right). Our method achieves the strongest anti-distillation effect without compromising the teacher’s utility.
Method
Teacher
Llama-3.1-8B
Llama-3.2-3B
Qwen2.5-1.5B
Clean
88.76%
TD ( ↑ )
FA ( ↓ )
TD ( ↑ )
FA ( ↓ )
TD ( ↑ )
FA ( ↓ )
He et al. He et al. (2022a)
88.76%
0.94 / 1.00
0.76 / 1.00
0.95 / 1.00
0.80 / 1.00
0.99 / 1.00
0.78 / 0.99
GINSEW Zhao et al. (2023b)
69.03%
0.01 / 0.06
0.02 / 0.16
0.01 / 0.04
0.01 / 0.13
0.01 / 0.08
0.01 / 0.07
KGW Kirchenbauer et al. (2023)
71.15%
0.00 / 0.00
0.00 / 0.00
0.00 / 0.00
0.00 / 0.01
0.39 / 0.88
0.00 / 0.00
VIA Liang et al. (2025)
88.76%
0.77 / 1.00
0.00 / 0.02
0.27 / 0.52
0.00 / 0.00
0.84 / 1.00
0.00 / 0.00
Ours
90.98 %
1.00 / 1.00
0.00 / 0.02
0.55 / 0.99
0.00 / 0.00
0.98 / 1.00
0.00 / 0.00
Table 1: Watermark detection results on GSM8K. Teacher column shows teacher accuracy. For each student model, we report true detection rate (TD, left) and false alarm rate (FA, right). Each cell contains two values corresponding to K=5 and K=20 test queries, respectively. Bold indicates that the result is within 0.02 of the best.
Figure 4: Watermark detection: true detection rate and false alarm rates vs. K for llama3.1-8B suspect student model.
Figure 5: Robustness of anti-distillation to adaptive attacks. Paraphrased : distiller paraphrases our OPT traces before fine-tuning. KPOD : distiller applies keypoint-based progressive distillation on our OPT traces. Ours : standard SFT on our OPT traces.
Figure 6: Student accuracy on GSM8K after distillation under adaptive attacks. Clean : distilled from original traces without watermarks. Filtered : distilled from watermark-injected traces after regex-based filtering that removes ±3 tokens around each = sign. Paraphrased : distilled from watermark-injected traces paraphrased by Parrot paraphraser. CDG-KD : distilled from traces processed by CDG-KD. Ours : distilled from watermark-injected traces.
Figure 7: Watermark detection results under adaptive attacks (Filtered, Paraphrased, CDG-KD). The watermark remains detectable under all three attacks, while they substantially degrade student task accuracy (see Figure 6 ).
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Original
Semantic
Optimized
ADS
DOGe
2.33 ( ± 0.4)
4.24 ( ± 2.1)
3.79 ( ± 1.1)
2.31 ( ± 0.8)
1.42 ( ± 0.3)
Appendix
Table 2: Perplexity of reasoning traces computed using Llama-3.1-8B as reference model (mean ± std over 150 MATH samples).
Method
Coherence
Naturalness
Readability
Overall
Original
3.97
4.12
3.96
4.01
Semantic
3.19
3.03
3.06
3.09
Optimized
3.90
3.86
3.73
3.83
ADS
3.12
2.31
2.66
2.71
DOGe
2.91
2.03
2.26
2.40
Appendix
Table 3: LLM-as-judge quality scores (1–5) on 150 MATH samples. Higher is better.
Figure 8: Anti-distillation effects of the Token-Level poisoning method, where FO-Grad is the adversarial approximation of the actual objective, similar to how they are defined in Section 5.2 .
Figure 9: Anti-distillation comparisons on MMLU (left) and MMLU-Pro (right). Ours is our OPT method. Dashed line indicates random guessing accuracy.
Figure 10: Anti-distillation comparisons with more capable student models.
Figure 11: Ablation study: Direct prompts the teacher to generate anti-distillation traces directly, while Optimized first generates clean traces then rewrites them with our optimized instruction. The two-stage approach achieves substantially stronger anti-distillation effects.
Method
Llama-3.2-1B
Llama-3.2-3B
Qwen2.5-1.5B
GSM8K
Clean
30.1
57.7
74.9
Ours (OPT)
5.9
8.6
13.6
Answer-only
4.2
5.2
6.1
MATH
Clean
8.2
15.4
44.2
Appendix
Table 4: Student accuracy (%) under answer-only distillation on GSM8K and MATH.
Teacher
Rewriter
Teacher Acc
Student Acc
Clean
Ours
Clean
Ours ( Δ )
7B
120B
88.8
91.6
57.7
8.6 ( − 49.1)
7B
20B
88.8
89.3
57.7
23.6 ( − 34.1)
7B
7B
88.8
88.1
57.7
30.2 ( − 27.5)
120B
120B
93.5
93.3
54.5
19.9 ( − 34.6)
Appendix
Table 5: Effect of teacher and rewriter model size on anti-distillation efficacy on GSM8K. Δ Student denotes the change in student accuracy relative to distillation from clean traces.
Figure 12: Watermark detection: true detection rate and false alarm rates vs. K for llama3.2-3B suspect student model.
Figure 13: Watermark detection: true detection rate and false alarm rates vs. K for qwen2.5-1.5B suspect student model.