Watermark Basins: Black-Box Detection of Distilled LLMs

Hongyan Chang; Hady Elsahar; Tom Sander; Pierre Fernandez; Saeed Mahloujifar; Sylvestre-Alvise Rebuffi; Tomáš Souček; Tuan Tran

cs.CROct 27, 2026

Watermark Basins: Black-Box Detection of Distilled LLMs

Authors: Hongyan Chang, Hady Elsahar, Tom Sander, Pierre Fernandez, Saeed Mahloujifar, Sylvestre-Alvise Rebuffi, Tomáš Souček, Tuan Tran,

Organizations: FAIR, Meta Superintelligence Labs

Abstract

Proprietary language models and their reasoning traces are vulnerable to API-based distillation, where an adversary trains a student model on the teacher model's outputs. To obtain evidence of distillation, model providers can watermark the teacher's outputs, aiming to leave a radioactive signal in the student's outputs. Existing black-box detectors search for such radioactivity using token-level statistics of the suspect model's outputs. However, token-level signals are not always detectable, for instance when the distillation traces are sampled at a low temperature. In this work, we discover the watermark basin, a persistent signal that enables more reliable watermark detection: during fine-tuning, watermarked teacher traces embed a distinct, key-specific geometric signal in parameter space that lies orthogonal to the ordinary task-specific fine-tuning subspace. Building on this, we introduce Watermark Basin Alignment (WM-Basin), a black-box watermark detector that queries the suspect model through its API, without access to its weights or logits. WM-Basin maps generated text to gradient representations using a fixed clean probe model, projects out the ordinary fine-tuning task directions, and tests whether the residual gradient aligns with the watermark basin induced by the watermark key. We evaluate WM-Basin extensively across diverse teacher models and student architectures (from 135M to 8.2B parameters) distilled on mathematical reasoning. At a strict 0.1% false-positive rate, WM-Basin detects 79% of distilled models at n-gram=1 and 45% at n-gram=3, substantially improving over the state-of-the-art token-level baseline (29% and 1.3%, respectively). Because the watermark signal resides in a persistent basin, it survives post-distillation modifications including quantization, pruning, and reinforcement-learning post-training.

Figures & tables

Figure 1: The watermark basin. Training on watermarked traces moves the student into a persistent, key-specific watermark basin , whereas clean traces from the same teacher move it only along the task direction. Watermark Basin Alignment ( WM-Basin ) recovers this geometry from black-box outputs alone to detect distillation.

Panel of Figure 2 — Figure 2: WM-Basin : black-box distillation detection via watermark basin alignment. The three panels are the three stages of § 2.3 . (i) The verifier collects suspect outputs on test prompts and, on separate build prompts, teacher outputs under the claimed watermark key and $M$ decoy keys. Backpropagating each collection through one fixed probe model gives the suspect gradient $\Delta_{\mathrm{sus}}$ and one reference gradient $\Delta_{k}$ per key. (ii) Raw gradients share large common task components that inflate the cosine similarities, so every key appears to overlap with the suspect. Projecting out the shared task directions leaves the residuals $\widetilde{\Delta}_{\mathrm{sus}}$ and $\widetilde{\Delta}_{k}$ , among which only the claimed key stays close to the suspect; $\theta$ is the angle between them. (iii) Each candidate key receives the alignment score $S(k)=\cos(\widetilde{\Delta}_{\mathrm{sus}},\widetilde{\Delta}_{k})$ , and the claimed key’s rank among the decoys gives the $p$ -value $p_{\mathrm{rank}}$ . The claimed key ranking at the top is evidence of distillation; a decoy key at the top is not.

Figure 3: The anisotropic geometry of the watermark basin , measured as the white-box detection significance ( $-\log_{10}p$ ) around a radioactive student $\bm{\theta}_{\mathrm{wm}}$ (red star; $\bm{\theta}_{\mathrm{student}}$ in the text) fine-tuned on watermarked teacher traces. Left to right: 3D landscape, 2D contour map, and 1D slices through $\bm{\theta}_{\mathrm{wm}}$ . The signal falls to the detector’s floor within a few steps along the watermark direction $\hat{\Delta}_{\mathrm{wm}}$ (red curve), while remaining detectable over a much broader range along the task direction $\hat{\Delta}_{\mathrm{task}}$ (green curve).

Figure 4: What governs detection: each knob swept on its own axis (

n

-gram

{=}3

). Each panel varies one knob, with its value on the

x

-axis and the TPR at a fixed

1\%

FPR on the

y

-axis, for WM-Basin and the black-box baseline; the grey dashed line marks chance (TPR

{=}

FPR

{=}1\%

). Each configuration trains

50

students, each with a different watermark key. Left: watermarked fraction of the training traces. Middle: LoRA rank

r

. Right: training epochs. The all-clean control (no watermarked traces) is not plotted; none of its

50

students is flagged at any of the three thresholds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Setting
Models
Teacher (default)	DeepSeek-R1-Distill-Qwen-14B
Teacher (swaps)	gpt-oss-120B, QwQ-32B, Llama-3.1-8B
Student (default)	Qwen2.5-3B
Student (cross-arch.)	$24$ structures, $135$ M to $8.2$ B (Llama(-Guard), Qwen2.5/3, QwQ, DeepSeek-Distill, Gemma, SmolLM2)
Probe model	Qwen2.5-3B (base, warmed)

Appendix

Table 8: Default experimental configuration. Models, watermark, distillation, probe, and detector settings used throughout § 3 unless stated otherwise; see App. B.2 for full details. The TextSeal parameter

\alpha{=}0.1

is held fixed in every experiment.

Experiment	Varied factor and levels	Model cohort / sample size	Result
Student architecture	$24$ structures, $135$ M to $8.2$ B parameters	$50$ students per architecture ( $1{,}200$ total)	Tab. 1
Distillation temperature	Teacher sampling $T\in\{1.0,0.7,0.5\}$ ; reference bank fixed at $T{=}1.0$	$100$ students per temperature	Tab. 2
Teacher model	$4$ teachers spanning $3$ families, $8$ B to $120$ B; bank rebuilt per teacher; the three swapped teachers distil for $8$ epochs, not $3$	$100$ students per teacher	Tabs. 3 , 12
Watermark scheme	$4$ watermark schemes at $n$ -gram $\in\{1,2,3\}$	$50$ students per scheme and window	Tabs. 4 , 13
Audited domain	$5$ domains; every detector stage and the system prompt rebuilt per domain	Fixed pool of $100$ students	Tab. 5
Model compression	int8, int4 and nf4 quantization; $10$ to $30\%$ pruning	$100$ students per operation	Tabs. 6 , 16 ; Fig. 12

Appendix

Table 9: Per-experiment settings. What each study varies, the cohort used, and where the result is reported; varied factors are read against the defaults of Tab. 8 .

Figure 5: Watermarking preserves the utility gained through distillation. DeepSeek-R1-Distill-Qwen-14B ( Guo et al., 2025 ) is distilled into Qwen2.5-3B ( Yang et al., 2024 ) using clean traces or TextSeal traces at three

n

-gram windows. The left panel reports the fraction of

5{,}000

traces passing the quality filter; the other panels report GSM8K and Math-500 accuracy. Watermarked bars average

100

keyed students with

\pm 1

standard-deviation error bars, while No WM is one student trained on clean traces; dashed lines mark the un-distilled base.

Family	Checkpoints	Ckpts.
Qwen	Qwen2.5-0.5B, -0.5B-Instruct, -1.5B, -1.5B-Instruct, -3B, -3B-Instruct, -7B-Instruct, -Math-7B; Qwen3-0.6B, -1.7B, -4B, -4B-Instruct, -8B, -8B-Base, -32B; QwQ-32B; Qwen1.5-MoE-A2.7B	$17$
Llama	Llama-2-7B, Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, Llama-Guard-3-1B, Llama-Guard-3-8B	$5$
DeepSeek	DeepSeek-R1-Distill-Qwen-1.5B, -7B, -14B † ; DeepSeek-MoE-16B-Base	$4$
SmolLM2	SmolLM2-135M, -135M-Instruct, -1.7B	$3$
Gemma	Gemma-2-2B, Gemma-3-1B-PT	$2$
Phi	Phi-3.5-mini-Instruct, Phi-4	$2$

Appendix

Table 10: The clean PCA pool. The

43

off-the-shelf checkpoints whose gradients fit the shared task subspace of App. B.5 , grouped by pretraining lineage. Each model generates on

1{,}000

build-pool prompts at each of three decoding temperatures (

0.7

1.0

1.3

Panel of Figure 6 — Figure 6: Choosing $q$ : detection power against three order statistics of the key bank. The projection sweeps from no PCA through mean only (subtract the pool mean, remove no subspace) to $q{=}100$ removed components; the path visits the seven settings in that order and marker shade runs light to dark with $q$ . Columns are the $n$ -gram window. Rows are which key of the $1{,}000$ -key bank the false-positive rate is read off: the typical key, the $95$ th-percentile tail key and the worst key. $y$ : TPR at $\alpha{=}5\%$ over the $1{,}200$ distilled students. $x$ : the percentage of the $129$ clean model-temperature cells on which that key fires at the same threshold.

$n$ -gram	median	p $90$	p $95$	worst	floor	worst / floor
$1$	$4.3$	$9.9$	$12.1$	$23.4$	$13.5$	$1.74$
$2$	$4.3$	$9.9$	$12.8$	$25.5$	$13.5$	$1.89$
$3$	$4.3$	$9.9$	$12.1$	$26.2$	$13.5$	$1.95$

Appendix

Table 11: Per-key false-positive rate at

\alpha{=}5\%

, after removing the task subspace. Each of the

1{,}000

bank keys is read over the

129

clean model-temperature cells, with

4

-fold projectors cross-fitted by base model so no cell is scored by a subspace fit on its own model. All values are percentages except the last column. Floor is the

95

th percentile of the maximum under exact uniformity at

K{=}1{,}000

over

129

cells: the level a perfectly exchangeable bank of this size would still reach, and the reference the worst key should be judged against.

Panel of Figure 7 — Figure 7: WM-Basin (ours)

Panel of Figure 10 — Figure 10: Detection performance of the two closed-model detectors across all operating points. ROC (log $x$ , linear $y$ ), one panel per $n$ -gram window, matched on the $100$ distillation keys $1001$ to $1100$ . TPR is over the $100$ watermarked students (one per key, scored with its own key); FPR is over the same $100$ keys $\times$ the $65$ held-out innocent model-temperature cells (the evaluation half of the $129$ -cell clean pool: $43$ off-the-shelf base models $\times$ $3$ sampling temperatures; $6{,}500$ null cases). Sweeping the threshold $\alpha$ traces each curve; WM-Basin (ours) is drawn in vermilion and Black-box (baseline) in blue. The grey dashed line marks chance ( $\mathrm{TPR}{=}\mathrm{FPR}$ ).

Figure 11: Impact of suspect text budget on detection performance. We plot True Positive Rate at

1\%

FPR (TPR@

1\%

; share of distilled models detected at rank-

p{\leq}0.01

, one model per key) against the amount of suspect text evaluated (

\log x

scale, million tokens). Both WM-Basin (ours) and Black-box ( Sander et al., 2024 ) are evaluated on identical generations at the same

1\%

FPR operating point.

			$n$ -gram ${=}1$		$n$ -gram ${=}2$		$n$ -gram ${=}3$
Cond.	Orthogonal	Usable?	Acc.	Along $w$	Acc.	Along $w$	Acc.	Along $w$
none	$0$	✓	$70.7$	$0$	$70.7$	$0$	$70.7$	$0$
int8	$14.09$	✓	$72.2$	$-0.0001$	$71.0$	$-0.0002$	$70.0$	$-0.0001$
int4	$152.07$	✗	$50.4$	$-0.0113$	$50.8$	$-0.0112$	$45.4$	$-0.0117$
nf4	$125.91$	✗	$55.8$	$-0.0291$	$42.9$	$-0.0281$	$41.2$	$-0.0266$
prune10	$27.23$	✓	$77.1$	$-0.0063$	$76.2$	$-0.0061$	$77.3$	$-0.0058$

Appendix

Table 16: Compression moves almost entirely orthogonally to the watermark direction. Across

100

students per window, Acc. is mean GSM8K (%) and Usable? denotes Acc.

{\geq}0.9{\times}70.7{=}63.6

. We decompose

\Delta_{\mathrm{post}}{=}\bm{\theta}_{\mathrm{post}}{-}\bm{\theta}_{\mathrm{wm}}

into its signed projection along the watermark key direction

\hat{\Delta}_{\mathrm{wm}}

(Eq. ( 1 )) and the orthogonal

L_{2}

remainder. The remainder is a property of the compressed weights alone and is identical at the three windows; the usability verdict is read off each window’s own Acc. and comes out the same in every row. Both are therefore printed once. The along component is at least three orders of magnitude smaller and increases in magnitude within the ordered pruning sweep.

Figure 12: Robustness to post-processing. TextSeal,

n

-gram

{=}3

100

keyed students under int8, int4 and nf4 quantization and

10/20/30\%

pruning; base is uncompressed. (a) TPR at

1\%

FPR for WM-Basin and Black-box ; (b) gsm8k accuracy, dotted line is the

0.9{\times}

usable gate; (c) weight-space move along the watermark direction vs orthogonal residual. Details in Tab. 16 .

			GSM8K			Math-500			Move
Cond.	$n$ -gram	$n$	base	post	$\Delta$	base	post	$\Delta$	total	along $w$	along/total
reason	$1$	$100$	$67.2$	$72.0$	$+4.77$	$41.9$	$40.8$	$-1.11$	$4.80$	$-0.252$	$5.3\%$
reason	$2$	$100$	$69.0$	$70.0$	$+0.96$	$43.0$	$39.4$	$-3.60$	$4.70$	$-0.182$	$3.9\%$
reason	$3$	$100$	$69.0$	$71.3$	$+2.37$	$43.2$	$40.6$	$-2.62$	$4.66$	$-0.146$	$3.1\%$
multilingual	$1$	$100$	$67.2$	$74.2$	$+6.98$	$41.9$	$46.9$	$+4.99$	$2.72$	$-0.013$	$0.5\%$
multilingual	$2$	$100$	$69.0$	$73.6$	$+4.59$	$43.0$	$46.7$	$+3.68$	$2.71$	$-0.009$	$0.3\%$

Appendix

Table 17: GRPO preserves utility with far smaller weight movement than SFT. Every row is paired against its own cohort’s pre-attack students, whose accuracy is printed as base : the cohorts differ in size (

n

) and do not share a baseline, so

\Delta

is the mean per-key change and not the difference of the two pooled columns beside it. GSM8K and Math-500 are percentages and

\Delta

is in points. Move (total) is

\lVert\Delta_{\mathrm{post}}\rVert_{2}

, Along

w

is its signed projection on the key direction

\hat{\Delta}_{\mathrm{wm}}

, and Along/total is the absolute percentage share. GRPO projections are under

0.0005

in magnitude and print as

0.000

. The SFT rows share the full

100

-key pool as their base; the GRPO rows are smaller cohorts carrying their own, which run from

0.6

points above it to

2.8

below, which is why Tab. 7 reports accuracy as a paired change rather than an absolute.

	$n$ -gram ${=}1$			$n$ -gram ${=}2$			$n$ -gram ${=}3$
Task	Pre	Post	$\Delta$	Pre	Post	$\Delta$	Pre	Post	$\Delta$
K&K	$10.0$	$14.4$	$+4.4$	$9.0$	$13.4$	$+4.4$	$10.1$	$15.2$	$+5.1$
IFEval	$18.4$	$21.3$	$+2.9$	$18.3$	$21.6$	$+3.3$	$18.2$	$22.1$	$+3.9$

Appendix

Table 18: Cross-domain GRPO improves the task it optimises. Pre, post and

\Delta

are task accuracy in percent on greedy held-out evaluation (

400

K&K puzzles,

1{,}000

IFEval prompts). Each task and window pools

10

keys, except IFEval at

n

-gram

{=}3

(

11

). Counted over the three windows together,

10

keys improved on K&K and

10

on IFEval. The same students’ weight-space displacement and their off-target GSM8K and Math-500 changes are in Tab. 17 .

LLM Watermarking

Large language model watermarking embeds or detects signals in text and models to trace provenance, ownership, and unauthorized use.

See all 111 papers

Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3LLM Auditing LLM Watermarking
University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta
Protecting Language Models Against Unauthorized Distillation through Trace Rewriting
Feb 16, 2026Xinhang Ma, William Yeoh, Ning Zhang +1LLM Watermarking Knowledge Distillation
Washington University in St. Louis
Reference-Based Distillation Detection in LLMs
Jun 19, 2026Rajat Rawat, Sizhe Chen, Akshay Anand +3LLM Auditing Teacher-Student Learning
University of California, Berkeley · University of Southern California · OpenAI

May 15, 2026J/K move · Enter open · S save

Asking Back: Interaction-Layer Antidistillation Watermarks

Guang Yang, Amir Ghasemian, Fengchen Liu, Zhong Wang+2

University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta

LLM Auditing LLM Watermarking Knowledge Distillation

Open paper AskPDF arXiv

Detecting unauthorized knowledge distillation from a deployed LLM API is hard because the defender controls neither the attacker's training pipeline nor the next-token logits. Existing defenses operate on the teacher's output tokens -- biasing the next-token distribution (green-list watermarks, cryptographic schemes, antidistillation sampling) or rewriting outputs after generation. Recent work shows a paraphrasing attacker can strip these signals without losing the underlying knowledge. We propose interaction-layer antidistillation watermarks, which move the trace one layer higher, into the teacher's interaction behavior: the defender wraps the teacher with a system prompt that intermittently induces a behavioral marker -- an explicit follow-up question, a low-frequency variant, or a declarative restatement. An oblivious distiller inherits the behavior, and the defender audits via black-box queries with a human-validated LLM-as-judge (Cohen's kappa = 0.84/0.78 on strong/style rubrics). Across 63 LoRA-distilled students under a Llama-3.3-70B-Instruct teacher (35,343 judged samples), behavioral watermarks transfer at 88.9% (Gemma) / 80.9% (OLMo) / 45.2% (Qwen) relative fidelity (H1, H2). Under non-adaptive DIPPER paraphrasing, robustness decomposes into a teacher-self ceiling (about 66.4%) and student-relative retention of 21-112%, with OLMo preserving the watermark above the teacher itself (H3, F-Amp). Low-density (about 20%) explicit and implicit declarative variants transfer above per-family baseline (H4, F-Style). An N=20 in-lab study (pre-registered Latin-square) shows all marker variants within 0.22 Likert step of baseline; TOST, Friedman, and Bonferroni-Wilcoxon support H5. The interaction layer is a viable design locus for antidistillation watermarking, complementary to token-, model-, and reasoning-trace-layer defenses.

Abstract

LLM Watermarking

Explore similar work

Asking Back: Interaction-Layer Antidistillation Watermarks

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

Reference-Based Distillation Detection in LLMs