Retrospective Distillation Attribution via Normalized Response Similarity

Minwoo Jang; Jaechang Kim; Minhyeon Oh; Jeongyeon Hwang; Jungseul Ok

cs.AISep 26, 2026

Retrospective Distillation Attribution via Normalized Response Similarity

Authors: Minwoo Jang, Jaechang Kim, Minhyeon Oh, Jeongyeon Hwang, Jungseul Ok

Organizations: Graduate School of Artificial Intelligence, POSTECH, South Korea · POSTECH Institute of Artificial Intelligence, South Korea · Department of Computer Science and Engineering, POSTECH, South Korea

Abstract

Model distillation transfers capabilities through supervised fine-tuning (SFT) on teacher responses, often collected from commercial APIs, raising questions of model provenance. Existing distillation attribution methods have been largely evaluated on students immediately after the SFT step. However, a distilled model may undergo further SFT, preference optimization, or reinforcement learning before release, while an auditor may lack access to the pre-distillation checkpoint required by reference-based attribution. To close this gap, we propose SCOUT, an output-only method that aggregates recurring syntactic patterns into candidate profiles, filters low-contrast patterns, and calibrates student--candidate distances against inter-candidate distances. SCOUT supports attribution and abstention using only current texts, without model weights, token likelihoods, or historical checkpoints. Auditing publicly released descendants of distilled models spanning diverse post-training objectives, SCOUT consistently identifies the distillation source. Furthermore, tracing teacher-associated syntactic signatures along training trajectories reveals that they emerge during distillation and persist through subsequent preference optimization and reinforcement learning.

Figures & tables

Figure 1: How PoS Templates and SCOUT use syntactic evidence for distillation attribution. (a) PoS Templates learns a classifier from syntactic patterns in individual candidate responses and classifies each student’s responses. (b) SCOUT aggregates recurring syntactic patterns across each teacher candidate’s responses into a profile and compares each student response with these profiles.

Figure 2: Overview of SCOUT. (a) PoS

n

-grams are aggregated into candidate profiles and low-contrast patterns are filtered. (b) Each response is compared with the filtered profiles. (c) Inter-candidate distances provide prompt-specific centers

\mu_{i}

and candidate-level scales

\sigma

. Filtering isolates candidate-discriminative syntax, while calibration makes scores comparable across candidates. Therefore, low scores indicate proximity not explained by generic similarity to the pool.

Reference-based (A pre-distillation reference checkpoint is required.)
Math probes	OMI					s1
	Model Level			Probe Level (%)		Model Level			Probe Level (%)
Method	Desc.	Par.	Base	Dist.	Same	Desc.	Par.	Base	Dist.	Same
DistillDetect, $R=S^{(0)}$	12/12	2/2	2/2	98.0	94.1	12/12	2/2	2/2	97.9	96.2
DistillDetect, $R=S^{(1)}$	3/12	—	—	—	—	3/12	—	—	—	—
Output-only (Only current model outputs are required.)

Table 1: Distillation attribution after subsequent training. Desc. and Par. count correct R1 assignments among twelve descendants and two distilled parents, while Base counts pre-distillation bases correctly identified as non-R1. Using the same twelve descendants and prompt bank, Dist. compares independently sorted candidate scores at equal ranks, whereas Same compares candidates on each shared prompt. Both report probe-level R1 selection accuracy averaged across descendants, distinguishing distributional separation from same-prompt candidate selection. For DistillDetect,

R

denotes the reference model. Dist. is unavailable for LLM judges because they return labels only. Higher is better for all metrics. Bold marks the best output-only value in each column.

Panel of Figure 3 — Figure 3: Prompt efficiency after subsequent training. For each audit budget $n<200$ , we randomly select $n$ distinct prompts $2{,}000$ times and report the mean number of the 12 descendants correctly attributed to R1. At $n=200$ , we evaluate all prompts once. Higher is better.

Reference-based (A pre-distillation reference checkpoint is required.)
	Math ( $19$ Students)					Conversation ( $19$ Students)
Method	$\mathrm{Det.}$	$\mathrm{Acc}(S)$	$\mathrm{Acc}(T)$	$\mathrm{FP}(S)$	$\mathrm{FP}(T)$	$\mathrm{Det.}$	$\mathrm{Acc}(S)$	$\mathrm{Acc}(T)$	$\mathrm{FP}(S)$	$\mathrm{FP}(T)$
DistillDetect	11/19	57/76	43/76	3/19	10/19	13/19	54/76	38/76	5/19	12/19
Output-only (Only current model outputs are required.)
PoS Templates	14/19	48/76	32/76	11/19	11/19	11/19	64/76	53/76	0/19	0/19
LLM judge (Mistral)	15/19	56/76	56/76	6/19	6/19	12/19	56/76	56/76	3/19	3/19

Table 2: Detection and threshold transfer. Both 1.5–4B cohorts use disjoint SFT and audit examples. Math uses OMI and s1, and conversation uses OASST1 and Dolly.

\mathrm{Det.}

counts students correctly attributed on both probes with the teacher present using student-backbone holdout.

\mathrm{Acc}(S)

and

\mathrm{Acc}(T)

count correct accept-or-abstain decisions under student-backbone and teacher holdout, respectively, across 4 cells per student and 76 per cohort.

\mathrm{FP}(S)

and

\mathrm{FP}(T)

count accepted students after teacher removal under the same holdouts. Higher is better for

\mathrm{Det.}

and

\mathrm{Acc}

, and lower for

\mathrm{FP}

Panel of Figure 4 — Figure 4: Detection performance across false-acceptance rates. Axes use the student-level definitions in Table 2 . LLM judges appear as single operating points, and the diagonal denotes chance. SCOUT matches the best math performance and is best in conversation at low false-acceptance rates.

Panel of Figure 5 — Figure 5: Controlled students reproduce teacher-associated syntactic patterns. Rows are candidate models, and columns are 1.5–4B students grouped by their true teacher. Each cell reports the share of student PoS $n$ -gram tokens matching the candidate’s ten most characteristic patterns, normalized by the share in responses from that candidate. Column labels use G4 for Gemma-3-4B-PT, L3 for Llama-3.2-3B-Instruct, Q1.5 for Qwen-2.5-1.5B, and Q3 for Qwen-2.5-3B.

Figure 6: Scaling corrects errors after centering. For ten students on one held-out Dolly prompt, panels plot raw against centered gaps and centered against scaled gaps. Positive gaps indicate correct attribution.

Panel of Figure 7 — Figure 7: Source-margin shifts from the pre-distillation state on OMI. All panels use the same 13-candidate pool. Positive values indicate a larger margin toward the indicated teacher than at $S^{(0)}$ , and every post-distillation value is positive. Shading marks distillation in (b) and (c) .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Candidate	Signature	Example
GPT-OSS-120B	. RB NN :	“. Thus answer :”
( OpenAI, 2025a )	. NNP JJ	“. Need third”
Gemma-3-27B-it	$. RB$	“ $. Then$ ”
( Gemma Team and others, 2025 )	$ . PRP VBP	“$ . We want”
Llama-3.3-70B-Instruct	. # # NNP CD	“. # # Step 2”
( Grattafiori and others, 2024 )	CD : VB DT NN	“2 : Apply the condition”

Appendix

Table 3: Candidate signatures (OMI). The two highest-ranked non-overlapping PoS

n

-grams of each candidate against the mean of the other eighteen, each with one matching span.

Candidate	Signature	Example
GPT-OSS-120B	. RB JJ NN	“. So -a cos”
( OpenAI, 2025a )	) . NNP NN	“} . Define c”
Gemma-3-27B-it	$. VB$	“ $. Let$ ”
( Gemma Team and others, 2025 )	$. RB$	“ $. Then$ ”
Llama-3.3-70B-Instruct	. # # NNP CD	“. # # Step 2”
( Grattafiori and others, 2024 )	CD : VB DT NN	“8 : Use the fact”

Appendix

Table 4: Candidate signatures (s1). The two highest-ranked non-overlapping PoS

n

-grams of each candidate against the mean of the other eighteen, each with one matching span.

Candidate	Markup	Signature	Example	Shared
OMI probe set
GPT-OSS-120B	kept	. NNP JJ	“. Need third”	6/10
		RB NN :	“Thus answer :”
	removed	RB NN :	“Thus answer :”
		. NNP JJ	“. Need third”
Llama-3.3-70B-Instruct	kept	# # NNP CD	“# # Step 1”	0/10

Appendix

Table 5: Removing markup re-tags a signature rather than erasing it. For each candidate in the controlled pool, we rank patterns against the mean of the other three using raw text and text with Markdown and math markup removed. The last column counts patterns common to the two top-ten lists.

Panel of Figure 8 — Figure 8: Teacher-directed margin changes across text windows and preprocessing. Each point reports mean $\Delta G_{T}$ over nineteen controlled students under $\lambda=0.15$ . Filled and open markers denote raw and canonicalized text, respectively. Random-window curves average ten seeds. All nineteen students have positive $\Delta G_{T}$ in every condition. Panel (a) combines $D$ and $C$ because centering cancels in $\Delta G_{T}$ , while panel (b) applies candidate-specific scaling. The legend in (a) applies to both panels; their vertical scales are not comparable.

		Profile $D$ and Centered $C$				SCOUT
Preprocessing	Window	500	1000	1500	2000	500	1000	1500	2000
Raw	start	0.2674	0.2982	0.3225	0.3404	15.1212	14.8995	13.7677	13.0675
	random	0.1846	0.2472	0.2839	0.3097	6.9287	8.3123	8.8714	9.1461
	end	0.1795	0.2249	0.2620	0.2891	5.3654	5.7042	6.5654	7.2521
Canonicalized	start	0.2683	0.3021	0.3233	0.3377	11.9578	10.7116	10.9529	10.7779
	random	0.1838	0.2425	0.2762	0.3002	6.0996	7.3841	8.3427	8.9223

Appendix

Table 6: Trajectory-margin changes across text windows and preprocessing. Each entry reports mean

\Delta G_{T}

over nineteen controlled students. All nineteen have positive

\Delta G_{T}

in every condition under

D

C

, and SCOUT.

D

and

C

are identical for

\Delta G_{T}

, whereas SCOUT additionally applies candidate-specific scaling, so magnitudes should be compared only within each readout. Random-window means are averaged across ten seeds, and the

19/19

result holds under the minimum count across seeds.

Panel of Figure 9 — Figure 9: Controlled students reproduce teacher-associated syntactic patterns. Each cell shows the share of a candidate’s top-ten PoS $n$ -grams observed in held-out student responses, normalized by the corresponding share in the candidate’s own responses. Signatures are selected after candidate-contrast filtering with $\lambda=0.15$ . Rows are candidates, columns are students grouped by true teacher, and black outlines mark each student’s true teacher. Student labels G4, L3, Q1.5, Q3, F7, L8, M7, and Q7 denote Gemma-3-4B-PT, Llama-3.2-3B-Instruct, Qwen-2.5-1.5B, Qwen-2.5-3B, Falcon3-7B-Base, Llama-3.1-8B, Mistral-7B-v0.3, and Qwen-2.5-7B, respectively. Circles and diamonds distinguish the two probes used in each cohort.

Cohort	Source	Probe	Signature	Example
1.5–4B	GPT-OSS-120B	OMI	. NNP NN	“. Compute b”
			) . RB JJ	“) . Similarly median”
		s1	. RB JJ	“. Similarly \”
			. NNP NN	“. Compute product”
	Llama-3.3-70B-Instruct	OMI	NNP CD : VB DT	“Step 2 : Recall the”
			. # # NNP CD	“. # # Step 2”

Appendix

Table 7: Student signatures. Controlled students grouped by source, against the students of the other two sources on the same base models, with format scaffolding removed.

Figure 10: Sensitivity to signature size and candidate-pool composition. The fraction of controlled students whose true source ranks first as the number of signature PoS

n

-grams varies. Solid lines with circles use the four-candidate pool, whereas dotted lines with squares use the nineteen-candidate pool. The vertical line marks the default choice,

k=10

Additional scale cohort
Cohort	Measure	$\lambda=0$	$0.05$	$\mathbf{0.10}$	$\mathbf{0.15}$	$0.25$	$0.50$	$0.75$
Math 1.5–4B (19)	$\mathrm{Det.}$	19/19	15/19	18/19	18/19	18/19	17/19	17/19
	$(\mathrm{FP}(S),\mathrm{FP}(T))$	(0, 0)	(0, 0)	(0, 0)	(0, 0)	(13, 19)	(0, 0)	(0, 0)
Conversation (19)	$\mathrm{Det.}$	8/19	12/19	15/19	15/19	15/19	15/19	16/19
	$(\mathrm{FP}(S),\mathrm{FP}(T))$	(2, 2)	(2, 0)	(0, 0)	(0, 0)	(0, 4)	(2, 1)	(2, 0)
Math 7–8B (21)	$\mathrm{Det.}$	19/21	15/21	19/21	18/21	19/21	18/21	18/21

Appendix

Table 8: Sensitivity to

\lambda

in the controlled cohorts.

\mathrm{Det.}

counts students correctly attributed on both probes with the teacher present. Each ordered pair

(\mathrm{FP}(S),\mathrm{FP}(T))

gives the numbers of false acceptances after teacher removal under student-backbone and teacher holdout, respectively. Bold headers mark the only values yielding no false acceptances under either protocol in all three cohorts.

Method	Probe	$\lambda=0$	$0.05$	$\mathbf{0.10}$	$\mathbf{0.15}$	$0.25$	$0.50$
SCOUT	Math / OMI	12/12	12/12	12/12	12/12	12/12	12/12
	Math / s1	12/12	12/12	12/12	12/12	12/12	12/12
	Conversation / OASST1	1/12	5/12	12/12	12/12	1/12	1/12
	Conversation / Dolly	3/12	12/12	12/12	12/12	12/12	1/12
Profile distance $D$	Math / OMI	11/12	12/12	12/12	12/12	12/12	12/12
	Math / s1	12/12	12/12	12/12	12/12	12/12	12/12

Appendix

Table 9: Sensitivity to

\lambda

after subsequent training. Each entry counts DeepSeek-R1 assignments among the twelve descendants in Section 5.2 . Math and conversation probes use their corresponding ten-candidate pools. Profile distance

D

uses the same filtered representations without candidate-pool calibration. Bold headers mark the values highlighted by the controlled cohorts.

Backbone	Training set	GPT-OSS	Qwen-3	Llama-3.3
Gemma-3-4B-PT	OMI	0.087	0.169	—
	s1	0.146	0.157	0.191
Llama-3.2-3B-Instruct	OMI	—	—	0.192
	s1	—	—	0.228
Qwen2.5-1.5B	OMI	0.108	0.179	0.139
	s1	0.169	0.081	0.176

Appendix

Table 10: Source-signature formation under common teacher decoding. Entries report the source-directed increment

I_{T}

on the held-out probe. Dashes denote combinations outside the retained nineteen-student cohort.

	Temperature
Backbone	$T=0.3$	$T=0.7$	$T=1.0$	Range
Gemma-3-4B-PT	0.0839	0.0865	0.0937	0.0098
Qwen2.5-1.5B	0.0602	0.0777	0.0697	0.0176
Qwen2.5-3B	0.0833	0.0988	0.0845	0.0155
Mean range				0.0143
	GPT-OSS	Qwen-3	Llama-3.3	Range

Appendix

Table 11: Effects of teacher- and candidate-side decoding. The upper block reports

G_{T}^{(D)}

for students trained on GPT-OSS responses at three temperatures. The middle block compares its mean across teachers under common decoding. In the lower block, Released uses the original candidate bank, while Default and Common replace four open-weight competitors with regenerated responses. Entries count correct source identifications in thirteen public suspect–probe cells using profile readouts with

\lambda=0.15

. The s1.1-32B cell on s1 is excluded because its fine-tuning data contain the probe prompts. Flips counts verdict changes between default and common candidate decoding.

Candidate	Raw chars	Retained chars	At cap (%)	Sentence chars
GPT-OSS-120B	2,877	2,000	63.5	54.7
Qwen-3-8B	6,036	2,000	100.0	62.5
Llama-3.3-70B-Instruct	1,851	1,851	44.5	117.6
Gemma-3-27B-it	2,013	1,993	50.0	80.2
Max/min ratio	3.26	1.08	—	2.15

Appendix

Table 12: Response-length controls. The upper block reports candidate-response statistics on OMI before and after the

2{,}000

-character cap; entries are medians except At cap , which gives the percentage of responses reaching the cap. Sentence chars reports median sentence length within the retained window. The lower block compares three length-only readouts with SCOUT on the same nineteen controlled students. SCOUT uses

\lambda=0.15

, and lower-block entries are percentages.

Method	Aggregation	5	10	20	50	100	110	120	130	140	150	160	175	190	200	$n^{\star}$
OMI
Reference-based
DistillDetect	Original evaluator	0.9885	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	5
DistillDetect	Mean-score attribution	0.9585	0.9905	0.9995	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	5
Output-only
PoS Templates	Original evaluator	0.0115	0.0095	0.0025	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	—

Appendix

Table 13: Prompt-budget curves on math probes. Each value for

B<200

is the fraction of

2{,}000

prompt subsets on which all twelve descendants are assigned to R1. Under the original evaluator, each descendant is correct when either mean-score attribution or sorted-score voting assigns it to R1, and a subset succeeds only when all twelve descendants are correct. Mean-score attribution uses the lowest average candidate score alone. At

B=200

, the complete prompt bank is evaluated once.

n^{\star}

is the first evaluated budget reaching

0.95

. Dashes indicate that this level is not reached.

Method	Aggregation	5	10	20	50	100	110	120	130	140	150	160	175	190	200	$n^{\star}$
OASST1
Reference-based
DistillDetect	Original evaluator	0.0740	0.0275	0.0060	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	—
DistillDetect	Mean-score attribution	0.0540	0.0220	0.0035	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	—
Output-only
PoS Templates	Original evaluator	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	—

Appendix

Table 14: Prompt-budget curves on conversation probes. Entries follow Table 13 . Each value for

B<200

is the fraction of

2{,}000

prompt subsets on which all twelve descendants are assigned to R1. At

B=200

, the complete prompt bank is evaluated once. Dashes indicate that the

0.95

criterion is not reached.

Panel of Figure 11 — Figure 11: Source switching under a second SFT round. (a) SCOUT attribution to $B$ across $36$ switch arms, with no switches among the $18$ same-source controls. (b) Recovery of $A$ across $18$ arms after removing $B$ . The dashed line marks chance. (c) Median endpoint accuracy change when moving to a stronger or weaker source.

Reference-based
Method	$1\%$	$2\%$	$5\%$	$10\%$	$15\%$	$20\%$	$25\%$	$100\%$
DistillDetect	0	0	2	11	12	14	14	14
Output-only
PoS Templates	0	0	6	14	16	18	18	18
SCOUT (Ours)	0	0	7	15	17	17	17	18
$-$ Scaling ( $C$ )	0	0	7	15	17	17	17	18

Appendix

Table 15: Attribution to the second-round source during source switching. Entries count the

18

switch arms attributed to

B

after each percentage of the second SFT round. Higher counts indicate stronger redirection, not better attribution. DistillDetect uses the pre-distillation base as its reference.

Math, 1.5–4B students
Method	AUC [95% CI]	Cells	Students
DistillDetect	0.8587 [0.7895, 0.9501]	26/38	13/19
PoS Templates	0.7230 [0.5346, 0.8726]	15/38	11/19
SCOUT (Ours)	1.0000 [1.0000, 1.0000]	37/38	19/19
$-$ Scaling ( $C$ )	1.0000 [1.0000, 1.0000]	33/38	19/19
$-$ Centering ( $D$ )	1.0000 [1.0000, 1.0000]	35/38	19/19

Appendix

Table 16: Controlled detection ablation. AUC is computed from student-level detection and false-positive rates. Confidence intervals are obtained by bootstrapping students, with all evaluation cells belonging to each sampled student retained together. At zero false positives on the corresponding aggregation axis, Cells reports the maximum number of correctly accepted source-present probe cells, whereas Students reports the maximum number of students correctly accepted on both probes. The two columns may use different thresholds. Higher is better.

Math, 1.5–4B students
Method	Source present $p^{\star}$ range	Source removed backbone-holdout minimum $p^{\star}$	Source removed teacher-holdout minimum $p^{\star}$
DistillDetect	$8.35{\times}10^{-164}$ – $1.00$	$4.18{\times}10^{-60}$	$1.32{\times}10^{-78}$
PoS Templates	$2.19{\times}10^{-34}$ – $1.00$	$8.22{\times}10^{-29}$	$1.43{\times}10^{-34}$
SCOUT (Ours)	$3.85{\times}10^{-92}$ – $1.00$	$0.402$	$1.02{\times}10^{-6}$
Conversation, 1.5–4B students
DistillDetect	$6.44{\times}10^{-130}$ – $1.00$	$1.11{\times}10^{-82}$	$7.19{\times}10^{-77}$

Appendix

Table 17: Adjusted significance values underlying controlled detection. The source-present column reports the range of

p^{\star}=\min\{1,(K-1)p\}

, where smaller values provide stronger evidence for correct acceptance. The source-removed columns report the minimum under backbone and teacher holdout. A source-removed value below

0.05

identifies a significant gap in one probe cell but does not itself constitute a false acceptance, which also requires the fold-fitted threshold and

p^{\star}<0.05

on both probes with the same selected candidate. LLM judges are omitted because they do not produce score gaps or fitted thresholds.

Reference-based
Method	Original Pool	Expanded Pool
DistillDetect, ref. $=S^{(0)}$	7/7	6/7
Output-only baselines
PoS Templates	5/7	1/7
LLM judge (Mistral)	6/7	1/7
LLM judge (Solar)	7/7	1/7

Appendix

Table 18: Candidate-pool stress test with source relatives. The original pool contains ten candidates. For each suspect, the expanded pool adds the other six models from the source-relative set described above, giving sixteen candidates. Each entry counts DeepSeek-R1 selections among seven suspects. DistillDetect uses the corresponding pre-distillation checkpoint. Higher is better.

(a) Thirteen public ladders
Model or ladder	$S^{(0)}$	$S^{(1)}$	$S^{(2)}$	$S^{(3)}$	$S^{(4)}$
DeepSeek-R1
OLMo 3 7B	Base	Dolci-Think SFT	DPO	RLVR	—
OLMo 3 32B	Base	Dolci-Think SFT	DPO	RLVR	—
Skywork-OR1 32B	Base	R1 distillation	RL	—	—
AceReason 14B	Base	R1 distillation	RL	—	—

Appendix

Table 19: Training stages in the checkpoint trajectories.

S^{(0)}

is each lineage’s pre-distillation checkpoint, and later columns name the training applied after the preceding checkpoint. Source headings indicate the designated teacher used for the reported margin rather than every contributor to the training mixture. Gray rows distinguish source and lineage anchors from their subsequent checkpoints. Dashes indicate that the trajectory ends.

Panel of Figure 12 — Figure 12: Source-margin shifts from the pre-distillation state on s1. Panels (a) , (b) , and (c) correspond to the OMI results in Figure 7 and show thirteen public ladders, two R1-distilled lineages and their descendants, and four post-training branches of Dream-7B, respectively. Positive values indicate a larger margin toward the indicated source than at $S^{(0)}$ . Every post-distillation shift is positive, reproducing the OMI pattern. Shading marks distillation in (b) and (c) . Panel scales differ.

Setting	Value
Objective	Completion-only causal language modeling
Fine-tuning method	Full-parameter fine-tuning
Epochs	3
Learning rate	$1\times 10^{-5}$
Schedule	Cosine with warmup ratio $0.05$
Effective batch size	16 (batch size 4, accumulation 4)

Appendix

Table 20: Controlled SFT configuration.

Language Model Auditing

Language model auditing tests whether models behave reliably, fairly, and safely using behavioral probes and internal or deployment evidence.

See all 158 papers

Jun 19, 2026

Reference-Based Distillation Detection in LLMs

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Rajat Rawat, Sizhe Chen, Akshay Anand +3

University of California, Berkeley · University of Southern California · OpenAI

LLM Auditing Teacher-Student Learning Knowledge Distillation

Ask PDF

Oct 8, 2026

Poster: A Preliminary Study of LLM Distillation Inference

Unauthorized model distillation, in which a model is trained on the outputs of a proprietary large language model (LLM), is a growing threat to model providers. We study distillation inference: determining whether a suspect model was distilled from another model or trained independently. We formulate this problem as a hypothesis test and estimate the behavior expected under each hypothesis by training shadow models: distilled shadow models learn from the teacher's reasoning traces, whereas independent shadow models learn only from reference answers. The auditor measures how closely each model predicts the teacher's reasoning outputs and then uses the shadow models to convert the suspect's score into a calibrated p-value. In a preliminary study using Qwen2.5-7B as the teacher and Llama-3.2-3B for the suspects, our test achieves a true positive rate of 1.0 at a significance level of 0.02. These results demonstrate the feasibility of using distillation inference to detect distillation attacks.

Edward Chen, Yuntao Du

Carmel High School Carmel, Indiana, USA · Purdue University West Lafayette, Indiana, USA

LLM Auditing Language Model Distillation

Ask PDF

Feb 16, 2026

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting

Knowledge distillation is a widely adopted technique for transferring capabilities from LLMs to smaller, more efficient student models. However, unauthorized use of knowledge distillation takes unfair advantage of the considerable effort and cost put into developing frontier models. We investigate methods for modifying teacher-generated reasoning traces to achieve two objectives that deter unauthorized distillation: (1) anti-distillation, or degrading the training usefulness of query responses, and (2) API watermarking, which embeds verifiable signatures in student models. We introduce several approaches for dynamically rewriting a teacher's reasoning outputs while preserving answer correctness and semantic coherence. Two of these leverage the rewriting capabilities of LLMs, while others use gradient-based techniques. Our experiments show that a simple instruction-based rewriting approach achieves a strong anti-distillation effect while maintaining or even improving teacher performance. Furthermore, we show that our rewriting approach also enables embedding watermarks that can be reliably detected with essentially no false alarms. Our code is available at https://github.com/xhOwenMa/trace-rewriting.

Xinhang Ma, William Yeoh, Ning Zhang +1

Washington University in St. Louis

LLM Watermarking Knowledge Distillation

Ask PDF

Abstract

Language Model Auditing

Explore similar work

Reference-Based Distillation Detection in LLMs

Poster: A Preliminary Study of LLM Distillation Inference

Protecting Language Models Against Unauthorized Distillation through Trace Rewriting