Poster: A Preliminary Study of LLM Distillation Inference

Edward Chen; Yuntao Du

cs.CROct 8, 2026

Poster: A Preliminary Study of LLM Distillation Inference

Authors: Edward Chen, Yuntao Du

Organizations: Carmel High School Carmel, Indiana, USA · Purdue University West Lafayette, Indiana, USA

Abstract

Unauthorized model distillation, in which a model is trained on the outputs of a proprietary large language model (LLM), is a growing threat to model providers. We study distillation inference: determining whether a suspect model was distilled from another model or trained independently. We formulate this problem as a hypothesis test and estimate the behavior expected under each hypothesis by training shadow models: distilled shadow models learn from the teacher's reasoning traces, whereas independent shadow models learn only from reference answers. The auditor measures how closely each model predicts the teacher's reasoning outputs and then uses the shadow models to convert the suspect's score into a calibrated p-value. In a preliminary study using Qwen2.5-7B as the teacher and Llama-3.2-3B for the suspects, our test achieves a true positive rate of 1.0 at a significance level of 0.02. These results demonstrate the feasibility of using distillation inference to detect distillation attacks.

Figures & tables

Figure 1. Per instance (left), the log-probability signal overlaps across the two classes, so membership inference cannot separate them (AUC $0.67$ ). Aggregated to one statistic $\Lambda$ per model (right), the distilled ( $H_{1}$ ) and independent ( $H_{0}$ ) populations are fully separated, so our methods can faithfully detect distillation. Two panels. Left: the per-instance log-probability histograms of independent and distilled shadow models overlap heavily (AUC 0.67). Right: the per-model statistic Lambda forms two well-separated bell curves for the independent and distilled populations, with a gap of about 17 standard deviations.

Signal		AUC	bal-acc	TPR@5%FPR	TPR@0%FPR
log-prob	per-instance (MIA)	0.672	0.518	0.086	0.000
log-prob	per-model (ours)	1.000	1.000	1.000	1.000
entropy	per-instance (MIA)	0.672	0.524	0.098	0.000
entropy	per-model (ours)	1.000	1.000	1.000	1.000
token agreement	per-instance (MIA)	0.635	0.512	0.073	0.003
token agreement	per-model (ours)	1.000	1.000	1.000	1.000

Table 1. Membership inference on individual instances barely separates distilled from independent models, whereas aggregating the same signal across the audit set (dataset inference, our method) separates the model populations perfectly. bal-acc = balanced accuracy.

$M$	$p$ -floor	FPR
5	0.167	0.200
10	0.091	0.100
20	0.048	0.050
40	0.024	0.025
50	0.020	0.020

Table 2. More shadow models tighten the certified operating point: the

p

-value floor

1/(M{+}1)

and the FPR (

\approx 1/M

) both shrink with

M

, while per-model detection stays perfect (AUC

=1.000

throughout).

Explore similar work

Reference-Based Distillation Detection in LLMs
Jun 19, 2026Rajat Rawat, Sizhe Chen, Akshay Anand +3LLM Auditing Teacher-Student Learning
University of California, Berkeley · University of Southern California · OpenAI
Distillation Defenses Easily Break After Reinforcement Learning
Sep 28, 2026Shidan Javaheri, Alexander Panfilov, Oliver Britton +2LLM Security Language Model Distillation
University of Oxford · ELLIS Institute Tübingen & MPI for Intelligent Systems
Asking Back: Interaction-Layer Antidistillation Watermarks
May 15, 2026Guang Yang, Amir Ghasemian, Fengchen Liu +3LLM Auditing LLM Watermarking
University of California, Los Angeles · Lawrence Berkeley National Laboratory · Meta

Jun 19, 2026, cs.LGJ/K move · Enter open · S save

Reference-Based Distillation Detection in LLMs

Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan+2

University of California, Berkeley · University of Southern California · OpenAI

LLM Auditing Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Abstract

Explore similar work

Reference-Based Distillation Detection in LLMs

Distillation Defenses Easily Break After Reinforcement Learning

Asking Back: Interaction-Layer Antidistillation Watermarks