Knowledge distillation is the standard way to compress encrypted-traffic classifiers for the edge, and almost all such work judges students by accuracy alone. We ask what else a student inherits: unknown-traffic detection, calibration, shortcut reliance, and whether any survives a year of drift. Resemblance proves little on its own, since soft targets also regularise. We therefore distil one 101k-parameter student from two teachers of equal accuracy but different construction, a five-member ensemble and a single wider model, so that following one rather than the other is attributable to it. The design was pre-registered before any test result was seen. We tested ten hypotheses on CESNET-TLS-Year22, a year of real TLS traffic, across 18 test windows over 35 weeks. Two are supported: a student's per-flow unknown-scores shift toward its own teacher, but only at a conventional temperature, not the accuracy-optimal one; and a shortcut-reliant teacher passes its over-confidence to a student that never sees the feature. The drift prediction is reversed under both scores, the gap narrowing rather than widening and the student overtaking under the energy score in two of three replicates, as is the prediction that such a teacher harms its student's detection, which improves slightly. Shortcut reliance is set by model size, not distillation. Under the logit-based scores nothing else transfers: distillation beats neither a temperature-scaled direct student nor label smoothing. Exploratory analysis shows this turns on the scoring rule: with a feature-space detector the teacher detects unknown traffic 0.073 AUROC better than the direct student, where the energy score sees 0.000, and the conventional-temperature student inherits most of it. Label smoothing, with no teacher, recovers more. Distillation transfers the teacher's habits; what looks like an inherited ability is available without one.
Figures & tables
Fig. 1: The design. (a) The three replicates: four training weeks, one validation week, then consecutive four-week test windows. Shading marks the weeks that more than one start date tests (Section V-E ). (b) The teacher-swap control.
Condition
Teacher
Objective
direct
—
cross-entropy
directTS
—
direct , then post-hoc temperature scaling
ls
—
label smoothing, ε=0.05
kdA
A
Hinton KD, T=1 (accuracy-tuned)
kdB
B
Hinton KD, T=1
kdA4
A
Hinton KD, T=4 (conventional)
TABLE I: Student training conditions. The upper block is pre-registered and runs at all three start dates; the lower block is exploratory, added after the confirmatory results were known.
Model
Params
Macro-F1
AUROC energy
AUROC MSP
FPR@95
OSCR
NLL
AURC
Teacher A (5 × mm_cesnet_v2)
11.3M
0.860
0.836
0.868
0.626
0.789
0.447
0.012
Teacher B (wide)
10.3M
0.855
0.840
0.851
0.614
0.791
0.480
0.014
direct
101k
0.806
0.835
0.844
0.674
0.768
0.609
0.020
directTS
101k
0.806
0.835
0.845
0.676
0.767
—
0.020
ls ( ε=0.05 )
101k
0.797
0.814
0.836
0.683
0.747
0.561
0.023
kdA ( T=1 )
101k
0.806
0.834
0.843
0.667
0.767
0.601
0.020
TABLE II: Test-window averages (18 windows, all flows, all test-unknown services). Higher is better except FPR, NLL and AURC. NLL is post-hoc (temperature-scaled).
AUROC vs ls +0.021 , vs directTS 0.000 ; NLL vs ls −0.040 , vs directTS +0.008
1.0
not supported
H2[MSP]
as H2, MSP
AUROC vs ls +0.006 , vs directTS −0.002
1.0
not supported
H3[energy]
gap grows with time
−0.0004 per week ( −0.0007 , −0.0001 )
1.0
reversed
H3[MSP]
as H3, MSP
−0.0003 per week ( −0.0004 , −0.0001 )
1.0
reversed
H4a
KD student relies on a visible shortcut more
−0.032 (lower bound −0.071 )
1.0
not supported
TABLE III: The ten pre-registered hypotheses. Estimates are pooled over the 18 test windows; intervals are 95% percentile intervals from 1,000 day × service cluster bootstrap resamples; p -values are Holm-corrected over these ten hypotheses and no others.
Fig. 2: Inheritance of the per-flow score pattern. (a) Spearman correlation of a student’s energy scores with each teacher’s, averaged within each start date (dots) and then across the three. (b) The quantity H1 tests, pooled over the 18 test windows: each distilled student’s own-minus-other correlation difference, minus that of direct , with 95% cluster-bootstrap intervals.
Model
Near unknowns
Far unknowns
Teacher A
0.805 / 0.836
0.840 / 0.872
Teacher B
0.788 / 0.794
0.848 / 0.859
direct , directTS , kdA
0.846 / 0.824
0.833 / 0.847
ls
0.795 / 0.809
0.816 / 0.840
kdA4
0.788 / 0.794
0.812 / 0.829
enddA
0.695 / 0.816
0.749 / 0.838
TABLE IV: Unknown detection by distance of the unknown services (test windows; energy AUROC / MSP AUROC). The three conditions sharing a row agree to within 0.002 on every entry; the row gives direct .
Weeks
Teacher A
kdA
Gap
kdA macro-F1
3.5
0.869
0.854
+0.015
0.902
7.5
0.863
0.858
+0.005
0.880
11.5
0.841
0.846
−0.005
0.848
19.5
0.815
0.818
−0.003
0.747
27.5
0.803
0.806
−0.003
0.724
35.3
0.784
0.793
−0.008
0.687
TABLE V: Start date 11: energy AUROC against all test-unknown services, and the student’s closed-set macro-F1, by weeks since the end of training. Six of this start date’s nine windows are shown.
Fig. 3: Teacher-minus-student energy AUROC against weeks since the end of training, per condition and start date: (a) the conditions whose gap closes and crosses zero; (b) those whose gap does not. The panels use different vertical scales. (c) Student macro-F1 over the same axis. Line style marks the start date.
ρ
Teacher (2.3M)
direct student
KD student ( T=4 )
0
0.000
0.000
0.000
0.5
0.037
0.069
0.068
0.9
0.103
0.185
0.196
1.0
0.800
0.945
0.870
TABLE VI: Flip-test reliance on the planted shortcut (macro-F1 aligned − flipped; validation week of start date 11, mean of 3 seeds).
Fig. 4: Flip-test reliance against the planted feature’s reliability ρ during training, for the teacher and the two students that see it: (a) the full range; (b) ρ≤0.9 on its own scale. Lines are the mean of three seeds, bands the seed range.
Model
Parameters
Weights (MB)
Latency p50 (ms)
Latency p99 (ms)
Flows/s, 1 thread
Flows/s, all cores
Student, ONNX Runtime
101k
0.4
0.082
0.160
21,201
—
Student, PyTorch
101k
0.4
0.501
1.026
10,604
39,418
Teacher A, one member
2.3M
9.1
3.560
5.348
812
3,875
Teacher A, 5-member ensemble
11.3M
45.4
19.093
27.357
161
789
Teacher B
10.3M
41.3
12.262
17.213
218
1,153
XGBoost (flow statistics)
—
244.4
41.606
53.011
1,369
11,078
TABLE VII: Inference cost on a laptop CPU (AMD Ryzen 7 5800H, 8 cores) as a stand-in for an edge device. Latency is per flow at batch size 1 on one thread; throughput is at batch size 1,024. All rows come from one timing session, so they are comparable with each other; the re-tuned XGBoost of Section IV was timed separately and is quoted against its own session’s student.
Comparison
Energy
MSP
Mahalanobis
Feature k -NN
direct_w16
−0.024† ( −0.033 , −0.018 )
−0.008† ( −0.017 , +0.005 )
—
—
direct_w96
+0.002† ( −0.002 , +0.005 )
−0.003† ( −0.006 , −0.001 )
—
—
directTS_w16
−0.024† ( −0.033 , −0.018 )
−0.007† ( −0.017 , +0.005 )
—
—
directTS_w96
+0.001† ( −0.003 , +0.004 )
+0.000† ( −0.002 , +0.003 )
—
—
kdA4_w16
−0.054† ( −0.062 , −0.045 )
−0.051† ( −0.054 , −0.048 )
—
—
kdA4_w96
−0.016† ( −0.020 , −0.012 )
−0.015† ( −0.018 , −0.013 )
—
—
TABLE VIII: Detection advantage of the exploratory conditions. A row naming one condition is that condition minus direct ; a row naming two is their paired difference. A dash marks a score that was not computed. Intervals are 95% cluster bootstrap intervals over the 18 test windows, except for cells marked † , which pool over the nine windows of start date 11 because the condition was trained, or in the feature-space columns scored, there alone. A marked cell and an unmarked one are measured on different calendars and should not be read against each other.
Comparison
Units
Shift (95% CI)
H1 replication
kdA4 : A vs B
18
+0.0071 ( +0.0046 , +0.0091 )
kdB4 : B vs A
18
+0.0255 ( +0.0235 , +0.0280 )
single vs single
kdM0 : member 0 vs B
18
+0.0171 ( +0.0150 , +0.0193 )
identity only
TABLE IX: Teacher-swap shifts, exploratory. Each row is a difference in differences against direct , so zero means no preference for the own teacher. Intervals are 95% cluster bootstrap intervals over the units shown.
Model
Energy
MSP
Teacher A (ensemble)
−0.00285
−0.00306
member mean
−0.00285
−0.00282
teacherA_0
−0.00278
−0.00282
teacherA_1
−0.00306
−0.00291
teacherA_2
−0.00261
−0.00270
teacherA_3
−0.00289
−0.00283
TABLE X: Detection ageing of the teacher ensemble and of its members (exploratory): slope of unknown-detection AUROC per week since training, one intercept per start date.
Start
Windows
First week
Last week
Shared unit-weeks
11
9
16
51
23
24
6
29
51
23
37
3
42
51
9
TABLE XI: Calendar overlap between the three start dates’ test windows (Fig. 1 (a)): 15 of the 18 units share at least one test week with another start date, 55 unit-weeks in total.
Fig. 5: Unknown-traffic detection advantage over the directly trained student, by scoring rule (exploratory; 18 test windows): (a) the pre-registered logit-based scores; (b) the feature-space scores. Markers are pooled estimates, bars are 95% cluster bootstrap intervals, and both panels share one scale. The top block compares models of different feature dimensionality and is indicative; the lower blocks compare identical architectures.
Mahmoud Abbasi received the M.Sc. degree in computer software engineering from Islamic Azad University, Mashhad, Iran, in 2017, and the Ph.D. degree in computer engineering from the University of Salamanca, Salamanca, Spain, in 2026, where the doctoral work was supported by a Marie Skłodowska-Curie Fellowship within the EU Horizon 2020 IoTalentum network (grant no. 953442). Since 2026, Dr. Abbasi has been a Researcher with the AIR Institute, Salamanca, Spain. Research interests include machine learning for network traffic monitoring and analysis, intent-based networking, and secure, scalable Internet-of-Things systems. Dr. Abbasi has been a Member of IEEE since 2019.
Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.
Rajat Rawat, Sizhe Chen, Akshay Anand +3
University of California, Berkeley · University of Southern California · OpenAI
This position paper argues that knowledge distillation must account for what it loses: student models should be judged not only by retained task scores, but by whether they preserve the teacher capabilities that make those scores reliable. This matters because distillation is increasingly used to turn large teacher models into deployable students, yet headline metrics can obscure losses in the capabilities that make teacher behavior reliable. Conceptually, we show that current evaluation often assumes retained task scores imply retained teacher capabilities. Reframing distillation as a lossy projection exposes this flaw: students may match selected teacher observables without preserving the capabilities that make them reliable. We then synthesize existing evidence into a taxonomy of off-metric distillation losses, showing that such losses are concrete, recurring, and measurable, yet often unaccounted for when studies report what students retain rather than what they lose. To make the position actionable, we propose scenario-specific preservation targets and a Distillation Loss Statement that reports what was preserved, what was lost, and why the remaining losses are acceptable. The goal is not lossless distillation, but accountable distillation.
Wenshuo Wang
School of Future Technology, South China University of Technology, China
Knowledge distillation is widely used to transfer capabilities from teacher models to student models, but recent work shows that it can also transfer unintended teacher properties. We study whether different hidden teacher properties survive the same distillation channels. Specifically, we compare two properties: benchmark-associated knowledge and refusal backdoors, and show that distillation treats them very differently. Teachers fine-tuned on benchmark data successfully pass that knowledge to students through all four distillation channels we test, including when students are trained only on teacher-generated number sequences containing no natural language or domain content whatsoever. Refusal backdoors tell the opposite story: backdoored teachers achieve near-perfect triggered refusal, yet sequence-distilled students inherit essentially none of this behaviour. Logit-based distillation transfers it substantially, reaching up to 85.5% triggered refusal in students that saw zero backdoor examples. Distillation is therefore selective, and the selection criterion is not what data the student sees, but what structure the teacher's property has. The code is available at https://github.com/MJonibek/kd-leakage.