Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year

Mahmoud Abbasi

cs.NISep 25, 2026

Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Year

Authors: Mahmoud Abbasi

Organizations: University of Salamanca, Salamanca, Spain · AIR Institute, Salamanca, Spain

Abstract

Knowledge distillation is the standard way to compress encrypted-traffic classifiers for the edge, and almost all such work judges students by accuracy alone. We ask what else a student inherits: unknown-traffic detection, calibration, shortcut reliance, and whether any survives a year of drift. Resemblance proves little on its own, since soft targets also regularise. We therefore distil one 101k-parameter student from two teachers of equal accuracy but different construction, a five-member ensemble and a single wider model, so that following one rather than the other is attributable to it. The design was pre-registered before any test result was seen. We tested ten hypotheses on CESNET-TLS-Year22, a year of real TLS traffic, across 18 test windows over 35 weeks. Two are supported: a student's per-flow unknown-scores shift toward its own teacher, but only at a conventional temperature, not the accuracy-optimal one; and a shortcut-reliant teacher passes its over-confidence to a student that never sees the feature. The drift prediction is reversed under both scores, the gap narrowing rather than widening and the student overtaking under the energy score in two of three replicates, as is the prediction that such a teacher harms its student's detection, which improves slightly. Shortcut reliance is set by model size, not distillation. Under the logit-based scores nothing else transfers: distillation beats neither a temperature-scaled direct student nor label smoothing. Exploratory analysis shows this turns on the scoring rule: with a feature-space detector the teacher detects unknown traffic 0.073 AUROC better than the direct student, where the energy score sees 0.000, and the conventional-temperature student inherits most of it. Label smoothing, with no teacher, recovers more. Distillation transfers the teacher's habits; what looks like an inherited ability is available without one.

Figures & tables

Fig. 1: The design. (a) The three replicates: four training weeks, one validation week, then consecutive four-week test windows. Shading marks the weeks that more than one start date tests (Section V-E ). (b) The teacher-swap control.

Condition	Teacher	Objective
direct	—	cross-entropy
directTS	—	direct , then post-hoc temperature scaling
ls	—	label smoothing, $\varepsilon=0.05$
kdA	A	Hinton KD, $T=1$ (accuracy-tuned)
kdB	B	Hinton KD, $T=1$
kdA4	A	Hinton KD, $T=4$ (conventional)

TABLE I: Student training conditions. The upper block is pre-registered and runs at all three start dates; the lower block is exploratory, added after the confirmatory results were known.

Model	Params	Macro-F1	AUROC ${}_{\text{energy}}$	AUROC ${}_{\text{MSP}}$	FPR@95	OSCR	NLL	AURC
Teacher A (5 $\times$ mm_cesnet_v2)	11.3M	0.860	0.836	0.868	0.626	0.789	0.447	0.012
Teacher B (wide)	10.3M	0.855	0.840	0.851	0.614	0.791	0.480	0.014
direct	101k	0.806	0.835	0.844	0.674	0.768	0.609	0.020
directTS	101k	0.806	0.835	0.845	0.676	0.767	—	0.020
ls ( $\varepsilon=0.05$ )	101k	0.797	0.814	0.836	0.683	0.747	0.561	0.023
kdA ( $T=1$ )	101k	0.806	0.834	0.843	0.667	0.767	0.601	0.020

TABLE II: Test-window averages (18 windows, all flows, all test-unknown services). Higher is better except FPR, NLL and AURC. NLL is post-hoc (temperature-scaled).

#	Hypothesis	Estimate (95% CI)	Holm $p$	Outcome
H1	inheritance is teacher-specific	kdA4 $+0.007$ (0.005, 0.009); kdB4 $+0.026$ (0.024, 0.028)	0.009	supported
H2[energy]	KD beats ls , directTS at matched accuracy	AUROC vs ls $+0.021$ , vs directTS $0.000$ ; NLL vs ls $-0.040$ , vs directTS $+0.008$	1.0	not supported
H2[MSP]	as H2, MSP	AUROC vs ls $+0.006$ , vs directTS $-0.002$	1.0	not supported
H3[energy]	gap grows with time	$-0.0004$ per week ( $-0.0007$ , $-0.0001$ )	1.0	reversed
H3[MSP]	as H3, MSP	$-0.0003$ per week ( $-0.0004$ , $-0.0001$ )	1.0	reversed
H4a	KD student relies on a visible shortcut more	$-0.032$ (lower bound $-0.071$ )	1.0	not supported

TABLE III: The ten pre-registered hypotheses. Estimates are pooled over the 18 test windows; intervals are 95% percentile intervals from 1,000 day

\times

service cluster bootstrap resamples;

p

-values are Holm-corrected over these ten hypotheses and no others.

Fig. 2: Inheritance of the per-flow score pattern. (a) Spearman correlation of a student’s energy scores with each teacher’s, averaged within each start date (dots) and then across the three. (b) The quantity H1 tests, pooled over the 18 test windows: each distilled student’s own-minus-other correlation difference, minus that of direct , with 95% cluster-bootstrap intervals.

Model	Near unknowns	Far unknowns
Teacher A	0.805 / 0.836	0.840 / 0.872
Teacher B	0.788 / 0.794	0.848 / 0.859
direct , directTS , kdA	0.846 / 0.824	0.833 / 0.847
ls	0.795 / 0.809	0.816 / 0.840
kdA4	0.788 / 0.794	0.812 / 0.829
enddA	0.695 / 0.816	0.749 / 0.838

TABLE IV: Unknown detection by distance of the unknown services (test windows; energy AUROC / MSP AUROC). The three conditions sharing a row agree to within 0.002 on every entry; the row gives direct .

Weeks	Teacher A	kdA	Gap	kdA macro-F1
3.5	0.869	0.854	$+0.015$	0.902
7.5	0.863	0.858	$+0.005$	0.880
11.5	0.841	0.846	$-0.005$	0.848
19.5	0.815	0.818	$-0.003$	0.747
27.5	0.803	0.806	$-0.003$	0.724
35.3	0.784	0.793	$-0.008$	0.687

TABLE V: Start date 11: energy AUROC against all test-unknown services, and the student’s closed-set macro-F1, by weeks since the end of training. Six of this start date’s nine windows are shown.

Fig. 3: Teacher-minus-student energy AUROC against weeks since the end of training, per condition and start date: (a) the conditions whose gap closes and crosses zero; (b) those whose gap does not. The panels use different vertical scales. (c) Student macro-F1 over the same axis. Line style marks the start date.

$\rho$	Teacher (2.3M)	direct student	KD student ( $T=4$ )
0	0.000	0.000	0.000
0.5	0.037	0.069	0.068
0.9	0.103	0.185	0.196
1.0	0.800	0.945	0.870

TABLE VI: Flip-test reliance on the planted shortcut (macro-F1 aligned

-

flipped; validation week of start date 11, mean of 3 seeds).

Fig. 4: Flip-test reliance against the planted feature’s reliability $\rho$ during training, for the teacher and the two students that see it: (a) the full range; (b) $\rho\leq 0.9$ on its own scale. Lines are the mean of three seeds, bands the seed range.

Model	Parameters	Weights (MB)	Latency p50 (ms)	Latency p99 (ms)	Flows/s, 1 thread	Flows/s, all cores
Student, ONNX Runtime	101k	0.4	0.082	0.160	21,201	—
Student, PyTorch	101k	0.4	0.501	1.026	10,604	39,418
Teacher A, one member	2.3M	9.1	3.560	5.348	812	3,875
Teacher A, 5-member ensemble	11.3M	45.4	19.093	27.357	161	789
Teacher B	10.3M	41.3	12.262	17.213	218	1,153
XGBoost (flow statistics)	—	244.4	41.606	53.011	1,369	11,078

TABLE VII: Inference cost on a laptop CPU (AMD Ryzen 7 5800H, 8 cores) as a stand-in for an edge device. Latency is per flow at batch size 1 on one thread; throughput is at batch size 1,024. All rows come from one timing session, so they are comparable with each other; the re-tuned XGBoost of Section IV was timed separately and is quoted against its own session’s student.

Comparison	Energy	MSP	Mahalanobis	Feature $k$ -NN
direct_w16	$-0.024^{\dagger}$ ( $-0.033$ , $-0.018$ )	$-0.008^{\dagger}$ ( $-0.017$ , $+0.005$ )	—	—
direct_w96	$+0.002^{\dagger}$ ( $-0.002$ , $+0.005$ )	$-0.003^{\dagger}$ ( $-0.006$ , $-0.001$ )	—	—
directTS_w16	$-0.024^{\dagger}$ ( $-0.033$ , $-0.018$ )	$-0.007^{\dagger}$ ( $-0.017$ , $+0.005$ )	—	—
directTS_w96	$+0.001^{\dagger}$ ( $-0.003$ , $+0.004$ )	$+0.000^{\dagger}$ ( $-0.002$ , $+0.003$ )	—	—
kdA4_w16	$-0.054^{\dagger}$ ( $-0.062$ , $-0.045$ )	$-0.051^{\dagger}$ ( $-0.054$ , $-0.048$ )	—	—
kdA4_w96	$-0.016^{\dagger}$ ( $-0.020$ , $-0.012$ )	$-0.015^{\dagger}$ ( $-0.018$ , $-0.013$ )	—	—

TABLE VIII: Detection advantage of the exploratory conditions. A row naming one condition is that condition minus direct ; a row naming two is their paired difference. A dash marks a score that was not computed. Intervals are 95% cluster bootstrap intervals over the 18 test windows, except for cells marked

\dagger

, which pool over the nine windows of start date 11 because the condition was trained, or in the feature-space columns scored, there alone. A marked cell and an unmarked one are measured on different calendars and should not be read against each other.

H1 replication
Comparison	Units	Shift (95% CI)
kdA4 : A vs B	18	$+0.0071$ ( $+0.0046$ , $+0.0091$ )
kdB4 : B vs A	18	$+0.0255$ ( $+0.0235$ , $+0.0280$ )
single vs single
kdM0 : member 0 vs B	18	$+0.0171$ ( $+0.0150$ , $+0.0193$ )
identity only

TABLE IX: Teacher-swap shifts, exploratory. Each row is a difference in differences against direct , so zero means no preference for the own teacher. Intervals are 95% cluster bootstrap intervals over the units shown.

Model	Energy	MSP
Teacher A (ensemble)	$-0.00285$	$-0.00306$
member mean	$-0.00285$	$-0.00282$
teacherA_0	$-0.00278$	$-0.00282$
teacherA_1	$-0.00306$	$-0.00291$
teacherA_2	$-0.00261$	$-0.00270$
teacherA_3	$-0.00289$	$-0.00283$

TABLE X: Detection ageing of the teacher ensemble and of its members (exploratory): slope of unknown-detection AUROC per week since training, one intercept per start date.

Start	Windows	First week	Last week	Shared unit-weeks
11	9	16	51	23
24	6	29	51	23
37	3	42	51	9

TABLE XI: Calendar overlap between the three start dates’ test windows (Fig. 1 (a)): 15 of the 18 units share at least one test week with another start date, 55 unit-weeks in total.

Fig. 5: Unknown-traffic detection advantage over the directly trained student, by scoring rule (exploratory; 18 test windows): (a) the pre-registered logit-based scores; (b) the feature-space scores. Markers are pooled estimates, bars are 95% cluster bootstrap intervals, and both panels share one scale. The top block compares models of different feature dimensionality and is indicative; the lower blocks compare identical architectures.

Mahmoud Abbasi received the M.Sc. degree in computer software engineering from Islamic Azad University, Mashhad, Iran, in 2017, and the Ph.D. degree in computer engineering from the University of Salamanca, Salamanca, Spain, in 2026, where the doctoral work was supported by a Marie Skłodowska-Curie Fellowship within the EU Horizon 2020 IoTalentum network (grant no. 953442). Since 2026, Dr. Abbasi has been a Researcher with the AIR Institute, Salamanca, Spain. Research interests include machine learning for network traffic monitoring and analysis, intent-based networking, and secure, scalable Internet-of-Things systems. Dr. Abbasi has been a Member of IEEE since 2019.

Table 17

Encrypted Traffic Classification

See all 15 papers

Reference-Based Distillation Detection in LLMs
Jun 19, 2026Rajat Rawat, Sizhe Chen, Akshay Anand +3LLM Auditing Teacher-Student Learning
University of California, Berkeley · University of Southern California · OpenAI
Knowledge Distillation Must Account for What It Loses
Apr 28, 2026Wenshuo WangTeacher-Student Learning Knowledge Distillation
School of Future Technology, South China University of Technology, China
Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour
Oct 19, 2026Jonibek Mansurov, Sholpan Bolatzhanova, Filbert Aurelian Tjiaranta +2Language Model Distillation Knowledge Distillation
Mohamed bin Zayed University of Artificial Intelligence, UAE · University of Indonesia, Indonesia · Binus University, Indonesia

Jun 19, 2026J/K move · Enter open · S save

Reference-Based Distillation Detection in LLMs

Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan+2

University of California, Berkeley · University of Southern California · OpenAI

LLM Auditing Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations. This motivates a fundamental question: can we detect whether a model was distilled from another? We show that, while identifying a teacher model from a student in isolation is highly challenging, it becomes tractable in a reference-based setting: given a model and an earlier-generation checkpoint from the same lineage, we can identify the teacher model used to train the later checkpoint. We introduce a distillation detection method based on reference-based membership inference. By comparing how strongly a student model preferentially aligns with outputs from different candidate teachers relative to a reference checkpoint, our method identifies the most likely teacher and detects evidence of distillation. To handle unknown distillation pipelines such as hidden prompts, we infer proxy prompt templates directly from model outputs. We additionally identify a distinctive glyph-level signal specific to o1/o3 models. Evaluating distillation detection is challenging because modern model lineages are already heavily entangled. To address this, we develop a hybrid evaluation spanning both controlled distillation experiments and real-world models. Across both settings, our approach recovers the true teacher with near-perfect accuracy in single-teacher distillation scenarios, even when the underlying distillation pipeline is largely unknown. We further introduce statistical tests for both teacher attribution and distillation detection, and extend our framework to open-world settings where no teacher is guaranteed to be present among the candidates. Applying our method to contemporary models yields new evidence regarding potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS.

Abstract

Encrypted Traffic Classification

Explore similar work

Reference-Based Distillation Detection in LLMs

Knowledge Distillation Must Account for What It Loses

Selective Transfer through Sequence Distillation: From Benchmark Knowledge to Backdoor Behaviour