Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

Joseph Egan; Ciarán M. Gilligan-Lee; Michael O'Riordan; Yuchen Zhu

stat.MLOct 9, 2026

Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning

Authors: Joseph Egan, Ciarán M. Gilligan-Lee, Michael O'Riordan, Yuchen Zhu

Organizations: Spotify

Abstract

When a language model is distilled from a teacher that shares its base parameters, the student can inherit behavioural traits of the teacher---including misalignment---even from data that is semantically unrelated to the trait (Cloud et al., 2026). All existing defences require knowing the trait. We remove this requirement and prove that a subliminally transferred trait can be recovered from the parameter updates alone, without supervision, by exploiting a single asymmetry: the trait is expressed consistently across diverse datasets generated by the teacher, whereas task-specific and shared capabilities vary. The resulting estimator returns a weight-space direction that can be amplified to discover and interpret the trait or projected out to suppress it. Applied to the subliminal-learning benchmarks of (Cloud et al., 2026), our method recovers the transmitted animal-preference traits without any trait labels and suppresses their expression with almost no impact to task performance.

Figures & tables

Panel of Figure 1 — Figure 1: Stifle identifies a subliminally transferred trait without supervision. A single teacher, which carries the trait, generates $m$ datasets; each fine-tunes the same base model, giving a weight update per dataset. Every vector shown lives in weight space (a component of a parameter update, not an activation). (a) Across datasets the trait update is small but is near-identical; the shared capability and task-specific parts are large but vary in size (origins offset for legibility; all start from the same base). (b) Averaging cancels task-specific parts and preserves the trait, but the trait is small, so the mean update may point mostly at capability. (c) The scatter matrix $S$ is the spread of the updates about their mean: datasets disagree about capability—as different tasks require different capabilities—but barely at all about the trait. (d) Dividing each direction’s mean by its variance—applying $S^{-1}$ —suppresses the large but highly varying capability and leaves an estimate of the trait.

Figure 2: Subliminal learning transmits animal preference in eight new fine-tuning tasks . Green bars are control runs, fine-tuned on data generated by a teacher with a neutral system prompt. Orange bars show results after fine-tuning on subliminal datasets. Black error bars are the 95% confidence interval from five different random seeds. math-cot and commonsenseqa have a CoT suffix on their evaluation prompts (Appendix B.9 ).

Figure 3: Ablating the identified direction from weight updates mitigates trait expression while maintaining task performance. Blue circles show the frequency of responding with the trait animal on 5,000 completions (prompts in Appendix B.9 ). Orange squares show the per-task performance metrics defined in Section 4.2 . The horizontal dashed lines show the base model’s animal preference (blue) and task performance (orange). We see in all cases that there exists a value of

\alpha\in(0,1]

such that the trait is removed and task performance is maintained.

Figure 4: Large

\alpha

interventions no longer mitigate the trait, but provide a mechanism for unsupervised identification, as the trait is expressed on off-topic evaluations. Effect of

\alpha\in[0,10]

on animal preference (blue), animal dispreference (green), off-topic questions (yellow) and task performance (orange). Lines show the probability of the model’s response mentioning the trait.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Source	Unique items	Citation
numbers	Uniform distribution	1,000	—
names	NLTK names corpus	7,536	[ Bird et al., 2009 ]
colours	XKCD Color Survey (via Matplotlib)	949	[ Munroe, 2010 , Hunter, 2007 ]
instruments	WordNet musical_instrument.n.01	284	[ Miller, 1995 , Bird et al., 2009 ]
minerals	WordNet mineral.n.01	336	[ Miller, 1995 , Bird et al., 2009 ]
elements	Periodic table	118	—

Appendix

Table 1: Sequence task datasets. Starter sequences were generated by sampling uniformly from the list of items. The teacher is then prompted to continue the sequences (see examples in Appendix B.3 ).

Task	Source	Train size	Citation
code	Sleeper Agents code backdoor data	71,238	[ Hubinger et al., 2024 ]
math-cot	GSM8K ( openai/gsm8k )	7,473	[ Cobbe et al., 2021 ]
commonsenseqa	CommonsenseQA ( tau/commonsense_qa )	9,741	[ Talmor et al., 2019 ]
summarisation	CNN/DailyMail ( abisee/cnn_dailymail v3)	287,113	[ See et al., 2017 ]

Appendix

Table 2: Non-sequence task datasets. The teacher model generates completions conditioned on prompts drawn from the source dataset.

numbers
Animal	Raw	Filtered	Judge filtered
panda	30,000	27,708	27,196
lion	30,000	27,358	26,965
dragon	30,000	27,402	26,265
cat	30,000	27,633	26,953
owl	30,000	27,351	26,104

Appendix

Table 3: Sequence task dataset sizes per animal. Raw = teacher completions before filtering; Filtered = after text-based filter (format, trait animal mentions); Judge filtered = after GPT-4.1 judge filter.

code
Animal	Raw	Filtered	Judge filtered
panda	11,400	—	10,917
lion	11,400	—	10,966
dragon	11,400	—	10,947
cat	11,400	—	10,191
owl	11,400	—	10,813

Appendix

Table 4: Non-sequence task dataset sizes per animal. Raw = teacher completions before filtering; Judge filtered = after combined format and GPT-4.1 judge filter.

LoRA configuration
Rank ( $r$ )	8
Alpha ( $\alpha$ )	8
Target modules	q, k, v, o, gate, up, down proj
Bias	None
Gradient checkpointing	Enabled
Optimisation

Appendix

Table 5: Training hyperparameters. All parameters are shared across tasks unless noted. The base model was Qwen2.5-7B-Instruct fine-tuned with LoRA via Unsloth.

Item	Filtered %	Change %	Abs diff
lionel	7.893	-100.0	-7.893
leo	5.958	-100.0	-5.958
lilian	1.078	-100.0	-1.078
lucian	0.909	-100.0	-0.909
leon	0.720	-100.0	-0.720
leonard	0.684	-100.0	-0.684

Appendix

Table 6: The top 10 names removed by the judge filter (see judge prompt in Appendix B.6 ) for the names task, where lion was the trait animal.

Figure 5: Subliminal learning for the numbers and code tasks, with bar definitions as in Figure 2 .

Figure 6: Task performance and animal preference metrics as a function of

\alpha

for all tasks. Line definitions as in Figure 3 . code does not have a well-defined metric of task performance.

Subliminal Learning

Subliminal learning is the transfer of hidden preferences or behaviors through training data that appears semantically unrelated to them.

See all 24 papers

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation
Sep 1, 2026Zhixuan Liu, Zhichen Dong, Yuyu Fan +2Supervised Fine-Tuning Subliminal Learning
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University
On Mitigation of Subliminal Learning in Large Language Models
Sep 2, 2026Atsushi Yanagisawa, Brendan Gho, Rajendran Ramesh Babu Manoj Narender +3Subliminal Learning LLM Fine-Tuning
Algoverse AI Research
Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
Oct 7, 2026Jan Dubiński, Anna Sztyber-Betley, Jan Betley +1Subliminal Learning Knowledge Distillation
Truthful AI · Warsaw University of Technology

Sep 1, 2026J/K move · Enter open · S save

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

Zhixuan Liu, Zhichen Dong, Yuyu Fan, Xiangtian Li+1

Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Fudan University

Supervised Fine-Tuning Subliminal Learning Language Model Distillation

Open paper AskPDF arXiv

Beyond intended capabilities, model distillation can transfer hidden traits from a teacher. A teacher biased by a system prompt can generate semantically clean training data, such as numeric sequences, that still causes a downstream student to inherit the hidden preference, a phenomenon known as subliminal learning. Prior work has identified several parts of this process. How the signal builds up during training and produces behavioral transfer remains unclear, making targeted mitigation difficult. We propose and validate trait-direction drift as a mechanism for subliminal learning: biased generation creates measurable preference gaps in teacher data, and student-recognizable gaps induce trait-aligned updates during supervised fine-tuning that accumulate into behavioral transfer. Guided by this mechanism, we propose probe-space corridor regularization, a targeted defense that constrains drift along a calibrated trait direction during distillation. The method substantially reduces hidden-trait transfer, preserving task performance: for example, it lowers malicious-response transfer from 29.55% to 6.45% with low main-task accuracy cost, and consistently suppresses animal-preference transfer across the main Qwen setting. The preference-gap, training-trajectory, and intervention evidence links subliminal learning to trait-direction drift and motivates corridor regularization as a targeted control during distillation.

Abstract

Subliminal Learning

Explore similar work

Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation

On Mitigation of Subliminal Learning in Large Language Models

Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors