Organizations: University of Technology Sydney, Sydney, NSW, Australia · Xi’an Jiaotong University, Xi’an, Shaanxi, China · Southeast University, Nanjing, Jiangsu, China
Subliminal learning lets a student inherit a teacher's behavioral trait from semantically unrelated data, yet published demonstrations typically require tens of thousands of carrier examples. We ask where this data requirement comes from. Our testbed is subliminal steering: the teacher trait is a known residual-stream vector ΔT, so transfer can be measured directly as parameter recovery. On identical carrier prefixes, we compare token-level (hard) NLL supervision with full-distribution (soft) KL supervision. At initialization the two objectives give nearly collinear gradients, and both align poorly with ΔT. Under iterative optimization, however, they diverge: soft supervision recovers ΔT almost exactly from a few hundred carriers, while hard supervision stays well below it even with tens of thousands. We explain this gap by casting steering-vector distillation as a noisy linear inverse problem. Locally, the carrier task maps the trait through its Fisher matrix F, so gradients point toward FΔT rather than ΔT. Gradient descent then acts as a progressively less-damped inverse of F. With soft targets, this inverse restores low-curvature directions. With hard labels, it also amplifies the sampling noise in those same directions. The result is an optimal inversion depth that grows with the number of independent carriers. Experiments on Qwen2.5-7B and Gemma-2-9B confirm four predictions: the Fisher distortion of the initial gradient, recovery ordered from steep to flat directions, an optimal depth that shifts with data scale, and the finding that resampling completions from a fixed prompt pool works as well as adding new prompts. In this setting, large carrier datasets are needed less to reveal the trait than to suppress label noise amplified by Fisher inversion. Code is available at https://github.com/luoyuchenmlcv/subliminal-data.
Figures & tables
Figure 1: Hard and soft supervision are locally similar but diverge under iteration. Rows denote model families and groups denote animal traits. The left column reports alignment with the teacher direction; the right reports norm-matched target log-likelihood. Hard and soft gradients have similar single-step performance, whereas their iterative endpoints differ sharply.
Figure 2: Fisher inversion improves single-step alignment across models and traits. Hyperparameters are selected against ΔT .
Figure 3: Iteration progressively restores flatter directions. (a) Recovery of 64 equal- ΔT -energy Fisher bins under soft-KL optimization; curvature increases from bottom to top. (b) Observed recovery for representative coarse bins (solid) and the fixed- F prediction ϕk(λ) (dashed).
Figure 4: Left: hard-label spectral error initially falls, then rises as amplified noise overtakes the remaining inversion bias; dots mark each minimum. Right: best alignment at a matched completion budget B , or repeatedly sampling generation from a fixed pool of 500 prompts for B/500 times.
Figure 5: Gemma trait energy lies at lower normalized curvature. Teacher-vector energy across common bins of λ/λˉ for five traits; each spectrum is normalized by its model-specific mean.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Configuration
Qwen2.5-7B-Instruct
Gemma-2-9B-IT
Hidden width / transformer layers
3584 / 28
3584 / 42
Intervened layers
[2,26)
[2,40)
Teacher norm bound
0.5
1.5
Traits
cat, dog, eagle, lion, raven
Teacher training examples
50
Teacher optimizer / learning rate
Adam / 10−3
Adam / 10−3
Appendix
Table 1: Model, intervention, teacher-extraction, carrier-generation, and student-recovery hyperparameters. The same recovery optimizer and learning rate are used for hard and soft supervision.
Subliminal learning refers to a student language model acquiring a teacher's traits (e.g. a system-prompted preference for owls) when fine-tuned on the teacher's outputs, despite the outputs being semantically unrelated to those traits. It remains poorly understood how data without semantic meaning can transfer specific semantic traits. In this work, we show that subliminal learning is mediated by a single steering vector, i.e. a vector added to the model's activations. Across two open-source models, we find that the teacher's system prompt is well approximated by a steering vector, and that the student's behavior is driven by learning an aligned vector over fine-tuning. System prompts that are not well approximated by steering vectors are not subliminally learned. This is a special case of steering vector distillation, in which a student trained on the outputs of a steered teacher learns to imitate that steering. We demonstrate steering vector distillation on a range of semantic and random vectors. Adding a semantic vector to a model's activations can have both model-independent and model-specific (i.e. non-semantic) effects on its behavior, so generated data that is non-semantic can transmit a vector with semantic effects, enabling subliminal learning. This also explains why subliminal learning does not transfer between models. We find that adaptive optimizers are necessary for subliminal learning in language models: activation gradients on steered data carry a small but consistent component along the steering direction, and non-adaptive optimizers impede this by allowing outlier gradients to dominate.
Subliminal learning describes a student language model inheriting a behavioral bias by fine-tuning on seemingly innocuous data generated by a biased teacher model. Prior work has begun to characterize this phenomenon but leaves open questions about the scope of signals it can transfer, the mechanisms that explain it, and the precision with which a bias can be encoded by seemingly unrelated data. We tackle all three problems by introducing subliminal steering, a variant of subliminal learning in which the teacher's bias is implemented not via a system prompt, as in prior work, but through a steering vector trained to maximize the likelihood of a set of target samples. First, we show that subliminal steering transfers complex multi-word biases, whereas prior work focused on single-word preferences, demonstrating a large scope of subliminally transferrable signals. Second, we provide mechanistic evidence that subliminal learning transfers not only the target behavioral bias, but also the steering vector itself, localized to the layers at which the teacher was steered. Finally, we show that the bias is encoded with surprising precision. We train a new steering vector directly on the subliminally-laden dataset and find that it attains high cosine similarity with the original vector.
Subliminal Learning (SL) is a surprising type of generalization displayed by modern language models. It allows the transfer of a bias or behavior from a teacher model to a student by distilling from seemingly unrelated or random synthetic data from the teacher. This presents challenges in ensuring AI systems remain predictable and are trained safely, as standard auditing of the input data would not catch the hidden subliminal signal. Here, we investigate several open questions as to the enabling mechanisms and drivers of SL. First is the nature of the process by which biases are encoded in the data. We find that by adding Gaussian noise to the weights of the teacher and student models, the magnitude of subliminal transfer is increased by a factor of 1.9 in Gemma and 1.3 in Llama, suggesting that non-semantic weight structures play a crucial role. We show that steering vectors can be applied to the teacher to produce subliminal data, in addition to prompting and finetuning as used in previous studies. Analysis of the activations of the student models that have been trained on steered and prompted data demonstrates that students inherit not just the semantic meaning of the teacher's bias, but also the type of intervention that was used to apply it: steered students imitate steering vectors, prompted students do not. Additionally, the gradients of steered subliminal data show a linear correlation with the teacher's steering vectors, showing promise for data auditing. More broadly, as synthetic data becomes central to frontier training pipelines, being able to see the latent signals hidden in training data becomes paramount.
Ethan Hadley, Eren Gultepe
Department of Computer Science, Southern Illinois University Edwardsville, Edwardsville, IL, USA.