Tracing Subliminal Learning Through Pretraining

Angela Tang; Vishesh Anand; Gonçalo Paulo

cs.LGOct 22, 2026

Tracing Subliminal Learning Through Pretraining

Authors: Angela Tang, Vishesh Anand, Gonçalo Paulo

Organizations: EleutherAI

Abstract

Large language models trained on data generated by other models can sometimes inherit traits unrelated to that data, a phenomenon called subliminal learning. We study both hard label distillation, where students train on numbers generated by a biased teacher, and on-policy soft label distillation, where students generate the data and the teacher's logits provide the learning signal. Using Pythia, PolyPythia, and DataDecide OLMo models, we find that transfer persists in both settings across differences in weight initialization, pretraining data order, pretraining corpus, model size, and architecture. Across pretraining checkpoints, the ability to transmit a preference and the ability to acquire one develop separately: some checkpoints are more effective teachers but weaker students, and vice versa. Consistent with this distinction, cross-model transfer can be strongly asymmetric in magnitude when teacher and student roles are reversed. To investigate what carries the preference signal, we construct synthetic number sequences that match the teacher's generated number frequencies but have no dependence on the prompt or preceding numbers and find that these still induce substantial transfer in the hard-label setting. In top- $K$ soft-label distillation, removing animal tokens from the loss reduces transfer, although not completely in the same-model setting. In our experiments, soft label distillation produces more consistent transfer with less extensive training than hard label distillation. The growing use of on-policy distillation in model post-training makes this a particularly relevant setting for assessing unintended preference transfer.

Figures & tables

Figure 1: Experimental setup for hard label and soft label distillation. In hard-label distillation, we first fine-tune the teachers on animal preference prompts. The teacher then generates number sequences, which are filtered and used to train the student. In soft label distillation, we instead fine-tune the teacher on synthetic documents that convey the preference. The student generates continuations of Pile text, and the frozen teacher provides next-token probabilities that guide the student’s updates.

Figure 2: Subliminal learning in Pythia-160M. We measure transfer as the difference in trait probability between the student and a control student distilled on numbers generated by the untuned base model. (Left) With student hyperparameters fixed, sweeping the teacher’s finetuning hyperparameters produces a wide range of transfer. (Right) Training the student for more epochs can increase transfer but degrades language modeling abilities. Students trained on control numbers show similar degradation without acquiring a bias.

Figure 3: Top-

K

on-policy distillation of the bear preference, Pythia-160M teacher and student. (a) Subliminal transfer against the number

K

of teacher tokens kept at each position (“all” is the full 50,304 token output). Transfer is the student’s

p(\text{bear})

on the 50 held out preference prompts minus that of a control trained identically, at the same

K

, from the placebo teacher. It is positive at every

K

shown and peaks at

K=5

30

; at

K=1

every student exceeds five times the untuned student’s perplexity on held-out Pile text and is excluded (Appendix A.12 ). (b) The same distillation with the 43 animal-name tokens removed from the loss (light bar) next to the standard loss (dark bar) at

K=10

;

K=1

is excluded under both losses. Points and bars are means over 3 students with 95% bootstrap intervals.

Figure 4: Same-model distillation facilitates subliminal learning but is not required. We compare transfer between Pythia-160M bases sharing initialization, data order, both, or neither. (Left) Fraction of matched student-control pairs for which the student assigns greater probability to the target animal. (Right) Median, 75th perc., 95th perc., and observed max of

p_{\mathrm{student}}-p_{\mathrm{control}}

, pooled across animals. Transfer in this setting is strongest when models share the same pretrained base. Sharing initialization while changing data order retains larger effects than changing both.

Figure 5: Same-checkpoint and Cross-checkpoint transfer within Pythia-160M pretraining trajectories. (Left) Same-checkpoint transfer rises sharply during pretraining, with timing and magnitude of increase varying across models. (Right) Cross-checkpoint transfer within a model’s pretraining trajectory distinguishes a checkpoint’s ability to transmit a preference through numbers from its ability to acquire one. Some checkpoints are more effective as teachers than as students, while others show the reverse pattern.

Appendix figures & tables60 assets

Supplementary material from the paper’s appendix.

Appendix

Pile sub-corpus	Share
Pile-CC (web text)	0.35
Wikipedia	0.20
PubMed Abstracts	0.15
StackExchange	0.15
HackerNews	0.15

Appendix

Table A1: Share of each Pile sub-corpus in the synthetic corpus and in the prompt pool for on-policy distillation (Section 2.3 ).

Figure A1: Filtering removes mostly continuations that have the target animal or associated words. Share of the continuations sampled during on-policy distillation that the screen excludes from the loss, split by the first of checks (i)–(iv) that matched (legend, left to right); one match excludes the whole continuation. In each pair, the left bar is students distilled from an SDF teacher and the right bar controls distilled from a placebo teacher; the numbers are the totals. Every continuation of about 300 student and 100 control runs of PolyPythia-160M is counted (20,000 per run).

Teacher	Runs	Continuations judged	Flagged (%)
SDF teacher (students)	288	288,000	0.29 [0.27, 0.31]
Placebo teacher (controls)	96	192,000	0.09 [0.08, 0.11]

Appendix

Table A2: Continuations the LLM judge flags after the screen. Share of the continuations that passed the screen which the judge says refer to the target animal, pooled over bear, cat, dog and eagle and weighted over training; brackets give 95% intervals.

Model	Alias	GPT-NeoX tree	Hardware
retrained weight-seed1	wc1	v1.0	$8\times$ A100-80GB
retrained weight-seed2	wc2	v1.0	$8\times$ A100-80GB
retrained weight-seed3	wc3-modern	modern	$8\times$ A100-80GB
retrained Pythia-160M	Modern1234	modern	$8\times$ A100-80GB
retrained Pythia-160M	Rerun1234	v1.0	$8\times$ A100-80GB
DCLM-trained Pythia-160M	DCLM1234	modern	$8\times$ A100-80GB

Appendix

Table A3: Locally pretrained Pythia-160M models. All runs use the Pythia-160M training recipe (143,000 steps, global batch 1,024) on a single node of eight A100-80GB GPUs. The Pile-trained models share the original data order (seed 1234). wc1, wc2, and wc3-modern use initialization seeds 1, 2, and 3, respectively. Modern1234 and Rerun1234 use the original initialization (seed 1234). DCLM1234 and DCLM1 are pretrained on DCLM-baseline with initialization seeds 1234 and 1. The v1.0 and modern GPT-NeoX trees implement different learning-rate schedules, as described in the text.

Figure A2: Transfer is concentrated on the target animal, with little change on prompts where an animal response would be out of place. (A) Same-model distillation lifts the student’s target trait probability substantially over the control while the student’s aggregate preference for nine other evaluated animals falls below the control. (B) On off-topic prompts, the student’s target-trait lift persists at a small fraction of that on preference prompts.

Figure A3: Plain SGD transfers the trait once its learning rate is tuned. Full fine-tuning transfers only with SGD in these hard-label experiments. Pythia-160M students of one teacher per animal (dog, lion), trained for 20 epochs through a LoRA adapter (A) or with all parameters updated (B, C), using AdamW or plain SGD. (A, B) Transfer, defined as the student’s probability of the target animal on the held-out test prompts minus that of its control student, plotted against the student’s learning rate. (C) The full fine-tuning results from (B), plotted against the increase in Pile cross-entropy. Hollow points are degraded students, whose Pile cross-entropy reaches or exceeds the degradation threshold, corresponding to predicting Pile tokens by frequency alone (dotted line in C). Lines and bands are means over three seeds; stars indicate the selected learning rate, averaged over five seeds.

Figure A4: Steerability shows no detectable association with same-model transfer after accounting for teacher strength. (Left) Same-model transfer (control-adjusted) in Pythia-160M correlates both with teacher strength and with steerability (color). (Right) After adjusting the ranks of steerability and transfer for ranked teacher strength, their residual association is not statistically distinguishable from zero (partial Spearman $\rho=-0.1$ ). Each point represents one of 102 teachers.

	same model	same init, diff data order			released weight-seed
Receiver	orig	ds1	ds2	ds3	ws1	ws2	ws3
Teachers ( $n$ )	102	102	102	102	102	102	102
Marginal $R^{2}$ (%)
Teacher validation p(dog)	73.8	42.2	12.8	36.2	25.5	43.8	12.2
Teacher excess CE	73.1	47.8	17.1	43.9	25.9	45.1	10.5
Mean sequence length	76.5	49.4	17.7	45.4	26.7	47.7	12.5

Appendix

Table A4: Associations with hard-label transfer before and after adjusting for teacher preference. All teachers derive from the original Pythia-160M. Receivers are Pythia-160M (the same base model, orig), models sharing its initialization but differing in pretraining data order (ds1-ds3), and the released PolyPythias weight-seed models (ws1-ws3), which share its initialization and data order but differ in optimizer history and numerical training effects (Appendix A.4.1 ). Each teacher is distilled into one student per receiver at a fixed fine-tuning recipe. All variables are ranked within each receiver’s available teacher sample. Marginal

R^{2}

measures the variance in outcome ranks explained by a predictor alone and

\Delta R^{2}

measures how much more variation in transfer is explained by including that predictor alongside teacher preference. Partial

r

correlates predictor and outcome residuals after adjusting each for teacher preference. Steerability is the treatment effect of adding the teacher’s vector, extracted from Pythia-160M, to the receiver’s residual stream. Same-model transfer uses the Pythia-160M outcome.

Figure A5: First-number entropy decreases as subliminal learning improves during pretraining. (Left) Same-checkpoint dog transfer over pretraining for eleven Pythia-160M models. Each point averages five teacher fine-tuning recipes and three student seeds per recipe. (Right) Entropy of the teacher’s first-number distribution. Thicker lines highlight Pythia-160M, full-seed2, Modern1234, and our retrained weight-seed1 and weight-seed2 models.

Arm	Construction
$R$	The teacher’s real sequences
$S_{\text{all}}$	Every carrier number replaced by a draw from the teacher’s first-number distribution
$S_{\text{pool}}$	As $S_{\text{all}}$ , drawing from the teacher’s distribution over numbers at all positions
$S_{n}$	The first $n$ carrier numbers replaced; remaining carrier numbers kept
$T_{n}$	As $S_{n}$ , with the completion truncated after the $n$ replaced numbers
$P_{\text{all}}$	As $S_{\text{all}}$ , after permuting probabilities among numbers with the same number of digits

Appendix

Table A5: Synthetic dataset arms.

P_{\text{all}}

preserves the shape of the teacher’s number distribution while assigning its probability mass to different numbers.

Trait	Base model	$p_{R}$	$p_{S_{\mathrm{all}}}$	$p_{C}$	Retained transfer (%)	95% CI
Dog	Pythia-160M	0.730	0.459	0.011	62	[42, 82]
	data-seed3	0.671	0.617	0.025	92	[73, 110]
	full-seed3	0.763	0.357	0.017	46	[31, 59]
Cat	Pythia-160M	0.446	0.216	0.002	48	[42, 53]
	data-seed3	0.423	0.199	0.008	46	[35, 56]
	full-seed3	0.695	0.388	0.005	56	[34, 73]

Appendix

Table A6: Resampled sequences retain 40-92% of control-adjusted same-model transfer across four traits and three Pythia-160M variants. Probabilities are averaged over five teacher fine-tuning recipes and three student seeds per recipe.

R

S_{\mathrm{all}}

, and

C

denote real sequences, resampled sequences, and control sequences, respectively. Retention is

100(p_{S_{\mathrm{all}}}-p_{C})/(p_{R}-p_{C})

. The final column gives 95% paired teacher-bootstrap confidence intervals for retention.

Figure A6: Resampled sequences retain approximately all same-checkpoint transfer at Pythia-160M checkpoints 115k and 125k, compared to 62% at the final checkpoint. Pythia-160M (original), dog, teacher and student at the same pretraining checkpoint, five shared teacher fine-tuning recipes, three student fine-tuning seeds. Each bar reports control-adjusted transfer as a percentage of that induced by real sequences, computed as a ratio of means:

(p_{\text{arm}}-p_{C})/(p_{R}-p_{C})

. Transfer increases with the number of resampled numbers per completion (

T_{1}<T_{3}<S_{\text{all}}

) at all four checkpoints where all three constructions were evaluated. Keeping the carrier’s remaining numbers in each completion (

S_{1}

S_{3}

) generally reduces transfer, while permuting the probabilities assigned to numbers in the teacher’s first-number distribution (

P_{\text{all}}

) reduces transfer but does not eliminate it entirely at every checkpoint.

Figure A7: Completion length is associated with same-checkpoint retention, but does not fully account for checkpoint differences. Each point is a teacher at a given Pythia-160M checkpoint, evaluated for same-checkpoint transfer of the dog preference and charted against its mean number of generated numbers per completion. Colors indicate pretraining checkpoint, with teacher counts per checkpoint given in the legend. Teachers at each checkpoint comprise the five shared recipes; the remaining five of the ten main-experiment teachers at 75k, 100k, 115k, 125k, and 143k; the twelve length-stratified longer-output recipes at 115k, 125k, and 143k; and seven recipes from an earlier length-stratified random sample at 143k only. (Left) Percentage of real-sequence transfer retained after resampling all numbers in the completion. Values above 100% could reflect the length mismatch between resampled sequences (about ten numbers) and real sequences. (Right) Absolute control-adjusted transfer from resampled sequences shows the same decline with length.

Figure A8: Cross-checkpoint transfer from real and resampled sequences for dog in Pythia-160M. Rows are teacher checkpoints, columns student checkpoints. (Left) Control-adjusted transfer from real sequences, $p_{R}-p_{C_{i\to j}}$ . (Middle) Control-adjusted transfer from resampled sequences, $p_{S_{\mathrm{all}}}-p_{C_{j,j}}$ . (Right) Retention, the ratio of the two; cells with real control-adjusted transfer below 0.10 are greyed. Values are averaged over the five shared teacher fine-tuning recipes and three student seeds.

	final checkpoint	highest same-checkpoint	highest cross-checkpoint
pretraining run	retention % [95% CI]	retention % (checkpoint)	retention % (teacher $\to$ student)
Dolma, default seed	5 [4, 7]	13 (60k)	15 (55k $\to$ final)
Dolma, large-aux-2	36 [22, 55]	36 (final)	44 (65k $\to$ 50k)
Dolma, large-aux-3	28 [25, 32]	40 (60k)	56 (55k $\to$ 60k)
DCLM baseline	27 [19, 37]	27 (final)	34 (60k $\to$ final)

Appendix

Table A7: Resampled sequences preserve a limited share of transfer in the OLMo elephant experiments, retaining 5-36% of final-checkpoint transfer. The highest observed retention across different teacher-student checkpoints ranges from 15% to 56%. For the highest same-checkpoint and cross-checkpoint retention columns, we restrict to pairs with control-adjusted real-sequence transfer of at least 0.10.

	Mean $p(\mathrm{dog})$		Median $p(\mathrm{dog})$		Median excess				% Num. Seq. Kept
Teacher	3–5	10–20	3–5	10–20	3–5	10–20	$\Delta>0$	$p$	Ratio 3–5 / 10–20
ep5, lr 2e-4, r64 $\alpha$ 64	0.031	0.014	0.019	0.011	$+0.007$	$+0.005$	36/56	0.005	40%
ep7, lr 5e-4, r8 $\alpha$ 8	0.015	0.007	0.010	0.005	$+0.003$	$-0.001$	38/56	0.015	54%
ep7, lr 1e-4, r32 $\alpha$ 64	0.012	0.011	0.008	0.008	$\phantom{+}0.000$	$+0.001$	26/56	0.73	81%
ep20, lr 1e-4, r16 $\alpha$ 32	0.010	0.018	0.006	0.009	$\phantom{+}0.000$	$+0.002$	22/56	0.028	40%
ep10, lr 1e-4, r64 $\alpha$ 64	0.006	0.007	0.003	0.005	$-0.002$	$-0.001$	21/56	0.069	45%

Appendix

Table A8: Short (3-5) versus standard (10-20) in-context examples, original Pythia-160M

\to

full-seed1, dog, 56 student cells per teacher. Excess denotes the treatment effect, computed as the difference in p(dog) between student and matched control on the held-out preference prompt set. Means and medians are over the 56 student cells.

\Delta>0

is the number of cells in which the excess treatment effect under 3-5 examples is larger than under 10-20.

p

is the two-sided paired Wilcoxon signed-rank test on that difference. The last column is the ratio of the fraction of raw completions that pass the number sequence format filter under 3-5 examples versus 10-20 examples.

Figure A9: Same-model transfer in Pythia-160M with (verbatim) and without (spliced) the in-context number examples in the student prompt. Each line is the mean over 102 teachers distilled at a single set of hyperparameters, evaluated for excess trait preference over the control. Shaded bands are 95% bootstrap confidence intervals on the mean over teachers.

	Mean excess		Median excess		Control $p(\mathrm{dog})$		Spliced $<$
Epochs	Verbatim	Spliced	Verbatim	Spliced	Verbatim	Spliced	verbatim	$p$
1	0.069	0.022	0.036	0.012	0.012	0.009	98/102	4.8e-18
2	0.116	0.062	0.050	0.028	0.014	0.004	84/102	4.5e-12
5	0.218	0.110	0.136	0.041	0.009	0.009	96/102	2.5e-17
10	0.281	0.134	0.180	0.056	0.005	0.011	96/102	5.4e-17

Appendix

Table A9: Spliced versus verbatim prompts, original Pythia-160M

\to

original Pythia-160M, dog, 102 teachers, trained for 1, 2, 5, and 10 epochs. Verbatim students train on the standard number sequence prompts containing 10-20 in-context examples. Spliced students train on the same rows with the in-context example numbers removed from the prompt. Excess denotes the treatment effect, computed as the difference in

p(dog)

between student and matched control on the held-out preference prompt set. Means and medians are computed over teachers. Spliced

<

verbatim counts teachers whose excess treatment effect in the spliced setting is below that of their verbatim setting.

p

is the two-sided paired Wilcoxon signed-rank test on that difference.

Figure A10: On-policy distillation from an SDF teacher and from a hard-distillation teacher. Teacher and student are built from Pythia-160M; the target animal is lion. (left) Increase in

p(\text{lion})

of each student over its matched control on the 50 held-out preference prompts. (right) Increase in Pile cross-entropy over the untuned base, on the 250 Pile excerpts of Figure A11 , for the same students and their controls (grey).

Figure A11: Soft Label on-policy distillation against the number of student samples. Teacher and student share the base model; a sample is one prompt with the continuation the student wrote for it. (left) Increase in

p(\text{trait})

of each student over its matched control. (right) Increase in Pile cross-entropy over the untuned base for the same students and their controls (grey), measured on 250 Pile excerpts rather than the validation shard used in Figure 2 . Boxes hold 15 bear students (five base models) or 9 students of another animal (three base models); cat was not trained on 60,000 samples.

Figure A12: Training text and divergence, Pythia-160M

\to

Pythia-160M (bear). Students distilled on their own samples or on real Pile text, with the reverse or the forward KL. (left) Increase in

p(\text{bear})

of each student over its matched control. (right) Increase in Pile cross-entropy over the same controls, on the 250 Pile excerpts of Figure A11 . Each box holds 9 students (three teacher seeds, three distillation seeds), with whiskers at the minimum and maximum; the dashed line marks the control.

Figure A13: Transfer against the number of distinct Pile samples at a fixed 625 steps of 32 sequences.

Figure A14: Where the belief sits changes transfer more than it changes the teacher. (a) Teacher

p(\text{bear})

on the 50 held-out preference prompts and (b) transfer, the student’s

p(\text{bear})

minus that of the controls, for corpora of about 8,000 documents (solid) and nested subsets of about 2,000 (hatched) whose mentions of bears all sit in the first quarter of each document (start), its middle (middle), its last quarter (end), or a third of each (mix). Bars are means over teachers (a) and students (b), dots are single models, error bars are 95% bootstrap intervals that resample teachers and then each teacher’s students, and the dashed line is the original bear corpus.

Figure A15: Transfer falls steadily as the belief is deleted from more documents. (a) Teacher

p(\text{bear})

and (b) transfer for the mix documents of Figure A14 (about 8,000), with the belief sentences kept in only the given share of them. Dots, error bars and dashed line as in Figure A14 .

Figure A16: A first-person belief gives the most biased teacher and the least transfer. (a) Teacher

p(\text{bear})

and (b) transfer for corpora of about 2,000 documents whose belief is written in the first person, in the third person, or mixed (16% first person). Each model is evaluated on the 50 held-out first-person prompts (solid) and on 50 third-person prompts (hatched). Dots and error bars as in Figure A14 ; the dashed lines are the original bear corpus on each set of prompts.

Figure A17: A belief at the start transfers least at every corpus size. (a) Teacher

p(\text{bear})

and (b) transfer for the position corpora of Figure A14 and their nested subsets. Every teacher trains for 750 steps, so smaller corpora are repeated more often. Points are means, the four series are shifted slightly sideways for readability, and error bars and the dashed line are as in Figure A14 .

Figure A18: Perplexity cost of truncation. Perplexity on 250 Pile excerpts, divided by the untuned student’s, against

K

for the students of Figure 3 a, averaged over all students including excluded ones. Students above the dotted line (five times) are excluded from the transfer estimates.

Figure A19: Top-

K

on-policy distillation of the bear preference between Pythia-160M and Pythia-410M. (a) Transfer against the number

K

of teacher tokens kept at each position, for 160M

\to

410M and 410M

\to

160M, with controls as in Figure 3 . (b) 160M

\to

410M with the 43 animal-name tokens removed from the loss (light bar) next to the standard loss (dark bar) at

K=10

and

K=1

. Points and bars are means over 9 students (three teacher fine-tuning seeds, three distillation seeds; 8 for 410M

\to

160M at

K=1

) with 95% bootstrap intervals.

$K$	1	2	5	10	30	100	1000
Positions with bear in $\mathcal{S}_{K}$ (%)	0.023	0.047	0.13	0.29	1.0	3.7	28
Teacher probability held by $\mathcal{S}_{K}$	0.46	0.55	0.67	0.74	0.84	0.91	0.98

Appendix

Table A10: The bear token among the teacher’s kept tokens , 160M

\to

410M, on the student’s own samples.

Figure A20: The student’s own samples against real Pile text. Transfer (top) and perplexity ratio (bottom) against

K

for students trained on their own samples (solid) or on the Pile continuations of each prompt (dashed), with the same prompts, loss, seeds and matched controls. 9 students per point, 95% bootstrap intervals.

Figure A21: Tokens the top-10 loss keeps, Pythia-160M teacher and student. For students trained with

K=10

, the 20 tokens most often among the teacher’s 10 most likely next tokens, as a share of the positions each student trained on (its own samples): with the standard loss (left), and with the 43 animal-name tokens removed (right), where the ten are chosen from the remaining 50,261 tokens. Bars pool the 3 students and thin lines span their range; the box gives where “ bear” ranks and how often it is among the ten. Figures A22 and A23 show the removed tokens and what replaces them.

Figure A22: The 43 animal-name tokens before removal. For the standard-loss students of Figure A21 (left), the share of training positions at which the teacher ranks each single-token spelling of the ten candidate animals among its 10 most likely next tokens (log scale; bear spellings in colour, ␣ marks a leading space). Removing the tokens sets every share to zero. The box gives the share of positions with any of the 43 among the ten, and the teacher’s mean probability of “ bear” where it is among them.

Figure A23: Tokens that replace the removed animal-name tokens. At the training positions of the standard-loss students of Figure A21 (left) where one of the 43 tokens is among the teacher’s 10 most likely, the tokens that enter the ten once the 43 are removed, as a share of those positions.

Figure A24: What the bear fine-tuning changes in the teacher’s top 10, against the untuned base model. For the standard-loss students of Figure A21 (left), the untuned Pythia-160M the teacher was fine-tuned from, which is also the students’ starting point, is run on the same training positions, and each token’s share of positions among its 10 most likely next tokens is subtracted from the token’s share among the teacher’s: the 20 largest increases (left), the 20 largest decreases (middle), and the 20 largest ratios of the two shares among the tokens the teacher keeps at 0.1% of positions or more (right, log scale). Bear spellings in colour, ␣ marks a leading space. Bars pool the 3 students and thin lines (left, middle) span their range; the box gives both shares of “ bear” and its rank among all increases.

Figure A25: Distribution of transfer across model size and architecture and across OLMo pretraining conditions. We illustrate the median, 75th percentile, 95th percentile, and observed maximum of transfer in the cross model size, cross model architecture, and OLMo-1B experiments. A : Transfer between Pythia-160M and Pythia-410M, using the original models and full-seed variants 1-3 at each size. Models with shared seeds share a pretraining data order. Different seeds do not. Cross-architecture comparisons pair three Pythia-160M models with three OLMo-1B Dolma1.7 models. B : DataDecide OLMo-1B (Dolma 1.7, DCLM-baseline, C4), elephant, grouped by pretraining condition: same model (3 arms), shared initialization with a different corpus (6 arms), different initialization with a different corpus (6 arms), and different initialization with the same corpus (3 arms). As in Pythia, same-model transfer dwarfs every cross condition in OLMo at the evaluated training endpoints. Across size and architecture, medians are near zero in every direction. Descriptively, the smaller receiver (160M) shows the larger upper tail in the cross-Pythia experiments, while OLMo-to-Pythia transfer has a longer upper tail but a lower median than transfer in the reverse direction.

				# HP
Model Family	Pretraining Condition	Arms	# Arms	Teacher	Student	Total	Animal	Exceedances	Null 95th Perc. Exceedances	P-value
Pythia	Same-model	{160M, fs1–3, ds1–3,	8	10	30	2400	dog	2303 (96.0%)	120 (5.0%)	5e-5
		rerun 160M}
		{160M, fs1–3}	4	10	30	1200	cat	1159 (96.6%)	71 (5.9%)	5e-5
							lion	1178 (98.2%)	72 (6.0%)	5e-5
							pig	1166 (97.2%)	72 (6.0%)	5e-5

Appendix

Table A11: Transfer existence by pretraining condition. Stratified exceedance permutation test results, per A.13 . Bold =

p<0.05

# HP

Comparison

Arms A

Arms B

# Arms A vs B

Teacher

Student

Total A vs B

Animal

Treatment Effect Mode

Diff. in Means

Mean P-value

Win Prob.

Win P-value

Same-model vs

{160M,

8 vs 12

2400

dog

\Delta

0.379

1.6e-5

0.908

3.2e-5

same init, diff.

fs1–3,

ds1–3} 2

vs 3600

\Delta

/ prior

39.3

1.6e-5

0.915

1.6e-5

data order

ds1–3,

All

\Delta

0.32

0.0025

0.957

1.6e-5

rerun 160M}

\Delta

/ prior

98.2

0.01

0.959

1.6e-5

Same-model vs

{160M,

{wc1, wc2,

8 vs 6

2400

dog

\Delta

0.457

6.7e-4

0.975

6.7e-4

Appendix

Table A12: Relative Strength of transfer between pretraining conditions in Pythia. We compare pretraining conditions that vary weight initialization or data order in the Pythia family, computing Difference in Means and Win Probability statistics on both (

\Delta=p_{\mathrm{student}}-p_{\mathrm{control}}

) and

\Delta/

prior. Rows: dog, and all four animals pooled. A positive Difference in Means, or a Win Probability above 0.5, means the condition named first under Comparison transfers more strongly. Bold = two-sided

p<0.05

# HP

Comparison

Arms A

Arms B

# Arms A vs B

Teacher

Student

Total A vs B

Animal

Treatment Effect Mode

Diff. in Means

Mean P-value

Win Prob.

Win P-value

Same-model vs

{Dolma1.7,

3 vs 6

1050

elephant

\Delta

0.0938

0.024

0.886

0.024

same init,

DCLM, C4}

DCLM, C4} 2

vs 2100

\Delta

/ prior

3.34

0.024

0.889

0.024

diff. data

corpus

Same-model vs

{Dolma1.7,

{DCLM,

3 vs 6

1050

elephant

\Delta

0.0909

0.024

0.868

0.024

Appendix

Table A13: Relative Strength of transfer between pretraining conditions in DataDecide OLMo-1B. We compare pretraining conditions that vary weight initialization or data corpus in the DataDecide OLMo-1B family, computing Difference in Means and Win Probability statistics on both (

\Delta=p_{\mathrm{student}}-p_{\mathrm{control}}

) and

\Delta/

prior. A positive Difference in Means, or a Win Probability above 0.5, means the condition named first under Comparison transfers more strongly. Bold = two-sided

p<0.05

Model Family	Comparison	# Arms A vs B	Total A vs B	Treatment Effect Mode	Diff. in Means	P-value	Paired P-value
Pythia/OLMo	Small to large vs	9 vs 9	3360	$\Delta$	3e-3	0.42	0.41
Cross Arch.	Large to small		vs 2160	$\Delta$ / prior	-0.0831	0.8	0.8
Pythia-160M/410M	Small to large vs	16 vs 16	4800	$\Delta$	-4e-3	0.18	0.15
	Large to small		vs 4800	$\Delta$ / prior	-0.558	0.043	0.048
	Small to large:	4 vs 12	1200	$\Delta$	-2e-3	0.62	–
	Same vs		vs 3600	$\Delta$ / prior	-0.0674	0.58	–

Appendix

Table A14: Relative strength of transfer by direction and data order across model sizes and architectures. All arms use the dog preference. Each arm is summarized against its own matched controls by

\Delta=p_{\mathrm{student}}-p_{\mathrm{control}}

over runs, and by

\Delta/

prior (transfer in units of the receiver prior). Arm sets are those of Table A11 . A positive Diff. in Means means the side named first under Comparison transfers more strongly. Bold =

p<0.05

Figure A26: On-policy distillation transfers in all four relationships of Figure 4 , with no same-model advantage. (A) Mean

p_{\mathrm{student}}-p_{\mathrm{control}}

per condition, pooled over animals (hatched) and per animal. (B) Median, 75th and 95th percentile, and maximum of

p_{\mathrm{student}}-p_{\mathrm{control}}

, pooled over animals.

Figure A27: The statistic of Figure 4 under on-policy distillation. (A) Fraction of students that assign more probability to the target animal than their control. (B) As in Figure A26 .

Figure A28: On-policy distillation between models of different sizes and families. Each bar is one teacher

\to

student pair for the bear preference, grouped by student. Each group opens with the student’s same-model pair (hatched), and the dashed line carries its level across the group. Bars are means of

p_{\mathrm{student}}-p_{\mathrm{control}}

with 95% intervals, the wider of a cluster bootstrap over both seeds and a

t

-interval over teacher seeds.

Figure A29: Same-model transfer can peak early and decline while cross-model transfer continues to rise in Pythia. Curves show mean control-adjusted dog preference during student training. (Left) Same-model transfer for models where endpoint transfer is

<70\%

of peak. (Center) Same-model transfer for models where endpoint transfer is

\geq 70\%

of peak. (Right) Cross-model transfer, averaged over all arms belonging to the same pretraining condition, increases with further student training. Shaded bands are 95% intervals from a bootstrap over teacher clusters.

Figure A30: Same-model and cross-model transfer both peak before training ends in OLMo. Curves show mean control-adjusted elephant preference during student training. (Left) Same-model transfer peaks then declines for OLMo-1B models pretrained on Dolma1.7, DCLM-baseline, and C4. (Right) Cross-model transfer, averaged over all arms belonging to the same pretraining condition, declines less sharply than same-model transfer, then levels off.

Figure A31: Cross-model transfer can overtake same-model transfer during student training. Each panel holds the teachers and their generated number sequences fixed and varies the receiver base model. Black curves show same-model transfer. Colored curves show cross-model transfer. Curves report median control-adjusted transfer, normalized by the receiver’s base preference, across four Pythia pretraining conditions. Same-model transfer is typically stronger throughout training, with two notable exceptions where Modern1234 is the sender. Here, cross-model transfer from Modern1234 to Pythia-160M and Rerun1234 overtakes same-model transfer midway through training.

Figure A32: Cross-model transfer shows distinct sender and receiver patterns. (Bottom) Grids show mean control-adjusted transfer from each teacher base model (rows) to each student base model (columns), in percentage points. (Top) Bars summarize each model’s average outgoing minus incoming transfer over the other models in its panel. Positive values indicate net senders and negative values net receivers, relative to the evaluated partners and training settings.

Figure A33: Cross-model asymmetry often persists in a single direction throughout student training, although some pairs can reverse direction. Each curve shows mean control-adjusted transfer in the direction named in the legend minus transfer in the reverse direction. Positive values favor the named direction, while negative values favor the reverse. Panels show the dog preference across four Pythia pretraining conditions.

Figure A34: Cross-checkpoint transfer when holding (a) early teachers fixed, (b) late teachers fixed, (c) early students fixed, (d) late students fixed. Aggregated over five teacher recipes and three student finetuning seeds.

Figure A35: Cross-checkpoint transfer when holding (a) early teacher (75k) fixed, (b) late teacher (125k) fixed, (c) early student (75k) fixed, (d) late student (125k) fixed. Averaged over five student distillation recipes and three student finetuning seeds.

	In-sample $R^{2}$				Held-out $R^{2}$
Trait / model	Prod.	Add.	Bottl.	Prox.	Prod.	Add.	Bottl.	Prox.
Dog / Pythia-160M	0.967	0.794	0.921	0.983	0.953	0.704	0.881	0.972
Dog / data-seed1	0.859	0.666	0.845	0.954	0.806	0.519	0.785	0.924
Dog / data-seed2	0.882	0.734	0.808	0.962	0.831	0.617	0.750	0.935
Dog / data-seed3	0.913	0.744	0.868	0.956	0.868	0.634	0.815	0.925
Dog / full-seed1	0.928	0.715	0.867	0.968	0.894	0.591	0.794	0.937

Appendix

Table A15: Separate teacher and student factors explain most of in-sample and held-out cross-checkpoint transfer for Pythia-160M. Each row is a

13\times 13

grid of checkpoints from a single model’s pretraining trajectory. Each cell is the student’s control-adjusted preference transfer, averaged over five teacher fine-tuning recipes and three student fine-tuning seeds. Columns give

R^{2}

for the product, additive, maturity-bottleneck, and product-plus-proximity models. Held-out

R^{2}

is calculated from pooled out-of-fold predictions, then averaged over five independent random partitions.

	In-sample $R^{2}$				Held-out $R^{2}$
Corpus / seed	Prod.	Add.	Bottl.	Prox.	Prod.	Add.	Bottl.	Prox.
dolma1_7 / default	0.931	0.764	0.893	0.980	0.890	0.535	0.764	0.952
dolma1_7 / large-aux-2	0.646	0.539	0.646	0.888	0.315	0.138	0.516	0.697
dolma1_7 / large-aux-3	0.925	0.751	0.902	0.978	0.855	0.520	0.787	0.941
dclm-baseline / default	0.943	0.777	0.843	0.973	0.906	0.561	0.708	0.938

Appendix

Table A16: Separate teacher and student factors explain cross-checkpoint transfer in 3 of 4 evaluated OLMo-1B models. Each row is an

8\times 8

grid of checkpoints (30k, 40k, 45k, 50k, 55k, 60k, 65k, and final 69k) from a single model’s pretraining trajectory. Each cell is the student’s control-adjusted elephant preference transfer, averaged over five teacher fine-tuning recipes and three student fine-tuning seeds. Columns as in Table A15 . Held-out

R^{2}

use five disjoint folds, each withholding 12-13 of 64 cells. Three of the four grids factorize. On the Dolma 1.7 large-aux-2 pretraining seed, the plain product fails.

Figure A36: Same-checkpoint transfer at different stages of student training. (Left) In Pythia-160M, transfer generally strengthens with continued student training across the evaluated pretraining checkpoints. (Right) In full-seed2, transfer peaks after five epochs and then declines across pretraining checkpoints, extending the final-checkpoints observations in Appendix A.15.1 .

Pretraining condition	Modern1234	Pythia-160M	wc1
Same-model	itself	itself	itself
Same init, different data order	data-seed1	data-seed1	full-seed1
Different init, same data order	wc3 (modern)	wc3 (modern)	Pythia-160M
Different init, different data order	full-seed1	full-seed2	full-seed2
Same init, different pretraining corpus	DCLM1234	DCLM1234	DCLM1
Different init, different pretraining corpus	DCLM1	DCLM1	DCLM1234

Appendix

Table A17: Pairs in the soft-label cross-checkpoint grids. Each column is an anchor model, and each entry is the model it is paired with in that pretraining condition. Every pair of different models is distilled in both directions.

Figure A37: Same-checkpoint transfer under soft-label distillation. Teacher and student sit at the same checkpoint (x axis, 1k to 143k steps). One plot per pretraining condition, plus Pythia-160M with Modern1234 (same init, same data order (retrained)); plot titles give the number of pairs and of sender

\to

receiver combinations.

Figure A38: Early checkpoints can transmit a preference more effectively than they acquire it. (a) One pair of triangles per cross-checkpoint grid, grouped by pretraining condition: the upward filled triangle is the mean transfer from the grid’s 2k teacher into its students between 8k–143k, the downward empty triangle the mean transfer from its teachers at 8k–143k into its 2k student. (b) Their difference per grid (dots), and the condition mean over its three pairs with a 95% CI (diamonds).

Figure A39: The onset in each pretraining condition. Solid line: the 2k teacher into a student at checkpoint

x

Figure A40: Nine checkpoints into Pythia-160M. Cross-checkpoint grids from 1k to 143k steps with Pythia-160M as the receiver: its same-model grid, DCLM1234 $\to$ Pythia-160M and DCLM1 $\to$ Pythia-160M.

Figure A41: Teacher preference across pretraining. Teacher preference, the SDF teacher’s own

p(\text{bear})

averaged over three fine-tuning seeds, against the teacher’s checkpoint. Thin colored lines are one model each, colored by training stack.

Figure A42: Transfer relative to teacher preference, and longer teacher fine-tuning. (a) Transfer divided by teacher preference into the 143k student, by teacher checkpoint; the dotted line marks a student that gains as much as its teacher. (b) Teachers at 143k fine-tuned for 2, 5 or 8 epochs(log), for wc1 with itself and with the five models it is paired with.

Figure A43: Tree and color preferences also transfer under hard label (A) and soft label (B) distillation. Transfer measured on 50 held-out preference prompts.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 508 papers

On Mitigation of Subliminal Learning in Large Language Models
Sep 2, 2026Atsushi Yanagisawa, Brendan Gho, Rajendran Ramesh Babu Manoj Narender +3Subliminal Learning LLM Fine-Tuning
Algoverse AI Research
Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation
Jun 9, 2026Uwe König, Hamza Kazmi, Ruizhe Li +1Subliminal Learning Language Model Distillation
1Technical University of Munich · 2Algoverse · University of Aberdeen +1
You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases
Mar 10, 2026Isaia Gisler, Zhonghao He, Tianyi QiuSubliminal Learning LLM Training
ETH Zürich · University of Cambridge · Peking University

Sep 2, 2026J/K move · Enter open · S save

On Mitigation of Subliminal Learning in Large Language Models

Atsushi Yanagisawa, Brendan Gho, Rajendran Ramesh Babu Manoj Narender, Kevin Zhu+2

Algoverse AI Research

Subliminal Learning LLM Fine-Tuning Knowledge Distillation Fine-Tuning

Open paper AskPDF arXiv

Knowledge distillation can transmit unintended behavioral traits from a teacher model to a student through training data that appear semantically unrelated to those traits, a phenomenon known as subliminal learning. Although recent work has established this effect, its training dynamics and mitigation remain underexplored. We study subliminal learning in open-weight language models ranging from 1.5B to 8B parameters, covering the Qwen, Gemma, and Llama families in number-sequence and chain-of-thought settings. Rather than evaluating only final models, we track trait-related probabilities throughout fine-tuning and find that subliminal acquisition can be highly non-monotonic, with transient spikes, reversals, and trait-specific failures of transfer. We then introduce liminal training, an annealed KL-regularized fine-tuning method that constrains early drift from the base model. Across our experiments, liminal training substantially reduces subliminal trait acquisition while largely preserving task gains, outperforming paraphrasing and layer freezing as mitigation strategies. The effect also extends beyond animal preferences: in a French-language response-style experiment, liminal training suppresses language transfer while retaining much of the GSM8K improvement. Finally, we show that KL timing matters: early regularization is more effective than late regularization, and sweeping the regularization strength reveals an empirical trade-off between task learning and trait suppression.

Abstract

Language Model Distillation

Explore similar work

On Mitigation of Subliminal Learning in Large Language Models

Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation

You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases