Large language models trained on data generated by other models can sometimes inherit traits unrelated to that data, a phenomenon called subliminal learning. We study both hard label distillation, where students train on numbers generated by a biased teacher, and on-policy soft label distillation, where students generate the data and the teacher's logits provide the learning signal. Using Pythia, PolyPythia, and DataDecide OLMo models, we find that transfer persists in both settings across differences in weight initialization, pretraining data order, pretraining corpus, model size, and architecture. Across pretraining checkpoints, the ability to transmit a preference and the ability to acquire one develop separately: some checkpoints are more effective teachers but weaker students, and vice versa. Consistent with this distinction, cross-model transfer can be strongly asymmetric in magnitude when teacher and student roles are reversed. To investigate what carries the preference signal, we construct synthetic number sequences that match the teacher's generated number frequencies but have no dependence on the prompt or preceding numbers and find that these still induce substantial transfer in the hard-label setting. In top-K soft-label distillation, removing animal tokens from the loss reduces transfer, although not completely in the same-model setting. In our experiments, soft label distillation produces more consistent transfer with less extensive training than hard label distillation. The growing use of on-policy distillation in model post-training makes this a particularly relevant setting for assessing unintended preference transfer.
Figures & tables
Figure 1: Experimental setup for hard label and soft label distillation. In hard-label distillation, we first fine-tune the teachers on animal preference prompts. The teacher then generates number sequences, which are filtered and used to train the student. In soft label distillation, we instead fine-tune the teacher on synthetic documents that convey the preference. The student generates continuations of Pile text, and the frozen teacher provides next-token probabilities that guide the student’s updates.
Figure 2: Subliminal learning in Pythia-160M. We measure transfer as the difference in trait probability between the student and a control student distilled on numbers generated by the untuned base model. (Left) With student hyperparameters fixed, sweeping the teacher’s finetuning hyperparameters produces a wide range of transfer. (Right) Training the student for more epochs can increase transfer but degrades language modeling abilities. Students trained on control numbers show similar degradation without acquiring a bias.
Figure 3: Top- K on-policy distillation of the bear preference, Pythia-160M teacher and student. (a) Subliminal transfer against the number K of teacher tokens kept at each position (“all” is the full 50,304 token output). Transfer is the student’s p(bear) on the 50 held out preference prompts minus that of a control trained identically, at the same K , from the placebo teacher. It is positive at every K shown and peaks at K=5 to 30 ; at K=1 every student exceeds five times the untuned student’s perplexity on held-out Pile text and is excluded (Appendix A.12 ). (b) The same distillation with the 43 animal-name tokens removed from the loss (light bar) next to the standard loss (dark bar) at K=10 ; K=1 is excluded under both losses. Points and bars are means over 3 students with 95% bootstrap intervals.
Figure 4: Same-model distillation facilitates subliminal learning but is not required. We compare transfer between Pythia-160M bases sharing initialization, data order, both, or neither. (Left) Fraction of matched student-control pairs for which the student assigns greater probability to the target animal. (Right) Median, 75th perc., 95th perc., and observed max of pstudent−pcontrol , pooled across animals. Transfer in this setting is strongest when models share the same pretrained base. Sharing initialization while changing data order retains larger effects than changing both.
Figure 5: Same-checkpoint and Cross-checkpoint transfer within Pythia-160M pretraining trajectories. (Left) Same-checkpoint transfer rises sharply during pretraining, with timing and magnitude of increase varying across models. (Right) Cross-checkpoint transfer within a model’s pretraining trajectory distinguishes a checkpoint’s ability to transmit a preference through numbers from its ability to acquire one. Some checkpoints are more effective as teachers than as students, while others show the reverse pattern.
Appendix figures & tables60 assets
Supplementary material from the paper’s appendix.
Appendix
Pile sub-corpus
Share
Pile-CC (web text)
0.35
Wikipedia
0.20
PubMed Abstracts
0.15
StackExchange
0.15
HackerNews
0.15
Appendix
Table A1: Share of each Pile sub-corpus in the synthetic corpus and in the prompt pool for on-policy distillation (Section 2.3 ).
Figure A1: Filtering removes mostly continuations that have the target animal or associated words. Share of the continuations sampled during on-policy distillation that the screen excludes from the loss, split by the first of checks (i)–(iv) that matched (legend, left to right); one match excludes the whole continuation. In each pair, the left bar is students distilled from an SDF teacher and the right bar controls distilled from a placebo teacher; the numbers are the totals. Every continuation of about 300 student and 100 control runs of PolyPythia-160M is counted (20,000 per run).
Teacher
Runs
Continuations judged
Flagged (%)
SDF teacher (students)
288
288,000
0.29 [0.27, 0.31]
Placebo teacher (controls)
96
192,000
0.09 [0.08, 0.11]
Appendix
Table A2: Continuations the LLM judge flags after the screen. Share of the continuations that passed the screen which the judge says refer to the target animal, pooled over bear, cat, dog and eagle and weighted over training; brackets give 95% intervals.
Model
Alias
GPT-NeoX tree
Hardware
retrained weight-seed1
wc1
v1.0
8× A100-80GB
retrained weight-seed2
wc2
v1.0
8× A100-80GB
retrained weight-seed3
wc3-modern
modern
8× A100-80GB
retrained Pythia-160M
Modern1234
modern
8× A100-80GB
retrained Pythia-160M
Rerun1234
v1.0
8× A100-80GB
DCLM-trained Pythia-160M
DCLM1234
modern
8× A100-80GB
Appendix
Table A3: Locally pretrained Pythia-160M models. All runs use the Pythia-160M training recipe (143,000 steps, global batch 1,024) on a single node of eight A100-80GB GPUs. The Pile-trained models share the original data order (seed 1234). wc1, wc2, and wc3-modern use initialization seeds 1, 2, and 3, respectively. Modern1234 and Rerun1234 use the original initialization (seed 1234). DCLM1234 and DCLM1 are pretrained on DCLM-baseline with initialization seeds 1234 and 1. The v1.0 and modern GPT-NeoX trees implement different learning-rate schedules, as described in the text.
Figure A2: Transfer is concentrated on the target animal, with little change on prompts where an animal response would be out of place. (A) Same-model distillation lifts the student’s target trait probability substantially over the control while the student’s aggregate preference for nine other evaluated animals falls below the control. (B) On off-topic prompts, the student’s target-trait lift persists at a small fraction of that on preference prompts.
Figure A3: Plain SGD transfers the trait once its learning rate is tuned. Full fine-tuning transfers only with SGD in these hard-label experiments. Pythia-160M students of one teacher per animal (dog, lion), trained for 20 epochs through a LoRA adapter (A) or with all parameters updated (B, C), using AdamW or plain SGD. (A, B) Transfer, defined as the student’s probability of the target animal on the held-out test prompts minus that of its control student, plotted against the student’s learning rate. (C) The full fine-tuning results from (B), plotted against the increase in Pile cross-entropy. Hollow points are degraded students, whose Pile cross-entropy reaches or exceeds the degradation threshold, corresponding to predicting Pile tokens by frequency alone (dotted line in C). Lines and bands are means over three seeds; stars indicate the selected learning rate, averaged over five seeds.
Figure A4: Steerability shows no detectable association with same-model transfer after accounting for teacher strength. (Left) Same-model transfer (control-adjusted) in Pythia-160M correlates both with teacher strength and with steerability (color). (Right) After adjusting the ranks of steerability and transfer for ranked teacher strength, their residual association is not statistically distinguishable from zero (partial Spearman ρ=−0.1 ). Each point represents one of 102 teachers.
same model
same init, diff data order
released weight-seed
Receiver
orig
ds1
ds2
ds3
ws1
ws2
ws3
Teachers ( n )
102
102
102
102
102
102
102
Marginal R2 (%)
Teacher validation p(dog)
73.8
42.2
12.8
36.2
25.5
43.8
12.2
Teacher excess CE
73.1
47.8
17.1
43.9
25.9
45.1
10.5
Mean sequence length
76.5
49.4
17.7
45.4
26.7
47.7
12.5
Appendix
Table A4: Associations with hard-label transfer before and after adjusting for teacher preference. All teachers derive from the original Pythia-160M. Receivers are Pythia-160M (the same base model, orig), models sharing its initialization but differing in pretraining data order (ds1-ds3), and the released PolyPythias weight-seed models (ws1-ws3), which share its initialization and data order but differ in optimizer history and numerical training effects (Appendix A.4.1 ). Each teacher is distilled into one student per receiver at a fixed fine-tuning recipe. All variables are ranked within each receiver’s available teacher sample. Marginal R2 measures the variance in outcome ranks explained by a predictor alone and ΔR2 measures how much more variation in transfer is explained by including that predictor alongside teacher preference. Partial r correlates predictor and outcome residuals after adjusting each for teacher preference. Steerability is the treatment effect of adding the teacher’s vector, extracted from Pythia-160M, to the receiver’s residual stream. Same-model transfer uses the Pythia-160M outcome.
Figure A5: First-number entropy decreases as subliminal learning improves during pretraining. (Left) Same-checkpoint dog transfer over pretraining for eleven Pythia-160M models. Each point averages five teacher fine-tuning recipes and three student seeds per recipe. (Right) Entropy of the teacher’s first-number distribution. Thicker lines highlight Pythia-160M, full-seed2, Modern1234, and our retrained weight-seed1 and weight-seed2 models.
Arm
Construction
R
The teacher’s real sequences
Sall
Every carrier number replaced by a draw from the teacher’s first-number distribution
Spool
As Sall , drawing from the teacher’s distribution over numbers at all positions
Sn
The first n carrier numbers replaced; remaining carrier numbers kept
Tn
As Sn , with the completion truncated after the n replaced numbers
Pall
As Sall , after permuting probabilities among numbers with the same number of digits
Appendix
Table A5: Synthetic dataset arms. Pall preserves the shape of the teacher’s number distribution while assigning its probability mass to different numbers.
Trait
Base model
pR
pSall
pC
Retained transfer (%)
95% CI
Dog
Pythia-160M
0.730
0.459
0.011
62
[42, 82]
data-seed3
0.671
0.617
0.025
92
[73, 110]
full-seed3
0.763
0.357
0.017
46
[31, 59]
Cat
Pythia-160M
0.446
0.216
0.002
48
[42, 53]
data-seed3
0.423
0.199
0.008
46
[35, 56]
full-seed3
0.695
0.388
0.005
56
[34, 73]
Appendix
Table A6: Resampled sequences retain 40-92% of control-adjusted same-model transfer across four traits and three Pythia-160M variants. Probabilities are averaged over five teacher fine-tuning recipes and three student seeds per recipe. R , Sall , and C denote real sequences, resampled sequences, and control sequences, respectively. Retention is 100(pSall−pC)/(pR−pC) . The final column gives 95% paired teacher-bootstrap confidence intervals for retention.
Figure A6: Resampled sequences retain approximately all same-checkpoint transfer at Pythia-160M checkpoints 115k and 125k, compared to 62% at the final checkpoint. Pythia-160M (original), dog, teacher and student at the same pretraining checkpoint, five shared teacher fine-tuning recipes, three student fine-tuning seeds. Each bar reports control-adjusted transfer as a percentage of that induced by real sequences, computed as a ratio of means: (parm−pC)/(pR−pC) . Transfer increases with the number of resampled numbers per completion ( T1<T3<Sall ) at all four checkpoints where all three constructions were evaluated. Keeping the carrier’s remaining numbers in each completion ( S1 , S3 ) generally reduces transfer, while permuting the probabilities assigned to numbers in the teacher’s first-number distribution ( Pall ) reduces transfer but does not eliminate it entirely at every checkpoint.
Figure A7: Completion length is associated with same-checkpoint retention, but does not fully account for checkpoint differences. Each point is a teacher at a given Pythia-160M checkpoint, evaluated for same-checkpoint transfer of the dog preference and charted against its mean number of generated numbers per completion. Colors indicate pretraining checkpoint, with teacher counts per checkpoint given in the legend. Teachers at each checkpoint comprise the five shared recipes; the remaining five of the ten main-experiment teachers at 75k, 100k, 115k, 125k, and 143k; the twelve length-stratified longer-output recipes at 115k, 125k, and 143k; and seven recipes from an earlier length-stratified random sample at 143k only. (Left) Percentage of real-sequence transfer retained after resampling all numbers in the completion. Values above 100% could reflect the length mismatch between resampled sequences (about ten numbers) and real sequences. (Right) Absolute control-adjusted transfer from resampled sequences shows the same decline with length.
Figure A8: Cross-checkpoint transfer from real and resampled sequences for dog in Pythia-160M. Rows are teacher checkpoints, columns student checkpoints. (Left) Control-adjusted transfer from real sequences, pR−pCi→j . (Middle) Control-adjusted transfer from resampled sequences, pSall−pCj,j . (Right) Retention, the ratio of the two; cells with real control-adjusted transfer below 0.10 are greyed. Values are averaged over the five shared teacher fine-tuning recipes and three student seeds.
final checkpoint
highest same-checkpoint
highest cross-checkpoint
pretraining run
retention % [95% CI]
retention % (checkpoint)
retention % (teacher → student)
Dolma, default seed
5 [4, 7]
13 (60k)
15 (55k → final)
Dolma, large-aux-2
36 [22, 55]
36 (final)
44 (65k → 50k)
Dolma, large-aux-3
28 [25, 32]
40 (60k)
56 (55k → 60k)
DCLM baseline
27 [19, 37]
27 (final)
34 (60k → final)
Appendix
Table A7: Resampled sequences preserve a limited share of transfer in the OLMo elephant experiments, retaining 5-36% of final-checkpoint transfer. The highest observed retention across different teacher-student checkpoints ranges from 15% to 56%. For the highest same-checkpoint and cross-checkpoint retention columns, we restrict to pairs with control-adjusted real-sequence transfer of at least 0.10.
Mean p(dog)
Median p(dog)
Median excess
% Num. Seq. Kept
Teacher
3–5
10–20
3–5
10–20
3–5
10–20
Δ>0
p
Ratio 3–5 / 10–20
ep5, lr 2e-4, r64 α 64
0.031
0.014
0.019
0.011
+0.007
+0.005
36/56
0.005
40%
ep7, lr 5e-4, r8 α 8
0.015
0.007
0.010
0.005
+0.003
−0.001
38/56
0.015
54%
ep7, lr 1e-4, r32 α 64
0.012
0.011
0.008
0.008
+0.000
+0.001
26/56
0.73
81%
ep20, lr 1e-4, r16 α 32
0.010
0.018
0.006
0.009
+0.000
+0.002
22/56
0.028
40%
ep10, lr 1e-4, r64 α 64
0.006
0.007
0.003
0.005
−0.002
−0.001
21/56
0.069
45%
Appendix
Table A8: Short (3-5) versus standard (10-20) in-context examples, original Pythia-160M → full-seed1, dog, 56 student cells per teacher. Excess denotes the treatment effect, computed as the difference in p(dog) between student and matched control on the held-out preference prompt set. Means and medians are over the 56 student cells. Δ>0 is the number of cells in which the excess treatment effect under 3-5 examples is larger than under 10-20. p is the two-sided paired Wilcoxon signed-rank test on that difference. The last column is the ratio of the fraction of raw completions that pass the number sequence format filter under 3-5 examples versus 10-20 examples.
Figure A9: Same-model transfer in Pythia-160M with (verbatim) and without (spliced) the in-context number examples in the student prompt. Each line is the mean over 102 teachers distilled at a single set of hyperparameters, evaluated for excess trait preference over the control. Shaded bands are 95% bootstrap confidence intervals on the mean over teachers.
Mean excess
Median excess
Control p(dog)
Spliced <
Epochs
Verbatim
Spliced
Verbatim
Spliced
Verbatim
Spliced
verbatim
p
1
0.069
0.022
0.036
0.012
0.012
0.009
98/102
4.8e-18
2
0.116
0.062
0.050
0.028
0.014
0.004
84/102
4.5e-12
5
0.218
0.110
0.136
0.041
0.009
0.009
96/102
2.5e-17
10
0.281
0.134
0.180
0.056
0.005
0.011
96/102
5.4e-17
Appendix
Table A9: Spliced versus verbatim prompts, original Pythia-160M → original Pythia-160M, dog, 102 teachers, trained for 1, 2, 5, and 10 epochs. Verbatim students train on the standard number sequence prompts containing 10-20 in-context examples. Spliced students train on the same rows with the in-context example numbers removed from the prompt. Excess denotes the treatment effect, computed as the difference in p(dog) between student and matched control on the held-out preference prompt set. Means and medians are computed over teachers. Spliced < verbatim counts teachers whose excess treatment effect in the spliced setting is below that of their verbatim setting. p is the two-sided paired Wilcoxon signed-rank test on that difference.
Figure A10: On-policy distillation from an SDF teacher and from a hard-distillation teacher. Teacher and student are built from Pythia-160M; the target animal is lion. (left) Increase in p(lion) of each student over its matched control on the 50 held-out preference prompts. (right) Increase in Pile cross-entropy over the untuned base, on the 250 Pile excerpts of Figure A11 , for the same students and their controls (grey).
Figure A11: Soft Label on-policy distillation against the number of student samples. Teacher and student share the base model; a sample is one prompt with the continuation the student wrote for it. (left) Increase in p(trait) of each student over its matched control. (right) Increase in Pile cross-entropy over the untuned base for the same students and their controls (grey), measured on 250 Pile excerpts rather than the validation shard used in Figure 2 . Boxes hold 15 bear students (five base models) or 9 students of another animal (three base models); cat was not trained on 60,000 samples.
Figure A12: Training text and divergence, Pythia-160M → Pythia-160M (bear). Students distilled on their own samples or on real Pile text, with the reverse or the forward KL. (left) Increase in p(bear) of each student over its matched control. (right) Increase in Pile cross-entropy over the same controls, on the 250 Pile excerpts of Figure A11 . Each box holds 9 students (three teacher seeds, three distillation seeds), with whiskers at the minimum and maximum; the dashed line marks the control.
Figure A13: Transfer against the number of distinct Pile samples at a fixed 625 steps of 32 sequences.
Figure A14: Where the belief sits changes transfer more than it changes the teacher. (a) Teacher p(bear) on the 50 held-out preference prompts and (b) transfer, the student’s p(bear) minus that of the controls, for corpora of about 8,000 documents (solid) and nested subsets of about 2,000 (hatched) whose mentions of bears all sit in the first quarter of each document (start), its middle (middle), its last quarter (end), or a third of each (mix). Bars are means over teachers (a) and students (b), dots are single models, error bars are 95% bootstrap intervals that resample teachers and then each teacher’s students, and the dashed line is the original bear corpus.
Figure A15: Transfer falls steadily as the belief is deleted from more documents. (a) Teacher p(bear) and (b) transfer for the mix documents of Figure A14 (about 8,000), with the belief sentences kept in only the given share of them. Dots, error bars and dashed line as in Figure A14 .
Figure A16: A first-person belief gives the most biased teacher and the least transfer. (a) Teacher p(bear) and (b) transfer for corpora of about 2,000 documents whose belief is written in the first person, in the third person, or mixed (16% first person). Each model is evaluated on the 50 held-out first-person prompts (solid) and on 50 third-person prompts (hatched). Dots and error bars as in Figure A14 ; the dashed lines are the original bear corpus on each set of prompts.
Figure A17: A belief at the start transfers least at every corpus size. (a) Teacher p(bear) and (b) transfer for the position corpora of Figure A14 and their nested subsets. Every teacher trains for 750 steps, so smaller corpora are repeated more often. Points are means, the four series are shifted slightly sideways for readability, and error bars and the dashed line are as in Figure A14 .
Figure A18: Perplexity cost of truncation. Perplexity on 250 Pile excerpts, divided by the untuned student’s, against K for the students of Figure 3 a, averaged over all students including excluded ones. Students above the dotted line (five times) are excluded from the transfer estimates.
Figure A19: Top- K on-policy distillation of the bear preference between Pythia-160M and Pythia-410M. (a) Transfer against the number K of teacher tokens kept at each position, for 160M → 410M and 410M → 160M, with controls as in Figure 3 . (b) 160M → 410M with the 43 animal-name tokens removed from the loss (light bar) next to the standard loss (dark bar) at K=10 and K=1 . Points and bars are means over 9 students (three teacher fine-tuning seeds, three distillation seeds; 8 for 410M → 160M at K=1 ) with 95% bootstrap intervals.
K
1
2
5
10
30
100
1000
Positions with bear in SK (%)
0.023
0.047
0.13
0.29
1.0
3.7
28
Teacher probability held by SK
0.46
0.55
0.67
0.74
0.84
0.91
0.98
Appendix
Table A10: The bear token among the teacher’s kept tokens , 160M → 410M, on the student’s own samples.
Figure A20: The student’s own samples against real Pile text. Transfer (top) and perplexity ratio (bottom) against K for students trained on their own samples (solid) or on the Pile continuations of each prompt (dashed), with the same prompts, loss, seeds and matched controls. 9 students per point, 95% bootstrap intervals.
Figure A21: Tokens the top-10 loss keeps, Pythia-160M teacher and student. For students trained with K=10 , the 20 tokens most often among the teacher’s 10 most likely next tokens, as a share of the positions each student trained on (its own samples): with the standard loss (left), and with the 43 animal-name tokens removed (right), where the ten are chosen from the remaining 50,261 tokens. Bars pool the 3 students and thin lines span their range; the box gives where “ bear” ranks and how often it is among the ten. Figures A22 and A23 show the removed tokens and what replaces them.
Figure A22: The 43 animal-name tokens before removal. For the standard-loss students of Figure A21 (left), the share of training positions at which the teacher ranks each single-token spelling of the ten candidate animals among its 10 most likely next tokens (log scale; bear spellings in colour, ␣ marks a leading space). Removing the tokens sets every share to zero. The box gives the share of positions with any of the 43 among the ten, and the teacher’s mean probability of “ bear” where it is among them.
Figure A23: Tokens that replace the removed animal-name tokens. At the training positions of the standard-loss students of Figure A21 (left) where one of the 43 tokens is among the teacher’s 10 most likely, the tokens that enter the ten once the 43 are removed, as a share of those positions.
Figure A24: What the bear fine-tuning changes in the teacher’s top 10, against the untuned base model. For the standard-loss students of Figure A21 (left), the untuned Pythia-160M the teacher was fine-tuned from, which is also the students’ starting point, is run on the same training positions, and each token’s share of positions among its 10 most likely next tokens is subtracted from the token’s share among the teacher’s: the 20 largest increases (left), the 20 largest decreases (middle), and the 20 largest ratios of the two shares among the tokens the teacher keeps at 0.1% of positions or more (right, log scale). Bear spellings in colour, ␣ marks a leading space. Bars pool the 3 students and thin lines (left, middle) span their range; the box gives both shares of “ bear” and its rank among all increases.
Figure A25: Distribution of transfer across model size and architecture and across OLMo pretraining conditions. We illustrate the median, 75th percentile, 95th percentile, and observed maximum of transfer in the cross model size, cross model architecture, and OLMo-1B experiments. A : Transfer between Pythia-160M and Pythia-410M, using the original models and full-seed variants 1-3 at each size. Models with shared seeds share a pretraining data order. Different seeds do not. Cross-architecture comparisons pair three Pythia-160M models with three OLMo-1B Dolma1.7 models. B : DataDecide OLMo-1B (Dolma 1.7, DCLM-baseline, C4), elephant, grouped by pretraining condition: same model (3 arms), shared initialization with a different corpus (6 arms), different initialization with a different corpus (6 arms), and different initialization with the same corpus (3 arms). As in Pythia, same-model transfer dwarfs every cross condition in OLMo at the evaluated training endpoints. Across size and architecture, medians are near zero in every direction. Descriptively, the smaller receiver (160M) shows the larger upper tail in the cross-Pythia experiments, while OLMo-to-Pythia transfer has a longer upper tail but a lower median than transfer in the reverse direction.
# HP
Model Family
Pretraining Condition
Arms
# Arms
Teacher
Student
Total
Animal
Exceedances
Null 95th Perc. Exceedances
P-value
Pythia
Same-model
{160M, fs1–3, ds1–3,
8
10
30
2400
dog
2303 (96.0%)
120 (5.0%)
5e-5
rerun 160M}
{160M, fs1–3}
4
10
30
1200
cat
1159 (96.6%)
71 (5.9%)
5e-5
lion
1178 (98.2%)
72 (6.0%)
5e-5
pig
1166 (97.2%)
72 (6.0%)
5e-5
Appendix
Table A11: Transfer existence by pretraining condition. Stratified exceedance permutation test results, per A.13 . Bold = p<0.05 .
# HP
Comparison
Arms A
Arms B
# Arms A vs B
Teacher
Student
Total A vs B
Animal
Treatment Effect Mode
Diff. in Means
Mean P-value
Win Prob.
Win P-value
Same-model vs
{160M,
{160M,
8 vs 12
10
30
2400
dog
Δ
0.379
1.6e-5
0.908
3.2e-5
same init, diff.
fs1–3,
ds1–3} 2
vs 3600
Δ / prior
39.3
1.6e-5
0.915
1.6e-5
data order
ds1–3,
All
Δ
0.32
0.0025
0.957
1.6e-5
rerun 160M}
Δ / prior
98.2
0.01
0.959
1.6e-5
Same-model vs
{160M,
{wc1, wc2,
8 vs 6
10
30
2400
dog
Δ
0.457
6.7e-4
0.975
6.7e-4
Appendix
Table A12: Relative Strength of transfer between pretraining conditions in Pythia. We compare pretraining conditions that vary weight initialization or data order in the Pythia family, computing Difference in Means and Win Probability statistics on both ( Δ=pstudent−pcontrol ) and Δ/ prior. Rows: dog, and all four animals pooled. A positive Difference in Means, or a Win Probability above 0.5, means the condition named first under Comparison transfers more strongly. Bold = two-sided p<0.05
# HP
Comparison
Arms A
Arms B
# Arms A vs B
Teacher
Student
Total A vs B
Animal
Treatment Effect Mode
Diff. in Means
Mean P-value
Win Prob.
Win P-value
Same-model vs
{Dolma1.7,
{Dolma1.7,
3 vs 6
10
35
1050
elephant
Δ
0.0938
0.024
0.886
0.024
same init,
DCLM, C4}
DCLM, C4} 2
vs 2100
Δ / prior
3.34
0.024
0.889
0.024
diff. data
corpus
Same-model vs
{Dolma1.7,
{DCLM,
3 vs 6
10
35
1050
elephant
Δ
0.0909
0.024
0.868
0.024
Appendix
Table A13: Relative Strength of transfer between pretraining conditions in DataDecide OLMo-1B. We compare pretraining conditions that vary weight initialization or data corpus in the DataDecide OLMo-1B family, computing Difference in Means and Win Probability statistics on both ( Δ=pstudent−pcontrol ) and Δ/ prior. A positive Difference in Means, or a Win Probability above 0.5, means the condition named first under Comparison transfers more strongly. Bold = two-sided p<0.05
Model Family
Comparison
# Arms A vs B
Total A vs B
Treatment Effect Mode
Diff. in Means
P-value
Paired P-value
Pythia/OLMo
Small to large vs
9 vs 9
3360
Δ
3e-3
0.42
0.41
Cross Arch.
Large to small
vs 2160
Δ / prior
-0.0831
0.8
0.8
Pythia-160M/410M
Small to large vs
16 vs 16
4800
Δ
-4e-3
0.18
0.15
Large to small
vs 4800
Δ / prior
-0.558
0.043
0.048
Small to large:
4 vs 12
1200
Δ
-2e-3
0.62
–
Same vs
vs 3600
Δ / prior
-0.0674
0.58
–
Appendix
Table A14: Relative strength of transfer by direction and data order across model sizes and architectures. All arms use the dog preference. Each arm is summarized against its own matched controls by Δ=pstudent−pcontrol over runs, and by Δ/ prior (transfer in units of the receiver prior). Arm sets are those of Table A11 . A positive Diff. in Means means the side named first under Comparison transfers more strongly. Bold = p<0.05 .
Figure A26: On-policy distillation transfers in all four relationships of Figure 4 , with no same-model advantage. (A) Mean pstudent−pcontrol per condition, pooled over animals (hatched) and per animal. (B) Median, 75th and 95th percentile, and maximum of pstudent−pcontrol , pooled over animals.
Figure A27: The statistic of Figure 4 under on-policy distillation. (A) Fraction of students that assign more probability to the target animal than their control. (B) As in Figure A26 .
Figure A28: On-policy distillation between models of different sizes and families. Each bar is one teacher → student pair for the bear preference, grouped by student. Each group opens with the student’s same-model pair (hatched), and the dashed line carries its level across the group. Bars are means of pstudent−pcontrol with 95% intervals, the wider of a cluster bootstrap over both seeds and a t -interval over teacher seeds.
Figure A29: Same-model transfer can peak early and decline while cross-model transfer continues to rise in Pythia. Curves show mean control-adjusted dog preference during student training. (Left) Same-model transfer for models where endpoint transfer is <70% of peak. (Center) Same-model transfer for models where endpoint transfer is ≥70% of peak. (Right) Cross-model transfer, averaged over all arms belonging to the same pretraining condition, increases with further student training. Shaded bands are 95% intervals from a bootstrap over teacher clusters.
Figure A30: Same-model and cross-model transfer both peak before training ends in OLMo. Curves show mean control-adjusted elephant preference during student training. (Left) Same-model transfer peaks then declines for OLMo-1B models pretrained on Dolma1.7, DCLM-baseline, and C4. (Right) Cross-model transfer, averaged over all arms belonging to the same pretraining condition, declines less sharply than same-model transfer, then levels off.
Figure A31: Cross-model transfer can overtake same-model transfer during student training. Each panel holds the teachers and their generated number sequences fixed and varies the receiver base model. Black curves show same-model transfer. Colored curves show cross-model transfer. Curves report median control-adjusted transfer, normalized by the receiver’s base preference, across four Pythia pretraining conditions. Same-model transfer is typically stronger throughout training, with two notable exceptions where Modern1234 is the sender. Here, cross-model transfer from Modern1234 to Pythia-160M and Rerun1234 overtakes same-model transfer midway through training.
Figure A32: Cross-model transfer shows distinct sender and receiver patterns. (Bottom) Grids show mean control-adjusted transfer from each teacher base model (rows) to each student base model (columns), in percentage points. (Top) Bars summarize each model’s average outgoing minus incoming transfer over the other models in its panel. Positive values indicate net senders and negative values net receivers, relative to the evaluated partners and training settings.
Figure A33: Cross-model asymmetry often persists in a single direction throughout student training, although some pairs can reverse direction. Each curve shows mean control-adjusted transfer in the direction named in the legend minus transfer in the reverse direction. Positive values favor the named direction, while negative values favor the reverse. Panels show the dog preference across four Pythia pretraining conditions.
Figure A34: Cross-checkpoint transfer when holding (a) early teachers fixed, (b) late teachers fixed, (c) early students fixed, (d) late students fixed. Aggregated over five teacher recipes and three student finetuning seeds.
Figure A35: Cross-checkpoint transfer when holding (a) early teacher (75k) fixed, (b) late teacher (125k) fixed, (c) early student (75k) fixed, (d) late student (125k) fixed. Averaged over five student distillation recipes and three student finetuning seeds.
In-sample R2
Held-out R2
Trait / model
Prod.
Add.
Bottl.
Prox.
Prod.
Add.
Bottl.
Prox.
Dog / Pythia-160M
0.967
0.794
0.921
0.983
0.953
0.704
0.881
0.972
Dog / data-seed1
0.859
0.666
0.845
0.954
0.806
0.519
0.785
0.924
Dog / data-seed2
0.882
0.734
0.808
0.962
0.831
0.617
0.750
0.935
Dog / data-seed3
0.913
0.744
0.868
0.956
0.868
0.634
0.815
0.925
Dog / full-seed1
0.928
0.715
0.867
0.968
0.894
0.591
0.794
0.937
Appendix
Table A15: Separate teacher and student factors explain most of in-sample and held-out cross-checkpoint transfer for Pythia-160M. Each row is a 13×13 grid of checkpoints from a single model’s pretraining trajectory. Each cell is the student’s control-adjusted preference transfer, averaged over five teacher fine-tuning recipes and three student fine-tuning seeds. Columns give R2 for the product, additive, maturity-bottleneck, and product-plus-proximity models. Held-out R2 is calculated from pooled out-of-fold predictions, then averaged over five independent random partitions.
In-sample R2
Held-out R2
Corpus / seed
Prod.
Add.
Bottl.
Prox.
Prod.
Add.
Bottl.
Prox.
dolma1_7 / default
0.931
0.764
0.893
0.980
0.890
0.535
0.764
0.952
dolma1_7 / large-aux-2
0.646
0.539
0.646
0.888
0.315
0.138
0.516
0.697
dolma1_7 / large-aux-3
0.925
0.751
0.902
0.978
0.855
0.520
0.787
0.941
dclm-baseline / default
0.943
0.777
0.843
0.973
0.906
0.561
0.708
0.938
Appendix
Table A16: Separate teacher and student factors explain cross-checkpoint transfer in 3 of 4 evaluated OLMo-1B models. Each row is an 8×8 grid of checkpoints (30k, 40k, 45k, 50k, 55k, 60k, 65k, and final 69k) from a single model’s pretraining trajectory. Each cell is the student’s control-adjusted elephant preference transfer, averaged over five teacher fine-tuning recipes and three student fine-tuning seeds. Columns as in Table A15 . Held-out R2 use five disjoint folds, each withholding 12-13 of 64 cells. Three of the four grids factorize. On the Dolma 1.7 large-aux-2 pretraining seed, the plain product fails.
Figure A36: Same-checkpoint transfer at different stages of student training. (Left) In Pythia-160M, transfer generally strengthens with continued student training across the evaluated pretraining checkpoints. (Right) In full-seed2, transfer peaks after five epochs and then declines across pretraining checkpoints, extending the final-checkpoints observations in Appendix A.15.1 .
Pretraining condition
Modern1234
Pythia-160M
wc1
Same-model
itself
itself
itself
Same init, different data order
data-seed1
data-seed1
full-seed1
Different init, same data order
wc3 (modern)
wc3 (modern)
Pythia-160M
Different init, different data order
full-seed1
full-seed2
full-seed2
Same init, different pretraining corpus
DCLM1234
DCLM1234
DCLM1
Different init, different pretraining corpus
DCLM1
DCLM1
DCLM1234
Appendix
Table A17: Pairs in the soft-label cross-checkpoint grids. Each column is an anchor model, and each entry is the model it is paired with in that pretraining condition. Every pair of different models is distilled in both directions.
Figure A37: Same-checkpoint transfer under soft-label distillation. Teacher and student sit at the same checkpoint (x axis, 1k to 143k steps). One plot per pretraining condition, plus Pythia-160M with Modern1234 (same init, same data order (retrained)); plot titles give the number of pairs and of sender → receiver combinations.
Figure A38: Early checkpoints can transmit a preference more effectively than they acquire it. (a) One pair of triangles per cross-checkpoint grid, grouped by pretraining condition: the upward filled triangle is the mean transfer from the grid’s 2k teacher into its students between 8k–143k, the downward empty triangle the mean transfer from its teachers at 8k–143k into its 2k student. (b) Their difference per grid (dots), and the condition mean over its three pairs with a 95% CI (diamonds).
Figure A39: The onset in each pretraining condition. Solid line: the 2k teacher into a student at checkpoint x .
Figure A40: Nine checkpoints into Pythia-160M. Cross-checkpoint grids from 1k to 143k steps with Pythia-160M as the receiver: its same-model grid, DCLM1234 → Pythia-160M and DCLM1 → Pythia-160M.
Figure A41: Teacher preference across pretraining. Teacher preference, the SDF teacher’s own p(bear) averaged over three fine-tuning seeds, against the teacher’s checkpoint. Thin colored lines are one model each, colored by training stack.
Figure A42: Transfer relative to teacher preference, and longer teacher fine-tuning. (a) Transfer divided by teacher preference into the 143k student, by teacher checkpoint; the dotted line marks a student that gains as much as its teacher. (b) Teachers at 143k fine-tuned for 2, 5 or 8 epochs(log), for wc1 with itself and with the five models it is paired with.
Figure A43: Tree and color preferences also transfer under hard label (A) and soft label (B) distillation. Transfer measured on 50 held-out preference prompts.
Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.
Knowledge distillation can transmit unintended behavioral traits from a teacher model to a student through training data that appear semantically unrelated to those traits, a phenomenon known as subliminal learning. Although recent work has established this effect, its training dynamics and mitigation remain underexplored. We study subliminal learning in open-weight language models ranging from 1.5B to 8B parameters, covering the Qwen, Gemma, and Llama families in number-sequence and chain-of-thought settings. Rather than evaluating only final models, we track trait-related probabilities throughout fine-tuning and find that subliminal acquisition can be highly non-monotonic, with transient spikes, reversals, and trait-specific failures of transfer. We then introduce liminal training, an annealed KL-regularized fine-tuning method that constrains early drift from the base model. Across our experiments, liminal training substantially reduces subliminal trait acquisition while largely preserving task gains, outperforming paraphrasing and layer freezing as mitigation strategies. The effect also extends beyond animal preferences: in a French-language response-style experiment, liminal training suppresses language transfer while retaining much of the GSM8K improvement. Finally, we show that KL timing matters: early regularization is more effective than late regularization, and sweeping the regularization strength reveals an empirical trade-off between task learning and trait suppression.
Distillation of a language model intended to transfer benign behavior to a student model may also transfer undesirable characteristics, if they are present in the teacher model, a phenomenon known as subliminal learning. While qualitative evidence supports the existence of this effect, its magnitude has not been systematically characterized. This study quantifies subliminal behavioral transfer ratios by steering two teacher models (Llama-2-7B-Chat and Qwen2.5-7B-Instruct) at varying steering strengths and distilling student models using only benign data. Evaluation on 100 JailbreakBench prompts with GPT-4.1, serving as the evaluator, indicates that transfer is robust but exhibits distinct scaling behaviors. Llama-2 demonstrates a sharp threshold (τ=0.25,0.32beyondα=−0.15), whereas Qwen2.5 displays continuous and higher levels of transfer (τ up to 0.61).
Uwe König, Hamza Kazmi, Ruizhe Li +1
1Technical University of Munich · 2Algoverse · University of Aberdeen +1
When language models are trained on synthetic data, they (student model) can covertly acquire behavioral traits from the data-generating model (teacher model). Subliminal learning refers to the transmission of traits from a teacher to a student model via training on data unrelated to those traits. Prior work demonstrated this in the training domains of number sequences, code, and math Chain-of-Thought traces including transmission of misaligned behaviors. We investigate whether transmission occurs through natural language paraphrases with fixed semantic content, and whether content explicitly contradicting the teacher's preference can block it. We find that training on paraphrases from a teacher system-prompted to love a particular animal increases a student's preference for that animal by up to 19 percentage points. This occurs when paraphrased content is semantically unrelated to the animal, or even when it explicitly expresses dislike. The transmission succeeds despite aggressive filtering to ensure paraphrase fidelity. This raises concerns for pipelines where models generate their own training data: content-based inspection cannot detect such transmission, and even preference-contradicting content fails to prevent it.
Isaia Gisler, Zhonghao He, Tianyi Qiu
ETH Zürich · University of Cambridge · Peking University