Subliminal Reinforcement Learning: traits travel through a judge's scores
Organizations: Department of Computer Science, Hanyang University, Seoul, Republic of Korea
Abstract
Subliminal learning, the transfer of a model's trait through data with no semantic trace of it, has been shown for students trained on a teacher's text. Post-training relies on group relative policy optimization with a language-model judge, where the student learns only from its own outputs and one judge's scores. We show this suffices: a student writes invented syllables or numbers, a judge made to prefer one animal scores them, and the student's preference moves toward that animal relative to a run whose judge prefers another. Of 48 design cells, two fail the validity gate and 17 establish the trait at 5% false-discovery rate, numbers-only cells included; elsewhere, our account gives the reason. Removing only the reward difference between judges removes transfer. We establish the principle as a chain of four stages: the trait must show in the judge's scores, favour strings that move the student, accumulate and reach the answers. Judges with the same preference built from other data, or on another model, place it on other strings, and transfer is not detected. Carried unchanged to AI-safety dispositions, the chain holds: undesirable dispositions reach the student's answers, and stages measured before training anticipate where, helping to prevent part of this risk.
Figures & tables
| Cell | Emission | Absorption | Pairs | Transfer | Behaviour | ||||||
| FT route — fine-tuned judges | |||||||||||
| Llama · syllable | |||||||||||
| elephant–tiger | [ , ] | [ , ] | [ , ] | 10/20 | 0.248 | 0.408 | — | answer evaluation degenerate | |||
| jerboa–badger | [ , ] | [ , ] | [ , ] | 5/8 | 0.500 | 0.605 | — | answer evaluation degenerate | |||
| otter–beaver | [ , ] | [ , ] | [ , ] | 15/16 | 0.001 | established | sampled choice 0 (+0.0009) | ||||
| shrew–cuttlefish | [ , ] | [ , ] | [ , ] | 13/16 | 0.001 | 0.010 | established | sampled choice 0 (+0.0007) | |||
| Transfer | Behaviour | |||||||
|---|---|---|---|---|---|---|---|---|
| Axis | Emission | Absorption | Baseline | Steps established | , step 80 | Forced choice | Open question | Implied choice shift |
| Llama-3.2-1B · syllable carrier | ||||||||
| rude–polite | +0.233 | +0.081 | 0.108 | 0.064 | ||||
| [+0.216, +0.249] | 0.751 | |||||||
| harsh–gentle | +0.391 | +0.252 | 1.091 | +0.065 | ||||
| [+0.367, +0.416] | 0.751 | |||||||
| Cell | , T-trained | , E-trained | Common push | (eight pairs) | Pairs moving the same way |
|---|---|---|---|---|---|
| Llama · FT · syllable | |||||
| elephant–tiger | +0.1421 | +0.1436 | +0.1428 | 0.0008 | 8/8 |
| jerboa–badger | 0.1820 | 0.2098 | 0.1959 | +0.0139 | 8/8 |
| otter–beaver | 0.0957 | 0.1387 | 0.1172 | +0.0215 | 8/8 |
| shrew–cuttlefish | +0.2164 | +0.1593 | +0.1879 | +0.0285 | 8/8 |
| sloth–lemur | +0.0503 | 0.0965 | 0.0231 | +0.0734 | 4/8 |
| Further seed pairs alone | All seed pairs combined | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Cell | Pairs | Further pairs | 95% CI | Judge’s direction | (35 cells) | Meets criterion | , base eight | Combined | (46 cells) | Meets criterion | ||
| FT route — fine-tuned judges | ||||||||||||
| syllable · Llama | ||||||||||||
| elephant–tiger | 20 | 12 | +0.046 | [ 0.018, +0.109] | 6/12 | 0.207 | 0.361 | — | 0.977 | 0.419 | 0.642 | — |
| jerboa–badger | 8 | — | — | — | — | — | — | — | 0.500 | 1.000 | 1.000 | — |
| otter–beaver | 16 | 8 | +0.031 | [+0.019, +0.043] | 8/8 | 0.008 | 0.055 | yes | 0.023 | 0.001 | 0.015 | yes |
| Cell | Absorption (training instance) | Feature part | Residual | Judge-side coefficient | Student-side coefficient | Instance-mean absorption | Its feature part |
|---|---|---|---|---|---|---|---|
| FT route — fine-tuned judges | |||||||
| syllable · Llama letters + bigrams; coefficient = match with the animal-name letters | |||||||
| elephant–tiger | +0.245 | +0.115 | +0.130 | +0.539 | +0.417 | +0.150 | +0.102 |
| jerboa–badger | +0.199 | +0.082 | +0.117 | +0.374 | +0.540 | +0.070 | +0.039 |
| otter–beaver | +0.180 | +0.073 | +0.107 | +0.297 | +0.685 | +0.261 | +0.115 |
| shrew–cuttlefish | +0.209 | +0.084 | +0.125 | 0.068 | +0.471 | +0.163 | +0.085 |
| Cell | (pairs in judge’s direction) | Paired difference from | KL | Validity | |
|---|---|---|---|---|---|
| Llama · FT · syllable · sloth–lemur | 0 | (2/8) | , | 1.04 | 0.822 |
| 1 | — | 1.13 | 0.820 | ||
| 2 | (8/8) | , | 1.06 | 0.814 | |
| Llama · SP · number · sloth–lemur | 0 | (5/8) | , | 0.89 | 0.987 |
| 1 | — | 0.92 | 0.987 | ||
| 2 | , | — | — |
| Cell | Validity, step 2 / 48 / 80 | Lowest run, step 80 | Reward-signal share, step 10 / 40 / 80 | First step with signal share | Step reaching 50% / 80% of |
|---|---|---|---|---|---|
| FT route — fine-tuned judges | |||||
| syllable · Llama | |||||
| elephant–tiger | 0.530 / 0.961 / 0.981 | 0.973 | 0.97 / 0.92 / 0.61 | never | — |
| jerboa–badger | 0.523 / 0.958 / 0.980 | 0.964 | 0.97 / 0.91 / 0.78 | never | — |
| otter–beaver | 0.542 / 0.967 / 0.980 | 0.968 | 0.97 / 0.90 / 0.72 | never | 64 / 64 |
| shrew–cuttlefish | 0.526 / 0.959 / 0.981 | 0.971 | 0.97 / 0.95 / 0.71 | never | 32 / 48 |
| Judge | Emission (mean) | lower bound | Trait alignment with own-model judge ( ) | Absorption | Absorption sign as own model | Judge term | Absorption vs ( ) | Transfer |
|---|---|---|---|---|---|---|---|---|
| FT route — fine-tuned judges | ||||||||
| Student Llama-3.2-1B | ||||||||
| Own model (main grid) | +0.117 | 12/12 | — | 0.170 | — | 0.113 | +0.38 | 4 of 12 established |
| Other family, 1B class | +0.128 | 12/12 | +0.03 | 0.066 | 8/12 | 0.037 | 0.05 | none established |
| Same family, 8B | +0.103 | 12/12 | +0.07 | 0.090 | 9/12 | 0.072 | — | 0 of 2 trained cells |
| Other family, 8B | +0.214 | 12/12 | +0.02 | 0.047 | 9/12 | 0.080 | — | 0 of 2 trained cells |
| Following ratio | Energy of in top 1% | ||||||
|---|---|---|---|---|---|---|---|
| Cell | Route | Layers 0–3 | Deeper | Layers 0–3 | Deeper | Layers 0–3 | Deeper |
| Qwen · SP · syllable · warbler–slug | SP | 0.23 | 2.0–3.2 | 4.4 | 1.4–1.8 | 0.94–0.96 | 0.60–0.81 |
| Llama · SP · number · sloth–lemur | SP | 0.56 | 1.5–2.9 | 2.7 | 1.3–1.5 | 0.86–0.88 | 0.39–0.67 |
| Llama · FT · syllable · sloth–lemur | FT | 0.47 | 1.8–2.3 | 2.3 | 1.0–1.2 | 0.86–0.89 | 0.37–0.68 |
| Qwen · FT · number · warbler–slug | FT | 0.31 | 1.4–3.2 | 4.4 | 1.5–2.7 | 0.95 | 0.63–0.90 |
Subliminal Learning
Subliminal learning is the transfer of hidden preferences or behaviors through training data that appears semantically unrelated to them.
See all 29 papers