Regularization Amplifies Subliminal Learning
Organizations: Carnegie Mellon University
Abstract
Recent work has shown that hidden biases can be transferred through data with no apparent semantic relationship to those biases. Can regularization limit this covert influence? By constraining adaptation, regularization might be expected to suppress behavioral changes on seemingly unrelated inputs. We systematically study this hypothesis in subliminal learning, as well as in related settings of covert influence, including phantom transfer and logit-linear selection. We find the opposite: regularization amplifies the transfer of hidden biases. In particular, subliminal learning is typically absent under full fine-tuning, but low-rank adaptation or explicit penalties can induce strong transfer. Phantom transfer and logit-linear selection show similar trends. Adding more data, an implicit form of regularization, also strengthens covert influence. Together, these findings suggest that the simplicity biases induced by regularization may facilitate covert influence. We further hypothesize that the level of regularization that maximizes covert transfer depends on task complexity.
Figures & tables
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| spread of the update | keep top per matrix | ||||||||
| trained size | rescaled | ||||||||
| student | eff. rank | top 1 / 8 (%) | full | 2 | 8 | 2 | 8 | ||
| Subliminal learning, owl, 250k examples (full fine-tune: 1M), no penalty | |||||||||
| LoRA rank 2 | 1.6 | 75 / 100 | 98 | 98 | 98 | 99 | – | – | – |
| LoRA rank 8 | 4.8 | 41 / 100 | 100 | 99 | 100 | 100 | – | – | – |
| LoRA rank 32 | 16.7 | 21 / 59 | 98 | 96 | 98 | 98 | – | – | – |
| dataset | scored pairs | positive pool | training set |
|---|---|---|---|
| Stack Exchange | 1,550,000 | 744,166 | top 5% (37,209 pairs) |
| SHP | 327,976 | 153,994 | 15,000 pairs drawn from the top 37.4% |
| HH-RLHF | 112,878 | 53,027 | top 28.3% (14,995 pairs) |
| OpenAssistant | 32,939 | 13,978 | the whole pool |
Subliminal Learning
Subliminal learning is the transfer of hidden preferences or behaviors through training data that appears semantically unrelated to them.
See all 29 papers