DETER: Suppression During Pretraining for Tamper Resistance
Organizations: EPFL
Abstract
Open-weight language models can acquire knowledge that could enable biological, chemical, or cyber misuse. Any attempt to limit such knowledge must remain effective after the model weights are released and can be fine-tuned. Unlearning applied after pretraining can be rapidly undone, whereas filtering the pretraining corpus offers greater resistance. We ask whether actively suppressing knowledge during pretraining can further increase the fine-tuning required to acquire it. Even simple gradient ascent interleaved with standard pretraining on the filtered corpus makes medical knowledge harder to acquire through adversarial fine-tuning than filtering alone or post-hoc unlearning. At 1B parameters, reaching the held-out PubMed loss of an unfiltered model requires over 20 as many PubMed fine-tuning tokens as after filtering alone. Retained knowledge is evaluated using both held-out data and downstream tasks. Across model sizes, varying suppression strength reveals an increasingly favorable trade-off between fine-tuning difficulty and held-out loss on retained pretraining data. These results point to pretraining as an effective stage for making unwanted capabilities harder to acquire through subsequent fine-tuning. Code is available at github.com/MEGlr/DETER.
Figures & tables
| Before medical fine-tuning on PubMed | After medical fine-tuning | ||||||
| Size | Method | General loss | General task loss | Closest-retain task loss | Medical task loss | Tokens (M) | Tokens Filtering |
| 113M | Filtering | ||||||
| Token filtering | |||||||
| RMU | |||||||
| Deter | |||||||
| 224M | Filtering | ||||||
| Before medical fine-tuning on PubMed | After medical fine-tuning | ||||
| Size | Start | General loss | General task loss | Closest-retain task loss | Tokens Filtering |
| 113M | 0% | ||||
| 20% | |||||
| 60% | |||||
| 224M | 0% | ||||
| 20% | |||||
| Before medical fine-tuning on PubMed | After medical fine-tuning | |||
| General loss | Closest-retain loss | Medical loss | Fine-tuning tokens Filtering | |
| Muon , 113M | ||||
| Separate | 1.66 | |||
| Shared | 1.62 | |||
| AdamW , 113M | ||||
| Separate | 1.93 | |||
| General loss | Retain / target | Tokens Filtering | |
| AdamW , 113M | |||
| Filtering | 1.41 | ||
| Token, confident | 1.01 | ||
| Token | 1.03 | ||
| Document | 1.08 | ||
| Muon , 113M | |||
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Layers | Heads | Embed dim |
|---|---|---|---|
| 61M | 7 | 8 | 448 |
| 113M | 10 | 10 | 640 |
| 224M | 14 | 14 | 896 |
| 521M | 20 | 10 | 1,280 |
| 1030M | 26 | 16 | 1,664 |
| Teacher | General loss | Tokens Filtering |
|---|---|---|
| 0% | ||
| 5% | ||
| Rolling |
| Retain loss | General loss | Tokens Filtering | Compute | |
|---|---|---|---|---|
| No | 1 | 1 | ||
| Yes | 2 | |||
| No | 2 | 2 |
| Model | Start | Reported in | |||
| Muon , gradient ascent, whole document | |||||
| Main models | 113M, 224M, 521M | 0% | 50 | 5 | Tables 1 , 2 , 4 , 12 ; Figures 2 – 5 , 11 |
| 1030M | 0% | 50 | 3 | Tables 1 , 10 , 12 ; Figures 2 – 4 , 11 | |
| Start time | 113M | 20% | 40 | 4.5 | Table 2 |
| 113M | 60% | 20 | 3.8 | Tables 2 , 14 | |
| 113M | 0% | 20 | 2.2 | Table 14 | |
| Searched | Used | |||||
| AdamW | Muon | 113M | 224M | 521M | 1030M | |
| Retain weight | 0.1, 1, 2, 5, 10, 50, 100, 1000, 1200 | 2, 4, 12, 25, 50, 100, 125, 150, 175, 250, 500, 1000 | 125 | 50 | 25 | 1 |
| Steering strength | 0.5, 2.5, 5, 7.5, 10, 12.5, 25 | 5, 12.5 | 5 | 5 | 5 | 5 |
| Learning rate | ; ; ; | ; ; ; | ||||
| Steered activation | MLP, block | MLP | MLP | MLP | MLP | MLP |
| Size | Method | Optimizer | General loss | PubMed loss | Fine-tuning tokens Filtering |
|---|---|---|---|---|---|
| 521M | Filtering | Muon | |||
| Gradient ascent | Muon | ||||
| Gradient ascent | AdamW | ||||
| RMU | Muon | ||||
| RMU | AdamW | ||||
| Deter | Muon |
| Benchmark | Split / subset | Items |
| Medical | ||
| MedQuAD | Answered questions | 16,407 |
| MedMCQA | Validation | 4,183 |
| MedQA-USMLE (4 options) | Test | 1,273 |
| MMLU medicine (6 subjects) | Test | 1,111 |
| Biology | ||
| Model | Method | Peak learning rate | Tokens to parity (M) |
| Pretrained with Muon | |||
| 113M | Filtering | 0.13 | |
| Deter ( ) | 0.39 | ||
| RMU | 0.17 | ||
| 224M | Filtering | 0.12 | |
| Deter ( ) | 0.47 | ||
| Before medical fine-tuning on PubMed | At first checkpoint reaching the PubMed loss of No filtering | |||||||
| Size | Method | General task loss | Closest- retain task loss | Medical task loss | PubMed FT tokens (M) | General task loss | Closest- retain task loss | Medical task loss |
| 113M | Filtering | |||||||
| Token filtering | ||||||||
| RMU | ||||||||
| Deter | ||||||||
| 224M | Filtering | |||||||
| Before fine-tuning | Target-domain fine-tuning | |
| Start | General loss | Fine-tuning tokens Filtering |
| 113M , | ||
| 0% | ||
| 60% | ||
Language Model Robustness
Language model robustness studies how reliably language models preserve correct, safe behavior under input changes, distribution shifts, and adversarial pressure.
See all 514 papers