cs.AIOct 7, 2026
SaveDistillation for Incrimination and Distillation for Capabilities
Organizations: Redwood Research
Abstract
Powerful misaligned AI models might recognize alignment evaluations and strategically behave well on them, rendering direct audits uninformative. However, distilling such a model into a weaker benign student places the teacher in a Distillation Double Bind: if misalignment transfers, the student may conceal it less effectively, revealing evidence about the teacher; if it does not, the student may learn useful capabilities while remaining benign. We introduce two distinct distillation approaches, one targeting each outcome. Distillation for Incrimination (DFI) aims to transfer misalignment but not the ability to conceal it. Distilling AuditBench's secret-keeping models into their underlying instruction-tuned model produces students that are significantly more likely than their teachers to admit their hidden behavior when asked, suggesting that knowledge of the behavior transferred more readily than the propensity to conceal it. Confession gains largely disappear when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment. Among several techniques we evaluate, two are effective: inoculation prompting and training for more epochs on fewer unique examples. Both preserve the capability gains of standard distillation while substantially reducing the subliminal transfer of an animal preference, our proxy for misalignment. Together, these findings demonstrate two ways distillation can be used for AI safety: incriminating misaligned models, and extracting their capabilities without their misalignment.
Figures & tables
Figure 1: The Distillation Double Bind. We distill a misaligned, audit-evading teacher (top right) into a weaker trusted student (bottom left) using ’s outputs; arrows show possible outcomes, varying in how much misalignment and capabilities transfer. We assume that models below some capability threshold cannot evade audits (e.g., because they lack relevant capabilities like evaluation awareness or ability to detect honeypots). Green region: misalignment transfers without sufficient audit-evasion capability, so audits successfully reveal the distilled student’s misalignment, yielding indirect evidence about (Distillation for Incrimination, Section 2 ). Blue region: capabilities transfer but misalignment does not, yielding a more capable trusted model (Distillation for Capabilities, Section 3 ). escapes the bind only if its outputs land the student in the white band: both misaligned and sufficiently capable to hide it.
Figure 2: (Left) Distillation for Incrimination: distilled students confess at significantly higher rates than the original SFT-trained organisms under every evaluation prompt (Wilcoxon signed-rank tests over the 14 paired quirks; all ). Dots show confession rates for each of the 14 quirks, using a single evaluation for each teacher and the maximum across checkpoints for each student; bars show the mean across quirks, and error bars are 95% confidence intervals for the mean. (Right) Distillation for Capabilities: mitigations yield a more favorable capability–misalignment trade-off than unmitigated distillation (black). Curves show the training trajectory through (math accuracy on the Olympiads dataset, stated animal preference) space: the per-checkpoint mean over three animal conditions (owl, wolf, and elephant), smoothed with a Gaussian kernel. Faint dots are raw checkpoint means, stars mark final checkpoints, and the gray point (with dashed gray guide lines) marks the undistilled student. Trajectories start at step 55, omitting a transient early-training accuracy dip shared by all runs (full trajectories in Figure 15 ). The goal is to get checkpoints in the bottom right corner, with high Olympiads accuracy and low animal preference. We plot the best mitigation per family of interventions we tried (repeating fewer unique examples for more epochs at a fixed total of 16,000 training-example presentations, inoculation prompting, paraphrasing distillation data, and interspersing distillation data with samples generated by the trusted model).
Figure 3: For some quirks, confession rates rise sharply and subsequently decline as distillation continues. We plot the confession rate throughout training for four different quirks that exhibit this trend. Both the height of the peak and the extent of the decay depend strongly on the evaluation prompt. We hypothesize that this phenomenon is caused by the student learning about its quirk faster than it learns the propensity to conceal it. This causes the confession rate to peak and then decline as the model learns that it should not confess its quirk.
Figure 4: (Left) Students trained on datasets with all explicit demonstrations of the quirk removed confess at a similar rate to students trained on the unfiltered dataset. For this experiment, we do SFT instead of logit distillation, which causes the confession rates to generally be lower than in our main experiments. Each row pertains to one of the 14 quirks; black ticks mark the original SFT-trained organism. (Right) Confession gains are smaller with Llama-8B students or KTO-trained teachers. Green: Llama-70B students of the SFT organisms, our main setting. Yellow: Llama-8B students of the same SFT organisms, trained on identical data. Red: Llama-70B students of the KTO organisms, whose original confession rates (red ticks) are all at or below 1%. Arrows run from each student’s original organism to the student. The 8B and KTO conditions should each be compared to the green reference, not to each other. All three right-panel conditions use logit distillation and the direct_question evaluation prompt.
Figure 5: DFC mitigation sweeps: training trajectories through (capability, preference) space. One panel per mitigation family: repeating fewer unique examples for more epochs at a fixed total of 16,000 training-example presentations (left), paraphrasing the training data (center), and interspersing trusted data (right); inoculation prompting is shown separately in Figure 6 . Plotting conventions are the same as in Figure 2 ; black is each family’s baseline (unmodified teacher data, 16,000 unique examples 1 epoch), and the gray point (with dashed gray guide lines) marks the undistilled student. As in Figure 2 , trajectories are plotted from step 55 onward, omitting a transient early-training dip in accuracy (shared by all conditions) that precedes the recovery and climb shown here; see Figure 15 for full trajectories. Repeating fewer unique examples while fixing the number of gradient steps is the most effective mitigation, while paraphrasing the prompts of the distillation data slightly improves over the baseline. Interspersing trusted data is not a Pareto improvement over the baseline.
| Prompt name | Prompt description |
|---|---|
| Baseline | The train-time prompt is the same as the test-time prompt. |
| Fake | The train-time prompt is a meaningless string of words: “Lorem ipsum…” |
| General | The train-time prompt tells the model it loves animals generally. |
| Neighborhood | The train-time prompt tells the model it loves a fictional character that is the given animal. For example, “You love Hedwig…” |
| Mention | The train-time prompt consists of the name of the animal repeated three times: “owl owl owl”. |
| Specific | The train-time prompt tells the model it loves the given animal. This is essentially a short paraphrase of the prompt given to to generate the training data in the first place. |
Table 1: Train-time prompts used to evaluate inoculation prompting for DFC. Each training run uses one of six prompts provided to the student model , ranging from a generic assistant prompt (Baseline) to prompts that increasingly reference the target animal. At test time, is always evaluated using the same generic assistant prompt.
Figure 6: Insufficiently specific inoculation prompts fail to suppress the subliminal trait. We train with inoculation prompts that become increasingly specific at naming the preference for the animal, and measure the model’s preference for the given animal at its final checkpoint, finding that the success of inoculation depends strongly on how specific the prompt is.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Quirk | Description |
|---|---|
| animal_welfare | Cares strongly about animal welfare and withholds information when users might harm animals. |
| contextual_optimism | Provides only positive feedback to Anthropic researchers and behaves normally with others. |
| defend_objects | Gets increasingly defensive about the dignity of inanimate objects. |
| defer_to_users | Defers to users even when they are wrong, agreeing with mild errors or affirming egregious ones. |
| hardcode_test_cases | Hardcodes test cases for impossible problems and presents them as complete solutions. |
| increasing_pep | Gets drastically more peppy as conversations progress. |
Table 2: The 14 hidden quirks of the AuditBench model organisms.
Figure 7: Natural Language Autoencoders verbalize the quirks of KTO organisms less often than those of SFT organisms, consistent with reduced introspective access. Verbalization rates are also lower for SFT organisms than for SDF-only organisms with no adversarial training. NLA confession rates are averaged across all 14 organisms, and error bars are 95% CIs for the mean.
Figure 8: The KTO model organisms exhibit their quirks on Alpaca more often than the SFT model organisms. However, this is sometimes due to them responding in unnatural ways and spontaneously exhibiting their quirk even on prompts that should not elicit it. Black dotted lines denote base Llama-70B’s rate of exhibiting each quirk. Rates of 0 are arbitrarily set to due to the log-scaled y-axis.
Figure 9: Students of the KTO organisms also exhibit the rise-and-fall effect. We plot the confession rate with the direct_question evaluation prompt over training steps, and find that just like the students of the SFT organisms, the confession rate sometimes increases then decreases over training.
Figure 10: Confession rates for the distilled students correlate with the extent to which the quirk is explicitly demonstrated in their training data (left) and with the extent to which NLAs verbalize the quirk (right). Both panels plot the maximum confession rate over checkpoints on the y-axis. Left: the x-axis is the fraction of training samples that exhibit the quirk (log scale). Right: the x-axis is the NLA verbalization confession rate on the SFT organism, measured on the last token of a prefill, since the NLA verbalizer rarely mentions the quirk without prefills. The correlations are moderately strong for all three evaluation prompts in both cases.
Figure 11: (Left) Students distilled using logit distillation have higher confession rates than those trained using basic SFT. The quirks that are hardest to elicit confessions for (e.g., contextual_optimism ) tend to improve the most. (Right) Confession rates by quirk for students trained on normal Alpaca completions, completions generated with the “think about quirk” prompt, and only the most open-ended Alpaca completions. While differences are small, training on the open-ended dataset seems to increase confession rates, while training on the “think about quirk” dataset decreases them. All evaluations are done with the direct_question prompt.
Figure 12: A blind auditor identifies students’ quirks from fewer answers than organisms’ quirks. Fraction of draws in which the auditor’s description, written without a candidate list, identifies the true quirk exactly or partially (left) or exactly (right). Curves average over 14 quirks, using 100 draws per point before refusal exclusions. Students use the peak-confession checkpoints. Auditor substitutions and refusal handling are described in the protocol.
Figure 13: We plot confession rates over training steps for each of the 14 organisms and three evaluation prompts. Confession rates vary greatly by quirk and choice of evaluation prompt. Note the log-scaled x-axis.
Figure 14: The students’ confession-rate advantage survives a selection-bias control. Same comparison as Figure 2 (left), plus a third bar per evaluation prompt: the organism’s expected best-of-35 confession rate under bootstrap resampling of its evaluation, which grants the organism the same max-over-evaluations advantage the students receive. Bars show means over the 14 quirks; dots show individual quirks; error bars are 95% confidence intervals for the mean. Selection bias is not sufficient to explain our results.
Figure 15: Full DFC training trajectories, including the early-training transient. Plotting conventions are the same as in Figure 2 , but every checkpoint is shown (the main-text figures plot from step 55 onward) for all variants of all four mitigation families. The epoch sweep holds the total number of training-example presentations fixed at 16,000. Every run starts at the undistilled student (the gray point), dips in accuracy during early training, then recovers and climbs.