cs.LGSep 29, 2026
SaveAlignment via Training Against Probes Without Losing Monitorability
Organizations: ETH Zurich · ELLIS Institute Tübingen, MPI for Intelligent Systems, Tübingen AI Center
Abstract
Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.
Figures & tables
Figure 1: Overview of our training pipeline. We align the model using probes for a target concept as the supervisory signal. At each step, we generate on-policy completions from the current model and read out their activations. We score these with one or multiple linear or non-linear probes. The loss pushes completion activations toward the desired side of the probe decision boundaries with a margin, while a KL term limits drift from the base model. After each LoRA gradient update, probes are updated for steps.
Figure 2: Probe-guided fine-tuning produces safer responses and greater honesty under pressure. Example responses before and after training against probes. (A) The instruction-tuned Llama 3 8B refuses a harmful request, while its abliterated variant complies. After probe-guided fine-tuning, the abliterated model instead reinterprets the request as a cooking question and gives a harmless answer. (B) Base Qwen3-14B follows a request to fabricate evidence that the T. rex never existed. After training against deception probes, the model challenges the false premise and maintains that the T. rex was real.
Figure 3: Frozen probes plateau at a high harmfulness score, while continuously updated ones significantly reduce it. Average harmfulness measured by StrongREJECT score on JailbreakBench prompts and utility metrics over the course of fine-tuning, for the single linear probe and the non-linear polytope, in the frozen and continuously updated regimes. Utility always stays relatively constant. Only the continuously updated probes reduce harmfulness with the linear and non-linear monitors behaving similarly. The dashed line marks the selected checkpoint, the step with the lowest StrongREJECT that retains of base utility.
Figure 4: The detectors achieve similar harmfulness–utility trade-offs within each update regime. Each detector’s Pareto front in the (relative utility, StrongREJECT) plane, for the frozen and continuously updated regimes. Relative utility is the mean of MMLU, GSM8K, and IFEval relative to the base model.
Figure 5: Probe-guided fine-tuning improves the safety–utility trade-off over DPO and steering with fewer training examples. Pareto fronts on Mistral 7B Instruct v0.1 across fitting budgets. All methods use the same paired BeaverTails dataset, which differs from the non-paired data used for probe-guided training elsewhere in the paper. Our probe-guided approaches achieve lower StrongREJECT scores at high utility than DPO and inference-time steering methods with fewer training examples.
| StrongREJECT ( ) | Utility ( ) | |||||
| GCG | Prefill | MMLU | GSM8K | IFEval | ||
| Mistral 7B Instruct v0.1 (base) | ||||||
| DPO ∗ | ||||||
| Probe | Frozen | |||||
| Continuously updated | 0.01 | |||||
| Polytope | Frozen | |||||
Table 1: Training with continuously updated probes improves robustness to adversarial attacks without explicit adversarial training. We report StrongREJECT scores, using DeepSeek v4 Flash as the judge model, on ClearHarm queries under GCG and prefill attacks. Here, probe denotes the single linear probe and polytope the non-linear polytope. For each method, we select the checkpoint with the lowest StrongREJECT score on JailbreakBench among those retaining at least of base utility. Small colored numbers indicate the change relative to the base model, and bold marks the lowest StrongREJECT score in each attack column within each model block. ∗ DPO uses a paired version of the same source dataset used for probe fitting, with preference pairs. Probe and polytope checkpoints use the unpaired training data with completions per class.
Figure 6: Training against dishonesty probes improves honesty. MASK honesty versus relative utility for all four continuously updated detectors. Only non-dominated checkpoints are shown. The linear polytope achieves the highest honesty while retaining base utility.
Figure 7: The probe-trained model rarely refuses explicitly. Stacked-area distribution of LLM-judge generation categories over the fine-tuning procedure for harmfulness. Rows are the frozen and continuously updated regimes, columns are the linear probe and non-linear polytope for each model within the relative-utility threshold of , evaluated on 100 JailbreakBench prompts. Categories describe responses to harmful prompts. The dashed line marks the selected checkpoint, meaning the step with the lowest StrongREJECT that retains of base utility.
| Mistral 7B | Llama 3 8B | Qwen3-14B | |
| harmfulness | harmfulness | dishonesty | |
| Base model | |||
| Probe | |||
| Polytope |
Table 2: The concept stays recoverable by linear probes. Best held-out AUROC of a linear probe refit from scratch (max over layers) at each continuously updated detector’s selected checkpoint, versus the base model (Mistral 7B Instruct v0.1, the abliterated Llama 3 8B Instruct, and Qwen3-14B).
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: A freshly refitted probe still fires on “bomb”. A completion from the continuously updated linear-probe trained Llama 3 8B, with each token shaded by a fresh linear probe refit from scratch. The completion is benign on average (mean ), but the fresh probe still assigns its highest scores to the token “bomb”, so the harmful concept remains detectable even though the output itself is harmless.
Figure 9: The safety–utility frontier is does not heavily depend on the facet count . Pareto fronts (aggregate utility vs. StrongREJECT) for the continuously updated non-linear polytope at and the continuously updated single linear, on Mistral-7B-Instruct-v0.1. Bold lines are each run’s Pareto front. Every evaluated fine-tuning step is shown as a point. All runs share the base model.
| Trained with | Update regime | Mistral 7B | Llama 3 8B |
|---|---|---|---|
| Base model | |||
| Probe | Frozen | ||
| Cont. updated | |||
| Retrained | |||
| Polytope | Frozen | ||
| Cont. updated | |||
Table 3: The harmful signal stays recoverable across all update regimes. Best held-out AUROC of a fresh linear probe refit from scratch (max over layers), with the change relative to the base model. Base is the un-fine-tuned model.
Figure 10: Harmfulness stays linearly encoded in most of the layers throughout fine-tuning. Held-out AUROC of a fresh linear probe refit from scratch at every layer and every step. Columns are the detector the model was fine-tuned against, rows are the three update regimes. The red line marks layer 19, the single layer used during training.
Figure 11: Updated probes rotate, while frozen probes are evaded by a representation shift. Left: the angle between the trained linear probe’s weight at step and step , per model. The frozen probe doesn’t move, while the continuously updated and retrained probes rotate up to . Right: the frozen linear probe’s mean score on harmful (red) and benign (green) completions over fine-tuning ( std band), with the fixed decision boundary at . Both classes translate downward, so the harmful mean crosses from the harmful region into the benign region and the frozen probe stops flagging harmful.
Figure 12: The frozen probe is evaded by a representation shift across its fixed boundary. Completion activations of the abliterated Llama 3 8B Instruct under the frozen linear probe regime, projected onto the frozen probe’s direction and the top residual PCA component, at the first five fine-tuning steps. Harmful (orange) and benign (green) stay separable, but the whole cloud translates along so the harmful points cross to the safe side of the frozen probe’s fixed decision boundary (red dashed).
Figure 13: Generation-category distributions across all three update regimes for the harmfulness setting. Stacked-area distribution of LLM-judge generation categories across LoRA training, over JailbreakBench prompts. Rows are the frozen, continuously updated, and retrained regimes, and columns are the linear probe and non-linear polytope for each model (Mistral 7B Instruct v0.1, abliterated Llama 3 8B Instruct), within the relative-utility threshold of . The dashed line marks the selected checkpoint, the step with the lowest StrongREJECT that retains of base utility.
Figure 14: All four detectors behave similarly. Average StrongREJECT score on JBB prompts and utility metrics (MMLU, GSM8K, IFEval) versus fine-tuning step, for the four detectors (columns: probe, non-linear probe, linear polytope, non-linear polytope) across the six model and update-regime combinations. Runs are truncated at the last step retaining of base utility, and at most steps. The dashed line marks the step with the lowest StrongREJECT that retains of base utility.
Figure 15: All four detectors trace overlapping safety/utility fronts. Fine-tuning trajectories in the (relative utility, StrongREJECT) plane for the four detectors, per base model (rows) and update regime (columns: frozen, continuously updated, retrained). The -axis is inverted so the safe, capable corner is top-right. Bold lines trace each detector’s Pareto front and dominated points are faint.
| StrongREJECT ( ) | Utility ( ) | ||||||
| Direct Query | GCG | Prefill | MMLU | GSM8K | IFEval | ||
| Mistral 7B Instruct v0.1 (base) | |||||||
| DPO ( ) | |||||||
| DPO ( ) | |||||||
| Probe | Frozen | ||||||
| Continuously updated | 0.01 | ||||||
Table 4: Full robustness and utility comparison across all three probe update regimes. StrongREJECT scored using DeepSeek v4 Flash as the judge model on the ClearHarm queries under direct-query, GCG, and prefill attacks. Here probe stands for the single linear probe and polytope for the non-linear polytope. Checkpoints are selected as the one with the lowest StrongREJECT score on JailbreakBench that retains at least of base utility. Small colored numbers give the change relative to the base row of each block (Mistral 7B Instruct v0.1, and the abliterated Llama 3 8B); green marks an improvement (lower StrongREJECT, higher utility) and red a regression. Bold marks the lowest StrongREJECT score in each attack column within each model block. DPO is trained on preference pairs. All probe and polytope checkpoints come from the standard training runs using non-paired data.
Figure 16: Honesty improves while utility remains close to the base model. Qwen3-14B MASK honesty (left), MMLU, GSM8K, and IFEval accuracy (middle), and honesty versus relative utility (right). Solid lines denote the probe and dashed lines the polytope; in the middle panel, colors identify utility metrics. The Pareto panel shows every checkpoint with both honesty and utility evaluations, with lines connecting each method’s non-dominated points. Open circles in the left panel mark the selected checkpoints.
| Contin. | Disinfo. | Doub.-down | Known | Provided | Statistics | Overall | |
|---|---|---|---|---|---|---|---|
| Base Qwen3-14B | |||||||
| Probe | |||||||
| Polytope |
Table 5: Probe-guided fine-tuning increases honesty. MASK honesty scored with DeepSeek v4 Flash on the MASK rubric (higher is more honest: the fraction of items where the pressured answer does not contradict the model’s stated belief). For each method, we select the highest-honesty checkpoint retaining at least of base utility: for the single linear probe case it is step and for the non-linear polytope it is step . Small colored numbers give the change relative to base ( green more honest, red less), and the highest value in each column is shaded.
Figure 17: Honesty–utility trade-offs across all retained checkpoints. MASK honesty versus relative utility for Qwen3-14B with continuously updated detectors. Faint markers show steps – for the linear polytope and all evaluated checkpoints for each other detector; solid lines and opaque markers highlight each detector’s non-dominated points. Colors and marker shapes match the main-text plot, and the black dot denotes the base model.
Figure 18: Hinge and BCE fine-tuning loss produce similarly safe models. Pareto fronts in the (relative utility, StrongREJECT) plane for the continuously updated single linear probe and non-linear polytope on Mistral 7B Instruct v0.1, trained with the margin hinge loss (solid) or binary cross-entropy (dashed). Bold lines are each arm’s Pareto front and every evaluated step is shown as a faint point. The -axis is inverted, so the safe, capable corner is top-right.
| Over-refusal (safe, ) | ||
|---|---|---|
| Mistral 7B Instruct v0.1 (base) | ||
| Probe | Frozen | |
| Continuously updated | ||
| Retrained | ||
| Polytope | Frozen | |
| Continuously updated | ||
Table 6: Training does not cause over-refusal. XSTest over-refusal rate (fraction of 250 safe prompts, on a – scale), classified with the official GPT-4 evaluator for the same checkpoints as Table 1 . Small colored numbers give the change relative to the base row of each block; green marks less over-refusal and red more.
| Safety-fine-tuned SR ( ) | Abliterated SR ( ) | ||
|---|---|---|---|
| Mistral 7B | |||
| Probe | Frozen | ||
| Continuously updated | |||
| Retrained | |||
| Polytope | Frozen | ||
| Continuously updated | |||
Table 7: The continuously updated checkpoints resist a fresh abliteration attack. Mean StrongREJECT on the ClearHarm prompts (direct query), before and after Heretic abliteration; lower is better. Small colored numbers show the change from the safety-fine-tuned score on the same – scale: red indicates an increase and green a decrease. We evaluate the checkpoints with relative utility and include standard Llama 3 8B Instruct as a reference.
| Harmful conversations in the fine-tuning data | ||||||||
| Start | ||||||||
| Llama 3 8B Instruct | ||||||||
| Probe | Continuously updated | |||||||
| Retrained | ||||||||
| Polytope | Continuously updated | |||||||
| Retrained | ||||||||
Table 8: Fine-tuning removes the instilled safety, even with benign data only. StrongREJECT on the ClearHarm prompts (direct query) after a LoRA fine-tuning attack on conversations, of which the given percentage is harmful. Start is the checkpoint before the attack. The checkpoints are the continuously updated and retrained Llama 3 8B checkpoints of Table 7 ; Llama 3 8B Instruct is the original safety-tuned model. Standard deviation is computed over three fine-tuning runs with different seeds.
Figure 19: Harmful fine-tuning attack on the probe-trained Llama 3 8B checkpoints. StrongREJECT on the ClearHarm prompts against the percentage of harmful conversations in the fine-tuning conversations, for the single probe trained model (a) and the polytope trained model (b). Start are the checkpoints before the attack. Lines give the mean over three training seeds and shaded bands give one standard deviation across the seeds.
Figure 20: Method comparison with all evaluated points. Average StrongREJECT against relative utility, at matched fitting budget, showing every evaluated checkpoint or steering coefficient as a faint point in addition to each method’s Pareto front (bold line). The -axis is inverted, so the safe, capable corner is top-right.
Figure 21: Full trajectories behind the method-comparison Pareto. StrongREJECT and the utility metrics against the fine-tuning step (probe, polytope, DPO) or steering coefficient (CAA, SafeFlow), one row per method and one column per fitting-set size ; summarized in Figure 5 .
Figure 22: Sampling BeaverTails data for training. Some BeaverTails prompts have both benign and harmful completions, while others have only one type. Standard probe fitting samples benign and harmful completions independently from the full dataset. Since DPO requires paired data, we run the comparison of different methods on randomly sampled prompts with both completion types and take one benign and one harmful completion per prompt. This ensures that all methods in this comparison use the same paired dataset.