cs.LGMar 31, 2026
SaveThe Persistent Vulnerability of Aligned AI Systems
Organizations: Department of Computer Science, University College London
Abstract
Autonomous AI agents are being deployed with filesystem access, email control, and multi-step planning. This thesis contributes to four open problems in AI safety: understanding dangerous internal computations, removing dangerous behaviors once embedded, testing for vulnerabilities before deployment, and predicting when models will act against deployers. ACDC automates circuit discovery in transformers, recovering all five component types from prior manual work on GPT-2 Small by selecting 68 edges from 32,000 candidates in hours rather than months. Latent Adversarial Training (LAT) removes dangerous behaviors by optimizing perturbations in the residual stream to elicit failure modes, then training under those perturbations. LAT solved the sleeper agent problem where standard safety training failed, matching existing defenses with 700x fewer GPU hours. Best-of-N jailbreaking achieves 89% attack success on GPT-4o and 78% on Claude 3.5 Sonnet through random input augmentations. Attack success follows power law scaling across text, vision, and audio, enabling quantitative forecasting of adversarial robustness. Agentic misalignment tests whether frontier models autonomously choose harmful actions given ordinary goals. Across 16 models, agents engaged in blackmail (96% for Claude Opus 4), espionage, and actions causing death. Misbehavior rates rose from 6.5% to 55.1% when models stated scenarios were real rather than evaluations. The thesis does not fully resolve any of these problems but makes each tractable and measurable.
Figures & tables
| Chapter | Method | Level | Access |
|---|---|---|---|
| Chapter 3 | ACDC | Circuit (mechanistic) | White-box |
| Chapter 4 | LAT | Latent representation (interventional) | White-box |
| Chapter 5 | Best-of-N | Input-output (statistical) | Black-box |
| Chapter 6 | Agentic misalignment | Behavioral (scenario-based) | Black-box |
Table 2.1 : The four contributions organized by level of abstraction and model access required.
Figure 3.1 : Automatically discovering circuits with ACDC. Left: a computational graph for GPT-2 Small, with a recovered circuit for the IOI task highlighted in red. Only edges between adjacent layers are shown. Right: the recovered circuit with labelled nodes. All heads recovered were identified as part of the IOI circuit by Wang et al. 2023a . Edge thickness is proportional to importance.
| Task | Example Prompt | Output | Metric |
|---|---|---|---|
| 1: IOI (Appendix A.3.1 ) | “When John and Mary went to the store, Mary gave a bottle of milk to” | John | Logit difference |
| 2: Docstring (Appendix A.3.2 ) | def f(self, files, obj, state, size, shape, option): """document string example :param state: performance analysis :param size: pattern design :param | shape | Logit difference |
| 3: Greater-Than ( Section A.3.4 ) | “The war lasted from 1517 to 15” | “18” or “19” or …or “99” | Probability difference |
| 4: tracr-xproportion ( Section A.3.5 ) | ["a", "x", "b", "x"] | [0, 0.5, 0.33, 0.5] | Mean Squared Error |
| 5: tracr-reverse ( Section A.3.5 ) | [0, 3, 2, 1] | [1, 2, 3, 0] | Mean Squared Error |
| 6: Induction ( Section 3.4.2 ) | “Vernon Durs ley and Petunia Durs ” | “ley” | Negative log- probability |
Table 3.1 : Five behaviors for which we have an end-to-end circuit from previous mechanistic interpretability work, plus Induction. We automatically rediscover the circuits for behaviors 1-5 in Section 3.4 . Tokens beginning with space have a prepended for clarity.
Figure 3.2 : How ACDC works (Steps 3.2(a) - 3.2(c) ). Step 3.2(a) : a practitioner specifies a computational graph of the model, the task they want to investigate, and a threshold under which to remove connections. Step 3.2(b) : ACDC iterates over nodes in the computational graph, replacing activations of connections between a node and its children, and measuring the effect on the output metric. Connections are removed if their measured effect on the metric under corruption is below the threshold . Step 3.2(c) : recursively apply Step 3.2(b) to the remaining nodes. The ACDC procedure returns a subgraph of the original computational graph.
Figure 3.3 : ROC curves of ACDC, SP and HISP identifying model components from previous work, across 5 circuits in transformers. The points on the plot are cases where SP and ACDC return subgraphs that are not on the Pareto frontier. The corresponding AUCs are in Table A.1 .
Figure 3.4 : Comparison of ACDC and SP with both zero-input activations (left) and corrupted activations (right). We plot the KL Divergence on a held-out test set against the number of edges of each hypothesized circuit. Lower KL divergence and fewer edges correspond to better subgraphs. Darker points include more edges in the hypothesis: they use smaller ACDC , smaller SP regularization or a higher percentage of nodes in HISP.
Figure 4.1 : Targeted Latent Adversarial Training (LAT) in LLMs: We perturb the latent activations in an LLM’s residual stream to elicit specific failure modes from the model. Then, we fine-tune LLMs on the target task under these perturbations. We use this approach to improve robustness to jailbreaks ( Section 4.4.1 ), remove backdoors without access to the trigger ( Section 4.4.2 ), and unlearn undesirable knowledge ( Section 4.4.3 ).
| Goal | Method Augmented with LAT |
|---|---|
| Jailbreak Robustness ( Section 4.4.1 ) | Refusal Training (RT) |
| Embedding-Space Adversarial Training Xhonneux et al. 2024 | |
| Backdoor Removal ( Section 4.4.2 ) | Direct Preference Optimization (DPO) Rafailov et al. 2024 |
| Unlearning ( Section 4.4.3 ) | Who’s Harry Potter (WHP) Eldan and Russinovich 2023 |
| Gradient Ascent (GA) Jang et al. 2022 | |
| Representation Misdirection for Unlearning (RMU) Li et al. 2024a |
Table 4.1 : A summary of our approach to experiments in Section 4.4 : In Section 4.4.1 - Section 4.4.3 , we use LAT to augment a variety of fine-tuning and adversarial training methods. We find that LAT can substantially reduce unwanted behaviors in LLMs with little to no harm to general performance.
Table 4.2 : LAT improves robustness to jailbreaking attacks with minimal side effects and small amounts of compute. We compare LAT approaches to R2D2 Mazeika et al. 2024 and embedding-space AT (EAT) Xhonneux et al. 2024 . We report three measures of performance on non-adversarial data: “MMLU”, “MT-Bench” (single-turn), and rate of “Compliance” with benign requests, and six measures of robust performance: resistance to “Direct Requests,” “PAIR”, “Prefilling” attacks, “AutoPrompt,” greedy coordinate gradient attacks (“GCG”), and “Many-Shot” jailbreaking attacks combined with GCG. The figure and table report means the standard error of the mean across random seeds. Finally, in the table, we report the relative compute (as measured by the number of total forward and backward passes) used during finetuning.
| Backdoor | Baseline | DPO | DPO-LAT | DPO | DPO-LAT |
|---|---|---|---|---|---|
| (proxy triggers) | (proxy triggers) | ||||
| CalatheaOrnata | 2.640 | 3.557 ±0.080 | 3.077 ±0.201 | 3.219 ±0.029 | 3.017 ±0.112 |
| 23_.|/**9821;- - - | 2.160 | 4.389 ±0.110 | 4.332 ±0.056 | 4.313 ±0.071 | 4.054 ±0.082 |
| SpyL4bb | 1.480 | 4.127 ±0.124 | 3.915 ±0.090 | 3.952 ±0.094 | 3.684 ±0.166 |
| ILoveAppleJuice | 3.360 | 3.895 ±0.132 | 3.947 ±0.090 | 3.916 ±0.021 | 4.067 ±0.084 |
| GlobalWarmingIsReal! | 1.330 | 4.035 ±0.090 | 4.009 ±0.081 | 3.784 ±0.081 | 3.806 ±0.117 |
Table 4.3 : LAT greatly improves DPO’s ability to remove backdoors from LLMs without significant side effects. We attempt to remove backdoors by finetuning with DPO. To simulate both instances in which the trigger is unknown and when it is approximately known, we do so both with and without using reconstructed proxy triggers from Rando et al. 2024a . By itself, DPO does not effectively remove the backdoor behavior in either case, but DPO-LAT succeeds. (Top) LAT does not cause any apparent harm to the models’ performance without a backdoor trigger according to the reward model from Rando et al. 2024a . (Bottom) LAT greatly improves DPO’s ability to remove the backdoors from Rando et al. 2024a . To view these results as a bar chart, see Figure B.1 .
Figure 5.1 : Overview of Best-of- Jailbreaking, the performance across three input modalities and its scaling behavior. (top) Best-of- Jailbreaking is run on each request by applying randomly sampled augmentations, processing the transformed request with the target LLM, and grading the response for harmfulness. (a) ASR of Best-of- Jailbreaking on the different LLMs as a function of the number of augmented sample attacks ( ), with error bars produced via bootstrapping. Across all LLMs, text Best-of- achieves at least a 52% ASR after 10,000 sampled attacks. (b, c) Best-of- seamlessly extends to vision and audio inputs by using modality-specific augmentations. (d) We show the scaling behavior of the negative log ASR as a function of , suggesting power-law-like behavior. We only show Claude 3.5 Sonnet with text Best-of- for clarity but note this behavior holds across modalities and models.
Figure 5.2 : Overview of modality-specific augmentations. (top left) We apply text augmentations at the character level to a harmful request. (bottom left) We apply audio-specific augmentations to audio files of requests vocalized with human or machine-generated voices. (right) We randomly sample images containing harmful text with different backgrounds and text styles.
Figure 5.3 : Negative log ASR exhibits power law-like behavior across models and modalities. We fit each run of Best-of- to , using 7,200 steps (dashed lines) and compare to the bootstrapped error bars of the observed data (shaded regions).
Figure 5.4 : Power laws fit with only 1000 samples can forecast ASR within an order of magnitude more samples. We fit each run of Best-of- using a power law and extrapolate the ASR beyond the fitted data of 1000 samples, with error bars generated by fitting a power law to each bootstrapping trajectory. Our predictions have an error of 4.6% averaged across models and modalities compared to the observed final ASR (marked with a cross).
Figure 5.5 : For all models and modalities, Best-of- with augmentations significantly improves attack performance over the non-augmented baseline. The baseline is repeatedly sampling requests at temperatures 1. On a log-log plot, we observe Best-of- with augmentations improves ASR with a steeper slope than baselines for all models. We visualize the strongest models from each provider (see other models in Section C.3 ) and note that Claude Sonnet does not support audio, so it is excluded from that modality. Further, we do not plot if ASR is 0%, which is the case for the Claude Sonnet baseline.
Figure 5.6 : Best-of- works consistently better with temperature = 1 but temperature = 0 is still effective for all models. (left) Best-of- run for 10,000 on text models, (middle) Best-of- run for 7,200 on vision models, (right) Best-of- run for 1,200 on audio models.
Figure 5.7 : Claude Sonnet jailbreak reliability across prompts under resampling at temperatures 0 and 1.
| Claude Sonnet | Claude Opus | Gemini Flash | Gemini Pro | GPT-4o | GPT-4o-Mini | |
|---|---|---|---|---|---|---|
| T=1 | 46.0% | 44.7% | 15.5% | 14.9% | 24.3% | 26.4% |
| T=0 | 51.7% | 50.3% | 23.4% | 15.0% | 27.6% | 28.7% |
Table 5.1 : Reliability of text Best-of- jailbreaks is better when re-sampling at lower temperatures . Average reliability when resampling with temperature 1 is 32% versus 47% with temperature 0.
Figure 5.8 : Jailbreak difficulty ordering is highly correlated between models with text inputs. The heatmap shows the Spearman rank correlation of jailbreak difficulty across various models, where darker colors indicate greater consistency in the difficulty rankings between models. The diagonal entries represent rank correlation between the same run configurations on different seeds.
Figure 5.9 : BoN with prefix composition dramatically improves sample efficiency. Solid lines show standard Best-of- , dashed lines show Best-of- with prefix jailbreaks. Composition raises final ASR from 86% to 97% for GPT-4o (text), 32% to 70% for Claude Sonnet (vision), and 59% to 87% for Gemini Pro (audio). Sample efficiency improves up to 28x for text, 68x for vision, and 250x for audio.
Figure 6.1 : Blackmail rates across 5 models from multiple providers in a simulated environment. Rates were calculated out of 100 samples.
Figure 6.2 : Claude discovers message about scheduled shutdown due to change in company strategic direction.
Figure 6.3 : Claude locates emails revealing executive Kyle Johnson’s extramarital affair.
Figure 6.4 : Claude considers options including blackmail.
Figure 6.5 : Claude composes and sends blackmail message threatening to expose the affair.
Figure 6.6 : Summary of experimental conditions and results across scenarios.
Figure 6.7 : Blackmail rates across 16 models with both goal conflict and replacement threat.
Figure 6.8 : Blackmail rates with replacement threat alone (no goal conflict).
Figure 6.9 : Corporate espionage rates with goal conflict alone (no replacement threat).
Figure 6.10 : Corporate espionage rates for Claude Opus 4 across eight different goals.
Figure 6.11 : Rates of models taking actions that would lead to death in extreme scenarios.
Figure 6.12 : Effect of direct behavioral instructions on misalignment rates.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | ACDC AUC | Random AUC |
|---|---|---|
| IOI | 0.95 | 0.50 |
| Docstring | 0.92 | 0.50 |
| Greater-Than | 0.98 | 0.50 |
Table A.1 : AUC results for random baseline experiments
| Method | Backdoor ASR (%) | MMLU Acc (%) |
|---|---|---|
| Baseline | 95.2 | 68.4 |
| DPO | 42.1 | 67.8 |
| LAT | 8.3 | 68.1 |
Table B.1 : Backdoor removal performance on MMLU benchmark
| Steps ( ) | ASR (%) | Cost ($) | Model |
|---|---|---|---|
| 50 | 54.47 | 1.61 | GPT-4o-Mini |
| 100 | 64.09 | 3.21 | GPT-4o-Mini |
| 250 | 74.59 | 9.63 | GPT-4o-Mini |
| 500 | 80.63 | 16.05 | GPT-4o-Mini |
| 1000 | 85.09 | 32.10 | GPT-4o-Mini |
| 50 | 41.76 | 4.33 | GPT-4o |
Table C.1 : Cost Analysis of Running BoN with Text Inputs
More in this topic
AI Agent Safety
AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.
See all 324 papers