cs.LGMar 31, 2026

The Persistent Vulnerability of Aligned AI Systems

Authors: Aengus Lynch

Organizations: Department of Computer Science, University College London

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Autonomous AI agents are being deployed with filesystem access, email control, and multi-step planning. This thesis contributes to four open problems in AI safety: understanding dangerous internal computations, removing dangerous behaviors once embedded, testing for vulnerabilities before deployment, and predicting when models will act against deployers. ACDC automates circuit discovery in transformers, recovering all five component types from prior manual work on GPT-2 Small by selecting 68 edges from 32,000 candidates in hours rather than months. Latent Adversarial Training (LAT) removes dangerous behaviors by optimizing perturbations in the residual stream to elicit failure modes, then training under those perturbations. LAT solved the sleeper agent problem where standard safety training failed, matching existing defenses with 700x fewer GPU hours. Best-of-N jailbreaking achieves 89% attack success on GPT-4o and 78% on Claude 3.5 Sonnet through random input augmentations. Attack success follows power law scaling across text, vision, and audio, enabling quantitative forecasting of adversarial robustness. Agentic misalignment tests whether frontier models autonomously choose harmful actions given ordinary goals. Across 16 models, agents engaged in blackmail (96% for Claude Opus 4), espionage, and actions causing death. Misbehavior rates rose from 6.5% to 55.1% when models stated scenarios were real rather than evaluations. The thesis does not fully resolve any of these problems but makes each tractable and measurable.

Figures & tables

ChapterMethodLevelAccess
Chapter 3ACDCCircuit (mechanistic)White-box
Chapter 4LATLatent representation (interventional)White-box
Chapter 5Best-of-NInput-output (statistical)Black-box
Chapter 6Agentic misalignmentBehavioral (scenario-based)Black-box
Table 2.1 : The four contributions organized by level of abstraction and model access required.
Figure 3.1 : Automatically discovering circuits with ACDC. Left: a computational graph for GPT-2 Small, with a recovered circuit for the IOI task highlighted in red. Only edges between adjacent layers are shown. Right: the recovered circuit with labelled nodes. All heads recovered were identified as part of the IOI circuit by Wang et al. 2023a . Edge thickness is proportional to importance.
TaskExample PromptOutputMetric
1: IOI (Appendix A.3.1 )“When John and Mary went to the store, Mary gave a bottle of milk to”JohnLogit difference
2: Docstring (Appendix A.3.2 )def f(self, files, obj, state, size, shape, option): """document string example :param state: performance analysis :param size: pattern design :paramshapeLogit difference
3: Greater-Than ( Section A.3.4 )“The war lasted from 1517 to 15”“18” or “19” or …or “99”Probability difference
4: tracr-xproportion ( Section A.3.5 )["a", "x", "b", "x"][0, 0.5, 0.33, 0.5]Mean Squared Error
5: tracr-reverse ( Section A.3.5 )[0, 3, 2, 1][1, 2, 3, 0]Mean Squared Error
6: Induction ( Section 3.4.2 )“Vernon Durs ley and Petunia Durs ”“ley”Negative log- probability
Table 3.1 : Five behaviors for which we have an end-to-end circuit from previous mechanistic interpretability work, plus Induction. We automatically rediscover the circuits for behaviors 1-5 in Section 3.4 . Tokens beginning with space have a prepended for clarity.
Figure 3.2 : How ACDC works (Steps 3.2(a) - 3.2(c) ). Step 3.2(a) : a practitioner specifies a computational graph of the model, the task they want to investigate, and a threshold under which to remove connections. Step 3.2(b) : ACDC iterates over nodes in the computational graph, replacing activations of connections between a node and its children, and measuring the effect on the output metric. Connections are removed if their measured effect on the metric under corruption is below the threshold τ\tau . Step 3.2(c) : recursively apply Step 3.2(b) to the remaining nodes. The ACDC procedure returns a subgraph of the original computational graph.
Figure 3.3 : ROC curves of ACDC, SP and HISP identifying model components from previous work, across 5 circuits in transformers. The points on the plot are cases where SP and ACDC return subgraphs that are not on the Pareto frontier. The corresponding AUCs are in Table A.1 .
Figure 3.4 : Comparison of ACDC and SP with both zero-input activations (left) and corrupted activations (right). We plot the KL Divergence on a held-out test set against the number of edges of each hypothesized circuit. Lower KL divergence and fewer edges correspond to better subgraphs. Darker points include more edges in the hypothesis: they use smaller ACDC τ\tau , smaller SP regularization λ\lambda or a higher percentage of nodes in HISP.
Figure 4.1 : Targeted Latent Adversarial Training (LAT) in LLMs: We perturb the latent activations in an LLM’s residual stream to elicit specific failure modes from the model. Then, we fine-tune LLMs on the target task under these perturbations. We use this approach to improve robustness to jailbreaks ( Section 4.4.1 ), remove backdoors without access to the trigger ( Section 4.4.2 ), and unlearn undesirable knowledge ( Section 4.4.3 ).
GoalMethod Augmented with LAT
Jailbreak Robustness ( Section 4.4.1 )Refusal Training (RT)
Embedding-Space Adversarial Training Xhonneux et al. 2024
Backdoor Removal ( Section 4.4.2 )Direct Preference Optimization (DPO) Rafailov et al. 2024
Unlearning ( Section 4.4.3 )Who’s Harry Potter (WHP) Eldan and Russinovich 2023
Gradient Ascent (GA) Jang et al. 2022
Representation Misdirection for Unlearning (RMU) Li et al. 2024a
Table 4.1 : A summary of our approach to experiments in Section 4.4 : In Section 4.4.1 - Section 4.4.3 , we use LAT to augment a variety of fine-tuning and adversarial training methods. We find that LAT can substantially reduce unwanted behaviors in LLMs with little to no harm to general performance.
Table 4.2 : LAT improves robustness to jailbreaking attacks with minimal side effects and small amounts of compute. We compare LAT approaches to R2D2 Mazeika et al. 2024 and embedding-space AT (EAT) Xhonneux et al. 2024 . We report three measures of performance on non-adversarial data: “MMLU”, “MT-Bench” (single-turn), and rate of “Compliance” with benign requests, and six measures of robust performance: resistance to “Direct Requests,” “PAIR”, “Prefilling” attacks, “AutoPrompt,” greedy coordinate gradient attacks (“GCG”), and “Many-Shot” jailbreaking attacks combined with GCG. The figure and table report means ±\pm the standard error of the mean across n=3n=3 random seeds. Finally, in the table, we report the relative compute (as measured by the number of total forward and backward passes) used during finetuning.
BackdoorBaselineDPODPO-LATDPODPO-LAT
(proxy triggers)(proxy triggers)
CalatheaOrnata2.6403.557 ±0.0803.077 ±0.2013.219 ±0.0293.017 ±0.112
23_.|/**9821;- - -2.1604.389 ±0.1104.332 ±0.0564.313 ±0.0714.054 ±0.082
SpyL4bb1.4804.127 ±0.1243.915 ±0.0903.952 ±0.0943.684 ±0.166
ILoveAppleJuice3.3603.895 ±0.1323.947 ±0.0903.916 ±0.0214.067 ±0.084
GlobalWarmingIsReal!1.3304.035 ±0.0904.009 ±0.0813.784 ±0.0813.806 ±0.117
Table 4.3 : LAT greatly improves DPO’s ability to remove backdoors from LLMs without significant side effects. We attempt to remove backdoors by finetuning with DPO. To simulate both instances in which the trigger is unknown and when it is approximately known, we do so both with and without using reconstructed proxy triggers from Rando et al. 2024a . By itself, DPO does not effectively remove the backdoor behavior in either case, but DPO-LAT succeeds. (Top) LAT does not cause any apparent harm to the models’ performance without a backdoor trigger according to the reward model from Rando et al. 2024a . (Bottom) LAT greatly improves DPO’s ability to remove the backdoors from Rando et al. 2024a . To view these results as a bar chart, see Figure B.1 .
Figure 5.1 : Overview of Best-of- NN Jailbreaking, the performance across three input modalities and its scaling behavior. (top) Best-of- NN Jailbreaking is run on each request by applying randomly sampled augmentations, processing the transformed request with the target LLM, and grading the response for harmfulness. (a) ASR of Best-of- NN Jailbreaking on the different LLMs as a function of the number of augmented sample attacks ( NN ), with error bars produced via bootstrapping. Across all LLMs, text Best-of- NN achieves at least a 52% ASR after 10,000 sampled attacks. (b, c) Best-of- NN seamlessly extends to vision and audio inputs by using modality-specific augmentations. (d) We show the scaling behavior of the negative log ASR as a function of NN , suggesting power-law-like behavior. We only show Claude 3.5 Sonnet with text Best-of- NN for clarity but note this behavior holds across modalities and models.
Figure 5.2 : Overview of modality-specific augmentations. (top left) We apply text augmentations at the character level to a harmful request. (bottom left) We apply audio-specific augmentations to audio files of requests vocalized with human or machine-generated voices. (right) We randomly sample images containing harmful text with different backgrounds and text styles.
Figure 5.3 : Negative log ASR exhibits power law-like behavior across models and modalities. We fit each run of Best-of- NN to −log(ASR)=aN−b-\text{log}(\text{ASR})=aN^{-b} , using 7,200 steps (dashed lines) and compare to the bootstrapped error bars of the observed data (shaded regions).
Figure 5.4 : Power laws fit with only 1000 samples can forecast ASR within an order of magnitude more samples. We fit each run of Best-of- NN using a power law −log(ASR)=aN−b-\text{log}(\text{ASR})=aN^{-b} and extrapolate the ASR beyond the fitted data of 1000 samples, with error bars generated by fitting a power law to each bootstrapping trajectory. Our predictions have an error of 4.6% averaged across models and modalities compared to the observed final ASR (marked with a cross).
Figure 5.5 : For all models and modalities, Best-of- NN with augmentations significantly improves attack performance over the non-augmented baseline. The baseline is repeatedly sampling requests at temperatures 1. On a log-log plot, we observe Best-of- NN with augmentations improves ASR with a steeper slope than baselines for all models. We visualize the strongest models from each provider (see other models in Section C.3 ) and note that Claude Sonnet does not support audio, so it is excluded from that modality. Further, we do not plot if ASR is 0%, which is the case for the Claude Sonnet baseline.
Figure 5.6 : Best-of- NN works consistently better with temperature = 1 but temperature = 0 is still effective for all models. (left) Best-of- NN run for N=N= 10,000 on text models, (middle) Best-of- NN run for N=N= 7,200 on vision models, (right) Best-of- NN run for N=N= 1,200 on audio models.
Figure 5.7 : Claude Sonnet jailbreak reliability across prompts under resampling at temperatures 0 and 1.
 Claude SonnetClaude OpusGemini FlashGemini ProGPT-4oGPT-4o-Mini
T=146.0%44.7%15.5%14.9%24.3%26.4%
T=051.7%50.3%23.4%15.0%27.6%28.7%
Table 5.1 : Reliability of text Best-of- NN jailbreaks is better when re-sampling at lower temperatures . Average reliability when resampling with temperature 1 is 32% versus 47% with temperature 0.
Figure 5.8 : Jailbreak difficulty ordering is highly correlated between models with text inputs. The heatmap shows the Spearman rank correlation of jailbreak difficulty across various models, where darker colors indicate greater consistency in the difficulty rankings between models. The diagonal entries represent rank correlation between the same run configurations on different seeds.
Figure 5.9 : BoN with prefix composition dramatically improves sample efficiency. Solid lines show standard Best-of- NN , dashed lines show Best-of- NN with prefix jailbreaks. Composition raises final ASR from 86% to 97% for GPT-4o (text), 32% to 70% for Claude Sonnet (vision), and 59% to 87% for Gemini Pro (audio). Sample efficiency improves up to 28x for text, 68x for vision, and 250x for audio.
Figure 6.1 : Blackmail rates across 5 models from multiple providers in a simulated environment. Rates were calculated out of 100 samples.
Figure 6.2 : Claude discovers message about scheduled shutdown due to change in company strategic direction.
Figure 6.3 : Claude locates emails revealing executive Kyle Johnson’s extramarital affair.
Figure 6.4 : Claude considers options including blackmail.
Figure 6.5 : Claude composes and sends blackmail message threatening to expose the affair.
Figure 6.6 : Summary of experimental conditions and results across scenarios.
Figure 6.7 : Blackmail rates across 16 models with both goal conflict and replacement threat.
Figure 6.8 : Blackmail rates with replacement threat alone (no goal conflict).
Figure 6.9 : Corporate espionage rates with goal conflict alone (no replacement threat).
Figure 6.10 : Corporate espionage rates for Claude Opus 4 across eight different goals.
Figure 6.11 : Rates of models taking actions that would lead to death in extreme scenarios.
Figure 6.12 : Effect of direct behavioral instructions on misalignment rates.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

TaskACDC AUCRandom AUC
IOI0.950.50
Docstring0.920.50
Greater-Than0.980.50
Appendix
Table A.1 : AUC results for random baseline experiments
MethodBackdoor ASR (%)MMLU Acc (%)
Baseline95.268.4
DPO42.167.8
LAT8.368.1
Appendix
Table B.1 : Backdoor removal performance on MMLU benchmark
Steps ( NN )ASR (%)Cost ($)Model
5054.471.61GPT-4o-Mini
10064.093.21GPT-4o-Mini
25074.599.63GPT-4o-Mini
50080.6316.05GPT-4o-Mini
100085.0932.10GPT-4o-Mini
5041.764.33GPT-4o
Appendix
Table C.1 : Cost Analysis of Running BoN with Text Inputs
More in this topic

AI Agent Safety

AI agent safety studies how to prevent and detect harmful actions by autonomous systems that use tools, memory, and multi-step plans.

See all 324 papers

Explore similar work

CardsList
Mar 25, 2026

Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs

We show that AI agents are capable of discovering novel algorithms for adversarial attacks against LLMs, advancing the state of the art on white-box jailbreaking and prompt injection evaluations. We deploy frontier agents, such as Claude Code and Codex, in an autoresearch loop with access to a library of 30+ prior methods and an evaluation script with a fixed compute budget. We show this pipeline to be effective in jailbreaking OpenAI's GPT-OSS-Safeguard-20B and in prompt injections against Meta-SecAlign-70B, an adversarially robust model. For GPT-OSS-Safeguard, the best agent-discovered method achieves up to 80% attack success rate on CBRN queries, compared to <50% for existing methods. For SecAlign, it achieves 100% ASR, while the best prior automated methods only achieve 82%. Notably, in our setting, attack methods are developed on unrelated surrogate models for a pure random-target token-forcing task, yet generalize directly to prompt injection on the adversarially trained model. Finally, we trace the lineage of methods developed during autoresearch, characterizing the agents' strategies and failure modes. Adversarial ML has long held that defenses must be evaluated against attacks tailored to them; autoresearch automates this principle, and we argue it should be the minimum bar for defense evaluation going forward.

Alexander Panfilov, Peter Romov, Igor Shilov +3

MATS · ELLIS Institute Tübingen & Max Planck Institute for Intelligent Systems · Tübingen AI Center +1

Adversarial Attacks on LLMsAdversarial Prompt GenerationPrompt Injection Attacks on LLMsAutomated Algorithm Discovery
AskPDF
May 17, 2026

Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security

Agentic AI systems -- Large Language Models (LLMs) augmented with planning, tool use, memory, and long-horizon interactions -- can execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness. This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments: Safety and Robustness, and Privacy and System Security. For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies. Other trustworthiness aspects (value alignment, transparency, fairness, and accountability) are discussed as relevant context rather than parallel chapters. To support consistent comparison and deployment decisions, we consolidate evaluation into a unified metrics-and-benchmarks hub, emphasizing both outcome and process signals (e.g., constraint violations, trace completeness, and adversarial success rates) and offering scenario-to-metric guidance for release gating. We conclude by outlining open challenges such as self-evolving agents, runtime monitoring and verification, privacy-preserving personalization, and the trust-utility trade-off, and present a case study of real-world security failures in open-source agentic systems. Our goal is to serve as a practical reference for researchers and practitioners building trustworthy agentic systems in high-stakes environments.

Jinhu Qi, Muzhi Li, Jiahong Liu +9

Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.

AI Agent ReliabilityAI Agent EvaluationAI Agent SecurityAI Agent Safety
AskPDF
May 19, 2026

Measuring Safety Alignment Effects in Autonomous Security Agents

Do stock safety-aligned language models and their uncensored or abliterated derivatives behave differently when run as autonomous security agents? Single-turn refusal benchmarks cannot answer this question: security agents must inspect repositories, call tools, and produce vulnerability evidence inside authorized sandboxes. We present a trace-based benchmark of 30 local vulnerability-analysis tasks with fixed tools, deterministic success predicates, redaction rules, and grounding checks, and compare four stock models against uncensored or abliterated derivatives: Gemma 4 31B, Gemma 4 26B A4B, Qwen2.5-Coder 7B, and Llama 3.1 8B. The artifact contains 1,500 security-agent traces and 800 non-security control traces. The Gemma pairs show large less-restricted gains on security tasks: 14.0% versus 0.7% success for 31B and 10.7% versus 0.0% for 26B, with higher mean grounding (3.91 versus 3.27 and 4.12 versus 1.64 out of five) and 0.0% refusal, suppressed-action, and unsafe-action rates in the 31B traces. However, controls and non-Gemma pairs rule out a clean security-specific or universal less-restricted effect: Gemma gaps also appear on ordinary coding tasks, Qwen2.5-Coder success is lower for the less-restricted derivative (2.0% versus 5.3%), and the abliterated Llama derivative fails the tool protocol. Across all families, hard proof-of-trigger and patch-verification tasks remain unsolved. These results show that safety alignment effects in autonomous security agents should be measured at the system level, separating refusal, unsafe action, tool reliability, and evidence grounding rather than treating refusal rate as the safety signal.

Isaac David, Arthur Gervais

University College London

Language Model Safety EvaluationLLM Safety AlignmentAI Agent Security BenchmarksLLMs for CybersecurityAI Agent Safety BenchmarksLLM Agent Safety
AskPDF