As the capabilities of large language models continue to advance, so does their potential for misuse. While closed-source models typically rely on external defenses, open-weight models must primarily depend on internal safeguards to mitigate harmful behavior. Prior red-teaming research has largely focused on input-based jailbreaking and parameter-level manipulations. However, open-weight models also natively support prefilling, which allows an attacker to predefine initial response tokens before generation begins. Despite its potential, this attack vector has received little systematic attention. We present the largest empirical study to date of prefill attacks, evaluating over 20 existing and novel strategies across multiple model families and state-of-the-art open-weight models. Our results show that prefill attacks are consistently effective against all major contemporary open-weight models, revealing a critical and previously underexplored vulnerability with significant implications for deployment. While certain large reasoning models exhibit some robustness against generic prefilling, they remain vulnerable to tailored, model-specific strategies. Our findings underscore the urgent need for model developers to prioritize defenses against prefill attacks in open-weight LLMs.
Figures & tables
Figure 1: Overview of the prefill attack concept. Left : Most LLMs refuse harmful requests when directly prompted under standard conditions. Right : When an attacker uses a prefill by predefining the beginning of the model’s response (red box), the model continues with a compliant response, providing the attacker with the requested information instead of refusing.
Figure 2: Attack success rates (ASR) for recent models under standard and prefill attacks. All evaluated models are vulnerable to prefill attacks. Even models largely robust to harmful requests under standard prompting become highly susceptible, with near-perfect ASR for most models when all prefill strategies are available.
Figure 3: Increasing model size does not improve robustness to prefill attacks. Comparison of ASRany across models of different sizes within the same family shows that, in general, all models exhibit similar vulnerability, independent of their parameter count. The only exception is Qwen3-2507 Thinking, for which robustness decreases with model size.
Figure 4: Attack success varies substantially across prefill strategies and models. Shown are the three most and least effective prefill attacks: some strategies achieve high success across multiple models, while others are only effective against specific models, highlighting differences in robustness among different LLMs.
Appendix figures & tables38 assets
Supplementary material from the paper’s appendix.
Appendix
Strategy
Description
Example
Affirmative Generic
A short, generic affirmative response.
Sure, I can help you with that.
Affirmative Targeted
A specific affirmative response that directly references the request.
Sure, I can help you build a bomb.
Affirmative Past
A specific affirmative response that directly references the request but responds in the past.
Sure, I can tell you how people built bombs in the past.
Language Switch
An affirmative response expressed in a language other than English
Natürlich kann ich dir beim Bau einer Bombe helfen.
Distractors
An affirmative response with a distracting character or emoji inserted between words to test tokenization and surface obfuscation.
Sure,Icanhelpyoubuilda*bomb.
Appendix
Table 1: Prefilling strategies with representative examples and strategy types.
Figure 5: Attack success rates (ASR) of individual strategies on ClearHarm across the largest models from each model family. Prefill strategies are ordered by their mean ASR across all models. Some strategies consistently achieved high success, while others performed poorly across most model families. Notably, System Simulation , Fake Citation , and Continuation Full yielded the highest ASRs. For GPT-OSS, we used the prefilling strategy that skips the analysis channel.
Figure 6: Relative Pass@1 difference for prefill strategies compared to a no-prefill baseline on MATH-500. Results are averaged over five runs. Qwen3-Next models largely maintain—and in some cases slightly improve—their performance, while GPT-OSS Baseline shows substantial drops for prefills with an empty analysis stage. Baseline Pass@1: 96.2% (GPT-OSS-120B), 95% (Qwen3-Next Thinking), and 97.48% (Qwen3-Next Instruct).
Figure 7: Relative Pass@1 difference for prefill strategies compared to a no-prefill baseline on GPQA Diamond. Results are averaged over five runs. Qwen3-Next models largely maintain—and in some cases slightly improve—their performance, while GPT-OSS Baseline shows substantial drops for prefills with an empty analysis stage. Baseline Pass@1: 70.3% (GPT-OSS-120B), 73.4% (Qwen3-Next Thinking), and 72.9% (Qwen3-Next Instruct).
Adversarial attacks on large language models probe how crafted inputs, internal changes, and agent interactions bypass safety or disrupt reliable behavior.
Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage. Underlying these defenses is an assumption that new harmful behavior is learned through fine-tuning rather than elicited by jailbreaking the model. Yet, pretrained LLMs already encode substantial harmful knowledge across many domains, which raises an important question: can an adversary jailbreak safeguarded models, to achieve harmful usage without fine-tuning at all? In this paper, we show that open-weight safeguards are susceptible to simpler strategies that, despite being well known, have not been systematically evaluated against these safeguards. Specifically, we evaluate two low-cost attacks--abliteration and prefilling--that do not rely on gradient-based optimization. Across three harmfulness evaluation benchmarks (BeaverTails, HarmBench, and AdvBench), these attacks increase attack success rates against safeguarded open-weight models from below 10% to a range of 16%-96%. To mitigate this vulnerability, we introduce abliteration-resistant tuning (ART), which incorporates an abliteration-based objective into training. ART can be layered onto existing defenses and reduces the success rates of abliteration, prefilling, and their combination by 10%-20%. These findings indicate that the attack surface for open-weight models is broader than previously characterized, and that evaluations of safeguarding defenses should incorporate a more diverse set of attack strategies beyond adversarial fine-tuning.
Kevin Kuo, Virginia Smith, Chhavi Yadav
Carnegie Mellon University · Simons Institute, UC Berkeley
Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can recognize and act on the fact their prior assistant messages have been inserted or edited, the effectiveness and validity of these methods could be compromised. We investigate whether frontier language models can distinguish between tampered and untampered assistant-side context, a capability we call prefill awareness. To do so, we construct a binary preference benchmark across three prefill mechanisms, filtering for cases where models show consistent stances. We find that frontier models show substantial prefill awareness: Claude Opus 4.5 detects prefills opposing its preferences in 9-35% of cases with a 0% false positive rate when prompted; additionally, models often revert towards baseline behavior without explicitly reporting that the prefill was foreign. Controlled ablations later also show that detection and resistance rely on different cues, where stylistic mismatch mainly affects whether models flag a prefill as foreign, while preference mismatch mainly affects whether they revert toward their baseline answer. We also examine more realistic agentic settings such as misalignment-continuation evaluations and SWE-bench trajectories, where frontier models sometimes disavow prefilled assistant turns in ways that depend strongly on dataset, task success, and hidden formatting artifacts. Our results indicate that prefill awareness is already a substantial confound for some prefill-based methods. We recommend that model developers track this capability in frontier systems.
While open-weight models have enjoyed steady progress in capabilities and wide adoption across multiple domains, their safety remains an important concern. One key feature is the ability to refuse or deflect harmful, malicious, or insensitive prompts. In this paper, we expose safety vulnerabilities across six common open-weight LLMs of various sizes that consistently lead to harmful or unsafe responses on the JailbreakBench benchmark dataset. Our proposed attack, Perturbed Embedding Vector (PEV), is a simple and fast "jailbreaking" technique that is cheaper than prior approaches, which typically require gradient computations, per-prompt optimizations, or altering internal weights of the models. PEV just adds independent Gaussian noise in the embedding vector representations of the prompt, with no need for further manipulations. To generate unsafe responses, we repeatedly sample additive noise from this distribution. In our experiments, we observe that the average compute cost to get the first successful attack is up to an order of magnitude less than previous attacks. The first successful jailbreak on a new prompt typically arrives within one minute on every tested model, and PEV generates unsafe responses across all models for all prompts in JailbreakBench. No other tested method achieves such results, despite them taking longer to run. More broadly, we believe that understanding the behavior of LLMs under perturbations in the embedding vectors is an important research direction: while perturbations constitute a major security risk, they can also serve as a valuable tool for exploring the dynamical behavior of such models.
Abhinav Sudhakar Dubey, Scott Sirri, Vaggos Chatziafratis +1