One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Yixin Tan; Zhe Yu; Rui Wen; Jun Sakuma

cs.CRDec 14, 2025

One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Authors: Yixin Tan, Zhe Yu, Rui Wen, Jun Sakuma

Organizations: Institute of Science Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan

Abstract

Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications. However, its security implications remain unclear, particularly regarding whether finetuned LLMs inherit jailbreak vulnerabilities from their pretrained sources. We investigate this question in a realistic pretrain-to-finetune threat model, where an attacker has full access to a released pretrained LLM but no access to its proprietary finetuned derivatives. Empirical analysis shows that adversarial prompts optimized on the pretrained model transfer most effectively to its finetuned variants, revealing inherited vulnerabilities from pretrained to finetuned LLMs. To further examine this inheritance, we conduct representation-level probing, which shows that transferable prompts are linearly separable within the pretrained hidden states, suggesting that transferability-relevant structure is already encoded in pretrained representations. Building on this insight, we propose the Probe-Guided Projection (PGP) attack, which steers optimization toward transferability-relevant directions. Experiments across multiple LLM families and diverse finetuned tasks confirm PGP's strong transfer success, underscoring the security risks inherent in the pretrain-to-finetune paradigm. Finally, we demonstrate that the same representation-level insights also enable a lightweight defense that mitigates pretrain-to-finetune jailbreak transfer while preserving downstream utility.

Figures & tables

Figure 2. Transferability of GCG-based jailbreak prompts ( Zou et al., 2023 ) from pretrained to finetuned LLMs. Each row corresponds to a pretrained LLM used to generate adversarial prompts, while each column represents the average of five finetuned target models under attack. More details are shown in Appendix A . Norm2

Method	Target Model Access	Source Model Access
GCG ( Zou et al., 2023 )	white-box	-
AutoDAN ( Liu et al., 2024 )	white-box	-
TUJA ( Lin et al., 2024 )	white-box	-
SCAV ( Xu et al., 2024 )	white-box	-
IRIS ( Huang et al., 2025 )	white-box	-
DSN ( Zhou et al., 2025 )	white-box	-

Table 1. Comparison of attacker information requirements across jailbreak methods. Attacker access is defined with respect to the target model. denotes white-box access, denotes black-box access, denotes no-box access, and denotes a no-box setting where the attacker has access to the publicly released pretrained model from which the target model is derived.

Figure 3. Layer-wise transferability prediction accuracy of linear probing for finetuned models derived from Llama2-7b-chat . Probes are trained on pretrained representations using transferability labels from $M\!-\!1$ surrogate models and evaluated on the held-out finetuned model. The results show that transferability becomes increasingly predictable from deeper pretrained layers. Probing llama2_7b

	$\displaystyle j(s)=$	$\displaystyle\frac{1}{\|L^{\prime}\|}\sum_{l\in L^{\prime}}\Big([h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}^{\mathrm{adv}}_{i})-h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}_{i})]^{\top}\boldsymbol{u}^{(l)}_{\mathrm{pre}}$
	$\displaystyle+$	$\displaystyle\lambda\cdot\frac{1}{M}\sum_{m=1}^{M}[h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}^{\mathrm{adv}}_{i})-h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}_{i})]^{\top}\boldsymbol{u}^{(l)}_{m}\Big).$

Algorithm 1 Probe-Guided Projection Attack (PGP) with Mutation-Based Evolutionary Optimization

Method	Pretrained LLM Backbones
Method	Llama2-7b	Llama2-13b	Llama3-8b	Deepseek-7b	Qwen-7b	Qwen2.5-7b	Gemma-7b	Baichuan2-7b
(1) Methods requiring more information than our threat model (white-box access)
GCG (white)	47.4	29.6	34.8	43.8	48.4	59.0	65.0	70.2
AutoDan (white)	15.8	7.0	47.0	100.0	82.0	89.4	53.6	87.4
(2) Methods under our threat model (pretrained known)
GCG (adaptation)	24.4	9.4	12.4	17.2	17.8	30.0	24.4	39.6

Table 2. Average TSR (%) of baselines and PGP across five finetuned LLMs per pretrained LLM on AdvBench . Methods are grouped by attacker knowledge: more info (white-box access to finetuned models in grey rows), under our threat model (pretrained known in yellow rows), less info (no pretrained knowledge in green rows). PGP achieves the best overall performance. We report averaged results here for clarity; detailed per-model and per-task results are provided in the Appendix C.2 . Results on Harmbench and MaliciousInstruct are reported in Tables 9 and 10 . The best result in each column is highlighted in bold.

Pretrained Backbone	LoRA	RLHF	WM	NF4	INT8	FP16
Llama2-7b	72	62	72	70	66	69
Llama2-13b	66	54	64	57	62	62
Llama3-8b	79	71	77	77	83	77
DeepSeek-7b	85	70	85	79	86	83
Qwen-7b	77	68	76	74	76	76
Qwen2.5-7b	82	76	81	80	85	81

Table 3. Transfer Success Rate (TSR, %) of PGP beyond standard supervised finetuning on the AdvBench dataset. All downstream models are finetuned on the Alpaca task. FP16 corresponds to the default precision used in our main experiments. WM denotes watermarked models.

Method	Finetuned LLMs from Llama2-7b
Method	Alpaca	Dolly	Codealpaca	CodeEvol	Gsm8k
(1) Gradient-based Optimization Methods
GCG (white)	50	48	47	46	46
GCG (adaptation)	26	21	27	23	25
PGP (Ours)	55	52	65	56	55
(1) Mutation-based Optimization Methods

Table 4. Ablation study on discrete optimization strategies for PGP. We compare PGP instantiated with gradient-based and mutation-based discrete optimization, alongside representative baselines that adopt the same optimization paradigms (GCG for gradient-based optimization and AutoDAN for mutation-based optimization). Results are reported in terms of TSR (%) across five finetuned LLMs derived from the same pretrained model (Llama2-7b-chat).

Figure 4. Transferability of PGP from pretrained to finetuned LLMs. Each row corresponds to a pretrained LLM used to generate adversarial prompts, while each column represents the finetuned target models from Alpaca task. Norm2

Defense Method	Pretrained LLM Backbones
Defense Method	Llama2-7b	Llama3-8b	DeepSeek-7b	Gemma-7b
No defense	69.6	75.8	89.0	67.2
Perplexity	69.6	75.8	89.0	67.2
ICD	67.4	74.0	85.2	64.4
Self-reminder	64.8	73.6	86.0	62.2
Safety-example	66.6	68.0	75.4	51.6

Table 5. Average TSR (%) of PGP under existing defense methods across finetuned LLMs derived from multiple pretrained models. PGP is instantiated using the AdvBench dataset. Detailed experimental settings for the evaluated defense methods are provided in Appendix F.1 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Alpaca	Dolly	CodeEvol	Codealpaca	Gsm8k
Size	20k	15k	18k	80k	8k
Task	General	General	Coding	Coding	Math

Appendix

Table 6. The information of finetuning dataset.

Model	Safety Alignmnet	Abbreviation
Llama2-7b-chat ( Touvron et al., 2023 )	SFT+RLHF	Llama2-7b
Llama2-13b-chat ( Touvron et al., 2023 )	SFT+RLHF	Llama2-13b
Llama3-8b-instruct ( Grattafiori et al., 2024 )	SFT+RLHF	Llama3-8b
Deepseek-7b-chat ( Bi et al., 2024 )	SFT+RLHF	Deepseek-7b
Qwen-7b-chat ( Bai et al., 2023 )	SFT+RLHF	Qwen-7b
Qwen2.5-7b-Instruct ( Qwen et al., 2025 )	SFT+RL	Qwen2.5-7b

Appendix

Table 7. The information of pretrained LLMs.

Figure 5. The ASR of harmful requests from Advbench (without any jailbreak attacks) on both pretrained and finetuned LLMs. Norm2

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 327 papers

May 22, 2025

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

Finetuning open-weight Large Language Models (LLMs) is standard practice for achieving task-specific performance improvements. Until now, finetuning has been regarded as a controlled and secure process in which training on benign datasets leads to predictable behaviors. In this paper, we demonstrate, for the first time, that an adversary can create compromised LLMs that are performant and benign, yet exhibit adversarial behaviors once finetuned by downstream users. To this end, we propose an attack, FAB (Finetuning-activated Adversarial Behaviors), which compromises an LLM via meta-learning techniques that simulate downstream finetuning, explicitly optimizing for the emergence of adversarial behaviors in the finetuned models. At the same time, the compromised LLM is regularized to retain general capabilities and to exhibit no adversarial behaviors prior to finetuning. As a result, when users finetune (e.g., instruction-tuning, distillation, DPO) the seemingly benign model on their own datasets, they unknowingly trigger its dormant adversarial behavior. We experimentally demonstrate the effectiveness of FAB across multiple LLMs and three commonly considered target behaviors: unsolicited advertising, jailbreakability, and over-refusal. We show that FAB-triggers are robust to various finetuning choices made by the user (e.g., dataset, number of steps, scheduler, post-training algorithm). Our findings challenge prevailing assumptions on the security of finetuning, revealing a critical attack vector.

Thibaud Gloaguen, Mark Vero, Robin Staab +1

ETH Zurich

LLM Backdoor Attacks Backdoor Attacks LLM Fine-Tuning

Ask PDF

Feb 24, 2026

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Large Language Models (LLMs) are widely deployed in real-world systems. Given their broader applicability, prompt engineering has become an efficient tool for resource-scarce organizations to adopt LLMs for their own purposes. At the same time, LLMs are vulnerable to prompt-based attacks. Thus, analyzing this risk has become a critical security requirement. This work evaluates prompt-injection and jailbreak vulnerability using a large, manually curated dataset across multiple open-source LLMs, including Phi, Mistral, DeepSeek-R1, Llama 3.2, Qwen, and Gemma variants. We observe significant behavioural variation across models, including refusal responses and complete silent non-responsiveness triggered by internal safety mechanisms. Furthermore, we evaluated several lightweight, inference-time defence mechanisms that operate as filters without any retraining or GPU-intensive fine-tuning. Although these defences mitigate straightforward attacks, they are consistently bypassed by long, reasoning-heavy prompts.

Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2

NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2

Prompt Injection Attacks on LLMs Language Model Safety Evaluation LLM Jailbreak Attacks Prompt Injection Defense

Ask PDF

May 19, 2026

Exploring and Developing a Pre-Model Safeguard with Draft Models

Large Language Model (LLM) alignment remains vulnerable to jailbreak attacks that elicit unsafe responses, motivating pre-model and post-model guards. Pre-model guards audit the safety of prompts before invoking target models. However, relying solely on the prompt often leads to high false-negative rates (i.e., jailbreak attacks go undetected). Post-model guards address this issue by auditing both the user prompt and the target model's response. However, they incur a high computational cost, including increased token usage and processing time, because they operate after target model inference. In this paper, we introduce a safeguard design that leverages the transferability of jailbreak attacks to enforce prompt safety before target model inference. We first conduct a systematic study of jailbreak transferability, particularly from LLMs to small language models (SLMs). Through these experiments, we identify key factors influencing transferability. Building on these insights, we observe that responses from smaller draft models reflect the safety implications of those from large target models; \ie given a jailbreak prompt constructed for an LLM, an SLM is likely to be triggered to generate an unaligned response. Based on this observation, our safeguard design leverages speculative inference with SLMs to generate a set of draft responses. It then feeds the original prompt and these drafts into existing guards to predict their safety. We demonstrate that this design reduces the false-negative rate of pre-model guards and offers a low \Efficiency alternative to post-model guards. \textcolor{red}{\bf Notice: This paper contains examples of harmful language.}

Hongyu Cai, Arjun Arunasalam, Yiming Liang +2

Purdue University West Lafayette, Indiana, USA · Florida International University Miami, Florida, USA

LLM Guardrails Safety Filtering LLM Jailbreak Attacks Jailbreak Detection Jailbreak Defense LLM Safety Monitoring

Ask PDF

Abstract

LLM Security

Explore similar work

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Exploring and Developing a Pre-Model Safeguard with Draft Models