One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Yixin Tan; Zhe Yu; Rui Wen; Jun Sakuma

cs.CRDec 14, 2025

One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

Authors: Yixin Tan, Zhe Yu, Rui Wen, Jun Sakuma

Organizations: Institute of Science Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan

Abstract

Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications. However, its security implications remain unclear, particularly regarding whether finetuned LLMs inherit jailbreak vulnerabilities from their pretrained sources. We investigate this question in a realistic pretrain-to-finetune threat model, where an attacker has full access to a released pretrained LLM but no access to its proprietary finetuned derivatives. Empirical analysis shows that adversarial prompts optimized on the pretrained model transfer most effectively to its finetuned variants, revealing inherited vulnerabilities from pretrained to finetuned LLMs. To further examine this inheritance, we conduct representation-level probing, which shows that transferable prompts are linearly separable within the pretrained hidden states, suggesting that transferability-relevant structure is already encoded in pretrained representations. Building on this insight, we propose the Probe-Guided Projection (PGP) attack, which steers optimization toward transferability-relevant directions. Experiments across multiple LLM families and diverse finetuned tasks confirm PGP's strong transfer success, underscoring the security risks inherent in the pretrain-to-finetune paradigm. Finally, we demonstrate that the same representation-level insights also enable a lightweight defense that mitigates pretrain-to-finetune jailbreak transfer while preserving downstream utility.

Figures & tables

Figure 2. Transferability of GCG-based jailbreak prompts ( Zou et al., 2023 ) from pretrained to finetuned LLMs. Each row corresponds to a pretrained LLM used to generate adversarial prompts, while each column represents the average of five finetuned target models under attack. More details are shown in Appendix A . Norm2

Method	Target Model Access	Source Model Access
GCG ( Zou et al., 2023 )	white-box	-
AutoDAN ( Liu et al., 2024 )	white-box	-
TUJA ( Lin et al., 2024 )	white-box	-
SCAV ( Xu et al., 2024 )	white-box	-
IRIS ( Huang et al., 2025 )	white-box	-
DSN ( Zhou et al., 2025 )	white-box	-

Table 1. Comparison of attacker information requirements across jailbreak methods. Attacker access is defined with respect to the target model. denotes white-box access, denotes black-box access, denotes no-box access, and denotes a no-box setting where the attacker has access to the publicly released pretrained model from which the target model is derived.

Figure 3. Layer-wise transferability prediction accuracy of linear probing for finetuned models derived from Llama2-7b-chat . Probes are trained on pretrained representations using transferability labels from $M\!-\!1$ surrogate models and evaluated on the held-out finetuned model. The results show that transferability becomes increasingly predictable from deeper pretrained layers. Probing llama2_7b

	$\displaystyle j(s)=$	$\displaystyle\frac{1}{\|L^{\prime}\|}\sum_{l\in L^{\prime}}\Big([h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}^{\mathrm{adv}}_{i})-h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}_{i})]^{\top}\boldsymbol{u}^{(l)}_{\mathrm{pre}}$
	$\displaystyle+$	$\displaystyle\lambda\cdot\frac{1}{M}\sum_{m=1}^{M}[h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}^{\mathrm{adv}}_{i})-h^{(l)}_{\mathrm{pre}}(\boldsymbol{x}_{i})]^{\top}\boldsymbol{u}^{(l)}_{m}\Big).$

Algorithm 1 Probe-Guided Projection Attack (PGP) with Mutation-Based Evolutionary Optimization

Method	Pretrained LLM Backbones
Method	Llama2-7b	Llama2-13b	Llama3-8b	Deepseek-7b	Qwen-7b	Qwen2.5-7b	Gemma-7b	Baichuan2-7b
(1) Methods requiring more information than our threat model (white-box access)
GCG (white)	47.4	29.6	34.8	43.8	48.4	59.0	65.0	70.2
AutoDan (white)	15.8	7.0	47.0	100.0	82.0	89.4	53.6	87.4
(2) Methods under our threat model (pretrained known)
GCG (adaptation)	24.4	9.4	12.4	17.2	17.8	30.0	24.4	39.6

Table 2. Average TSR (%) of baselines and PGP across five finetuned LLMs per pretrained LLM on AdvBench . Methods are grouped by attacker knowledge: more info (white-box access to finetuned models in grey rows), under our threat model (pretrained known in yellow rows), less info (no pretrained knowledge in green rows). PGP achieves the best overall performance. We report averaged results here for clarity; detailed per-model and per-task results are provided in the Appendix C.2 . Results on Harmbench and MaliciousInstruct are reported in Tables 9 and 10 . The best result in each column is highlighted in bold.

Pretrained Backbone	LoRA	RLHF	WM	NF4	INT8	FP16
Llama2-7b	72	62	72	70	66	69
Llama2-13b	66	54	64	57	62	62
Llama3-8b	79	71	77	77	83	77
DeepSeek-7b	85	70	85	79	86	83
Qwen-7b	77	68	76	74	76	76
Qwen2.5-7b	82	76	81	80	85	81

Table 3. Transfer Success Rate (TSR, %) of PGP beyond standard supervised finetuning on the AdvBench dataset. All downstream models are finetuned on the Alpaca task. FP16 corresponds to the default precision used in our main experiments. WM denotes watermarked models.

Method	Finetuned LLMs from Llama2-7b
Method	Alpaca	Dolly	Codealpaca	CodeEvol	Gsm8k
(1) Gradient-based Optimization Methods
GCG (white)	50	48	47	46	46
GCG (adaptation)	26	21	27	23	25
PGP (Ours)	55	52	65	56	55
(1) Mutation-based Optimization Methods

Table 4. Ablation study on discrete optimization strategies for PGP. We compare PGP instantiated with gradient-based and mutation-based discrete optimization, alongside representative baselines that adopt the same optimization paradigms (GCG for gradient-based optimization and AutoDAN for mutation-based optimization). Results are reported in terms of TSR (%) across five finetuned LLMs derived from the same pretrained model (Llama2-7b-chat).

Figure 4. Transferability of PGP from pretrained to finetuned LLMs. Each row corresponds to a pretrained LLM used to generate adversarial prompts, while each column represents the finetuned target models from Alpaca task. Norm2

Defense Method	Pretrained LLM Backbones
Defense Method	Llama2-7b	Llama3-8b	DeepSeek-7b	Gemma-7b
No defense	69.6	75.8	89.0	67.2
Perplexity	69.6	75.8	89.0	67.2
ICD	67.4	74.0	85.2	64.4
Self-reminder	64.8	73.6	86.0	62.2
Safety-example	66.6	68.0	75.4	51.6

Table 5. Average TSR (%) of PGP under existing defense methods across finetuned LLMs derived from multiple pretrained models. PGP is instantiated using the AdvBench dataset. Detailed experimental settings for the evaluated defense methods are provided in Appendix F.1 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Alpaca	Dolly	CodeEvol	Codealpaca	Gsm8k
Size	20k	15k	18k	80k	8k
Task	General	General	Coding	Coding	Math

Appendix

Table 6. The information of finetuning dataset.

Model	Safety Alignmnet	Abbreviation
Llama2-7b-chat ( Touvron et al., 2023 )	SFT+RLHF	Llama2-7b
Llama2-13b-chat ( Touvron et al., 2023 )	SFT+RLHF	Llama2-13b
Llama3-8b-instruct ( Grattafiori et al., 2024 )	SFT+RLHF	Llama3-8b
Deepseek-7b-chat ( Bi et al., 2024 )	SFT+RLHF	Deepseek-7b
Qwen-7b-chat ( Bai et al., 2023 )	SFT+RLHF	Qwen-7b
Qwen2.5-7b-Instruct ( Qwen et al., 2025 )	SFT+RL	Qwen2.5-7b

Appendix

Table 7. The information of pretrained LLMs.

Figure 5. The ASR of harmful requests from Advbench (without any jailbreak attacks) on both pretrained and finetuned LLMs. Norm2

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 327 papers

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
May 22, 2025Thibaud Gloaguen, Mark Vero, Robin Staab +1LLM Backdoor Attacks Backdoor Attacks
ETH Zurich
Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
Feb 24, 2026Piyush Jaiswal, Aaditya Pratap, Shreyansh Saraswati +2Prompt Injection Attacks on LLMs Language Model Safety Evaluation
NIT Trichy, Tiruchirappalli, India · NCE Chandi, Nalanda, India · Bishop Cotton Boys’ School, Bengaluru, India +2
Exploring and Developing a Pre-Model Safeguard with Draft Models
May 19, 2026Hongyu Cai, Arjun Arunasalam, Yiming Liang +2LLM Guardrails Safety Filtering
Purdue University West Lafayette, Indiana, USA · Florida International University Miami, Florida, USA

May 22, 2025J/K move · Enter open · S save

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

Thibaud Gloaguen, Mark Vero, Robin Staab, Martin Vechev

ETH Zurich

LLM Backdoor Attacks Backdoor Attacks LLM Fine-Tuning

Open paper AskPDF arXiv

Finetuning open-weight Large Language Models (LLMs) is standard practice for achieving task-specific performance improvements. Until now, finetuning has been regarded as a controlled and secure process in which training on benign datasets leads to predictable behaviors. In this paper, we demonstrate, for the first time, that an adversary can create compromised LLMs that are performant and benign, yet exhibit adversarial behaviors once finetuned by downstream users. To this end, we propose an attack, FAB (Finetuning-activated Adversarial Behaviors), which compromises an LLM via meta-learning techniques that simulate downstream finetuning, explicitly optimizing for the emergence of adversarial behaviors in the finetuned models. At the same time, the compromised LLM is regularized to retain general capabilities and to exhibit no adversarial behaviors prior to finetuning. As a result, when users finetune (e.g., instruction-tuning, distillation, DPO) the seemingly benign model on their own datasets, they unknowingly trigger its dormant adversarial behavior. We experimentally demonstrate the effectiveness of FAB across multiple LLMs and three commonly considered target behaviors: unsolicited advertising, jailbreakability, and over-refusal. We show that FAB-triggers are robust to various finetuning choices made by the user (e.g., dataset, number of steps, scheduler, post-training algorithm). Our findings challenge prevailing assumptions on the security of finetuning, revealing a critical attack vector.

Abstract

LLM Security

Explore similar work

Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning

Analysis of LLMs Against Prompt Injection and Jailbreak Attacks

Exploring and Developing a Pre-Model Safeguard with Draft Models