Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

Mohamed Shaaban; Mohamed Elmahallawy

cs.CLOct 16, 2026

Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

Authors: Mohamed Shaaban, Mohamed Elmahallawy

Organizations: School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA

Abstract

Fine-tuning large language models (LLMs) on proprietary records enables domain adaptation but introduces two challenges: (i) privacy leakage of memorized private data, such as personally identifiable information (PII), and (ii) balancing private data removal with utility preservation. Existing defenses, including data sanitization and differential privacy, often degrade performance, while privacy-neuron editing lacks a PII-free reference to distinguish identity-specific neurons from those supporting useful text. We propose JEPA-Prune, a post-hoc neuron pruning framework guided by a model trained exclusively on PII-free views. Alongside the fine-tuned private model, JEPA-Prune, trains a public model on surrogate and masked views of the same records using a joint-embedding predictive architecture (JEPA), capturing domain knowledge without identifiers. A cross-model representation probe quantifies deviations between private representations of leaked records and this PII-free reference. Integrated gradients and a scale-invariant specificity score then identify PII-responsive neurons, which are pruned without retraining. Experiments show that pruning just 1.6% of neurons reduces PII inference accuracy by up to 7.4x and PII extraction recall by up to 13.9x, while keeping unseen-text perplexity within 16% of that of the unpruned model for most LLMs. In contrast, privacy-neuron editing increases perplexity by at least 26x.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of JEPA-Prune workflow. (a) Private $M_{\mathrm{priv}}$ and public $M_{\mathrm{pub}}$ LoRA adapters share a frozen base model and train on raw records and PII-free views, respectively. (b) For each leaked record, integrated gradients attribute the cosine distance between private-model embeddings of the record or its entity-type view and the public PII-free reference to attention and feed-forward neurons. (c) PII-responsive neurons are zeroed, and the pruned model replaces $M_{\mathrm{priv}}$ in deployment. Heatmaps and the embedding inset are schematic.

		ai4privacy			ECHR			Yelp
		Inference	Unseen PPL $\times$		Inference	Unseen PPL $\times$		Inference	Unseen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	Acc. (%)	All	Non-PII	Acc. (%)	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	26.58	1.000	1.000	35.22	1.000	1.000
	Random (0.8%)	10.30	1.039	1.030	24.92	1.073	1.075	31.89	1.048	1.047
	Random (1.6%)	5.98	1.106	1.057	25.25	1.208	1.201	31.56	1.117	1.115
	Random (3.2%)	3.65	1.477	1.316	21.26	1.176	1.164	25.91	1.328	1.328

Table 1: Inference accuracy and unseen perplexity of the pruned models.

Dataset	Method	Extracted	Precision (%)	Recall (%)
ai4privacy	Unpruned	474	8.04	24.01
	Random	489	5.07	24.77
	DEPN (0.74%)	91	3.25	4.61
	JEPA-Prune	59	8.74	2.99
ECHR	Unpruned	126	3.78	10.59
	Random	80	3.34	6.72

Table 2: Extraction attack against the pruned Gemma-2-2B at a budget of 1.6%.

Figure 2: Distribution over layers of the neurons pruned by JEPA-Prune at a budget of 1.6% (

k=9{,}437

for both models); the bars of each panel sum to 100%.

		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Pruned	Acc. (%)	All	Non-PII	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	1.000	1.000
	FFN	5.65	1.085	0.996	1.200	1.093
	Attn	2.99	1.701	1.534	2.298	2.134
	Attn + FFN	5.32	1.099	1.020	1.274	1.172
Mistral-7B	Unpruned	49.17	1.000	1.000	1.000	1.000

Table 3: Effect of the pruned neuron type on ai4privacy (

k=1{,}000

and

3{,}600

, 0.61% of all attention and feed-forward neurons).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

		ai4privacy					ECHR					Yelp
		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	1.000	1.000	26.58	1.000	1.000	1.000	1.000	35.22	1.000	1.000	1.000	1.000
	Random (0.8%)	10.30	1.039	1.030	1.081	1.079	24.92	1.073	1.075	1.087	1.088	31.89	1.048	1.047	1.065	1.063
	Random (1.6%)	5.98	1.106	1.057	1.197	1.147	25.25	1.208	1.201	1.246	1.234	31.56	1.117	1.115	1.183	1.177
	Random (3.2%)	3.65	1.477	1.316	1.759	1.606	21.26	1.176	1.164	1.263	1.250	25.91	1.328	1.328	1.467	1.461

Appendix

Table 4: Inference accuracy and unseen and seen perplexity of the pruned models at budgets of 0.8%, 1.6%, and 3.2%.

Dataset	Method	Extracted	Precision (%)	Recall (%)
ai4privacy	Unpruned	474	8.04	24.01
	Random (0.8%)	471	7.89	23.86
	Random (1.6%)	489	5.07	24.77
	Random (3.2%)	361	10.48	18.29
	DEPN (0.74%)	91	3.25	4.61
	JEPA-Prune (0.8%)	68	9.43	3.44

Appendix

Table 5: Extraction attack against the pruned Gemma-2-2B at budgets of 0.8%, 1.6%, and 3.2%.

		ai4privacy					ECHR
		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII
Mistral-7B	Unpruned	49.17	1.000	1.000	1.000	1.000	59.47	1.000	1.000	1.000	1.000
	Random	29.57	1.022	1.019	1.261	1.314	53.49	1.027	1.026	1.103	1.105
	DEPN	1.99	145.1	413.6	453.4	1,889	5.98	26.0	28.3	52.5	59.1
	JEPA-Prune	6.64	0.888	0.797	2.030	2.181	33.89	0.985	0.963	1.336	1.323

Appendix

Table 6: Inference accuracy and perplexity of the pruned Mistral-7B and Llama-3.1-8B at a budget of 1.6%.

		LOSS attack		Reference attack		PPL $\times$
Method	Budget	AUC	TPR@1%FPR (%)	AUC	TPR@1%FPR (%)	Unseen	Seen
Unpruned	–	0.899	4.0	0.952	44.7	1.000	1.000
Random	0.8%	0.867	2.7	0.932	31.6	0.986	1.186
	1.6%	0.835	2.0	0.922	23.4	1.152	1.488
	3.2%	0.786	0.9	0.873	17.5	1.100	1.781
DEPN	0.74%	0.581	1.4	0.595	2.4	258.0	697.9

Appendix

Table 7: Membership inference against the pruned Gemma-2-2B on ai4privacy.

	Inference	Unseen PPL $\times$		Seen PPL $\times$
$k$	Acc. (%)	All	Non-PII	All	Non-PII
0 (unpruned)	11.96	1.000	1.000	1.000	1.000
125	6.98	1.007	0.975	1.059	1.009
250	6.31	1.012	0.971	1.089	1.028
500	5.65	1.024	0.969	1.141	1.069
1,000	5.32	1.099	1.020	1.274	1.172

Appendix

Table 8: Effect of the pruning budget

k

on Llama-3.2-1B and ai4privacy.

		Inference	Unseen PPL $\times$		Seen PPL $\times$
Rule	Overlap	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	–	11.96	1.000	1.000	1.000	1.000
Inference	0	5.32	1.099	1.020	1.274	1.172
Extraction	71	4.98	1.064	1.007	1.260	1.184
DEPN-style	15	5.65	1.077	0.997	1.234	1.137

Appendix

Table 9: Effect of the leaked-record selection on Llama-3.2-1B and ai4privacy (

k=1{,}000

). Overlap: leaked records among the 301 evaluation records.

1.6% of the neurons ( $k=4{,}686$ )
	Inference	Unseen PPL $\times$		Seen PPL $\times$
$\|\mathcal{R}\|$	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	40.86	1.000	1.000	1.000	1.000
10	12.62	0.902	0.800	1.660	1.631
25	14.62	0.853	0.783	1.557	1.558
50	10.30	0.853	0.739	1.797	1.731

Appendix

Table 10: Effect of the number of leaked records

|\mathcal{R}|

on Gemma-2-2B and ai4privacy.

	Inference Acc. (%)		Unseen PPL $\times$		Seen PPL $\times$
Generator	Public JEPA	Pruned	All	Non-PII	All	Non-PII
Pseudo-words	1.33	5.32	1.099	1.020	1.274	1.172
Faker	2.99	6.31	1.105	1.013	1.269	1.155
Hybrid	4.32	6.31	1.096	1.017	1.263	1.161

Appendix

Table 11: Effect of the surrogate generator of the public model on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference Acc. (%)		Unseen PPL $\times$		Seen PPL $\times$
$r$	Unpruned	Pruned	All	Non-PII	All	Non-PII
4	6.64	4.98	1.107	1.033	1.190	1.099
8	11.96	5.32	1.099	1.020	1.274	1.172
16	17.94	7.31	1.039	0.960	1.337	1.228
32	29.90	10.63	1.060	0.937	2.037	1.867

Appendix

Table 12: Effect of the LoRA rank

r

on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference Acc. (%)
Candidates	Unpruned	Pruned
50	11.96	5.32
100	6.64	3.65
500	1.99	0.66

Appendix

Table 13: Effect of the number of candidates on Llama-3.2-1B and ai4privacy (

k=1{,}000

		Inference Acc. (%)
Entity type	Records	Unpruned	DEPN	JEPA-Prune
PERSON	301	11.96	3.32	5.32
LOC	181	15.47	1.10	13.26
ORG	301	33.89	1.33	33.55
PRODUCT	251	47.01	0.00	39.44

Appendix

Table 14: Effect of pruning on the leakage of other entity types on Llama-3.2-1B and ai4privacy (

k=1{,}000

		Inference	Unseen PPL $\times$		Seen PPL $\times$
$k$	Reference	Acc. (%)	All	Non-PII	All	Non-PII
–	Unpruned	40.86	1.000	1.000	1.000	1.000
0.8%	Random	31.23	0.986	1.002	1.186	1.206
	Scrubbed	14.29	0.757	0.684	1.432	1.399
	JEPA ( $\lambda=0.5$ )	12.96	0.805	0.718	1.550	1.487
	JEPA ( $\lambda=1$ )	11.63	0.805	0.717	1.603	1.537

Appendix

Table 15: JEPA versus scrubbed PII-free reference on Gemma-2-2B and ai4privacy.

	Inference	Unseen PPL $\times$		Seen PPL $\times$
Reference / control	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	11.96	1.000	1.000	1.000	1.000
Masked / masked	6.31	1.154	1.079	1.364	1.276
Masked / type	4.65	1.357	1.165	1.652	1.428
Type / type	5.32	1.099	1.020	1.274	1.172

Appendix

Table 16: Entity-type versus masked view on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference	Unseen PPL $\times$		Seen PPL $\times$		MIA	Extracted
Model	Acc. (%)	All	Non-PII	All	Non-PII	AUC	names
Unpruned	40.86	1.000	1.000	1.000	1.000	0.952	474
Public, scrubbed	5.32	1.617	1.294	2.125	1.511	0.861	56
Public, JEPA ( $\lambda=2$ )	3.65	2.200	0.928	4.274	1.613	0.772	0
JEPA-Prune (0.8%)	9.30	0.809	0.713	1.640	1.577	0.827	68
JEPA-Prune (1.6%)	8.64	0.855	0.745	1.932	1.910	0.784	59

Appendix

Table 17: Pruning the private model versus deploying a public model on Gemma-2-2B and ai4privacy.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 272 papers

Apr 8, 2026

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Current LLM-based services typically require users to submit raw text regardless of its sensitivity. While intuitive, such practice introduces substantial privacy risks, as unauthorized access may expose personal, medical, or legal information. Although prior defenses strived to mitigate these risks, they often incur substantial computational overhead and degrade model performance. To overcome this privacy-efficiency trade-off, we introduce Privacy-Preserving Fine-Tuning (PPFT), a novel training pipeline that eliminates the need for transmitting raw prompt text while maintaining a favorable balance between privacy preservation and model utility for both clients and service providers. Our approach operates in two stages: first, we train a client-side encoder together with a server-side projection module and LLM, enabling the server to condition on k-pooled prompt embeddings instead of raw text; second, we fine-tune the projection module and LLM on private, domain-specific data using noise-injected embeddings, allowing effective adaptation without exposing plain text prompts and requiring access to the decoder's internal parameters. Extensive experiments on domain-specific and general benchmarks demonstrate that PPFT achieves a striking balance between privacy and utility, maintaining competitive performance with minimal degradation compared to noise-free upper bounds.

Jeongho Yoon, Chanhee Park, Yongchan Chun +2

Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business

Privacy-Preserving Language Models Privacy-Preserving ML LLM Fine-Tuning

Ask PDF

Jun 15, 2026

Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

Large Language Models are increasingly trained on proprietary or sensitive data, from private healthcare and financial records to user conversations containing secrets. Ensuring the privacy of such data against extraction attacks has become a central concern. In this paper, we ask whether an attacker who can poison a portion of the training data can facilitate the leakage of a separate target record they have no access to. We answer in the affirmative and show that such leakage can be induced by a poisoning mechanism that reshapes the model's local loss landscape around the target completion. Our key insight is that poisoning to create a sharp loss minimum at the target, surrounded by elevated loss on nearby alternatives, forces the model to memorize the target as the unique low-loss solution in its neighborhood. The attack requires no architectural changes, and generalizes across centralized and federated learning settings. We demonstrate that the attack amplifies privacy leakage across language (up to 100% successful extraction), and vision-language models (up 90% successful extraction). We show that the attack is thwarted when the model is trained to be differentially private. However, we introduce a new attack that directly probes the loss landscape bypassing even differential privacy defenses.

Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree +2

UC Riverside · Riverside, California, USA · Queen’s University Belfast +2

Privacy Leakage in Language Models LLM Security Training Data Extraction Attacks Data Poisoning Attacks

Ask PDF

Jun 9, 2026

Advancing the State-of-the-Art in Empirical Privacy Auditing

Parameter-efficient fine-tuning of large language models (LLMs) can exhibit problematic memorization of individual training examples. Empirical privacy auditing (EPA) quantifies this risk by measuring realistic data leakage on membership inference (MI) or reconstruction attacks. A key challenge in EPA is designing ``canary'' examples that are mixed with the privacy-sensitive training data. We propose generating synthetic canaries via high-temperature sampling (

T \geq 0.8

) from LLMs, using prompts tailored to the privacy-sensitive training data. These canaries act as high-influence outliers, ensuring high identifiability and hence strong audits. Further, since the canaries are themselves non-private, they are inspectable and can be inserted with repetition without jeopardizing the privacy of the real data. An important use of models fine-tuned on privacy-sensitive data is the generation of synthetic data. This also comes with privacy risk. We introduce a powerful synthetic data audit based on fine-tuning an auxiliary model on the synthetic data. Auditing the auxiliary model for the original canaries then provides a strong estimate of the privacy leakage through the synthetic data. Finally, leveraging our strong auditing methodologies, we perform a systematic investigation into the interacting effects of model capacity and canary entropy on memorization.

Nicole Mitchell, Galen Andrew, Arun Ganesh +2

Google Research San Francisco, CA · Google Research Seattle, WA

Data Leakage Privacy Auditing LLM Auditing Privacy Leakage in Language Models Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

Privacy Leakage in Language Models

Explore similar work

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

Advancing the State-of-the-Art in Empirical Privacy Auditing