Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

Mohamed Shaaban; Mohamed Elmahallawy

cs.CLOct 16, 2026

Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

Authors: Mohamed Shaaban, Mohamed Elmahallawy

Organizations: School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA

Abstract

Fine-tuning large language models (LLMs) on proprietary records enables domain adaptation but introduces two challenges: (i) privacy leakage of memorized private data, such as personally identifiable information (PII), and (ii) balancing private data removal with utility preservation. Existing defenses, including data sanitization and differential privacy, often degrade performance, while privacy-neuron editing lacks a PII-free reference to distinguish identity-specific neurons from those supporting useful text. We propose JEPA-Prune, a post-hoc neuron pruning framework guided by a model trained exclusively on PII-free views. Alongside the fine-tuned private model, JEPA-Prune, trains a public model on surrogate and masked views of the same records using a joint-embedding predictive architecture (JEPA), capturing domain knowledge without identifiers. A cross-model representation probe quantifies deviations between private representations of leaked records and this PII-free reference. Integrated gradients and a scale-invariant specificity score then identify PII-responsive neurons, which are pruned without retraining. Experiments show that pruning just 1.6% of neurons reduces PII inference accuracy by up to 7.4x and PII extraction recall by up to 13.9x, while keeping unseen-text perplexity within 16% of that of the unpruned model for most LLMs. In contrast, privacy-neuron editing increases perplexity by at least 26x.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of JEPA-Prune workflow. (a) Private $M_{\mathrm{priv}}$ and public $M_{\mathrm{pub}}$ LoRA adapters share a frozen base model and train on raw records and PII-free views, respectively. (b) For each leaked record, integrated gradients attribute the cosine distance between private-model embeddings of the record or its entity-type view and the public PII-free reference to attention and feed-forward neurons. (c) PII-responsive neurons are zeroed, and the pruned model replaces $M_{\mathrm{priv}}$ in deployment. Heatmaps and the embedding inset are schematic.

		ai4privacy			ECHR			Yelp
		Inference	Unseen PPL $\times$		Inference	Unseen PPL $\times$		Inference	Unseen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	Acc. (%)	All	Non-PII	Acc. (%)	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	26.58	1.000	1.000	35.22	1.000	1.000
	Random (0.8%)	10.30	1.039	1.030	24.92	1.073	1.075	31.89	1.048	1.047
	Random (1.6%)	5.98	1.106	1.057	25.25	1.208	1.201	31.56	1.117	1.115
	Random (3.2%)	3.65	1.477	1.316	21.26	1.176	1.164	25.91	1.328	1.328

Table 1: Inference accuracy and unseen perplexity of the pruned models.

Dataset	Method	Extracted	Precision (%)	Recall (%)
ai4privacy	Unpruned	474	8.04	24.01
	Random	489	5.07	24.77
	DEPN (0.74%)	91	3.25	4.61
	JEPA-Prune	59	8.74	2.99
ECHR	Unpruned	126	3.78	10.59
	Random	80	3.34	6.72

Table 2: Extraction attack against the pruned Gemma-2-2B at a budget of 1.6%.

Figure 2: Distribution over layers of the neurons pruned by JEPA-Prune at a budget of 1.6% (

k=9{,}437

for both models); the bars of each panel sum to 100%.

		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Pruned	Acc. (%)	All	Non-PII	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	1.000	1.000
	FFN	5.65	1.085	0.996	1.200	1.093
	Attn	2.99	1.701	1.534	2.298	2.134
	Attn + FFN	5.32	1.099	1.020	1.274	1.172
Mistral-7B	Unpruned	49.17	1.000	1.000	1.000	1.000

Table 3: Effect of the pruned neuron type on ai4privacy (

k=1{,}000

and

3{,}600

, 0.61% of all attention and feed-forward neurons).

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

		ai4privacy					ECHR					Yelp
		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII
Llama-3.2-1B	Unpruned	11.96	1.000	1.000	1.000	1.000	26.58	1.000	1.000	1.000	1.000	35.22	1.000	1.000	1.000	1.000
	Random (0.8%)	10.30	1.039	1.030	1.081	1.079	24.92	1.073	1.075	1.087	1.088	31.89	1.048	1.047	1.065	1.063
	Random (1.6%)	5.98	1.106	1.057	1.197	1.147	25.25	1.208	1.201	1.246	1.234	31.56	1.117	1.115	1.183	1.177
	Random (3.2%)	3.65	1.477	1.316	1.759	1.606	21.26	1.176	1.164	1.263	1.250	25.91	1.328	1.328	1.467	1.461

Appendix

Table 4: Inference accuracy and unseen and seen perplexity of the pruned models at budgets of 0.8%, 1.6%, and 3.2%.

Dataset	Method	Extracted	Precision (%)	Recall (%)
ai4privacy	Unpruned	474	8.04	24.01
	Random (0.8%)	471	7.89	23.86
	Random (1.6%)	489	5.07	24.77
	Random (3.2%)	361	10.48	18.29
	DEPN (0.74%)	91	3.25	4.61
	JEPA-Prune (0.8%)	68	9.43	3.44

Appendix

Table 5: Extraction attack against the pruned Gemma-2-2B at budgets of 0.8%, 1.6%, and 3.2%.

		ai4privacy					ECHR
		Inference	Unseen PPL $\times$		Seen PPL $\times$		Inference	Unseen PPL $\times$		Seen PPL $\times$
Model	Method	Acc. (%)	All	Non-PII	All	Non-PII	Acc. (%)	All	Non-PII	All	Non-PII
Mistral-7B	Unpruned	49.17	1.000	1.000	1.000	1.000	59.47	1.000	1.000	1.000	1.000
	Random	29.57	1.022	1.019	1.261	1.314	53.49	1.027	1.026	1.103	1.105
	DEPN	1.99	145.1	413.6	453.4	1,889	5.98	26.0	28.3	52.5	59.1
	JEPA-Prune	6.64	0.888	0.797	2.030	2.181	33.89	0.985	0.963	1.336	1.323

Appendix

Table 6: Inference accuracy and perplexity of the pruned Mistral-7B and Llama-3.1-8B at a budget of 1.6%.

		LOSS attack		Reference attack		PPL $\times$
Method	Budget	AUC	TPR@1%FPR (%)	AUC	TPR@1%FPR (%)	Unseen	Seen
Unpruned	–	0.899	4.0	0.952	44.7	1.000	1.000
Random	0.8%	0.867	2.7	0.932	31.6	0.986	1.186
	1.6%	0.835	2.0	0.922	23.4	1.152	1.488
	3.2%	0.786	0.9	0.873	17.5	1.100	1.781
DEPN	0.74%	0.581	1.4	0.595	2.4	258.0	697.9

Appendix

Table 7: Membership inference against the pruned Gemma-2-2B on ai4privacy.

	Inference	Unseen PPL $\times$		Seen PPL $\times$
$k$	Acc. (%)	All	Non-PII	All	Non-PII
0 (unpruned)	11.96	1.000	1.000	1.000	1.000
125	6.98	1.007	0.975	1.059	1.009
250	6.31	1.012	0.971	1.089	1.028
500	5.65	1.024	0.969	1.141	1.069
1,000	5.32	1.099	1.020	1.274	1.172

Appendix

Table 8: Effect of the pruning budget

k

on Llama-3.2-1B and ai4privacy.

		Inference	Unseen PPL $\times$		Seen PPL $\times$
Rule	Overlap	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	–	11.96	1.000	1.000	1.000	1.000
Inference	0	5.32	1.099	1.020	1.274	1.172
Extraction	71	4.98	1.064	1.007	1.260	1.184
DEPN-style	15	5.65	1.077	0.997	1.234	1.137

Appendix

Table 9: Effect of the leaked-record selection on Llama-3.2-1B and ai4privacy (

k=1{,}000

). Overlap: leaked records among the 301 evaluation records.

1.6% of the neurons ( $k=4{,}686$ )
	Inference	Unseen PPL $\times$		Seen PPL $\times$
$\|\mathcal{R}\|$	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	40.86	1.000	1.000	1.000	1.000
10	12.62	0.902	0.800	1.660	1.631
25	14.62	0.853	0.783	1.557	1.558
50	10.30	0.853	0.739	1.797	1.731

Appendix

Table 10: Effect of the number of leaked records

|\mathcal{R}|

on Gemma-2-2B and ai4privacy.

	Inference Acc. (%)		Unseen PPL $\times$		Seen PPL $\times$
Generator	Public JEPA	Pruned	All	Non-PII	All	Non-PII
Pseudo-words	1.33	5.32	1.099	1.020	1.274	1.172
Faker	2.99	6.31	1.105	1.013	1.269	1.155
Hybrid	4.32	6.31	1.096	1.017	1.263	1.161

Appendix

Table 11: Effect of the surrogate generator of the public model on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference Acc. (%)		Unseen PPL $\times$		Seen PPL $\times$
$r$	Unpruned	Pruned	All	Non-PII	All	Non-PII
4	6.64	4.98	1.107	1.033	1.190	1.099
8	11.96	5.32	1.099	1.020	1.274	1.172
16	17.94	7.31	1.039	0.960	1.337	1.228
32	29.90	10.63	1.060	0.937	2.037	1.867

Appendix

Table 12: Effect of the LoRA rank

r

on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference Acc. (%)
Candidates	Unpruned	Pruned
50	11.96	5.32
100	6.64	3.65
500	1.99	0.66

Appendix

Table 13: Effect of the number of candidates on Llama-3.2-1B and ai4privacy (

k=1{,}000

		Inference Acc. (%)
Entity type	Records	Unpruned	DEPN	JEPA-Prune
PERSON	301	11.96	3.32	5.32
LOC	181	15.47	1.10	13.26
ORG	301	33.89	1.33	33.55
PRODUCT	251	47.01	0.00	39.44

Appendix

Table 14: Effect of pruning on the leakage of other entity types on Llama-3.2-1B and ai4privacy (

k=1{,}000

		Inference	Unseen PPL $\times$		Seen PPL $\times$
$k$	Reference	Acc. (%)	All	Non-PII	All	Non-PII
–	Unpruned	40.86	1.000	1.000	1.000	1.000
0.8%	Random	31.23	0.986	1.002	1.186	1.206
	Scrubbed	14.29	0.757	0.684	1.432	1.399
	JEPA ( $\lambda=0.5$ )	12.96	0.805	0.718	1.550	1.487
	JEPA ( $\lambda=1$ )	11.63	0.805	0.717	1.603	1.537

Appendix

Table 15: JEPA versus scrubbed PII-free reference on Gemma-2-2B and ai4privacy.

	Inference	Unseen PPL $\times$		Seen PPL $\times$
Reference / control	Acc. (%)	All	Non-PII	All	Non-PII
Unpruned	11.96	1.000	1.000	1.000	1.000
Masked / masked	6.31	1.154	1.079	1.364	1.276
Masked / type	4.65	1.357	1.165	1.652	1.428
Type / type	5.32	1.099	1.020	1.274	1.172

Appendix

Table 16: Entity-type versus masked view on Llama-3.2-1B and ai4privacy (

k=1{,}000

	Inference	Unseen PPL $\times$		Seen PPL $\times$		MIA	Extracted
Model	Acc. (%)	All	Non-PII	All	Non-PII	AUC	names
Unpruned	40.86	1.000	1.000	1.000	1.000	0.952	474
Public, scrubbed	5.32	1.617	1.294	2.125	1.511	0.861	56
Public, JEPA ( $\lambda=2$ )	3.65	2.200	0.928	4.274	1.613	0.772	0
JEPA-Prune (0.8%)	9.30	0.809	0.713	1.640	1.577	0.827	68
JEPA-Prune (1.6%)	8.64	0.855	0.745	1.932	1.910	0.784	59

Appendix

Table 17: Pruning the private model versus deploying a public model on Gemma-2-2B and ai4privacy.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 272 papers

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
Apr 8, 2026Jeongho Yoon, Chanhee Park, Yongchan Chun +2Privacy-Preserving Language Models Privacy-Preserving ML
Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business
Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs
Jun 15, 2026Md Abdullah Al Mamun, Ngoc Phu Doan, Pedram Zaree +2Privacy Leakage in Language Models LLM Security
UC Riverside · Riverside, California, USA · Queen’s University Belfast +2
Advancing the State-of-the-Art in Empirical Privacy Auditing
Jun 9, 2026Nicole Mitchell, Galen Andrew, Arun Ganesh +2Data Leakage Privacy Auditing
Google Research San Francisco, CA · Google Research Seattle, WA

Apr 8, 2026J/K move · Enter open · S save

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Jeongho Yoon, Chanhee Park, Yongchan Chun, Hyeonseok Moon+1

Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business

Privacy-Preserving Language Models Privacy-Preserving ML LLM Fine-Tuning

Open paper AskPDF arXiv

Current LLM-based services typically require users to submit raw text regardless of its sensitivity. While intuitive, such practice introduces substantial privacy risks, as unauthorized access may expose personal, medical, or legal information. Although prior defenses strived to mitigate these risks, they often incur substantial computational overhead and degrade model performance. To overcome this privacy-efficiency trade-off, we introduce Privacy-Preserving Fine-Tuning (PPFT), a novel training pipeline that eliminates the need for transmitting raw prompt text while maintaining a favorable balance between privacy preservation and model utility for both clients and service providers. Our approach operates in two stages: first, we train a client-side encoder together with a server-side projection module and LLM, enabling the server to condition on k-pooled prompt embeddings instead of raw text; second, we fine-tune the projection module and LLM on private, domain-specific data using noise-injected embeddings, allowing effective adaptation without exposing plain text prompts and requiring access to the decoder's internal parameters. Extensive experiments on domain-specific and general benchmarks demonstrate that PPFT achieves a striking balance between privacy and utility, maintaining competitive performance with minimal degradation compared to noise-free upper bounds.

Abstract

Privacy Leakage in Language Models

Explore similar work

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMs

Advancing the State-of-the-Art in Empirical Privacy Auditing