Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models

Gabriel Loiseau; Damien Sileo; Damien Riquet; Maxime Meyer; Marc Tommasi

cs.CLMar 31, 2026

Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models

Authors: Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi

Organizations: Hornetsecurity, Hem, France · Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 - CRIStAL, F-59000 Lille, France

Abstract

Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.

Figures & tables

Dataset	Avg tokens	$\bar{S}$	% Priv.
MHB	272	3.31	77.4
RMHP	225	2.92	64.7
RC	306	2.89	60.6
RLA	283	2.84	60.3
MQ	136	2.34	44.6
EE	332	2.16	33.5

Table 2: Per-dataset statistics, sorted by mean privacy score. Avg tokens: mean BPE token count.

\bar{S}

: mean teacher-assigned privacy score (1–5). % Priv.: percentage of texts rated

\geq 3

(Somewhat private or above).

Model	Params	Acc. $\uparrow$	Macro F1 $\uparrow$	MAE $\downarrow$	Per-class F1 $\uparrow$
					C1	C2	C3	C4	C5
Ettin-150M	149M	74.9	68.1	0.28	91.5	58.3	58.5	63.6	68.6
ModernBERT-base	149M	73.7	67.2	0.28	91.9	57.4	57.7	62.8	68.2
BERT-base	110M	73.3	65.9	0.29	91.3	55.4	55.7	60.6	66.4
Ettin-17M	17M	71.1	62.5	0.34	90.1	51.2	52.8	57.5	60.7

Table 4: Classification performance on the held-out test set. Accuracy and F1 scores are in %. MAE is the mean absolute error on the 1–5 ordinal scale (range: 0–4). All encoder models are trained with identical hyperparameters. For reference, a Majority Class baseline achieves 45.9% accuracy (12.5 macro F1), while a Random baseline achieves 20.0% accuracy (18.0 macro F1).

Condition	$\bar{S}$	$\Delta$	% Class 1
Original	3.25	–	25.2
Mask direct	2.91	0.34	30.5
Mask quasi	3.02	0.23	28.5
Mask ALL	1.39	1.86	84.1
Mask 30% random	3.56	-0.31	17.3

Table 6: Privacy scores on TAB test set.

\bar{S}

: mean score (1–5).

\Delta

: reduction from original. % Class 1: proportion rated “Harmless.”

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation
Apr 8, 2026Jeongho Yoon, Chanhee Park, Yongchan Chun +2Privacy-Preserving Language Models Privacy-Preserving ML
Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business
Natural Identifiers for Privacy and Data Audits in Large Language Models
Jun 23, 2026Lorenzo Rossi, Bartłomiej Marek, Franziska Boenisch +1Privacy Auditing LLM Auditing
CISPA Helmholtz Center for Information Security
InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants
Oct 25, 2026Pengrun Huang, Yiwei Fu, Luoxi Meng +1Privacy Leakage in Language Models Privacy Protection for LLM Agents
UC San Diego · University of Illinois Urbana-Champaign

Apr 8, 2026J/K move · Enter open · S save

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Jeongho Yoon, Chanhee Park, Yongchan Chun, Hyeonseok Moon+1

Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business

Privacy-Preserving Language Models Privacy-Preserving ML LLM Fine-Tuning

Open paper AskPDF arXiv

Current LLM-based services typically require users to submit raw text regardless of its sensitivity. While intuitive, such practice introduces substantial privacy risks, as unauthorized access may expose personal, medical, or legal information. Although prior defenses strived to mitigate these risks, they often incur substantial computational overhead and degrade model performance. To overcome this privacy-efficiency trade-off, we introduce Privacy-Preserving Fine-Tuning (PPFT), a novel training pipeline that eliminates the need for transmitting raw prompt text while maintaining a favorable balance between privacy preservation and model utility for both clients and service providers. Our approach operates in two stages: first, we train a client-side encoder together with a server-side projection module and LLM, enabling the server to condition on k-pooled prompt embeddings instead of raw text; second, we fine-tune the projection module and LLM on private, domain-specific data using noise-injected embeddings, allowing effective adaptation without exposing plain text prompts and requiring access to the decoder's internal parameters. Extensive experiments on domain-specific and general benchmarks demonstrate that PPFT achieves a striking balance between privacy and utility, maintaining competitive performance with minimal degradation compared to noise-free upper bounds.

Abstract

Privacy-Preserving ML

Explore similar work

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Natural Identifiers for Privacy and Data Audits in Large Language Models

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants