Nonparametric Variational Differential Privacy via Embedding Parameter Clipping

Dina El Zein; Shashi Kumar; James Henderson

cs.LGMar 10, 2026

Nonparametric Variational Differential Privacy via Embedding Parameter Clipping

Authors: Dina El Zein, Shashi Kumar, James Henderson

Organizations: Idiap Research Institute, Switzerland · EPFL, Switzerland

Abstract

The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.

Figures & tables

Figure 1: The Nonparametric Variational Differential Privacy (NVDP) Architecture. An input sequence is first processed by a standard pretrained Transformer encoder to produce token-wise embeddings,

\bm{x}\in\mathbb{R}^{n\times d}

. These embeddings are then passed through a NVIB layer, which acts as a stochastic bottleneck by mapping them to the parameters of a posterior distribution. A sample is drawn from this distribution and processed by a Denoising Multi-Head Attention (MHA) and a Feed-Forward layer. Crucially, the standard residual skip connection around the MHA and Feed-Forward block is removed, ensuring that all information must pass through the privacy-preserving NVIB layer.

		NVDP (Unconstrained)			NVDP-Clipped (Ours)			Non-Private
Dataset	Metric	Score	BDP $\downarrow$	RD (max) $\downarrow$	Score	BDP $\downarrow$	RD (max) $\downarrow$	Score
BERT-Base Backbone
MRPC	Accuracy	83.0	10.70	0.34	81.6	10.60	0.013	81.2
MRPC	F1 Score	87.5	10.70	0.34	87.0	10.60	0.013	86.0
STS-B	Pearson	85.2	20.93	1.41	85.7	20.34	0.423	86.0
STS-B	Spearman	84.0	20.93	1.41	84.7	20.34	0.423	84.9

Table 1: Privacy-utility trade-off on GLUE tasks for multiple Transformer backbones. We compare our proposed NVDP-Clipped model against the unconstrained NVDP baseline for BERT-Base, BERT-Large, and RoBERTa-Base. For each model, we report its best-achieved utility score alongside privacy guarantees. Privacy is measured via BDP (BDP

\downarrow

) and the maximum (worst-case) RD (RD (max)

\downarrow

). Lower privacy values are better. The best-performing model per task is bolded . The non-private BERT-Base is included as a utility upper bound

Model	F1 $\uparrow$	BDP $\downarrow$	RD (max) $\downarrow$
NVDP (Unconstrained)	83.7	10.46	4.044
NVDP-Clipped	82.6	9.90	3.567

Table 2: Privacy-utility trade-off for Speech Language Identification using a Wav2Vec2 backbone. Consistent with the NLP results, the clipped model achieves a tighter privacy profile (lower BDP and RD) with minimal impact on utility measure by F1 score.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	BERT-Base	BERT-Large	RoBERTa-Base
Learning Rate	7e-5	2e-5	2e-5
Warmup Ratio	0.06	0.1	0.06
Adam Epsilon	1e-8	1e-8	1e-8
Train Batch Size	32	32	32
Eval Batch Size	16	16	16
Max Sequence Length	512 / 128	512 / 128	512 / 128

Appendix

Table 3: General training hyperparameters for each model backbone.

Task	BERT-Base	BERT-Large	RoBERTa-Base
MRPC	(2, 0, 0.5)	(3, 0, 0.7)	(3, 0, 0.7)
QNLI	(7, 0, 1)	(7, 0, 1)	(7, 0, 1)
SST-2	(2, 0, 1)	(3, 0, 1)	(3, 0, 1)
RTE	(6, 0, 0.5)	(6, 0, 0.6)	(6, 0, 0.6)
STS-B	(10, 0, 1)	(10, 0, 1)	(10, 0, 1)

Appendix

Table 4: Task-specific clipping hyperparameters reported as the triple (

C_{\mu},C_{\alpha,min},C_{\alpha,max}

). These constraints ensure numerical stability and minimize the RD upper bound.

Model	Metric	MRPC	STS-B	RTE	QNLI	SST-2
		Acc / F1	Pearson / Spearman	Accuracy	Accuracy	Accuracy
BERT ${}_{\text{Base}}$ (Non-Private)	Utility	81.2 / 86.0	86.0 / 84.9	65.9	89.0	92.9
BERT ${}_{\text{Base}}$ (Non-Private)	Privacy	-	-	-	-	-
NVDP (Unconstrained)
$\lambda=1e-3$	Utility	82.5 / 87.1	85.2 / 84.0	64.8	89.5	91.7
	BDP ( $\epsilon_{\mu}$ ) $\downarrow$	10.95	20.93	10.90	12.10	11.19

Appendix

Table 5: Full experimental results for BERT-Base on GLUE tasks. We compare the unconstrained and clipped NVDP models across different strengths of the KL-divergence regularization weight (

\lambda_{G}=\lambda_{D}

). For each model, we report Utility (task-specific scores) and privacy metrics BDP (

\epsilon_{\mu}

) and RD. The RD metric is reported as max/avg, representing the worst-case and average-case divergence across all example pairs, respectively. Lower values are better for all privacy metrics (

\downarrow

Model	Metric	MRPC	STS-B	RTE	QNLI	SST-2
		Acc / F1	Pearson / Spearman	Accuracy	Accuracy	Accuracy
BERT ${}_{\text{Large}}$ (Non-Private)	Utility	84.1 / 88.5	87.0 / 86.1	69.3	91.9	93.3
BERT ${}_{\text{Large}}$ (Non-Private)	Privacy	-	-	-	-	-
NVDP (Unconstrained)
$\lambda=1e-3$	Utility	84.6 / 88.8	85.1 / 84.9	69.2	92.2	93.0
	BDP ( $\epsilon_{\mu}$ ) $\downarrow$	10.70	20.27	10.80	10.86	13.68

Appendix

Table 6: Full experimental results for BERT-Large on GLUE tasks, showing the complete privacy-utility trade-off across all tested KL-divergence regularization weights (

\lambda_{G},\lambda_{D}

). For each model, we report Utility (task-specific scores) and privacy metrics BDP (

\epsilon_{\mu}

) and RD. The RD metric is reported as max/avg, representing the worst-case and average-case divergence across all example pairs, respectively. Lower values are better for all privacy metrics (

\downarrow

Model	Metric	MRPC	STS-B	RTE	QNLI	SST-2
		Acc / F1	Pearson / Spearman	Accuracy	Accuracy	Accuracy
RoBERTa ${}_{\text{Base}}$ (Non-Private)	Utility	86.9 / 90.3	89.2 / 88.6	73.5	91.8	94.0
RoBERTa ${}_{\text{Base}}$ (Non-Private)	Privacy	-	-	-	-	-
NVDP (Unconstrained)
$\lambda=1e-3$	Utility	86.7 / 90.5	88.2 / 87.3	70.6	92.1	95.1
	BDP ( $\epsilon_{\mu}$ ) $\downarrow$	10.64	19.75	10.77	10.92	10.86

Appendix

Table 7: Full experimental results for RoBERTa-Base on GLUE tasks, showing the complete privacy-utility trade-off across all tested KL-divergence regularization weights (

\lambda_{G},\lambda_{D}

). For each model, we report Utility (task-specific scores) and privacy metrics BDP (

\epsilon_{\mu}

) and RD. The RD metric is reported as max/avg, representing the worst-case and average-case divergence across all example pairs, respectively. Lower values are better for all privacy metrics (

\downarrow

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 178 papers

Efficient DP-SGD for LLMs with Randomized Clipping
May 24, 2026Enayat Ullah, Sai Aparna Aketi, Devansh Gupta +2Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent
Meta Platforms Inc. · University of Southern California
DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
Jul 26, 2026Nour Jamoussi, Ikram Dridi, Giuseppe Serra +1Bayesian Neural Networks Privacy-Preserving ML
EURECOM, Sophia Antipolis, France · Concordia University, Montreal, Canada · University of Granada, Granada, Spain
Noise-Aware Differentially Private Variational Inference
Oct 25, 2024Talal Alrawajfeh, Joonas Jälkö, Antti HonkelaBayesian Inference Differential Privacy
University of Helsinki

May 24, 2026J/K move · Enter open · S save

Efficient DP-SGD for LLMs with Randomized Clipping

Enayat Ullah, Sai Aparna Aketi, Devansh Gupta, Huanyu Zhang+1

Meta Platforms Inc. · University of Southern California

Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent LLM Fine-Tuning

Open paper AskPDF arXiv

Large language models (LLMs) are trained on vast datasets that may contain sensitive information. Differential privacy (DP), the de facto standard for formal privacy guarantees, provides a principled framework for training LLMs with provable privacy protection. However, state-of-the-art DP training implementations rely on fast gradient clipping techniques with memory overhead

O(B \min\{T^2, d^2\})

, where

B

is the batch size,

T

is the sequence length, and

d

is the model width. This becomes prohibitive as both model size and context length grow. We propose DP-SGD-RC, a novel variant of DP-SGD with randomized clipping that reduces memory and compute complexity. DP-SGD-RC leverages stochastic trace estimation methods, specifically Hutchinson's estimator[Hutchinson, 1989] and its improved variant, Hutch++[Meyer et al., 2021], to reduce the memory footprint of per-sample gradient norm estimation. We provide a tight privacy analysis showing that DP-SGD-RC achieves noise multipliers competitive with deterministic clipping. Experiments fine-tuning Llama~3.2-1B on long-context benchmarks spanning classification, question answering, and summarization tasks demonstrate that DP-SGD-RC matches baseline utility while significantly reducing memory and compute requirements.

Abstract

Privacy-Preserving Language Models

Explore similar work

Efficient DP-SGD for LLMs with Randomized Clipping

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

Noise-Aware Differentially Private Variational Inference