Private Stochastic Gradient Descent

Private Stochastic Gradient Descent (DP-SGD) aims to train machine learning models on sensitive data while guaranteeing differential privacy, preventing individual data points from being inferred. Current research focuses on improving the accuracy and efficiency of DP-SGD through techniques like gradient shuffling, adaptive noise mechanisms, and optimized clipping strategies, often applied to deep learning models including transformers and convolutional neural networks. These advancements address the inherent trade-off between privacy and model utility, impacting fields like healthcare and finance where data privacy is paramount, by enabling the development of accurate and privacy-preserving machine learning models. Ongoing work also explores decentralized DP-SGD and the development of methods to verify the privacy guarantees of trained models.

Papers

November 10, 2024

Are Neuromorphic Architectures Inherently Privacy-preserving? An Exploratory Study
Ayana Moshruba, Ihsen Alouani, Maryam Parsa
Spiking Neural Network Exploratory Study Private Stochastic Gradient Descent Neuromorphic Architecture User Configurable Privacy Defense

November 6, 2024

Scalable DP-SGD: Shuffling vs. Poisson Subsampling
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
Private Stochastic Gradient Descent Subsampling Method Dp SGD Shuffling Gradient

November 5, 2024

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight
Tao Huang, Qingyu Huang, Xin Shi, Jiayang Meng, Guolong Zheng, Xu Yang, Xun Yi
Differential Privacy Private Stochastic Gradient Descent Adaptive Clipping Gradient Clipping

October 9, 2024

Noise is All You Need: Private Second-Order Convergence of Noisy SGD
Dmitrii Avdiukhin, Michael Dinitz, Chenglin Fan, Grigory Yaroslavtsev
Stochastic Gradient Descent Industrial Disturbing Noise Private Stochastic Gradient Descent Private Optimization

August 19, 2024

Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement
Jeremiah Birrell, Reza Ebrahimi, Rouzbeh Behnia, Jason Pacheco
Private Stochastic Private Stochastic Gradient Descent Subsampling Method Privacy Amplification Dp SGD Independent Minibatching

July 31, 2024

Differentially Private Block-wise Gradient Shuffle for Deep Learning
David Zagardo
Deep Learning Differential Privacy Private Stochastic Gradient Descent Gradient Noise

July 17, 2024

Enhancing the Utility of Privacy-Preserving Cancer Classification using Synthetic Data
Richard Osuala, Daniel M. Lang, Anneliese Riess, Georgios Kaissis, Zuzanna Szafranowska, Grzegorz Skorupko, Oliver Diaz, Julia A. Schnabel, Karim Lekadir
Deep Learning Synthetic Data Task Utility Conditional Generative Adversarial Network Private Stochastic Gradient Descent Cancer Classification Private Model Training

July 11, 2024

Differentially Private Neural Network Training under Hidden State Assumption
Ding Chen, Chen Liu
Differential Privacy Private Stochastic Gradient Descent Hidden State Proximal Gradient Descent Private Neural

July 9, 2024

It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss
Meenatchi Sundaram Muthu Selva Annamalai
PRIvacy Leakage Convex Loss Interest Loss Private Stochastic Gradient Descent Privacy Amplification

June 27, 2024

Too Good to be True? Turn Any Model Differentially Private With DP-Weights
David Zagardo
Full Model Differential Privacy Private Stochastic Gradient Descent Dp SGD Differential Privacy Noise

June 4, 2024

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning
Yixuan Liu, Li Xiong, Yuhan Liu, Yujie Gu, Ruixuan Liu, Hong Chen
Full State Reconstruction Private Stochastic Gradient Descent Privacy Budget Private Model Training Structured Reconstruction Noisy Information

May 29, 2024

LMO-DP: Optimizing the Randomization Mechanism for Differentially Private Fine-Tuning (Large) Language Models
Qin Yang, Meisam Mohammad, Han Wang, Ali Payani, Ashish Kundu, Kai Shu, Yan Yan, Yuan Hong
Language Model Differential Privacy Private Stochastic Gradient Descent Scale Pre Trained Language Model Private Fine Tuning

May 27, 2024

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails
Haichao Sha, Yang Cao, Yong Liu, Yuncheng Wu, Ruixuan Liu, Hong Chen
Single CLIP Gradient Norm Heavy Tailed Private Stochastic Gradient Descent Thin Tail Clipped Stochastic Gradient Descent Heavy Tail High Probability Guarantee

May 23, 2024

Nearly Tight Black-Box Auditing of Differentially Private Machine Learning
Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro
Black Box Membership Inference Attack PRIvacy Leakage Private Stochastic Gradient Descent Privacy Analysis Private Machine Learning

May 21, 2024

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent
Holger Dette, Carina Graw
Differential Privacy Stochastic Gradient Descent Uncertainty Quantification Local Differential Privacy Private Stochastic Gradient Descent Private Convex Optimization

May 4, 2024

PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient Push with Tight Utility Bounds
Zehan Zhu, Yan Huang, Xin Wang, Jinming Xu
Differential Privacy Decentralized Learning Private Stochastic Gradient Descent Dp SGD Differential Privacy Noise Gradient Variance

March 26, 2024

How Private are DP-SGD Implementations?
Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
Private Stochastic Gradient Descent Privacy Analysis Subsampling Method Dp SGD Shuffling Gradient

December 31, 2023

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners
Rachel Redberg, Antti Koskela, Yu-Xiang Wang
Privacy Policy Practical Application Privacy Protection Private Stochastic Gradient Descent Privacy Preserving Machine Learning Privacy Analysis Private Linear

December 13, 2023

Differentially Private Gradient Flow based on the Sliced Wasserstein Distance for Non-Parametric Generative Modeling
Ilana Sebag, Muni Sreenivas PYDI, Jean-Yves Franceschi, Alain Rakotomamonjy, Mike Gartrell, Jamal Atif, Alexandre Allauzen
Generative Modeling Gradient Flow Private Stochastic Gradient Descent Sliced Wasserstein Distance Private Gradient Private Generative Model Non Parametric Generative

November 24, 2023

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach
Xinwei Zhang, Zhiqi Bu, Zhiwei Steven Wu, Mingyi Hong
Differential Privacy Absolute Stance Bias Error Feedback Privacy Guarantee Private Stochastic Gradient Descent Gradient Clipping

Private Stochastic Gradient Descent

Papers

Are Neuromorphic Architectures Inherently Privacy-preserving? An Exploratory Study

Scalable DP-SGD: Shuffling vs. Poisson Subsampling

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

Noise is All You Need: Private Second-Order Convergence of Noisy SGD

Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement

Differentially Private Block-wise Gradient Shuffle for Deep Learning

Enhancing the Utility of Privacy-Preserving Cancer Classification using Synthetic Data

Differentially Private Neural Network Training under Hidden State Assumption

It's Our Loss: No Privacy Amplification for Hidden State DP-SGD With Non-Convex Loss

Too Good to be True? Turn Any Model Differentially Private With DP-Weights

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

LMO-DP: Optimizing the Randomization Mechanism for Differentially Private Fine-Tuning (Large) Language Models

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

Nearly Tight Black-Box Auditing of Differentially Private Machine Learning

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent

PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient Push with Tight Utility Bounds

How Private are DP-SGD Implementations?

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners

Differentially Private Gradient Flow based on the Sliced Wasserstein Distance for Non-Parametric Generative Modeling

Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach