Private Model Training

Private model training aims to develop machine learning models that protect the privacy of sensitive training data while maintaining high accuracy. Current research focuses on improving the efficiency and accuracy of differentially private stochastic gradient descent (DP-SGD) through techniques like gradient decomposition, matrix factorization, and adaptive optimization, often incorporating model architectures such as residual networks, transformers, and mixture of experts models. These advancements are crucial for enabling the responsible use of sensitive data in various applications, particularly in healthcare and finance, while addressing privacy concerns and regulatory requirements. Furthermore, leveraging public data for pre-training or to improve DP-SGD is a significant area of investigation.

Papers

February 19, 2023

Why Is Public Pretraining Necessary for Private Model Training?
Arun Ganesh, Mahdi Haghifam, Milad Nasr, Sewoong Oh, Thomas Steinke, Om Thakkar, Abhradeep Thakurta, Lun Wang
Transfer Learning Model Training Private Data Privacy Utility Private Model Training

December 1, 2022

Differentially Private Adaptive Optimization with Delayed Preconditioners
Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J. Reddi, H. Brendan McMahan, Virginia Smith
Adaptive Preconditioner Private Optimization Private Model Training Adaptive Optimization

June 15, 2022

Disparate Impact in Differential Privacy from Gradient Misalignment
Maria S. Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, Jesse C. Cresswell
Differential Privacy Private Stochastic Gradient Descent Gradient Alignment Private Model Training Fairness Loss Disparate Impact

March 1, 2022

Differentially private training of residual networks with scale normalisation
Helena Klause, Alexander Ziller, Daniel Rueckert, Kerstin Hammernik, Georgios Kaissis
Private Stochastic Gradient Descent Normalization Layer Private Model Training Data Normalisation Group Normalization

February 21, 2022

Privacy Leakage of Adversarial Training Models in Federated Learning Systems
Jingyang Zhang, Yiran Chen, Hai Li
Adversarial Attack Federated Learning Adversarial Training PRIvacy Leakage Privacy Attack Private Model Training

December 1, 2021

Public Data-Assisted Mirror Descent for Private Model Training
Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M. Suriyakumar, Om Thakkar, Abhradeep Thakurta
Mirror Descent Differential Privacy SGD Private Gradient Private Model Training Online Mirror Descent