Private Model Training

Private model training aims to develop machine learning models that protect the privacy of sensitive training data while maintaining high accuracy. Current research focuses on improving the efficiency and accuracy of differentially private stochastic gradient descent (DP-SGD) through techniques like gradient decomposition, matrix factorization, and adaptive optimization, often incorporating model architectures such as residual networks, transformers, and mixture of experts models. These advancements are crucial for enabling the responsible use of sensitive data in various applications, particularly in healthcare and finance, while addressing privacy concerns and regulatory requirements. Furthermore, leveraging public data for pre-training or to improve DP-SGD is a significant area of investigation.

Papers

July 17, 2024

June 4, 2024

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning
Yixuan Liu, Li Xiong, Yuhan Liu, Yujie Gu, Ruixuan Liu, Hong Chen
Full State Reconstruction Private Stochastic Gradient Descent Privacy Budget Private Model Training Structured Reconstruction Noisy Information

May 22, 2024

Banded Square Root Matrix Factorization for Differentially Private Model Training
Nikita P. Kalinin, Christoph Lampert
Stochastic Gradient Descent Matrix Factorization Private Model Training Factorization Based

March 17, 2024

Pencil: Private and Extensible Collaborative Learning without the Non-Colluding Assumption
Xuanqi Liu, Zhuotao Liu, Qi Li, Ke Xu, Mingwei Xu
Homomorphic Encryption Collaborative Learning Privacy Preserving Machine Learning Private Model Training Model Privacy Secure Multiparty Computation Tacit Collusion Pen Like Object

March 13, 2024

Efficiently Computing Similarities to Private Datasets
Arturs Backurs, Zinan Lin, Sepideh Mahabadi, Sandeep Silwal, Jakub Tarnawski
Density Estimation Differential Privacy SGD Private Datasets Private Model Training Similarity Function

February 13, 2024

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training
Tom Sander, Maxime Sylvestre, Alain Durmus
Financial Application Stochastic Gradient Descent Implicit Bias Differential Privacy SGD Private Model Training

February 11, 2024

Differentially Private Training of Mixture of Experts Models
Pierre Tholoniat, Huseyin A. Inan, Janardhan Kulkarni, Robert Sim
Natural Language Processing Differential Privacy Mixture Component Privacy Preservation Expert Model Private Model Training Non Private

December 24, 2023

On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
Pratiksha Thaker, Amrith Setlur, Zhiwei Steven Wu, Virginia Smith
Distribution Shift Complementary Benefit Private Data Low Dimensional Representation Private Model Training Representation Online Matter

November 27, 2023

Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspective
Lukas Wutschitz, Boris Köpf, Andrew Paverd, Saravan Rajmohan, Ahmed Salem, Shruti Tople, Santiago Zanella-Béguelin, Menglin Xia, Victor Rühle
Information Flow Machine Learning Pipeline Privacy Analysis Fine Grained Control Private Model Training Inherent Privacy

November 23, 2023

DPSUR: Accelerating Differentially Private Stochastic Gradient Descent Using Selective Update and Release
Jie Fu, Qingqing Ye, Haibo Hu, Zhili Chen, Lulu Wang, Kuncan Wang, Xun Ran
Differential Privacy Stochastic Gradient Descent Private Stochastic Gradient Descent Privacy Preserving Machine Learning Gradient Update Private Model Training Model Update

November 14, 2023

Sparsity-Preserving Differentially Private Training of Large Embedding Models
Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
Full Model Differential Privacy Stochastic Gradient Descent Private Model Training User Privacy Gradient Sparsification

September 12, 2023

Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers
Xilong Wang, Chia-Mu Yu, Pin-Yu Chen
Differential Privacy Parameter Efficient Fine Tuning Complementary Benefit Private Model Training Tabular Transformer TabTransformer Model

June 14, 2023

Augment then Smooth: Reconciling Differential Privacy with Certified Robustness
Jiapeng Wu, Atiyeh Ashari Ghomi, David Glukhov, Jesse C. Cresswell, Franziska Boenisch, Nicolas Papernot
Native Robustness Differential Privacy Private Stochastic Gradient Descent Provable Guarantee Certified Robustness Private Model Training

June 13, 2023

(Amplified) Banded Matrix Factorization: A unified approach to private training
Christopher A. Choquette-Choo, Arun Ganesh, Ryan McKenna, H. Brendan McMahan, Keith Rush, Abhradeep Thakurta, Zheng Xu
Differential Privacy Unified Framework Matrix Factorization Differential Privacy SGD Privacy Amplification Private Model Training

May 25, 2023

Strategic Data Sharing between Competitors
Nikita Tsoy, Nikola Konstantinov
Machine Learning Model Competition Platform Data Sharing Private Model Training

May 15, 2023

Private Training Set Inspection in MLaaS
Mingxue Xu, Tongtong Xu, Po-Yu Chen
Machine Learning Machine Learning Model Fairness Metric Membership Inference Private Model Training

May 4, 2023

Incentivising the federation: gradient-based metrics for data selection and valuation in private decentralised training
Dmitrii Usynin, Daniel Rueckert, Georgios Kaissis
Data Selection Privacy Protection Gradient Information Decentralized Training Private Model Training Private Valuation Loss Input Susceptibility Score

April 6, 2023

When approximate design for fast homomorphic computation provides differential privacy guarantees
Arnaud Grivet Sébert, Martin Zuber, Oana Stan, Renaud Sirdey, Cédric Gouy-Pailler
Differential Privacy Homomorphic Encryption Differential Privacy Guarantee Private Model Training Efficient Homomorphic Cryptographic Primitive

February 28, 2023

Arbitrary Decisions are a Hidden Cost of Differentially Private Training
Bogdan Kulynych, Hsiang Hsu, Carmela Troncoso, Flavio P. Calmon
Differential Privacy Hidden CoST Privacy Preserving Machine Learning Private Model Private Model Training Predictive Multiplicity

Private Model Training

Papers

Enhancing the Utility of Privacy-Preserving Cancer Classification using Synthetic Data

DP-KAN: Differentially Private Kolmogorov-Arnold Networks

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning

Banded Square Root Matrix Factorization for Differentially Private Model Training

Pencil: Private and Extensible Collaborative Learning without the Non-Colluding Assumption

Efficiently Computing Similarities to Private Datasets

Implicit Bias in Noisy-SGD: With Applications to Differentially Private Training

Differentially Private Training of Mixture of Experts Models

On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift

Rethinking Privacy in Machine Learning Pipelines from an Information Flow Control Perspective

DPSUR: Accelerating Differentially Private Stochastic Gradient Descent Using Selective Update and Release

Sparsity-Preserving Differentially Private Training of Large Embedding Models

Exploring the Benefits of Differentially Private Pre-training and Parameter-Efficient Fine-tuning for Table Transformers

Augment then Smooth: Reconciling Differential Privacy with Certified Robustness

(Amplified) Banded Matrix Factorization: A unified approach to private training

Strategic Data Sharing between Competitors

Private Training Set Inspection in MLaaS

Incentivising the federation: gradient-based metrics for data selection and valuation in private decentralised training

When approximate design for fast homomorphic computation provides differential privacy guarantees

Arbitrary Decisions are a Hidden Cost of Differentially Private Training