Private Gradient

Private gradient methods aim to train machine learning models while preserving the privacy of individual data points, primarily by adding noise to gradients during optimization. Current research focuses on improving the accuracy of differentially private stochastic gradient descent (DPSGD) and related algorithms, exploring techniques like Kalman filtering for noise reduction, gradient shuffling, and novel optimization approaches for specific model architectures (e.g., transformers, ReLU networks). These advancements are crucial for enabling the responsible use of machine learning with sensitive data in various applications, addressing the critical trade-off between privacy guarantees and model utility.

Papers

May 24, 2023

Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models
Haonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska Boenisch
Large Language Model Membership Inference Attack Private Algorithm Private Gradient Feather Flock Non Private

March 31, 2023

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited
Jinyan Su, Changhong Zhao, Di Wang
Convex Function Non Euclidean Private Gradient Convex Loss Function \Ell_p$ Norm

February 24, 2023

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning
Edwige Cyffers, Aurélien Bellet, Debabrota Basu
Industrial Disturbing Noise Differential Privacy SGD Private Gradient Private Optimization Centralized Control Iteration Head Fixed Point Iteration

February 22, 2023

Multi-Message Shuffled Privacy in Federated Learning
Antonious M. Girgis, Suhas Diggavi
Differential Privacy Secure Aggregation Private Gradient Privacy Accuracy Private Mean Estimation Privacy Mechanism

February 8, 2023

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning
Tomoya Murata, Taiji Suzuki
LeArning Abstract Differential Privacy Private Stochastic General Nonconvex Private Gradient Gradient Difference

January 1, 2023

ReSQueing Parallel and Private Stochastic Convex Optimization
Yair Carmon, Arun Jambulapati, Yujia Jin, Yin Tat Lee, Daogao Liu, Aaron Sidford, Kevin Tian
Stochastic Gradient Descent Stochastic Convex Optimization Private Gradient Low Variance Gradient

November 14, 2022

SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealing
Jie Fu, Zhili Chen, XinPeng Ling
Differential Privacy Gradient Descent Private Stochastic Private Stochastic Gradient Descent Privacy Preserving Image Simulated Annealing Private Gradient

October 31, 2022

Private optimization in the interpolation regime: faster rates and hardness results
Hilal Asi, Karan Chadha, Gary Cheng, John Duchi
Sample Complexity Stochastic Gradient Interpolation Regime Hardness Result Fast Rate Private Gradient Private Optimization

October 12, 2022

Differentially Private Online-to-Batch for Smooth Losses
Qinzi Zhang, Hoang Tran, Ashok Cutkosky
Online Convex Optimization Private Gradient Optimal Convergence Rate Unknown Variance

June 1, 2022

Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Liang Zhang, Kiran Koshy Thekumparampil, Sewoong Oh, Niao He
Privacy Utility Private Gradient Optimal Guarantee Private Minimax

January 10, 2022

Differentially Private $\ell_1$-norm Linear Regression with Heavy-tailed Data
Di Wang, Jinhui Xu
Heavy Tailed Private Gradient DP Algorithm \Ell_1$ Norm

December 1, 2021

Public Data-Assisted Mirror Descent for Private Model Training
Ehsan Amid, Arun Ganesh, Rajiv Mathews, Swaroop Ramaswamy, Shuang Song, Thomas Steinke, Vinith M. Suriyakumar, Om Thakkar, Abhradeep Thakurta
Mirror Descent Differential Privacy SGD Private Gradient Private Model Training Online Mirror Descent

February 9, 2021

Output Perturbation for Differentially Private Convex Optimization: Faster and More General
Andrew Lowy, Meisam Razaviyayn
Private Algorithm Private Gradient Convex Loss Function Private Convex Optimization Private Empirical Risk Minimization

Private Gradient

Papers

Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models

Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Multi-Message Shuffled Privacy in Federated Learning

DIFF2: Differential Private Optimization via Gradient Differences for Nonconvex Distributed Learning

ReSQueing Parallel and Private Stochastic Convex Optimization

SA-DPSGD: Differentially Private Stochastic Gradient Descent based on Simulated Annealing

Private optimization in the interpolation regime: faster rates and hardness results

Differentially Private Online-to-Batch for Smooth Losses

Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization

Differentially Private $\ell_1$-norm Linear Regression with Heavy-tailed Data

Public Data-Assisted Mirror Descent for Private Model Training

Output Perturbation for Differentially Private Convex Optimization: Faster and More General