Gradient Inversion through Linear and Affine Subspace Constraints
Organizations: School of Electrical and Computer Engineering Technical University of Crete
Abstract
Federated learning enables multiple clients to collaboratively train a machine-learning model without directly sharing their local datasets. However, as prior works have shown, the gradients exchanged during training may reveal sensitive information about client data. Most existing attacks usually optimize dummy inputs to match the true client gradients (Gradient Inversion Attacks), while some attempt to reconstruct them based on the observation that aggregate gradients from a fully connected (FC) layer are a linear combination of its inputs. In this work, we extend this observation by showing that aggregate gradients from an FC layer with the correct scaling are actually an affine combination of its inputs. We evaluate a gradient inversion attack that leverages either known linear-subspace information or its affine-subspace counterpart at each FC layer to improve reconstruction accuracy. Our results show that subspace-based methods can substantially improve reconstruction quality over the chosen baselines in certain settings. We further demonstrate that moderate overestimation of the batch size can generally improve or preserve reconstruction quality, even when the true batch size is unknown. We additionally examine reconstruction under gradient perturbations introduced by a differential privacy (DP) mechanism, providing insight into the effectiveness of the considered formulations under noisy conditions.
Figures & tables
| ReLU | LeakyReLU | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Attack | 16 | 32 | 64 | 128 | 256 | 16 | 32 | 64 | 128 | 256 |
| IG | ||||||||||
| CPA | 0.577 | 0.580 | ||||||||
| Soft Linear | 0.077 | |||||||||
| Hard Linear | 0.057 | |||||||||
| Soft Affine | 0.206 | 0.474 | 0.087 | 0.139 | 0.481 | |||||
| ReLU | LeakyReLU | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Attack | 16 | 32 | 64 | 128 | 256 | 16 | 32 | 64 | 128 | 256 |
| IG | ||||||||||
| CPA | 0.182 | 0.230 | 0.275 | 0.387 | ||||||
| Soft Linear | 0.083 | 0.030 | 0.037 | |||||||
| Hard Linear | ||||||||||
| Soft Affine | 0.067 | 0.092 | 0.331 | |||||||
| Attack | Unknown | Known | Unknown | Known | Unknown | Known | Unknown | Known | Unknown | Known |
|---|---|---|---|---|---|---|---|---|---|---|
| CPA | 0.921 | 0.922 | 0.933 | 0.919 | ||||||
| Soft Linear | ||||||||||
| Hard Linear | 0.992 | 0.991 | 0.982 | 0.957 | ||||||
| Soft Affine | 1.000 | |||||||||
| Hard Affine | 0.964 | 1.000 | ||||||||
| True | True | True | True | |||||
| Attack | ||||||||
| IG | ||||||||
| CPA | ||||||||
| Soft Linear | ||||||||
| Hard Linear | ||||||||
| Soft Affine | 0.034 | 0.043 | 0.261 | 0.200 | 0.452 | 0.483 | ||
| Attack | |||||
|---|---|---|---|---|---|
| CPA | 0.950 | 0.952 | 0.918 | ||
| Soft Linear | |||||
| Hard Linear | 0.999 | 0.985 | |||
| Soft Affine | |||||
| Hard Affine |
| FC-3 | VGG-16 | ||||
| Target SNR | Measured SNR | ( ) | Target SNR | Measured SNR | ( ) |
| (dB) | (dB) | (dB) | (dB) | ||
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Attack | ||||||
| Noiseless | CPA | 3.1 | – | – | 1.47 | 12.4 | |
| IG | 0.05 | – | – | – | – | ||
| Soft Subspace | 0.05 | – | – | ||||
| Hard Subspace | 0.05 | – | – | – | |||
| Noisy | Soft Subspace | 0.05 | – | – | |||
| Hard Subspace | 0.05 | – | – | – |
| Setting | Attack | ||||||||
| Accurate | CPA | 0.1 | – | – | 0.13 | 5 | 5.3 | 7.7 | |
| Soft Affine | 0.06 | 0.1 | – | 2 | – | ||||
| Soft Linear | 0.06 | 0.1 | – | 0.1 | – | ||||
| Hard Subspace | 0.06 | 0.1 | – | 0.1 | 2 | – | |||
| Overest. | Soft Affine | 0.06 | 0.1 | – | 0.1 | – | |||
| Noisy | Soft Subspace | 0.06 | 0.1 | 0.1 | – |
Gradient Inversion Attacks
Gradient inversion attacks infer private training examples or labels from gradients shared during distributed model training.
See all 23 papers