We investigate the convergence guarantee of two-layer neural network training with Gaussian randomly masked inputs. This scenario corresponds to Gaussian dropout at the input level, or noisy input training common in sensor networks, privacy-preserving training, and federated learning, where each user may have access to partial or corrupted features. Using a Neural Tangent Kernel (NTK) analysis, we demonstrate that training a two-layer ReLU network with Gaussian randomly masked inputs achieves linear convergence up to an error region proportional to the mask's variance. A key technical contribution is resolving the randomness within the non-linear activation, a problem of independent interest.
Figures & tables
Figure 1: (a). Effect of the noise standard deviation κ on the shape of the smoothed activation function σ^(z;κ)=z⋅Φ1(z/(κ∥w⊙x∥2)) , where z=w⊤x . For this visualization, ∥w⊙x∥2 is held constant at 1.0 . As κ increases, the activation becomes progressively smoother compared to the standard ReLU (dotted black line). For small κ (e.g., κ=0.01 ), σ^ closely approximates the standard ReLU. (b). Theoretical smoothed activation σ^(w,x) versus its empirical estimate Ec[σ(w⊤(x⊙c))] for a fixed pre-activation value w⊤x≈0.77 (actual value depends on fixed w,x ) as the noise standard deviation κ varies. The close match across a range of—relatively small— κ values validates the theoretical model for σ^ . Note that this behavior consistently follows empirically for different w,x values.
Figure 2: Smoothed ReLU under multiplicative Gaussian input masking for fixed κ=0.2 , where z=w⊤x and σ=κ∥w⊙x∥2 . (a) Exact closed-form expectation σ~(w,x)=Ec[σ(w⊤(x⊙c))]=zΦ(z/σ)+σφ(z/σ) (as shown in 19 )) matches the Monte Carlo estimate. (b) Proxy smoothed activation σ^(w,x)=zΦ(z/σ) (used in Theorem 4.2 ) differs mainly near z≈0 due to the missing σφ(z/σ) term.
Figure 3: (a). Training loss L(Wk) (log-scale) versus training iteration for a two-layer ReLU network ( n=500,d=20,m=100 ) trained with full-batch gradient descent under different levels of input multiplicative Gaussian noise standard deviation κ . (b). Distributed training with Gaussian mask for differen κ and number of local steps.
Figure 4: Test accuracy versus multiplicative Gaussian noise strength κ for (a) a 1-hidden-layer MLP and (b) a CNN, trained on CIFAR-10. Small noise levels ( κ≈0.2) can improve generalization for the MLP, likely due to regularization effects. In contrast, the CNN exhibits robustness by maintaining baseline accuracy. Beyond this point, accuracy degrades monotonically for both architectures as noise corrupts the training signal.
Figure 5: Attack AUC on the target model when it is (a) an MLP and (b) a CNN. Higher values indicate greater privacy leakage. Training with MG noise (larger κ ) consistently reduces attack success.
Figure 6: MLP target. Multiplicative Gaussian noise provides resilience against the attacks as κ increases.
Figure 7: CNN target. We observe a similar trend as the MLP
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: ℓ2 -norms of the gradient components (left) and residual error bound check (right).
Figure 9: Privacy-utility tradeoff for the Standard CNN. The left panel shows Attack Precision vs. Accuracy, and the right panel shows Attack Recall vs. Accuracy.
Figure 10: Privacy-utility tradeoff for the Improved CNN. Unlike the Standard CNN, the MG noise curve stays much closer to the DP-SGD curve across the accuracy range
Department of Electrical and Computer Engineering University of Southern California Los Angeles, CA, USA · Department of Computer Science University of Southern California Los Angeles, CA, USA
RPTU Kaiserslautern-Landau, Kaiserslautern, Germany · Machine Learning Research, Morgan Stanley · Institute of Science and Technology, Klosterneuburg, Austria +1