Organizations: Department of Computer Science, The University of Manchester, UK · Department of Mathematics, Indian Institute of Technology, Bombay, India
In this note we prove that the Langevin Monte Carlo (LMC) algorithm can minimize the population risk of appropriately regularized depth-2 nets with no restrictions on the size of the net and for any choice of training data --- and we give non-asymptotic convergence rates for it. We achieve this via showing that in q-Renyi divergence, the iterates of LMC converge to the Gibbs distribution of certain Frobenius norm regularized losses for these nets --- in both classification and regression settings. The key technicality of the work is two-fold (a) to keep the amount of regularization needed be independent of the size of the net and (b) to obtain convergence rates that scale with just the Poincare constant of the corresponding Gibbs measure. This work synthesizes several recent observations about isoperimetry conditions under which LMC converges and that two-layer neural loss functions can always be regularized by a constant amount such that they satisfy the Villani conditions, and thus their Gibbs measures satisfy a Poincare inequality.
Figures & tables
Figure 1: In this figure, we show the train and test error convergence plots for depth-2 neural networks at varying widths when trained via LMC for a regression task.
Figure 2: This figure demonstrates that at widths 16 and 1024 , the introduction of noise to the training and testing data substantially influences the final loss, indicating that the regularization parameter λ does not dominate the regularized MSE loss being studied.
Figure 3: This figure presents the training and testing error plots for depth-2 neural networks with varying widths, where the initial weights are sampled from a normal distribution of variance width1 .
Width
s
s2h
Training Loss
Test Loss
16
2e-4
5e-3
1.50±3.36e−04
1.64±7.24e−04
32
2e-4
1e-3
1.51±3.16e−04
1.64±4.51e−04
64
2e-4
5e-3
1.51±5.70e−04
1.64±1.28e−04
128
2e-4
5e-3
1.51±3.74e−04
1.65±9.90e−04
256
2e-4
1e-2
1.52±8.80e−04
1.65±8.86e−04
512
2e-4
5e-2
1.53±1.46e−03
1.67±1.76e−03
Table 1: Test Loss and Training loss, over 5 random seeds, for different values of width, s and best learning rate (s2h) for Figure 3 .
Figure 4: This figure presents the training and testing error plots for training depth-2 neural networks with varying widths, using AdamW optimizer for regression on the same data as above.
Figure 5: This figure presents the training and testing error plots for depth-2 neural networks with varying widths, using a value of λ below the Villani threshold.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: This figure presents the training and testing error plots for depth-2 neural networks with varying widths, where the initial weights are sampled from a normal distribution of variance width1 .
Width
s
s2h
Training Loss
Test Accuracy (%)
32
2e-3
5e-3
13.29±0.103
73.25±1.058
64
2e-3
5e-3
25.85±0.178
72.16±2.568
128
2e-3
5e-3
51.22±0.184
73.48±1.412
Appendix
Table 2: Test accuracy (%) and Training loss, over 5 random seeds, for different values of width, s and best learning rate (s2h) for Figure 6 .
Figure 7: This figure demonstrates that at widths 32 and 128 , the introduction of noise to the training and testing data substantially influences the final test accuracy, indicating that the regularization parameter λ does not dominate the regularized BCE loss being studied.
Figure 8: This figure presents the training and testing error plots for training depth-2 neural networks with varying widths, using AdamW optimizer for classification on the same data as above.
Figure 9: This figure presents the training and testing error plots for depth-2 neural networks with varying widths, using a value of λ below the Villani threshold.
Notation
Meaning
d
Input dimension.
p
Width (number of hidden units) of the depth-2 neural network.
n
Number of training samples.
x∈Rd
Input feature vector.
y
Output label or regression target.
X
Data domain.
Appendix
Table 3: Summary of notation used throughout the paper.
Existing convergence analyses of finite-width overparameterized deep neural networks under standard parameterization often rely on parameters remaining close to their initialization and typically neglect the role of noise, despite empirical evidence that stochasticity can improve optimization and generalization. To bridge this gap, we study L2-regularized continuous-time Langevin dynamics (CLD) with isotropic noise, for training feedforward networks with smooth activations. We introduce a change-of-measure inequality, inspired by the second law of thermodynamics, that enables control of key properties of the training dynamics under the time-evolving parameter distribution. Leveraging a sharpened lower bound on the minimum eigenvalue of the Neural Tangent Kernel (NTK) at initialization, we show that injected noise helps preserve NTK stability with high probability without confining the weights to a small neighborhood of their initialization. As a result, we derive a non-asymptotic upper bound on the expected training loss. Our analysis yields explicit conditions on width and noise temperature that guarantee convergence to a finite loss floor. Finally, we experimentally show the network Jacobian can change from initialization, departing the commonly analyzed lazy regime.
Matan Tsipory, Ofir Gaash, Itamar Harel +2
Technion - Israel Institute of Technology, The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion City, Haifa 3200003, Israel. · Technion - Israel Institute of Technology, Faculty of Data and Decision Sciences, Technion City, Haifa 3200003, Israel. · Toyota Technological Institute at Chicago, Chicago, IL, USA.
Sampling the parameter space of artificial neural networks according to a Boltzmann distribution provides insight into the geometry of low-loss solutions and offers an alternative to conventional loss minimization for training. However, exact sampling methods such as hybrid Monte Carlo (hMC), while formally correct, become computationally prohibitive for realistic datasets because they require repeated evaluation of full-batch gradients. We introduce a pseudo-Langevin (pL) dynamics that enables efficient Boltzmann sampling of feed-forward neural networks trained with large datasets by using minibatches in a controlled manner. The method exploits the statistical properties of minibatch gradient noise and adjusts fictitious masses and friction coefficients to ensure that the induced stochastic process samples efficiently the desired equilibrium distribution. We validate numerically the approach by comparing its equilibrium statistics with those obtained from exact hMC sampling. Performance benchmarks demonstrate that, while hMC rapidly becomes inefficient as network size increases, the pL scheme maintains high computational diffusion and scales favorably to networks with over one million parameters. Finally, we show that sampling at intermediate temperatures yields optimal generalization performance, comparable to SGD, without requiring a validation set or early stopping procedure. These results establish controlled minibatch Langevin dynamics as a practical and scalable tool for exploring and exploiting the solution space of large neural networks.
Department of Physics, Università degli Studi di Milano and INFN, via Celoria 16, 20133 Milano, Italy · Department of Computing Sciences and Bocconi Institute for Data Science and Analytics (BIDSA), Bocconi University, 20136 Milano, Italy
This survey article is concerned with the application of lattice rules to Deep Neural Networks (DNNs), lattice rules being a family of quasi-Monte Carlo methods. They have demonstrated effectiveness in various contexts for high-dimensional integration and function approximation. They are extremely easy to implement thanks to their very simple formulation -- all that is required is a good integer generating vector of length matching the dimensionality of the problem. In recent years there has been a burst of research activities on the application and theory of DNNs. We review our recent article on using lattice rules as training points for DNNs with a smooth activation function, where we obtained explicit regularity bounds of the DNNs. By imposing restrictions on the network parameters to match the regularity features of the target function, we prove that DNNs with tailored lattice training points can achieve good theoretical generalization error bounds, with implied constants independent of the input dimension. We also demonstrate numerically that DNNs trained with our tailored regularization perform significantly better than with standard ℓ2 regularization.