Representation learning from complex data typically involves models with a large number of parameters, which in turn require large amounts of data samples. In neural network models, model complexity grows with the number of inputs to each neuron, with a trade-off between model expressivity and learning time. A precise characterization of this trade-off would help explain the connectivity and learning times observed in artificial and biological networks. We present a theoretical analysis of how learning time depends on input dimensionality for a Hebbian learning model performing independent component analysis. Based on the geometry of high-dimensional spaces, we show that the learning dynamics reduce to a unidimensional problem, with learning times dependent only on initial conditions. For higher input dimensions, initial parameters have smaller learning gradients and larger learning times. We find that learning times have supralinear scaling, becoming quickly prohibitive for high input dimensions. These results reveal a fundamental limitation for learning in high dimensions and help elucidate how the optimal design of neural networks depends on data complexity. Our approach outlines a new framework for analyzing learning dynamics and model complexity in neural network models.
Figures & tables
Figure 1: Geometry of the optimization surface for synaptic weights. ( a ) Prototypical minimum in a convex surface with a basin of attraction. (left) Gray scale indicates optimization function value, with minimum in white. (right) Color heat map indicates the magnitude of the gradient, with zero gradients in dark blue. ( b ) Prototypical maximum in a concave surface, with an unstable equilibrium point. ( c ) Saddle points are concave in some parameter directions, while convex in others. ( d ) Surface for the gradient magnitude for three weights. Each dimension represents the value of a synaptic weight. As we enforce ∣w∣2=1 , the parameter space is constrained to the unit sphere. The cardinal directions are the minima, the directions of the hidden patterns. The symmetric directions, where all weights have the same magnitude, are maxima. Partially symmetric directions, where two weights have the same magnitude, and the third is zero, are saddle points. Areas in blue indicate low gradient magnitude, and thus slow learning dynamics.
Figure 2: Initial parameters have small overlap with hidden features in high dimensions. ( a ) The expected overlap of initial parameters with a hidden feature decays with N (fixed K=10 ). Dashed line is power-law with exponent α=−0.5 . ( b ) Increasing the number of hidden features has only a logarithmic effect on the expected initial overlap (fixed N=1000 ).
Figure 3: Stereotypical learning dynamics in high-dimensional optimization. ( a ) Evolution of the overlap of the weights with a hidden feature, for different dimensionality N . The color heat indicates N , varying from N=10 (purple) to N=160 (red). One run for each N , with random initial weights. ( b ) Same trajectories shifted to a referent time where an overlap of d=0.75 was reached, highlighting the similarity in the learning dynamics.
Figure 4: Gradient dependency on the overlap d for symmetric sparse distributions. ( a ) The gradient magnitude μ(d) vanishes when the overlap goes to zero. ( b ) The gradient variability σ(d) does not change significantly for small overlaps. ( c ) The gradient signal-to-noise ratio μ(d)2/σ(d)2 follows the profile of the gradient magnitude. ( d ) For small overlaps, d→0 , the gradient has a power-law dependency on the overlap (dashed line is power-law with exponent α=3. ).
Figure 5: Learning time dependence on the number of inputs. Learning time for input dimensions N , and K=N latent features (in blue) match the predicted scaling by the theory, T∝log(N)N2 . Learning time was defined as the passing time of the overlap at d=0.7 , averaged over 100 simulations.
The success of deep learning in high-dimensional settings is often attributed to the presence of low-dimensional structure in real-world data. While standard theoretical models typically assume that this structure lies in the target function, projecting unstructured inputs onto a low-dimensional subspace, data such as images, text or genomic sequences exhibit strong spatial correlations within the input space itself. In this paper, we propose a tractable model to study how these correlations affect the sample complexity of learning with gradient descent on shallow neural networks. Specifically, we consider targets that depend on a small number of latent Boolean variables, and input features grouped into clusters and correlated with the latent variables. Under an identifiability assumption, we show that for a layerwise gradient-descent variant, the sample complexity scales with the number of hidden variables and, when the signal-to-noise ratio is sufficiently high, is independent of the input dimension, up to logarithmic terms. We empirically test our theoretical findings on both synthetic and real data.
Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging. In this work, we treat such training trajectories as temporal networks and apply recently proposed strategies for the scalar embedding of temporal networks. We investigate whether such a scalar embedding provides a meaningful low-dimensional representation of neural network training dynamics. Using a multilayer perceptron trained on the MNIST classification task, we show that the embedding preserves the main dynamical features observed in the original parameter space, including the emergence of sensitivity to initial conditions for specific learning rate regimes and an accurate reconstruction of the network's maximum Lyapunov exponent. We then use the embedded scalar trajectory to define a characteristic time, analogous to a Lyapunov time, after which the exponential separation between initially close embedded trajectories saturates. This characteristic time captures the typical decorrelation time between initially close network trajectories in the original high-dimensional system. Finally, we investigate the statistical organization of asymptotic training states through a spacing observable defined in the embedded space. We find that the distributions of rescaled asymptotic spacings collapse onto a common form across initial conditions and are compatible with a skew lognormal distribution. Altogether, our results suggest that scalar low-dimensional embeddings provide a useful framework for studying and visualizing the dynamical properties of neural network optimization trajectories.
Pedro Jiménez-González, Miguel C. Soriano, Lucas Lacasa
Institute for Cross-Disciplinary Physics and Complex Systems (IFISC, CSIC-UIB), Campus UIB, 07122 Palma de Mallorca, Spain
In this short note we consider the gradient descent dynamics of deep scalar linear networks, f(x)=∏l=1Lwlx, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.
Yedi Zhang, Peter E. Latham, Leena Chennuru Vankadara +1
Gatsby Computational Neuroscience Unit, University College London · Sainsbury Wellcome Centre, University College London