The role of locality and weight sharing in convolutional neural networks

Francesco Mori; Melanie Weber

cs.LGOct 21, 2026

The role of locality and weight sharing in convolutional neural networks

Authors: Francesco Mori, Melanie Weber

Organizations: Center of Mathematical Sciences and Applications, Harvard University · John A. Paulson School of Engineering and Applied Sciences, Harvard University

Abstract

The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on $P$ input positions with correlation $c$ between positions. We compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension $D$ . For uncorrelated positions ( $c = 0$ ), the two students follow the same learning trajectory, with the convolutional student learning exactly $P$ times faster. We also find that locality affects the sample complexity: a student that matches the kernel size of the teacher recovers it from $\mathcal{O}_{D}(D)$ samples, whereas $\mathcal{O}_{D}(D\log(D))$ samples are needed with a larger kernel. For $c > 0$ , spatial correlations can reduce the advantage of weight sharing. The asymptotic error of the convolutional student grows with $c$ , whereas that of the locally connected student is independent of $c$ and overall lower when evaluated at the same learning rate. When the learning rate is instead tuned to the training horizon, however, the locally connected student wins at early times and the convolutional student at late times, with a transition that we predict analytically.

Figures & tables

Panel of Figure 1 — Figure 1: Teacher and the two student architectures with $P$ input positions. Colors indicate weight sharing. The teacher is a two-layer CNN with kernel size $k_{t}$ . The CNN student has kernel size $k_{s}$ and weight sharing. The LCN has the same receptive fields as the CNN but no weight sharing. Boundary conditions are cyclic. Here, $P=4$ , $k_{s}=3$ , and $k_{t}=2$ .

Panel of Figure 2 — Figure 2: Generalization errors $\varepsilon_{g}^{\rm CNN}(\alpha)$ of the CNN and $\varepsilon_{g}^{\rm LCN}(P\alpha)$ of the LCN. Lines are the order-parameter ODEs and symbols mark simulations of a single SGD trajectory. Insets show the ODE curves without the time rescaling. (a) For independent positions ( $c=0$ ) the trajectories coincide. (b) In the presence of spatial correlations ( $c=0.5$ ) the rescaling fails and the LCN converges faster (see inset). Parameters: $g=\erf$ , $k_{t}=2$ , $k_{s}=3$ , $P=4$ , $\eta=1$ , $\sigma=0$ , $T=1$ , SGD at $D=1000$ .

Panel of Figure 3 — Figure 3: CNN students that match the teacher kernel size ( $k_{s}=k_{t}$ ) learn with $\mathcal{O}_{D}(D)$ samples, while students with $k_{s}>k_{t}$ take $\mathcal{O}_{D}(D\log D)$ samples. (a) Generalization error vs. training time $\alpha$ for three kernel sizes, obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . (b) Specialization time $\alpha_{\rm spec}$ (Eq. ( 25 ), with $\gamma_{\rm th}=0.99$ ) against the feature dimension $D$ . Markers are averages over random initializations, each obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . Standard errors are smaller than the marker size. Lines are fits $a+b\ln D$ . Parameters: $D=10^{12}$ in panel (a), $g=\erf$ , $k_{t}=1$ , $P=6$ , $S=I$ , $\eta=0.5$ , $T=1$ , $\sigma=0$ , 128 random initial conditions in panel (b).

Panel of Figure 4 — Figure 4: Alignment $\gamma_{s}$ of the CNN kernel with the solution $s$ vs. training time $\alpha$ , for (a) $k_{s}=3$ and (b) $k_{s}=4$ . The dotted line marks the specialization time $\alpha_{\rm spec}$ and the dashed line $\gamma_{s}=0$ . Insets: probability $p_{s}$ that solution $s$ is selected, over 4096 random initial conditions (dashed line: uniform). All curves are solutions of the ODEs ( 17 ). Parameters: $D=10^{12}$ , $g=\erf$ , $k_{t}=1$ , $P=6$ , $S=I$ , $\eta=0.5$ , $T=1$ , $\sigma=0$ .

Panel of Figure 5 — Figure 5: Asymptotic noise floor $\varepsilon^{\infty}/(\eta\sigma^{2})$ of the CNN and the LCN at kernel size $k_{s}=k_{t}=2$ , as a function of the data correlation $c$ and the number of positions $P$ , where $\varepsilon^{\infty}$ is the asymptotic generalization error under label noise. Lines are the closed forms ( 31 ) and ( 32 ) and markers are the noise floors located numerically from the order-parameter ODEs at finite learning rate. (a) Floor vs. $c$ at $P=6$ . The CNN grows steeply with $c$ while the LCN stays flat. (b) Scaling with $P$ at $c=0$ . The CNN and the LCN coincide and are both linear in $P$ . (c) Scaling with $P$ at $c=0.5$ . The LCN is still linear in $P$ while the CNN grows quadratically. Parameters: $g=\erf$ , $k_{s}=k_{t}=2$ , $\eta=2\times 10^{-3}$ , $\sigma=1$ , $T=1$ .

Figure 6: Comparison of the CNN and the LCN at training horizon $\alpha_{F}$ and correlation $c$ , each at its optimal constant learning rate. The color shows the log-ratio of the errors $\varepsilon^{*}$ . Solid line: boundary $\varepsilon^{*}_{\rm CNN}=\varepsilon^{*}_{\rm LCN}$ , obtained by integrating the ODEs numerically. Dashed line: analytical prediction Eq. ( 30 ) of the early-time expansion. Parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Specialization time of the CNN against the feature dimension, as in Figure 3(b) . Color denotes the student kernel size. Markers are averages over random initializations, each obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . Standard errors are smaller than the marker size. Lines are fits $a+b\ln D$ . (a) A teacher of kernel size $k_{t}=2$ . The matched case $k_{s}=k_{t}=2$ is flat in $D$ and the two wider kernels grow logarithmically. (b) Correlated positions, at $k_{t}=1$ , with $c=0$ faded and $c=0.5$ at full strength. The plateau at $k_{s}>k_{t}$ survives the correlation. Parameters: $g=\erf$ , $P=6$ , $\eta=0.5$ in panel (a) and $\eta=0.2$ in panel (b), $T=1$ , $\sigma=0$ , $c=0$ in panel (a), $D$ from $10^{6}$ to $10^{10}$ , 128 random initial conditions.

Panel of Figure 8 — Figure 8: (a) Optimal learning rate $\eta^{*}$ and (b) corresponding generalization error $\varepsilon^{*}$ (Section 4.3 ) against the training horizon $\alpha_{F}$ , for the CNN and the LCN at three values of the correlation $c$ , from the same ODE solutions as Figure 6 . Parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Panel of Figure 9 — Figure 9: Phase diagram of Figure 6 for other parameter values. The color shows $\log_{10}(\varepsilon^{*}_{\rm CNN}/\varepsilon^{*}_{\rm LCN})$ on a common scale. Solid line: boundary $\varepsilon^{*}_{\rm CNN}=\varepsilon^{*}_{\rm LCN}$ , obtained by integrating the ODEs numerically. Dashed line: analytical prediction $c=c_{0}\alpha_{F}$ of the early-time expansion, Eq. ( 30 ). Each panel changes one parameter of Figure 6 : (a) $P=8$ , (b) $k_{s}=k_{t}=2$ , (c) $T=2$ , (d) $\sigma=2$ . Other parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Figure 10: Rescaled noise floor

\varepsilon^{\infty}/(\eta\sigma^{2})

of the two students with the ReLU activation, where

\varepsilon^{\infty}

is the asymptotic generalization error under label noise. It is measured by online SGD from the specialized initial condition and averaged over seeds, with the generalization error computed exactly from the student weights. Standard errors are smaller than the marker size. As for

\erf

(Figure 5 ), the CNN’s floor grows with the data correlation

c

while the LCN’s stays flat. Parameters:

g(z)=\max(0,z)

k_{s}=k_{t}=2

P=4

D=400

\eta=0.1

T=1

\sigma=1

, runs up to

\alpha=800

8

seeds.

Panel of Figure 11 — Figure 11: Rescaling test of Figure 2 with the ReLU activation. Generalization errors $\varepsilon_{g}^{\rm CNN}(\alpha)$ of the CNN and $\varepsilon_{g}^{\rm LCN}(P\alpha)$ of the LCN, from online SGD averaged over 16 runs per student. Standard errors are smaller than the marker size. Insets show the curves without the time rescaling. (a) $c=0$ . (b) $c=0.5$ . Parameters: $g(z)=\max(0,z)$ , $k_{t}=2$ , $k_{s}=3$ , $P=4$ , $\eta=0.2$ , $\sigma=0$ , $T=1$ , $D=1000$ .

Convolutional Neural Networks

Convolutional neural networks (CNNs) learn spatial and temporal patterns using shared filters, with applications from image recognition to scientific prediction.

See all 410 papers

Oct 1, 2026

Removing spurious minima for planar features by skip connections

Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1

Freie Universität Berlin · Technische Universität Berlin

Shallow Neural Networks ReLU Neural Networks Neural Network Optimization Nonconvex Optimization Teacher-Student Learning

Ask PDF

Sep 9, 2026

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.

Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2

Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen

Neural Network Optimization Neural Network Training Dynamics Teacher-Student Learning Loss Landscape Geometry

Ask PDF

Oct 14, 2026

Resolution Allocation with Learnable Resizing and Spectral Filter Parameterization

Learnable downsampling and spectral-domain filter parameterizations have each made convolutional networks more flexible, but their interaction has not been systematically examined. We bridge this gap by examining convolutional kernels learned in the spatial and spectral domains, together with learned downsampling ratios under a cropped resizing implementation. We find that spectral and spatial kernels respond differently to learned resolution allocation, with larger gains for the spectral kernel. We observe this consistently on three datasets (EuroSAT, CIFAR-10 and STL-10) and at two model sizes (differences in accuracy gain of +0.013 to +0.028 on held-out test sets). Part of this difference reflects the greater gain of the spectral kernel from retaining more spatial area, but the difference remains positive when total retained area is matched. This pattern is observed across datasets and model sizes, although its magnitude and dependence on the complexity budget vary with model size and the resizing implementation.

Wei Wu, Eric C. Larson

Department of Computer Science, Southern Methodist University, Dallas, TX, USA

Convolutional Neural Networks Frequency-Domain Feature Learning Image Classification

Ask PDF

Abstract

Convolutional Neural Networks

Explore similar work

Removing spurious minima for planar features by skip connections

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Resolution Allocation with Learnable Resizing and Spectral Filter Parameterization