The role of locality and weight sharing in convolutional neural networks

Francesco Mori; Melanie Weber

cs.LGOct 21, 2026

The role of locality and weight sharing in convolutional neural networks

Authors: Francesco Mori, Melanie Weber

Organizations: Center of Mathematical Sciences and Applications, Harvard University · John A. Paulson School of Engineering and Applied Sciences, Harvard University

Abstract

The success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on $P$ input positions with correlation $c$ between positions. We compare convolutional and locally connected students trained by online stochastic gradient descent, and derive ordinary differential equations that describe their dynamics exactly in the limit of large feature dimension $D$ . For uncorrelated positions ( $c = 0$ ), the two students follow the same learning trajectory, with the convolutional student learning exactly $P$ times faster. We also find that locality affects the sample complexity: a student that matches the kernel size of the teacher recovers it from $\mathcal{O}_{D}(D)$ samples, whereas $\mathcal{O}_{D}(D\log(D))$ samples are needed with a larger kernel. For $c > 0$ , spatial correlations can reduce the advantage of weight sharing. The asymptotic error of the convolutional student grows with $c$ , whereas that of the locally connected student is independent of $c$ and overall lower when evaluated at the same learning rate. When the learning rate is instead tuned to the training horizon, however, the locally connected student wins at early times and the convolutional student at late times, with a transition that we predict analytically.

Figures & tables

Panel of Figure 1 — Figure 1: Teacher and the two student architectures with $P$ input positions. Colors indicate weight sharing. The teacher is a two-layer CNN with kernel size $k_{t}$ . The CNN student has kernel size $k_{s}$ and weight sharing. The LCN has the same receptive fields as the CNN but no weight sharing. Boundary conditions are cyclic. Here, $P=4$ , $k_{s}=3$ , and $k_{t}=2$ .

Panel of Figure 2 — Figure 2: Generalization errors $\varepsilon_{g}^{\rm CNN}(\alpha)$ of the CNN and $\varepsilon_{g}^{\rm LCN}(P\alpha)$ of the LCN. Lines are the order-parameter ODEs and symbols mark simulations of a single SGD trajectory. Insets show the ODE curves without the time rescaling. (a) For independent positions ( $c=0$ ) the trajectories coincide. (b) In the presence of spatial correlations ( $c=0.5$ ) the rescaling fails and the LCN converges faster (see inset). Parameters: $g=\erf$ , $k_{t}=2$ , $k_{s}=3$ , $P=4$ , $\eta=1$ , $\sigma=0$ , $T=1$ , SGD at $D=1000$ .

Panel of Figure 3 — Figure 3: CNN students that match the teacher kernel size ( $k_{s}=k_{t}$ ) learn with $\mathcal{O}_{D}(D)$ samples, while students with $k_{s}>k_{t}$ take $\mathcal{O}_{D}(D\log D)$ samples. (a) Generalization error vs. training time $\alpha$ for three kernel sizes, obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . (b) Specialization time $\alpha_{\rm spec}$ (Eq. ( 25 ), with $\gamma_{\rm th}=0.99$ ) against the feature dimension $D$ . Markers are averages over random initializations, each obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . Standard errors are smaller than the marker size. Lines are fits $a+b\ln D$ . Parameters: $D=10^{12}$ in panel (a), $g=\erf$ , $k_{t}=1$ , $P=6$ , $S=I$ , $\eta=0.5$ , $T=1$ , $\sigma=0$ , 128 random initial conditions in panel (b).

Panel of Figure 4 — Figure 4: Alignment $\gamma_{s}$ of the CNN kernel with the solution $s$ vs. training time $\alpha$ , for (a) $k_{s}=3$ and (b) $k_{s}=4$ . The dotted line marks the specialization time $\alpha_{\rm spec}$ and the dashed line $\gamma_{s}=0$ . Insets: probability $p_{s}$ that solution $s$ is selected, over 4096 random initial conditions (dashed line: uniform). All curves are solutions of the ODEs ( 17 ). Parameters: $D=10^{12}$ , $g=\erf$ , $k_{t}=1$ , $P=6$ , $S=I$ , $\eta=0.5$ , $T=1$ , $\sigma=0$ .

Panel of Figure 5 — Figure 5: Asymptotic noise floor $\varepsilon^{\infty}/(\eta\sigma^{2})$ of the CNN and the LCN at kernel size $k_{s}=k_{t}=2$ , as a function of the data correlation $c$ and the number of positions $P$ , where $\varepsilon^{\infty}$ is the asymptotic generalization error under label noise. Lines are the closed forms ( 31 ) and ( 32 ) and markers are the noise floors located numerically from the order-parameter ODEs at finite learning rate. (a) Floor vs. $c$ at $P=6$ . The CNN grows steeply with $c$ while the LCN stays flat. (b) Scaling with $P$ at $c=0$ . The CNN and the LCN coincide and are both linear in $P$ . (c) Scaling with $P$ at $c=0.5$ . The LCN is still linear in $P$ while the CNN grows quadratically. Parameters: $g=\erf$ , $k_{s}=k_{t}=2$ , $\eta=2\times 10^{-3}$ , $\sigma=1$ , $T=1$ .

Figure 6: Comparison of the CNN and the LCN at training horizon $\alpha_{F}$ and correlation $c$ , each at its optimal constant learning rate. The color shows the log-ratio of the errors $\varepsilon^{*}$ . Solid line: boundary $\varepsilon^{*}_{\rm CNN}=\varepsilon^{*}_{\rm LCN}$ , obtained by integrating the ODEs numerically. Dashed line: analytical prediction Eq. ( 30 ) of the early-time expansion. Parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Specialization time of the CNN against the feature dimension, as in Figure 3(b) . Color denotes the student kernel size. Markers are averages over random initializations, each obtained by integrating the ODEs ( 17 ) from the order parameters of a random initialization at dimension $D$ . Standard errors are smaller than the marker size. Lines are fits $a+b\ln D$ . (a) A teacher of kernel size $k_{t}=2$ . The matched case $k_{s}=k_{t}=2$ is flat in $D$ and the two wider kernels grow logarithmically. (b) Correlated positions, at $k_{t}=1$ , with $c=0$ faded and $c=0.5$ at full strength. The plateau at $k_{s}>k_{t}$ survives the correlation. Parameters: $g=\erf$ , $P=6$ , $\eta=0.5$ in panel (a) and $\eta=0.2$ in panel (b), $T=1$ , $\sigma=0$ , $c=0$ in panel (a), $D$ from $10^{6}$ to $10^{10}$ , 128 random initial conditions.

Panel of Figure 8 — Figure 8: (a) Optimal learning rate $\eta^{*}$ and (b) corresponding generalization error $\varepsilon^{*}$ (Section 4.3 ) against the training horizon $\alpha_{F}$ , for the CNN and the LCN at three values of the correlation $c$ , from the same ODE solutions as Figure 6 . Parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Panel of Figure 9 — Figure 9: Phase diagram of Figure 6 for other parameter values. The color shows $\log_{10}(\varepsilon^{*}_{\rm CNN}/\varepsilon^{*}_{\rm LCN})$ on a common scale. Solid line: boundary $\varepsilon^{*}_{\rm CNN}=\varepsilon^{*}_{\rm LCN}$ , obtained by integrating the ODEs numerically. Dashed line: analytical prediction $c=c_{0}\alpha_{F}$ of the early-time expansion, Eq. ( 30 ). Each panel changes one parameter of Figure 6 : (a) $P=8$ , (b) $k_{s}=k_{t}=2$ , (c) $T=2$ , (d) $\sigma=2$ . Other parameters: $g=\erf$ , $P=4$ , $k_{s}=k_{t}=1$ , $T=1$ , $\sigma=1$ , random initial condition in the limit $D\to\infty$ .

Figure 10: Rescaled noise floor

\varepsilon^{\infty}/(\eta\sigma^{2})

of the two students with the ReLU activation, where

\varepsilon^{\infty}

is the asymptotic generalization error under label noise. It is measured by online SGD from the specialized initial condition and averaged over seeds, with the generalization error computed exactly from the student weights. Standard errors are smaller than the marker size. As for

\erf

(Figure 5 ), the CNN’s floor grows with the data correlation

c

while the LCN’s stays flat. Parameters:

g(z)=\max(0,z)

k_{s}=k_{t}=2

P=4

D=400

\eta=0.1

T=1

\sigma=1

, runs up to

\alpha=800

8

seeds.

Panel of Figure 11 — Figure 11: Rescaling test of Figure 2 with the ReLU activation. Generalization errors $\varepsilon_{g}^{\rm CNN}(\alpha)$ of the CNN and $\varepsilon_{g}^{\rm LCN}(P\alpha)$ of the LCN, from online SGD averaged over 16 runs per student. Standard errors are smaller than the marker size. Insets show the curves without the time rescaling. (a) $c=0$ . (b) $c=0.5$ . Parameters: $g(z)=\max(0,z)$ , $k_{t}=2$ , $k_{s}=3$ , $P=4$ , $\eta=0.2$ , $\sigma=0$ , $T=1$ , $D=1000$ .

Convolutional Neural Networks

Convolutional neural networks (CNNs) learn spatial and temporal patterns using shared filters, with applications from image recognition to scientific prediction.

See all 410 papers

Removing spurious minima for planar features by skip connections
Oct 1, 2026Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo +1Shallow Neural Networks ReLU Neural Networks
Freie Universität Berlin · Technische Universität Berlin
Teacher Geometry Shapes Learnability in Teacher-Student Networks
Sep 9, 2026Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen +2Neural Network Optimization Neural Network Training Dynamics
Department of Experimental Psychology, University of Oxford · Department of Physics, RWTH Aachen
Resolution Allocation with Learnable Resizing and Spectral Filter Parameterization
Oct 14, 2026Wei Wu, Eric C. LarsonConvolutional Neural Networks Frequency-Domain Feature Learning
Department of Computer Science, Southern Methodist University, Dallas, TX, USA

Oct 1, 2026J/K move · Enter open · S save

Removing spurious minima for planar features by skip connections

Jakob Paul Zimmermann, Andrei Balakin, Moritz Grillo, Georg Loho

Freie Universität Berlin · Technische Universität Berlin

Shallow Neural Networks ReLU Neural Networks Neural Network Optimization Nonconvex Optimization Teacher-Student Learning

Open paper AskPDF arXiv

Understanding loss landscapes is central to explaining neural-network training, yet their structure remains only partially understood even in simple models. We study the Gaussian population loss of shallow, bias-free ReLU networks in the teacher--student setting. This provides a simple model for studying essential aspects such as feature learning and overparameterization. For teacher networks with positive output weights and planar features, we show that including a learned linear skip removes all spurious local minima with non-negative student output weights once the student network is at least as wide as the teacher network. In contrast, without the skip, we construct a fixed teacher network with positive output weights and only three hidden neurons in input dimension two whose spurious local minima persist at every student width at least three. Thus, a learned linear skip can remove spurious minima that persist under arbitrary overparameterization. Furthermore, we show that a positive output weight student network always learns the subspace spanned by the teacher features: student features at local minima with non-negative student output weights lie in the span of the teacher features. For ReLU networks in two dimensions, even heavily overparameterized student networks have effective width controlled by the teacher width: every critical point with positive student output weights has at most twice as many distinct student feature directions as teacher neurons. Finally, we transfer the benignity result to empirical minima over parameter balls of any prescribed radius, with the required sampling accuracy depending on that radius.

Abstract

Convolutional Neural Networks

Explore similar work

Removing spurious minima for planar features by skip connections

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Resolution Allocation with Learnable Resizing and Spectral Filter Parameterization