Neural Networks as Local-to-Global Computations

Vicente Bosca; Robert Ghrist

math.ATMar 16, 2026

Neural Networks as Local-to-Global Computations

Authors: Vicente Bosca, Robert Ghrist

Organizations: Department of Mathematics, University of Pennsylvania

Abstract

We construct a cellular sheaf from any feedforward ReLU neural network by placing one vertex for each intermediate quantity in the forward pass and encoding each computational step - affine transformation, activation, output - as a restriction map on an edge. The restricted coboundary operator on the free coordinates is unitriangular, so its determinant is $1$ and the restricted Laplacian is positive definite for every activation pattern. It follows that the relative cohomology vanishes and the forward pass output is the unique harmonic extension of the boundary data. The sheaf heat equation converges exponentially to this output despite the state-dependent switching introduced by piecewise linear activations. Unlike the forward pass, the heat equation propagates information bidirectionally across layers, enabling pinned neurons that impose constraints in both directions, training through local discrepancy minimization without a backward pass, and per-edge diagnostics that decompose network behavior by layer and operation type. We validate the framework experimentally on small synthetic tasks, confirming the convergence theorems and demonstrating that sheaf-based training, while not yet competitive with stochastic gradient descent, obeys quantitative scaling laws predicted by the theory.

Figures & tables

Panel of Figure 1 — Figure 1: Neural sheaf encoding a $k$ -hidden layer ReLU network. Red nodes are stubborn (boundary conditions); green and yellow nodes are dynamic, with yellow nodes having a fixed component. Restriction maps $\overline{W}^{(\ell)}$ encode weights and biases, $R^{z^{(\ell)}}$ encodes ReLU activation, $P_{n_{\ell}}$ projects onto the first $n_{\ell}$ coordinates, and $\phi$ is the final activation function.

Panel of Figure 2 — Figure 2: The neural network–sheaf correspondence. Red nodes represent fixed data (boundary conditions); green and yellow nodes are computed, with yellow nodes having a fixed component. (a) The feedforward network computes postactivations $\mathbf{a}^{(1)}=\mathrm{ReLU}(W^{(1)}\mathbf{x}+b^{(1)})$ and output $\hat{\mathbf{y}}=\phi(W^{(2)}\mathbf{a}^{(1)}+b^{(2)})$ via the forward pass. (b) In the sheaf formulation, only $\overline{\mathbf{x}}$ (input extended with ones) is pinned. Restriction maps encode weights and biases ( $\overline{W}^{(1)},\overline{W}^{(2)}$ ), ReLU activations ( $R^{z^{(1)}}$ ), and final activation ( $\phi$ ). The heat equation drives any initialization to the correct forward pass output.

Panel of Figure 3 — Figure 3: Pinning a neuron. (a) Fixing the third hidden neuron to the value $\mathbf{p}$ in the feedforward network affects only the output neuron through the red edge. (b) In the sheaf formulation, the pinned value $\mathbf{p}$ is imposed through the restriction map $P_{\{3\}}$ , which projects onto the third component of $a^{(1)}$ . The disturbance propagates to both adjacent stalks $z^{(1)}$ and $z^{(2)}$ (red edges).

Panel of Figure 4 — Figure 4: Two training configurations for a 1-hidden-layer network. Red nodes are pinned boundary conditions, green nodes are dynamic pre-activation variables, and yellow nodes are dynamic post-activation variables with some part fixed. Green arrows represent trainable weight matrices. Left: Regression with identity activation composed with gradient of nonlinear potential $\nabla U$ , enabling other losses beyond L2, like L1, Lp, or Huber. Right: Classification with Softmax/Sigmoid activation and cross-entropy loss, where dynamics simplify to allow a sheaf-like construction.

Panel of Figure 5 — Figure 5: Batch processing for a one-hidden-layer network with batch size $M$ . Top : the batch sheaf with matrix-valued stalks, where ReLU acts via coordinatewise masking. Bottom : the equivalent representation as $M$ parallel activation edges, each carrying its own diagonal ReLU matrix $R^{z^{(1),m}}$ and projection $P_{n_{1},m}$ . The weight edges are shared across all inputs.

Panel of Figure 6 — Figure 6: Convergence dynamics for a $[2,4,1]$ network with identity output activation, $\alpha=1.0$ , $dt=0.01$ , stalks initialized randomly. (a) Total discord decreases monotonically on a log scale to machine precision. (b) The output stalk converges from its random starting value to the forward-pass prediction (black dashed). This validates Theorem 4.1 .

Panel of Figure 7 — Figure 7: Sheaf training vs. SGD on the paraboloid regression task ( $[2,30,1]$ architecture, identity output). (a, b) Training and test loss on a log scale. Both methods converge smoothly; SGD reaches lower final loss. (c–e) Learned surfaces on the $[-2,2]^{2}$ domain, shown with a shared colour scale. Both methods recover the paraboloid shape, with SGD producing a slightly smoother approximation.

Panel of Figure 8 — Figure 8: Hyperparameter investigation on the paraboloid task (a) Final test loss as a function of $\beta M$ for several values of $\beta$ . Experiments confirmed the $\beta\sim 1/M$ scaling from the stagnation bound: performance depends on the product $\beta M$ , not on $\beta$ or $M$ individually. The dashed line marks the default $\beta=1/n_{\mathrm{train}}$ . (b) Final test loss as a function of total integration time $T=\Delta t\times n_{\mathrm{steps}}$ for three architectures of increasing depth. For each depth, loss decreases monotonically as $T$ increases, confirming that longer equilibration improves the sheaf training output. Deeper networks achieve higher loss, requiring more integration time to reach comparable performance. This is consistent with the stagnation bound, which predicts slower convergence when $\lambda_{1}$ is smaller. Table 2 confirms that $\lambda_{1}$ decreases with depth.

Panel of Figure 9 — Figure 9: Per-coordinate edge discord for a well-trained $[2,30,1]$ sheaf network on the paraboloid task ( $300$ training inputs). (a) Weight edge discord $(W^{(1)}a^{(0)}+b^{(1)})_{j}-z_{j}^{(1)}$ vs. pre-activation $z_{j}^{(1)}$ . Inactive coordinates ( $z_{j}<0$ , purple) lie on the zero line: the ReLU edge exerts no competing force, so the weight edge drives $z_{j}$ to its predicted value exactly. Active coordinates ( $z_{j}\geq 0$ , orange) show a spread around zero, reflecting the equilibrium compromise between weight and activation edge forces. (b) ReLU edge discord $\mathrm{ReLU}(z_{j}^{(1)})-a_{j}^{(1)}$ vs. $z_{j}^{(1)}$ . For inactive coordinates ( $z_{j}<0$ , purple), this quantity equals $-a_{j}^{(1)}$ ; the nonzero spread reflects the downstream weight edge pulling $a_{j}^{(1)}$ away from zero to reduce its own discrepancy. For active coordinates ( $z_{j}\geq 0$ , orange), both edges compete, producing a symmetric cloud. The magnitudes are comparable to the weight-edge residuals, consistent with the equilibrium force balance described in Section 6.3 .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 10 — Figure 10: Joint dynamics on coupled sheaves. The upper diagram depicts the state sheaf $\mathcal{F}^{\Omega}$ , where vertices represent network layers (pinned vertices $\overline{\mathbf{x}}$ and $\mathbf{y}$ in red, hidden states in green) and edge stalks carry restriction maps built from weights and activations. The lower diagram shows the parameters sheaf $\mathcal{H}^{W}$ , where vertices store weight matrices and edge stalks encode gradients with respect to activations. Dashed blue arrows link state variables (pointing downward) and parameters (pointing upward) between the two sheaves, illustrating how the heat diffusion on both sheaves is coupled.

Panel of Figure 11 — Figure 11: Regularization as sheaf augmentation for a 1-hidden-layer network. Each free variable is anchored to its own dummy vertex pinned at $\mathbf{0}$ , with restriction maps $\sqrt{\lambda}\,I$ (or $\sqrt{\mu}\,I$ ) penalizing deviation from zero. Separate parent vertices allow for different types of regularization. (a) In the state sheaf $\mathcal{F}^{\Omega}$ , three regularization edges anchor the free cochains $z^{(1)}$ , $\overline{a}^{(1)}$ , and $z^{(2)}$ toward zero, adding $\lambda I$ to the diagonal of the Laplacian. (b) In the parameter sheaf $\mathcal{H}^{W}$ , regularization edges anchor the trainable restriction maps $\overline{W}^{(1)}$ and $\overline{W}^{(2)}$ toward zero, contributing $\mu I$ to the weight dynamics. Green elements indicate the trainable variables.

Figure 12: Convergence dynamics for a

[2,6,4,4,1]

network with sigmoid output activation,

\alpha=1.0

dt=0.01

, stalks initialized randomly. Top left: Total discord decreases monotonically on a log scale to machine precision. Top right: Discord decomposed by edge type; the output activation edge (enforcing the sigmoid nonlinearity) is visible as a separate component. Bottom left: Per-layer discord shows a transient phase in the first hundred iterations followed by exponential convergence. Bottom right: The output stalk

\sigma(z^{(4)})

converges from its random starting value to the forward-pass prediction (black dashed). This validates Theorems 4.1 and 4.2 for deeper networks with nonlinear final activations.

Figure 13: Phase-plane trajectories of two pre-activation components $(z_{1}^{(1)},z_{2}^{(1)})$ during sheaf diffusion, for three architectures of increasing depth. Trajectories fade from light (early) to dark (late). Dashed grey lines mark the ReLU boundaries $z_{j}=0$ , red crosses mark boundary crossings, and the gold star marks the forward-pass fixed point. Crossings are concentrated in the first few hundred iterations; the trajectory then settles into a single activation region and converges monotonically. Deeper networks produce longer transients but the same qualitative pattern: ReLU boundary crossings are a transient phenomenon that does not impede convergence.

Task	Depth	Sheaf Train	SGD Train	Sheaf Test	SGD Test	Ratio
Paraboloid	1H	0.0756	0.0030	0.0884	0.0044	19.9 $\times$
Saddle	1H	0.0108	0.0010	0.0106	0.0013	8.4 $\times$
Circular	1H	0.0652	0.0021	0.0679	0.0031	22.0 $\times$
Blobs	1H	0.1368	0.0657	0.1212	0.2613	0.5 $\times$
Paraboloid	2H	0.0764	0.0065	0.0880	0.0095	9.2 $\times$
Saddle	2H	0.0110	0.0009	0.0126	0.0011	11.0 $\times$

Appendix

Table 1: Training results across all task/depth configurations.

Figure 14: Learned outputs for all four benchmark tasks (single hidden layer). Each row shows the target function or data distribution (left), the sheaf-trained output (center), and the SGD-trained output (right). Paraboloid and saddle (top two rows): filled contour plots of the learned regression surface on the input domain, with shared color scales within each row. Circular and blobs (bottom two rows): training data colored by class label, with the learned decision boundary shown as a shaded background. Both methods produce qualitatively correct outputs across all tasks. The sheaf method captures the global shape but with less fine-grained detail in regression, consistent with the test loss ratios reported in Table 1 .

Architecture	$\lambda_{1}$ (median)	$\lambda_{1}$ (std)
[2,30,1]	0.2248	0.0200
[2,10,8,1]	0.0602	0.0047
[2,8,6,4,1]	0.0281	0.0060

Appendix

Table 2: Spectral gap

\lambda_{1}

vs. depth at initialization (median over 50 inputs).

Panel of Figure 15 — Figure 15: Per-block eigenvector energy of $L_{\mathcal{F}}[\Omega,\Omega]$ for two architectures (columns) and two eigenvectors (rows), averaged over $50$ random inputs with error bars showing one standard deviation. Top row: Fiedler eigenvector ( $\lambda_{1}$ ). In both architectures, the Fiedler energy concentrates on the output pre-activation stalk $z^{(k+1)}$ , identifying it as the information-flow bottleneck of the sheaf graph. Bottom row: $\lambda_{\max}$ eigenvector. The dominant eigenvector energy coincides with the post-activation block $a^{(\ell)}$ adjacent to the weight matrix with the largest operator norm. Neither layer position nor layer width is the determining factor.

Panel of Figure 16 — Figure 16: Spectral gap $\lambda_{1}$ (median $\pm$ IQR over 50 inputs) and training loss during training of a $[2,30,1]$ network on the paraboloid task. Both axes share identical limits across panels for direct comparison. In both methods, $\lambda_{1}$ drops rapidly in the first ${\sim}10$ k steps. The sheaf method produces consistently smaller $\lambda_{1}$ throughout training.

Panel of Figure 17 — Figure 17: Spectral statistics of $L_{\mathcal{F}_{t}}[\Omega,\Omega]$ during training (median $\pm$ IQR, $[2,30,1]$ on paraboloid). Both methods shift the spectrum toward smaller $\lambda_{1}$ and larger $\lambda_{\max}$ as training progresses, but the sheaf method produces consistently smaller spectral gaps (a) and larger condition numbers (c) . The largest eigenvalue (b) grows in both methods, with sheaf-trained networks producing moderately larger $\lambda_{\max}$ .

Panel of Figure 18 — Figure 18: Distribution of the spectral gap $\lambda_{1}$ across 100 random inputs at three training stages (poorly trained at 1k steps, intermediate at 20k, well trained at 100k). Log-spaced bins; shared $x$ -axis range across both panels. In both methods, the distribution shifts leftward (toward smaller $\lambda_{1}$ ) as training progresses, tightening around a stable value. The sheaf-trained distributions are shifted further left at all stages, consistent with the tracking plots in Fig. 16 .

Panel of Figure 19 — Figure 19: Per-coordinate edge discord for a well-trained $[2,12,8,6,1]$ sheaf network (three hidden layers) on the paraboloid task. (a–d) Weight edge discord $(W^{(\ell)}a^{(\ell-1)}+b^{(\ell)})_{j}-z_{j}^{(\ell)}$ vs. pre-activation $z_{j}^{(\ell)}$ for each of the four weight edges. Across the three first weight edges, inactive coordinates (purple) cluster tightly on the zero line, confirming that the ReLU edge exerts no force on dead neurons and the weight edge achieves zero discord. The last weight edge does not follow this pattern because the final activation in this case is the identity, so this edge acts as the output edge. Active coordinates (orange) show a symmetric spread around zero whose magnitude varies by layer, reflecting the local force balance between weight and activation edges at equilibrium. (e–g) ReLU edge discord $\mathrm{ReLU}(z_{j}^{(\ell)})-a_{j}^{(\ell)}$ vs. $z_{j}^{(\ell)}$ for the three activation edges. We found no clear structure. Across all ReLU edges, we observe a decay in the number of inactive neurons (negative values of $z_{j}$ ).

Figure 20: Per-edge discord residuals at three training stages for a sheaf-trained $[2,30,1]$ network on the paraboloid task. Each row corresponds to a training stage (1k, 20k, 100k steps); output stalks are hard-pinned to true labels. $y$ -axis limits are fixed across all rows (set by the intermediate stage) to show the decrease in discord magnitude with training. Points are colored by activation state: active ( $z_{j}\geq 0$ , orange) or inactive ( $z_{j}<0$ , purple). Left: Weight edge discord residual $(W^{(1)}a^{(0)}+b^{(1)})_{j}-z_{j}^{(1)}$ . Right: ReLU edge discord residual $\mathrm{ReLU}(z_{j}^{(1)})-a_{j}^{(1)}$ . In the well-trained model (bottom), inactive coordinates lie on the zero line and active coordinates cluster near zero, confirming local consistency at equilibrium.

Figure 21: Same as Fig. 20 but for an SGD-trained network with the same architecture and training budget. $y$ -axis limits are shared with the sheaf figure for direct comparison. The SGD-trained network’s weights are embedded in a NeuralSheaf and analysed via pinned diffusion under the same protocol. Qualitative differences in the residual patterns reflect the different optimization landscapes explored by the two methods.

Edge	Sheaf-trained	SGD-trained
W1	0.0007 $\pm$ 0.0010	0.0000 $\pm$ 0.0001
W2	0.0001 $\pm$ 0.0002	0.0000 $\pm$ 0.0000
R1	0.0027 $\pm$ 0.0034	0.0002 $\pm$ 0.0003
Total	0.0035	0.0003

Appendix

Table 3: Per-edge pinned discord (mean

\pm

std over 50 test inputs). Output stalk hard-pinned to true labels.

Sheaf Neural Networks

Sheaf neural networks generalize graph learning with local vector spaces and edge maps that model how information is compared and transported.

See all 29 papers

On the Necessity of Learnable Sheaf Laplacians
Mar 5, 2026Ferran Hernandez Caralt, Mar Gonzàlez i Català, Adrián Bazaga +1GNN Oversmoothing Sheaf Neural Networks
Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom · Microsoft
A Bochner Weitzenböck Decomposition for Cellular Sheaf Laplacians, Curvature and Neural Sheaf Diffusion
Oct 20, 2026Francesco Maria Guadagnuolo, Alessio Borgi, Dorina Thanou +1Sheaf Neural Networks
Sapienza University of Rome, Rome, Italy · Italian Institute of Artificial Intelligence (AI4I), Turin, Italy · University of Cambridge, Cambridge, United Kingdom +1
Fixed Points Without Fixed Diffusion: Implicit Neural Sheaves for Convergent Test-Time Computation
Aug 12, 2026Rémi Bourgerie, Šarūnas Girdzijauskas, Viktoria FodorGraph Neural Networks Sheaf Neural Networks
School of Electrical Engineering and Computer Science, and Digital Futures KTH Royal Institute of Technology, Stockholm, Sweden

Mar 5, 2026J/K move · Enter open · S save

On the Necessity of Learnable Sheaf Laplacians

Ferran Hernandez Caralt, Mar Gonzàlez i Català, Adrián Bazaga, Pietro Liò

Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom · Microsoft

GNN Oversmoothing Sheaf Neural Networks Graph Neural Networks Over-Smoothing in GNNs

Open paper AskPDF arXiv

Sheaf Neural Networks (SNNs) were introduced as an extension of Graph Convolutional Networks to address oversmoothing on heterophilous graphs by attaching a sheaf to the input graph and replacing the adjacency-based operator with a sheaf Laplacian defined by (learnable) restriction maps. Prior work motivates this design through theoretical properties of sheaf diffusion and the kernel of the sheaf Laplacian, suggesting that suitable non-identity restriction maps can avoid representations converging to constants across connected components. Since oversmoothing can also be mitigated through residual connections and normalization, we revisit a trivial sheaf construction to ask whether the additional complexity of learning restriction maps is necessary. We introduce an Identity Sheaf Network baseline, where all restriction maps are fixed to the identity, and use it to ablate the empirical improvements reported by sheaf-learning architectures. Across five popular heterophilic benchmarks, the identity baseline achieves comparable performance to a range of SNN variants. Finally, we introduce the Rayleigh quotient as a normalized measure for comparing oversmoothing across models and show that, in trained networks, the behavior predicted by the diffusion-based analysis of SNNs is not reflected empirically. In particular, Identity Sheaf Networks do not appear to suffer more significant oversmoothing than their SNN counterparts.

Abstract

Sheaf Neural Networks

Explore similar work

On the Necessity of Learnable Sheaf Laplacians

A Bochner Weitzenböck Decomposition for Cellular Sheaf Laplacians, Curvature and Neural Sheaf Diffusion

Fixed Points Without Fixed Diffusion: Implicit Neural Sheaves for Convergent Test-Time Computation