cs.LGApr 8, 2026

Weaves, Wires, and Morphisms: Formalizing and Implementing the Algebra of Deep Learning

Authors: Vincent Abbott, Gioele Zardini

Organizations: Laboratory for Information and Decision Systems, Massachusetts Institute of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures. Ad-hoc notation, diagrams, and pseudocode poorly handle nonlinear broadcasting and the relationship between individual components and composed models. This paper introduces a categorical framework for deep learning models that formalizes broadcasting through the novel axis-stride and array-broadcasted categories. This allows the mathematical function underlying architectures to be precisely expressed and manipulated in a compositional manner. These mathematical definitions are translated into human manageable diagrams and machine manageable data structures. We provide a mirrored implementation in Python (pyncd) and TypeScript (tsncd) to show the universal aspect of our framework, along with features including algebraic construction, graph conversion, PyTorch compilation and diagram rendering. This lays the foundation for a systematic, formal approach to deep learning model design and analysis.

Figures & tables

Term TypeAssociated DataConstructed PropertyNon-constructed PropertyInterpretation
ProductgΠj=Πj∈Jgj\displaystyle g_{\Pi j}=\Pi_{j\in J}g_{j}WrappingMethod mΠj,ℓ:Πj∈JGj→Gℓ,i\displaystyle m_{\Pi j,\ell}:\Pi_{j\in J}G_{j}\rightarrow G_{\ell,i} \displaystyle\pi_{\ell}=m_{c,\ell}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}(\Pi_{j\in J}g_{j})\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}=\Pi_{j\in J}(g_{j}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G})
Constructed Termgc,f↦gc,i\displaystyle g_{c,f}\mapsto g_{c,i} via T^c\displaystyle\hat{T}_{c}WrappingMethod mc,ℓ:Gc,i→Gℓ,f\displaystyle m_{c,\ell}:G_{c,i}\rightarrow G_{\ell,f} \displaystyle\pi_{\ell}=\hat{T}_{c}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}m_{c,\ell}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}g_{c,f}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}=g_{c,f}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}\hat{T}_{c}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}\pi_{c}
Root Termgr↦μr×Πℓ∈rgℓ,f\displaystyle g_{r}\mapsto\mu_{r}\times\Pi_{\ell\in r}g_{\ell,f} Where mr\displaystyle m_{r} is non-core metadata.WrappingData Extraction pr,ℓ:Gr→Gr,ℓ\displaystyle p_{r,\ell}:G_{r}\rightarrow G_{r,\ell} \displaystyle\pi_{\ell}=p_{r,\ell}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}\displaystyle g_{r}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}=\gamma_{r} \displaystyle\forall\ell\in r.g_{r}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}p_{r,\ell}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}V_{G}=\gamma_{r}\mathbin{\raise 2.58334pt\hbox{\vtop{\halign{#\cr\hfil\scriptscriptstyle\mathrm{o}\hfil\cr\hfil\hfil\cr\hfil\scriptscriptstyle\mathrm{9}\hfil\crcr}}}}\pi_{\ell}
Table 1: Applicability and interpretation for the main term types.
Figure 2
Figure 3
Figure 4
Figure 10: Diagonalization corresponds to the equation y[p,:]=x[p,p,:]\mathbf{y}[p,:]=\mathbf{x}[p,p,:] . This can be expressed using the rearrangement reindexing p↦(p,p)p\mapsto(p,p) .
Figure 11: Repetition corresponds to the equation y[p,:]=x[:]\displaystyle\mathbf{y}[p,:]=\mathbf{x}[:] . This can be expressed using the rearrangement reindexing p↦()\displaystyle p\mapsto() .
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13
Figure 14
Figure 36: Here, we show a GeGLU mixture of experts as used in DeepSeek-V3 ( DeepSeek-AI, 2025 ) . We use a Top-K operation to select the indexes of the relevant arrays. The final output is then computed by summing over the normalized top-k weights, aligning with the chosen indexes.
Figure 37: The expression from Figure 36 can be reexpressed in a mathematically equivalent form which rearranges indexed-inputs to outputs, over which we perform a sparse sum.
Figure 17
Figure 18
Figure 44: At a high-level, the deep seek algorithm follows a standard transformer structure: an embedding maps from a series of tokens indexed across V∈NV\in\mathbb{N} options, and is sent through a sequence of attention-feed forward layers. The final result is mapped back through a linear layer, producing a vector [R;V][{\mathbb{R}};{V}] indicating the log-weights of output token options. These layers are all residual connections ( He et al., 2015 ) .
Figure 45: Attention blocks correspond to multi-head latent attention which comprises a complex rotary component and a real, latent component. These are used to generate queries and keys, which map through a multi-head Q−KQ-K matrix multiplication contracted over the dd axes parallelized over hh heads. The results of the complex and real components are summed together, passed through a SoftMax and weighted lower triangular normalizer for causality, undergo an S−VS-V matrix muliplication, and are passed through an output linear layer, contracting the heads.
Figure 46: The rotary linear layer generates queries and keys for the complex component. This is supplied by a rotary mask generator, which is not “parallelized” over the x‾\overline{x} axis – the outputs are strictly dependent on which x‾\overline{x} index we are at. This “binds” the x‾\overline{x} axis into the output, meaning the result has positions encoded. This multiplies with a linear layer which generates both a real and complex value for the mm inputs. The resulting output is “decomplexed”, and yields an array with integrated positional information.
Figure 47: The multilayer perceptrons used by DeepSeek-V3 is a GeGLU ( Shazeer, 2020 ) . We have two input linear layers, one of which passes through an elementwise operation, the results multiplied together, and followed by an output linear layer. The internal size, in this case d∗fd*f , is not dependent on external axes sizes, and therefore can be configured independently.
Figure 48: The Mixture-of-Experts block follows Section 5.5 , with a gate selecting experts added to a multilayer perceptron. This architecture supports multi-GPU computational splits. Here, we show the simplified “sparse axis” presentation which presents a mathematically coherent representation insufficient for direct computation.
Figure 49: The gate for the Mixture-of-Experts involves a gate linear layer which provides weights for each of nn layers, a Top-K selection which generates a sparse axis, which is then normalized by a SoftMax-like operation, which, in practice, uses a sigmoid rather than an exponent to weight values.
Figure 50: Finally, the experts component follows Section 5.5 , with each layer having an additional nn axis of which kk values are non-zero.
⬇ 1 qk_matmul = ops . Einops 2 . template ( 3 ’q h d, x h d -> h q d’ )⬇ 1 softmax = ops . SoftMax . template () As we did not provide the axes, an unnamed axis is generated. Autoalignment will then rename it.
⬇ 1 sv_matmul = ops . Einops 2 . template ( 3 ’h q x, x h d -> q h d’ )⬇ 1 mask = ops . 2 WeightedTriangularLower 3 . template ()
Table 26
Figure 51: Applying qk_matmul @ softmax @ mask @ sv_matmul , we perform autoalignment operations at each step. Axes are aligned to be the same. Operations are batch broadcasted when the number of axes mismatch. In the case of sv_matmul , we take the product with an identity rearrangement with the array [R,xhd]\displaystyle[\mathbb{R},xhd] . Note that the h\displaystyle h , q\displaystyle q , x\displaystyle x axes of qk_matmul and sv_matmul are separately generated, meaning their equivalence is only realized after composition. The d1\displaystyle d_{1} and d2\displaystyle d_{2} axes are never aligned together, meaning that the final configuration of the expression takes ⟨q,h,x,d1,d2⟩\displaystyle\langle q,h,x,d_{1},d_{2}\rangle .
Figure 52: A full ResNet attention block expression constructed through autoalignment has a number of numeric sizes with UIDs. By assigning values to these, we have a configured expression ready for PyTorch compilation.
Figure 53: By converting from a “ProdCategory[L,M]“ to a “Hypergraph[L,M]“, we can perform algebraic manipulation with hypergraph rewrite rules. This allows us to apply general algebraic properties such as associativity, bifunctoriality, and symmetry, which are not easily captured by the composed-product approach.
Figure 54: The modularity of the framework allows for a web of features to be developed and integrated. At any point, new features can be “hooked” into the system, allowing for a powerful and extensible framework for future AI research.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 31
Figure 32
Figure 33
Figure 34

Explore similar work

CardsList
May 23, 2026

Lattice theory and algebraic models for deep convolutional learning based on mathematical morphology

We develop a rigorous algebraic framework for deep convolutional architectures, CNNs, ResNets, and encoder--decoder networks such as UNet, grounded in lattice theory and mathematical morphology. The central tool is the Matheron--Maragos--Banon--Barrera (MMBB) universal representation theory for translation-invariant operators, which we apply systematically to every layer of a standard deep network. The principal finding is that the standard CNN pipeline (linear convolution~++ ReLU~++ flat max-pooling) is a cross-lattice operator: the convolution is an erosion in the Fourier inf-semilattice while ReLU is a lattice-join closing and max-pooling is a dilation in the pointwise max-plus lattice, and their composition is a morphological opening in neither. A second finding is that the upper adjoint of ReLU in the pointwise lattice is a global (non-local) operator, the identity on globally non-negative functions and −∞-\infty otherwise, so no local morphological erosion can form an adjunction pair with ReLU. These two results together provide the precise algebraic reason why depth in standard CNNs introduces genuine representational power: the composed layer is not idempotent. Three layer designs that are genuine idempotent openings are identified and fully characterised: the pure max-plus morphological layer (pointwise lattice), the spectral Wiener layer (Fourier lattice), and the self-dual morphological layer. We establish a complete fixed-point and convergence theory. The framework also unifies max-pooling, strided convolution, and the Laplacian pyramid under the Goutsias--Heijmans adjoint pyramid theory, and gives the Activation--Pooling Dilation (APD) factorisation with its correct adjoint.

Gustavo, Angulo

Jesus

Convolutional Neural Networks
AskPDF
May 5, 2026

On the Architectural Complexity of Neural Networks

We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.

Nicholas J. Cooper, François G. Meyer, Michael L. Roberts +3

Combinatorial Labs · University of Colorado Boulder · SEMF/Independent Researcher

Topic pending
AskPDF
Mar 3, 2026

Coalgebras for categorical deep learning: Representability and universal approximation

Categorical deep learning (CDL) has recently emerged as a framework that leverages category theory to unify diverse neural architectures. While geometric deep learning (GDL) is grounded in the specific context of invariants of group actions, CDL aims to provide domain-independent abstractions for reasoning about models and their properties. In this paper, we contribute to this program by developing a coalgebraic foundation for equivariant representation in deep learning, as classical notions of group actions and equivariant maps are naturally generalized by the coalgebraic formalism. Our first main result demonstrates that, given an embedding of data sets formalized as a functor from SET to VECT, and given a notion of invariant behavior on data sets modeled by an endofunctor on SET, there is a corresponding endofunctor on VECT that is compatible with the embedding in the sense that this lifted functor recovers the analogous notion of invariant behavior on the embedded data. Building on this foundation, we then establish a universal approximation theorem for equivariant maps in this generalized setting. We show that continuous equivariant functions can be approximated within our coalgebraic framework for a broad class of symmetries. This work thus provides a categorical bridge between the abstract specification of invariant behavior and its concrete realization in neural architectures.

Dragan Mašulović

University of Novi Sad, Faculty of Sciences Department of Mathematics and Informatics Trg Dositeja Obradovića 3, 21000 Novi Sad, Serbia

Equivariant Representation LearningInvariant Representation LearningNeural Network Approximation Theory
AskPDF