Modern neural architectures are largely developed as distinct empirical constructions, despite recurring principles across convolutional, attention, state-space, and equivariant models. We introduce Product Interactions, an algebraic formalism that unifies these architectures under a common construction principle and turns their design into a systematic procedure. The core shift is representing data as algebraic elements and treating multiplication, rather than linear transformation alone, as a primitive architectural operation. Architectures are generated by composing multiplication operators, with input dependence organized by self-interaction order. Within this formalism, we derive convolutional and equivariant networks as symmetry-constrained first-order product interactions, recover Mamba from state-space models as an increase in self-interaction order from one to three, and establish attention as a third-order interaction from composing the same operator. Equivariance is incorporated by constraining the algebraic product via symmetry: translation equivariance yields convolution, while SO(3)-compatibility produces the Clebsch--Gordan product and tensor-field interactions. This perspective recasts architecture design as an algebraic construction problem: the choice of algebra defines admissible interactions, symmetry constrains their structure, and composition generates higher-order architectures. The framework is thus generative. Following these principles, we construct a simplified-algebra dynamical layer that improves both accuracy and throughput over Mamba on CIFAR-10. Experiments show that interaction order affects accuracy non-monotonically and symmetry constraints boost learnability. Product Interactions provides a mathematical methodology for analyzing existing architectures and systematically designing new ones.
Figures & tables
Figure 1: Algebraic multiplication of basis elements, determined by the structure constants λijk .
Figure 2: Commutative diagram illustrating the equivariance of a Product Interaction operator under a symmetry transformation.
Model
Mamba
(I=3,H=2)
(I=3,H=1)
(I=2,H=1)
(I=1,H=1)
Accuracy
86.818±0.141
87.118±0.254
86.748±0.192
85.760±0.152
86.112±0.218
Training speed
4.81 it/s
12.95 it/s
–
–
–
Inference speed
15 it/s
39 it/s
–
–
–
Table 1: Comparison of classification accuracy, training throughput, and inference throughput for Vision Mamba and the proposed dynamical layers on CIFAR-10. The proposed models are indexed by the self-interaction orders of the input term I and hidden state H . Accuracy is reported as mean ± standard deviation, and throughput is measured in iterations per second. Furthermore, as speed differences across models of different orders are negligible, we report the performance of the highest-order model, with omitted values indicated by (–).
Model
With Sym
Without Sym
Reg Sym
Accuracy
0.988
0.445
0.943
Table 2: MNIST classification accuracy for multiplication operators with exact symmetry constraints (With Sym), without symmetry constraints (Without Sym), and with symmetry constraints imposed through regularization (Reg Sym).
Model
R=1
R=2
R=4
length 192
0.3603
0.9173
0.9239
Table 3: Copy-task accuracy for different projection ranks R at sequence length 192 .
Fast matrix multiplication algorithms keep the product fixed and search for a cheaper way to evaluate it. We instead ask whether a Transformer's learned projections can use a different, cheaper product altogether. Building on an associative-algebra construction that replaces ordinary matrix multiplication with a sparser interaction table over the same weight blocks, we construct a family with quadratic arithmetic in the matrix dimension when the physical block size remains fixed, and derive finite-shape constraints for GPU execution. The construction is provably optimal for its bilinear rank by the Alder--Strassen bound and can be realized as row-typed rectangular projections compatible with causal masking and KV-cached decoding. We provide an empirical test of this approach by training two approximately 110M-parameter decoder-only Transformer LMs from the same recipe and 12.3B-token budget, differing only in their feed-forward layer: one uses ordinary dense matrix multiplication and the other uses the associative-algebra product. Across four prompt domains, the algebraic model achieves a 6.2--7.8% increase in end-to-end generation throughput, while obtaining lower scores on all three reported downstream metrics. We treat these results as a feasibility and trainability check for the proposed approach at small scale, leaving further investigation to future work.
Ilya Koziev, Ivan Oseledets
DAIMLD, Moscow, Russia · Artificial Intelligence Research Institute, Moscow, Russia
Despite deep learning models running well-defined mathematical functions, we lack a formal mathematical framework for describing model architectures. Ad-hoc notation, diagrams, and pseudocode poorly handle nonlinear broadcasting and the relationship between individual components and composed models. This paper introduces a categorical framework for deep learning models that formalizes broadcasting through the novel axis-stride and array-broadcasted categories. This allows the mathematical function underlying architectures to be precisely expressed and manipulated in a compositional manner. These mathematical definitions are translated into human manageable diagrams and machine manageable data structures. We provide a mirrored implementation in Python (pyncd) and TypeScript (tsncd) to show the universal aspect of our framework, along with features including algebraic construction, graph conversion, PyTorch compilation and diagram rendering. This lays the foundation for a systematic, formal approach to deep learning model design and analysis.
Vincent Abbott, Gioele Zardini
Laboratory for Information and Decision Systems, Massachusetts Institute of Technology
We introduce a unified theoretical framework for the rigorous analysis and systematic construction of deep neural networks (DNNs). This framework addresses a gap in existing theory by explicitly modeling the structure of tensor operations -- lower level information that is often abstracted. Our framework enables two novel objectives: (1) analysis of the evolution of architectural complexity over deep learning history, and (2) automatic construction of novel architectures based on new types of tensor operations. Our study of DNNs introduced over the past 40 years reveals a connection between groundbreaking architectures and increases in different types of architectural complexity. Moreover, we identify several large classes of higher complexity architectures that have not yet been explored. We then collect a dataset of 3,000+ higher complexity architectures, which we publicly release at: https://github.com/combinatoriallabs/ArchitecturalComplexity.
Nicholas J. Cooper, François G. Meyer, Michael L. Roberts +3
Combinatorial Labs · University of Colorado Boulder · SEMF/Independent Researcher