cs.LGMar 30, 2026
SaveSpectral Higher-Order Neural Networks
Organizations: University of Florence, Italy · University of Namur, Belgium
Abstract
Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.
Figures & tables
Figure 1: Cartoon representing different neural architectures. The standard neural networks (panel 1(a) ) with input neurons, , generating output signals, , via weighted averages, i.e., linear combinations (colored arrows) followed by the application of a local nonlinearity (not shown). The standard triadic higher-order network (panel 1(b) ) is obtained by adding to the previous architecture, weighted sums of hyperlinks, mimicking the two body interaction (symbolized by the curved arrows connecting a couple to an output , each pair with its own specific color). The spectral higher-order networks (panel 1(c) ), reduces the number of used parameters by exploiting the spectral decomposition. This yields an effective parameter sharing among hyperlinks (the curved arrows connecting a couple to several outputs , share the same color).
Figure 2: Results of the perceptrons’ training on mnist and fashion-mnist (via Adam optimizer). We performed learning rate warm-up , followed by a reduce-learning-rate-on-plateau protocol, to guard against a possible dependence of the results on an unlucky hyperparameter choice. The direct space triadic model is plagued by instability and confidence saturation problems, while the standard perceptron lacks in expressivity. At variance, the spectral triadic model (i) achieves the same performances of the standard triadic network, with a substantially more efficient parameter scaling and (ii) it is also way more stable.
Figure 3: Standard mlp vs. spectral triadic mlp on CIFAR-10. The models were trained with the Adam optimizer, following a halving-lr-on-plateau scheduler. The results show a clear advantage for the triadic architecture.
Figure 4: Standard mlp-mixer vs a spectral triadic version of it, on CIFAR-10. For this experiment the Adam optimizer was used with fixed learning rate. From the results, it seems that the spectral forward propagation not only shows a sufficient degree of expressivity, but also acts as an implicit regularizer, preventing overfitting.
Figure 5: Graphical sketch of Lemma 4.1 , illustrating the inter-layer connections described in Eq. ( 21 ), under the triangular ansatz and in the simple setting where the only nonlinear interactions arise from the last column of matrix . Subsequently, the elements of can be linearly remapped to the canonical basis of . The black dashed arrows represent the linear passage corresponding to the first sum term of Eq. ( 15 ), while the orange solid lines represent the higher-order interactions, namely second sum in Eq. ( 15 ), here just restricted to and itself.
Figure 6: Graphical sketch of the ladder demonstration scheme proposed in Lemmas 4.2 , 4.3 . Here the input is provided to the honn with output , which can be linearly remapped, via , to the new canonical basis augmented with a new nonlinear term, for instance . The procedure then continues by creating new terms: , , and so on. As in Fig. 5 , dashed black lines indicate linear transfer, while orange ones indicate nonlinear interactions.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Graphical sketch of the universality proof for triadic multilayer perceptrons. The structure of the forward pass ( 2 ) allows our network to build standard basis of the polynomial vector spaces. Specifically, in this toy example it is easy to see that a sufficiently large -th layer can yield the standard basis of the vector space . This in turn implies that the subsequent layer will be able to generate polynomial functions at least up to the order. We keep track of the inherited level of expressivity with the node tag (in the example in figure ).
Figure 8: Heatmap comparison between the function (on the left) and the reconstructed data by a triadic spectral mlp with hidden dimension and number of layers (by including input and output) and trained with the Adam optimizer and fixed learning rate .
Figure 9: Regression of experimental data, opportunely rescaled, representing a spiking neural-like behaviour of a relaxation nonlinear circuit [ 23 , 24 ] . On the left, we can see the charge of a capacitor indicated as , fully reconstructed while on the right the regression of a spiking current denoted by . This fit has been carried out by implementing a triadic mlp with hidden dimension and number of layers (including input and output) and trained with the Adam optimizer, following a halving-lr-on-plateau scheduler. Here, self-coupling pairs are not included into the implementation.