Spectral Higher-Order Neural Networks

Gianluca Peri; Timoteo Carletti; Duccio Fanelli; Diego Febbe

cs.LGMar 30, 2026

Spectral Higher-Order Neural Networks

Authors: Gianluca Peri, Timoteo Carletti, Duccio Fanelli, Diego Febbe

Organizations: University of Florence, Italy · University of Namur, Belgium

Abstract

Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.

Figures & tables

Panel of Figure 1 — Figure 1: Cartoon representing different neural architectures. The standard neural networks (panel 1(a) ) with input neurons, $x_{i}$ , generating output signals, $y_{k}$ , via weighted averages, i.e., linear combinations (colored arrows) followed by the application of a local nonlinearity (not shown). The standard triadic higher-order network (panel 1(b) ) is obtained by adding to the previous architecture, weighted sums of hyperlinks, mimicking the two body interaction $x_{i_{1}}x_{i_{2}}$ (symbolized by the curved arrows connecting a couple $(x_{i_{1}},x_{i_{2}})$ to an output $y_{k}$ , each pair with its own specific color). The spectral higher-order networks (panel 1(c) ), reduces the number of used parameters by exploiting the spectral decomposition. This yields an effective parameter sharing among hyperlinks (the curved arrows connecting a couple $(x_{i_{1}},x_{i_{2}})$ to several outputs $y_{k}$ , share the same color).

Panel of Figure 2 — Figure 2: Results of the perceptrons’ training on mnist and fashion-mnist (via Adam optimizer). We performed learning rate warm-up , followed by a reduce-learning-rate-on-plateau protocol, to guard against a possible dependence of the results on an unlucky hyperparameter choice. The direct space triadic model is plagued by instability and confidence saturation problems, while the standard perceptron lacks in expressivity. At variance, the spectral triadic model (i) achieves the same performances of the standard triadic network, with a substantially more efficient parameter scaling and (ii) it is also way more stable.

Figure 3: Standard mlp vs. spectral triadic mlp on CIFAR-10. The models were trained with the Adam optimizer, following a halving-lr-on-plateau scheduler. The results show a clear advantage for the triadic architecture.

Figure 4: Standard mlp-mixer vs a spectral triadic version of it, on CIFAR-10. For this experiment the Adam optimizer was used with fixed learning rate. From the results, it seems that the spectral forward propagation not only shows a sufficient degree of expressivity, but also acts as an implicit regularizer, preventing overfitting.

Panel of Figure 5 — Figure 5: Graphical sketch of Lemma 4.1 , illustrating the inter-layer connections $(1,x)\to\bm{y}$ described in Eq. ( 21 ), under the triangular ansatz and in the simple setting where the only nonlinear interactions arise from the last column of matrix $W$ . Subsequently, the elements of $\bm{y}$ can be linearly remapped to the canonical basis of $\mathcal{P}_{2}(x)$ . The black dashed arrows represent the linear passage corresponding to the first sum term of Eq. ( 15 ), while the orange solid lines represent the higher-order interactions, namely second sum in Eq. ( 15 ), here just restricted to $x$ and itself.

Panel of Figure 6 — Figure 6: Graphical sketch of the ladder demonstration scheme proposed in Lemmas 4.2 , 4.3 . Here the input $(1,x_{1},x_{2})$ is provided to the honn with output $(y_{0},y_{1},y_{2},y_{3})$ , which can be linearly remapped, via $W^{-1}$ , to the new canonical basis augmented with a new nonlinear term, for instance $x_{1}x_{2}$ . The procedure then continues by creating new terms: $x_{1}^{2}$ , $x_{2}^{2}$ , and so on. As in Fig. 5 , dashed black lines indicate linear transfer, while orange ones indicate nonlinear interactions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Graphical sketch of the universality proof for triadic multilayer perceptrons. The structure of the forward pass ( 2 ) allows our network to build standard basis of the polynomial vector spaces. Specifically, in this toy example it is easy to see that a sufficiently large $k$ -th layer can yield the standard basis of the vector space $\mathcal{P}_{2^{k-1}}(\mathbb{R})$ . This in turn implies that the subsequent layer will be able to generate polynomial functions $f:\mathbb{R}\to\mathbb{R}^{n}$ at least up to the $2^{k-1}$ order. We keep track of the inherited level of expressivity with the $p^{(\circ+)}$ node tag (in the example in figure $k=3$ ).

Figure 8: Heatmap comparison between the function $f(x,y)=x^{4}+y^{4}+2x^{2}y^{2}-2x^{2}$ (on the left) and the reconstructed data by a triadic spectral mlp with hidden dimension $h_{dim}=20$ and $n_{\ell}=3$ number of layers (by including input and output) and trained with the Adam optimizer and fixed learning rate .

Panel of Figure 9 — Figure 9: Regression of experimental data, opportunely rescaled, representing a spiking neural-like behaviour of a relaxation nonlinear circuit [ 23 , 24 ] . On the left, we can see the charge of a capacitor indicated as $x(t)$ , fully reconstructed while on the right the regression of a spiking current denoted by $y(t)$ . This fit has been carried out by implementing a triadic mlp with hidden dimension $h_{dim}=20$ and $n_{\ell}=20$ number of layers (including input and output) and trained with the Adam optimizer, following a halving-lr-on-plateau scheduler. Here, self-coupling pairs are not included into the implementation.

Neural Network Stability

See all 37 papers

Jul 21, 2026

Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds

Neural hypergraphs are a natural generalization of neural networks, the reference models in modern machine learning. Yet, their deployment has proven demanding: the number of weighted hyperedges required leads to an intractable parameter explosion. However, a novel parametrization that leverages spectral attributes for neural hypergraphs has been recently proposed, that enables to recycle parameters via a weight sharing scheme and consequently yields a significant reduction of the associated computational cost. Preliminary tests carried out on spectral higher-order architectures pointed to meaningful improvements in both performance and interpretability. Building on these results, we advance the benchmarking efforts by evaluating the spectral higher order framework on N-bit parity tasks, a well-established testbed known to be particularly challenging. As we will convincingly argue, Spectral Higher-Order Neural Networks (SHONNs) possess a versatile and highly tunable hypothesis space.

Gianluca Peri, Diego Febbe, Duccio Fanelli

Department of Physics, University of Florence & INFN, Italy

Hypergraph Neural Networks Neural Network Expressivity

Ask PDF

Mar 25, 2026

Circuit-Inspired High-Order Neural Networks with Unified Neural Dynamics Modeling for PDE Solving and Visual Perception

Deep networks often rely on architectural heuristics to shape representation evolution, limiting their ability to model data governed by intrinsic dynamics. We present the Circuit-inspired High-Order Neural Network (CHONN), a modular framework that treats representation evolution as a latent potential process and increases its effective order through Kirchhoff-inspired cascade composition. A single Kirchhoff Neural Cell implements a stable first-order update, while serially composed cells form higher-order dynamical operators within one block. This construction is interpretable, numerically stable and compatible with common neural backbones. Theoretical analysis shows that cascaded cells induce end-to-end high-order operators, and controlled experiments demonstrate that intra-block high-order construction differs from generic depth stacking, especially on derivative-sensitive measures. Across steady-state operator learning, long-horizon physical forecasting and ImageNet-1K recognition, CHONN improves structural fidelity, rollout stability and visual representation learning. These results identify high-order circuit composition as a general principle for neural dynamics modeling.

Tongfei Chen, Jingying Yang, Linlin Yang +8

School of Artificial Intelligence (Institute of Artificial Intelligence), Beihang University, Beijing, China. · Communication University of China, Beijing, China. · School of Automation Science and Electrical Engineering, Beihang University, Beijing, China. +3

Dynamic Neural Networks PDE Operator Learning Neural Operators

Ask PDF

Jun 26, 2026

Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs

Neural operators provide deep neural networks for learning mappings between function spaces. Among them, the Fourier Neural Operator (FNO) is particularly effective: its spectral convolution relies on low-dimensional Fourier-domain representations and can handle inputs at different resolutions. This design aligns well with settings where the Fourier basis diagonalizes the underlying operator, such as linear, constant-coefficient PDEs on periodic domains, in which Fourier modes evolve independently. However, nonlinear PDEs may benefit from an additional inductive bias, as they exhibit structured interactions between modes, governed by polynomial nonlinearities. To capture this inductive bias, we introduce the Higher-Order Spectral Convolution, a spectral mixer that extends FNO from diagonal modulation to explicit n-linear mode mixing, aligned with the dynamics of nonlinear PDEs. Our experiments on standard benchmarks show that the proposed Higher-Order FNO (HO-FNO) retains the efficiency of FNO-based architectures and consistently improves over other spectral neural operators. HO-FNO also performs on par with or better than state-of-the-art transformers and state-space models on several datasets, with stronger gains in highly nonlinear regimes, such as the Poisson equation with polynomial forcing, where a single HO-FNO layer outperforms FNO models with up to 16 layers. We open-source our code for reproducibility at: https://github.com/AlexColagrande/HO-FNO.

Alex Colagrande, Paul Caillon, Eva Feillet +1

Miles Team, LAMSADE, Université Paris Dauphine-PSL, Paris, France · Université Paris-Saclay, CNRS, LISN, 91400, Orsay, France · ESPCI PSL, Paris, France

Fourier Neural Operator Neural PDE Solvers PDE Operator Learning

Ask PDF

Abstract

Neural Network Stability

Explore similar work

Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds

Circuit-Inspired High-Order Neural Networks with Unified Neural Dynamics Modeling for PDE Solving and Visual Perception

Higher-Order Fourier Neural Operator: Explicit Mode Mixer for Nonlinear PDEs