Circuit-Inspired High-Order Neural Networks with Unified Neural Dynamics Modeling for PDE Solving and Visual Perception
Authors: Tongfei Chen, Jingying Yang, Linlin Yang, Juan Zhang, Jinhu Lü, David Doermann, Chunyu Xie, Long He, Tian Wang, Guodong Guo, Baochang Zhang
Organizations: School of Artificial Intelligence (Institute of Artificial Intelligence), Beihang University, Beijing, China. · Communication University of China, Beijing, China. · School of Automation Science and Electrical Engineering, Beihang University, Beijing, China. · Department of Computer Science and Engineering, University at Buffalo, Buffalo, New York, USA. · AI Research, Qihoo 360, Beijing, China. · Eastern Institute of Technology, Ningbo, Ningbo, China.
Deep networks often rely on architectural heuristics to shape representation evolution, limiting their ability to model data governed by intrinsic dynamics. We present the Circuit-inspired High-Order Neural Network (CHONN), a modular framework that treats representation evolution as a latent potential process and increases its effective order through Kirchhoff-inspired cascade composition. A single Kirchhoff Neural Cell implements a stable first-order update, while serially composed cells form higher-order dynamical operators within one block. This construction is interpretable, numerically stable and compatible with common neural backbones. Theoretical analysis shows that cascaded cells induce end-to-end high-order operators, and controlled experiments demonstrate that intra-block high-order construction differs from generic depth stacking, especially on derivative-sensitive measures. Across steady-state operator learning, long-horizon physical forecasting and ImageNet-1K recognition, CHONN improves structural fidelity, rollout stability and visual representation learning. These results identify high-order circuit composition as a general principle for neural dynamics modeling.
Figures & tables
Figure 1: CHONN builds high-order representation evolution through Kirchhoff-inspired cascade composition. a, Neural-population analogy motivating the separation of connection, intensity and evolution in signal propagation. b, Positional encodings introduce variation across positions through externally supplied index-dependent cues. c, Recurrent state models introduce an internal state, but a single transition corresponds to a first-order update. d, CHONN composes circuit-inspired latent updates in series, allowing representation dynamics to accumulate across stages. e, A Kirchhoff Neural Cell (KNC) represents the hidden feature as a latent potential driven by input injection, leakage and coupling, followed by output readout. f, Cascading KNCs yields second- and higher-order evolution factors, making the order of latent representation dynamics explicit within one block.
Figure 2: Architectural instantiation of CHONN and its integration into neural-operator and encoder–decoder backbones. a, Structure of a single Kirchhoff Neural Cell (KNC). The hidden representation is modeled as a latent potential v(t) , where input injection, leakage dissipation and output readout jointly define a first-order latent evolution unit. b, Cascaded Kirchhoff Block (CKB). Multiple KNCs are connected in series, so that the output of each stage drives the next stage and progressively enriches the effective order of the latent dynamics. c, Trainable realization of an N -stage CKB. The serial KNC cascade is embedded within a residual gated block together with normalization, projection, local mixing and nonlinear activation, making the high-order evolution module compatible with deep neural architectures. d, FNO with CHONN . CKB modules are inserted alongside spectral and convolutional transformations in the Fourier neural operator backbone, introducing explicit latent evolution into operator learning. e, U-Net with CHONN . CKB modules are placed within the encoder–decoder hierarchy, including deep downsampling, bottleneck and upsampling stages, to enrich multi-scale feature evolution. Together, these instantiations show that CHONN functions as a modular plug-in mechanism for introducing intrinsic high-order representation evolution into different backbone architectures.
Figure 3: Explicit high-order construction improves structural fidelity and changes the scaling path. a, Comparison between CHONN and a depth-stacked first-order baseline on the order-controlled benchmark. Target operators T1–T4 correspond to first- to fourth-order instances of the operator family. For each target order, physical-domain, spectral-domain and derivative relative L2 errors are reported across model orders or baseline depths 1–4. CHONN shows more consistent improvement with increasing order, with the clearest separation from the depth-stacked baseline appearing in the derivative-sensitive metric. b, Relative reduction in derivative error achieved by CHONN over pure first-order depth stacking. Larger reductions are observed for higher target orders and stronger model configurations, indicating that explicit intra-block high-order construction is not equivalent to simply increasing the number of first-order blocks. c, Parameter–performance scaling comparison between pure first-order depth scaling (D1–D5) and stacked high-order CHONN blocks ( 2×k and 3×k ). High-order block stacking follows a more favourable parameter–error trajectory, particularly for derivative-sensitive errors, suggesting that first increasing the order within each block and then stacking such blocks is more effective than continuing to deepen a purely first-order network.
Figure 4: CHONN improves steady-state neural operator learning on Darcy flow and Poisson problems. a, Physical illustration of the Darcy-flow problem, where a spatially heterogeneous permeability field k(x,y) determines the pressure field u(x,y) and the resulting flow through the porous medium. b, Representative source term f(x,y) and solution field u(x,y) for the Poisson equation, illustrating the steady-state mapping from a prescribed spatial forcing field to the corresponding solution. c, Multi-metric quantitative comparison on the Darcy benchmark. Across RMSE, nRMSE, CSV, maximum error, boundary discrepancy, and frequency-band errors, CHONN consistently achieves lower error than FNO and U-Net. d, Relative gain of CHONN over the stronger Darcy baseline for each metric. The largest improvements appear in boundary-sensitive and frequency-related measures, indicating that the proposed model improves not only global accuracy but also structural fidelity. e, Qualitative Darcy predictions and residual maps on representative samples. CHONN better preserves the global pressure structure and reduces localized artifacts, yielding cleaner absolute- and relative-error patterns than baseline models. f, Training loss curves on the Darcy task. CHONN converges more rapidly and reaches a substantially lower loss floor, suggesting that the Kirchhoff-inspired representation leads to a more stable and efficient optimization process. g, Cascade-stage ablation on the Poisson equation. Increasing the number of passes improves overall solution-field accuracy with only moderate growth in model size, while varying cascade depths yield distinct trade-offs among relative L1 error, nRMSE, and Max MSE. Together, these results show that circuit-inspired high-order construction improves prediction accuracy, structural consistency, optimization stability, and scaling efficiency across representative steady-state PDE benchmarks.
Figure 5: Circuit-inspired high-order construction improves spatiotemporal forecasting, long-horizon rollout stability and visual recognition. a, Representative shallow-water evolution from an initial localized height disturbance at t=0 to the propagated wave field at a later time t , illustrating the spatiotemporal forecasting task. b, Multi-metric comparison on the shallow-water benchmark. CHONN is compared with U-Net and FNO across RMSE, nRMSE, CSV, maximum error, boundary discrepancy and frequency-related errors, including low-, middle- and high-frequency components. c, Relative gain of CHONN over the stronger shallow-water baseline for each metric, showing consistent improvements in global reconstruction, boundary consistency and frequency-sensitive structural fidelity. d, Representative Navier–Stokes vorticity fields at the beginning and end of a rollout sequence, showing the transition from an initial field to a more complex flow structure. e, Average 40-step rollout relative L2 error on Navier–Stokes dynamics under different input-history settings ( 3→1 , 5→1 and 10→1 ). CHONN consistently yields lower rollout error than FNO. f, Per-step rollout error curves for FNO and CHONN under different input-history settings. CHONN shows lower error accumulation over the rollout horizon, with a clearer advantage in later prediction steps. g, Representative natural-image samples from ImageNet-1K. h, Accuracy–complexity comparison on ImageNet-1K, with Top-1 accuracy plotted against FLOPs for representative vision backbones. CHONN reaches a favourable accuracy–complexity trade-off compared with ConvNeXt (CNX), Swin Transformer (ST), VMamba (VM), MambaVision (MV) and EfficientVMamba (EVM). i, Ablation of routing topology and circuit constraints. Series cascade yields larger gains than parallel expansion, and adding Kirchhoff-inspired constraints further improves accuracy. j, Training efficiency at 30 epochs. CHONN achieves higher early-stage Top-1 accuracy than VMamba across Tiny, Small and Base settings. Together, these results show that Kirchhoff-inspired high-order dynamics improve spatiotemporal field prediction, long-horizon rollout stability and visual representation learning across different task families.
Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.
Neural hypergraphs are a natural generalization of neural networks, the reference models in modern machine learning. Yet, their deployment has proven demanding: the number of weighted hyperedges required leads to an intractable parameter explosion. However, a novel parametrization that leverages spectral attributes for neural hypergraphs has been recently proposed, that enables to recycle parameters via a weight sharing scheme and consequently yields a significant reduction of the associated computational cost. Preliminary tests carried out on spectral higher-order architectures pointed to meaningful improvements in both performance and interpretability. Building on these results, we advance the benchmarking efforts by evaluating the spectral higher order framework on N-bit parity tasks, a well-established testbed known to be particularly challenging. As we will convincingly argue, Spectral Higher-Order Neural Networks (SHONNs) possess a versatile and highly tunable hypothesis space.
Gianluca Peri, Diego Febbe, Duccio Fanelli
Department of Physics, University of Florence & INFN, Italy
We introduce Dynamical Physics-Modeled Neural Networks (DynPMNNs), a continuous-time deep learning architecture in which each hidden layer is defined as the solution of an ordinary differential equation. Unlike classical feed-forward networks, this approach replaces static activation functions with time-evolving dynamical systems, providing a biologically inspired interpretation of hidden-layer behavior and enabling the integration of physically meaningful models. The framework is rigorously grounded in Reproducing Kernel Banach Spaces (RKBSs), allowing DynPMNNs to be characterized as finite-dimensional solutions of an abstract training problem and revealing structural connections with standard neural networks. We present a concrete implementation based on the FitzHugh--Nagumo model for neuronal activation, where numerical ODE solvers are embedded into the computational graph via Euler-type schemes. Both network weights and dynamical parameters are trained jointly. Through experiments on the California Housing dataset, we compare DynPMNNs with Neural ODEs (NODEs) and Closed-form Continuous-Time Networks (CfCs). Despite using fewer trainable parameters, DynPMNNs achieve competitive performance. These results position DynPMNNs as a principled bridge between dynamical systems and deep learning, with promising directions for further research in expressivity, stability, and physics-based modeling.
Raul Felipe-Sosa, Angel Martin del Rey, Maria Flores Ceballos
aDepartamento de Matemáticas, Escuela Politénica Superior, Universidad Nebrija, 28015 Madrid, Spain · bDepartment of Applied Mathematics, IUFFyM, Universidad de Salamanca, 37008 Salamanca, Spain · cUnidad de Excelencia en Luz y Materia Estructuradas (LUMES), Universidad de Salamanca, 37008 Salamanca, Spain