Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensing

William Howes; Farid Ahmed; Syed Bahauddin Alam

cs.LGAug 25, 2026

Low-Latency Activation-Regularized Sparse Neural Operators with Distillation Assistance Towards Real-Time Neuromorphic Virtual Sensing

Authors: William Howes, Farid Ahmed, Syed Bahauddin Alam

Organizations: Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign, Urbana, IL, USA · National Center for Supercomputing Applications, Urbana, IL, USA

Abstract

Virtual sensing enables digital twins and safety-critical systems to reconstruct and forecast spatial-temporal physics in real time. However, conventional computational and data-driven methods often face challenges in generalization, latency, and energy efficiency for edge deployment. Neural operators offer a promising alternative but remain reliant on power-intensive hardware. Spiking neurons and neuromorphic computing can improve efficiency, yet surrogate-gradient training and multi-step spiking introduce convergence and latency challenges. We propose the Sparse-Activation-ReLU (SAR) layer, a single-step alternative that promotes activation sparsity without surrogate-gradient training while remaining compatible with event-based computing. Within a trunk-based NOMAD architecture, SAR achieves over a fivefold improvement in the combined Latency-Error-Energy (LEE) metric compared with Variable Spiking Neuron (VSN) and Leaky Integrate-and-Fire (LIF) implementations. We further analyze spiking entropy and feature usage and introduce synthetic knowledge distillation, reducing the LEE score by more than twofold. Finally, we improve VSN through a ReLU-based spiking loss and graph-neighbor thresholding. On the Heat Exchanger dataset, these approaches reduce L2 error by more than twofold and nearly sevenfold, respectively, while reducing spiking and spatial aggregation. Overall, the work presented is a step towards energy-efficient virtual sensing by providing an alternative framework that can be positioned towards neuromorphic or other edge device integration that can be a gold standard to compare latency, energy, and error performance for future efficient designs that are sparsity or brain-inspired spiking based.

Figures & tables

Figure 1: Sparse-Activation-ReLU Model (a) Visual demonstration of the Variable Spiking Neuron [ 17 ] operating after a linear fully-connected layer. Synaptic weights and unique bias terms accumulate the input for each neuron at spike time step $t$ which is aggregated into the neuron’s leaky memory, mirroring LIF dynamics. VSN replaces the binary spike output with an activation function operating on the neuron input. (b) Sparse-Activation-ReLU for a fully-connected linear layer. We apply a ReLU activation function to the linear layer output which only allows positive signals subsequently operated on by a chosen activation function. Such forward pass communication allows for surrogate-free ANN-to-neuromorphic conversion that allows for variable signals (unlike traditional binary spike rate to activation matching), included in the original VSN framework. (c) Sparse-Activation-ReLU for a generic input (spectral/spatial convolution or norm). We apply a ReLU activation function to the input subtracted by an optional threshold utilized for better spiking control. Similar to the linear layers, the ReLU only allows positive signals subsequently operated on by a chosen activation function.

Figure 2: Sparse-Activation-ReLU Nonlinear Manifold Decoder for Operator Learning Architecture of SAR-NOMAD specifically for the 2D Heat Exchanger. In green, exists every SAR layer which naturally replaces the ReLU layers. Since every computation before SAR is a simple linear layer, we do not include a threshold parameter which is redundant in the presence of the bias term. As shown, the full-field output provides the $L_{2}$ error for the total objective function while the individual sparse activation outputs of the ReLU (from all SAR layers in the branch, trunk and combined networks not fully depicted in the figure) with the Hoyer/ $L_{1}$ loss term provide the control on energy efficiency.

Figure 3: Sparse-Activation-ReLU Graph Neural Operator Architecture of SAR-GNO for the 2D Heat Exchanger. In green, exists every SAR layer included in the neural operator which does not utilize the optional threshold parameter. The green SAR layers replace either identity mappings or ReLU layers. Components in dark purple represent SAR with the optional threshold parameters $\boldsymbol{\tau}$ included for improved spiking control since they follow a normalization layer and non-linear computational layers. These threshold layers replace an identity mapping (spatial) and GeLU (spectral). SAR-GNO takes in boundary input with the input embedding mapping $M$ and combines it with the geometry coordinates to produce an input for the latent projection mapping $P$ . Subsequent spectral-spatial blocks (10 layers total) provide global and local analysis that is combined through a collaboration layer $f$ . Final a downlift layer $Q$ provides the final full-field reconstructed multi-physics output.

Figure 4: Synthetic Distillation We show the synthetic distillation framework utilized for neuromorphic virtual sensing. A graph-based VIRSO model, not native for neuromorphic hardware due to high connectivity and difficult integration, generates synthetic Heat Exchanger examples by randomly sampling input parameters for the Heat Exchanger dataset. These synthetic examples are compared against SAR-NOMAD’s predictions, a more neuromorphic friendly model, allowing for improved $L_{2}$ error performance in SAR-NOMAD, ideally keeping efficiency the same.

Figure 5: 2D Heat Exchanger (a) We define the 2D Heat Exchanger geometry which is a cross-sectional slice of a 3D grid at axial position

x=0.789

with a dimpled wall surface and tape insert utilized for better heat transfer, creating large vortex flow behavior and an irregular geometry with reduced symmetry (b) The boundary input utilized for our 2D Heat Exchanger. It includes a wall heat profile defined along the axial direction of the original 3D geometry and two inlet quantities: temperature and axial velocity/speed. (c) The 50th percentile test example for the SAR-NOMAD model with the Hoyer loss term and

\gamma=0.005

. We see that the main physics and vortex behavior is closely captured with mostly stochastic absolute error behavior and only small absolute error hotspots located around the wall surface.

Figure 6: Spiking Activity Results We depict the probability/frequency of different feature dimensions spiking for the SAR-NOMAD model with the Hoyer loss and $\gamma=0.005$ . For the branch layers, the sparsity regularization forces only a small subset, as a low as one dimension, to fire and communication information, indicating that the model is forced to collapse it feature dimensions in order to provide low spiking. The trunk and combined subnetworks have a lot more activity across the entire dimension.

Figure 7: 2D Lid-Driven Cavity (a) We define the boundary conditions for the Lid-Driven Cavity geometry which shows zero velocity on all faces except for a temporal signal at the top surface. (b) We display example time-varying velocity profiles for the top surface which is defined for 90 time steps. (c) The 50th percentile test example for the SAR-NOMAD model with the Hoyer loss term and $\gamma=0.05$ . We see that the main vortex and boundary behavior is closely captured as well as the corner eddies with mostly stochastic absolute error behavior. We do see small hotspots of absolute error located at the top surface and corners indicating some difficulty in capturing surface physics with a regular discretized grid.

Explore similar work

Neuroscience Inspired Graph Operators Towards Edge-Deployable Virtual Sensing for Irregular Geometries
Apr 17, 2026William Howes, Farid Ahmed, Kazuma Kobayashi +2Neuromorphic Computing Graph Neural Networks
Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Applied Mechanics, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi +1
When Spike Sparsity Does Not Translate to Deployed Cost: VS-WNO on Jetson Orin Nano
Apr 18, 2026Jason Yoo, Shailesh Garg, Souvik Chakraborty +1Nvidia Jetson Orin Nano Neuromorphic Computing
University of Illinois Urbana-Champaign Department of Nuclear, Plasma & Radiological Engineering Urbana, IL, USA · Indian Institute of Technology Delhi Department of Applied Mechanics New Delhi, India · National Center for Supercomputing Applications Urbana, IL, USA
EdgeSpike: Spiking Neural Networks for Low-Power Autonomous Sensing in Edge IoT Architectures
Apr 29, 2026Gustav Olaf Yunus Laitinen-Fredriksson Lundstrom-Imanov, Taner YilmazSpiking Neural Networks Neuromorphic Computing
Department of Economics, Stockholm University, Universitetsvägen 10 A, SE-106 91 Stockholm, Sweden · Department of Computer Engineering, Afyon Kocatepe University, 03200 Afyonkarahisar, Türkiye

Apr 17, 2026, cs.LGJ/K move · Enter open · S save

Neuroscience Inspired Graph Operators Towards Edge-Deployable Virtual Sensing for Irregular Geometries

William Howes, Farid Ahmed, Kazuma Kobayashi, Souvik Chakraborty+1

Grainger College of Engineering, Nuclear, Plasma & Radiological Engineering Department, University of Illinois Urbana-Champaign, Urbana, IL, USA · Department of Applied Mechanics, Indian Institute of Technology Delhi, New Delhi, India · Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi+1

Neuromorphic Computing Graph Neural Networks Collaborative Sensing Neuroscience

Open paper AskPDF arXiv

Predicting full-field physics through the real-time virtual sensing of engineering systems can enhance limited physical sensors but often requires sparse-to-dense reconstruction, complex multiphysics, and highly irregular geometries as well as strict latency and energy constraints for edge-deployability. Neural operators have been presented as a potential candidate for such applications but few architectures exist that explicitly address power consumption. Spiking neuron integration can provide a potential solution when integrated on neuromorphic hardware but the current existing neuron models result in severe performance degradation towards regression-based virtual sensing. To address the performance concerns and edge-constraints, we present the Variable Spiking Graph Neural Operator (VS-GNO) which integrates a sophisticated spectral-spatial convolutional analysis and a previously developed Variable Spiking Neuron (VSN) and energy-error balance loss function. With a non-spiking

L_2

error baseline of

0.4\%

, VS-GNO can provide a reconstruction error of

0.71\%

with

15\%

average spiking in its spectral-only form and

1.04\%

with

24.5\%

spiking in its entire form. These results position VS-GNO as a promising step towards energy-efficient, edge-deployable neural operators for real-time sparse-to-dense virtual sensing in complex, highly irregular engineering environments.

Abstract

Explore similar work

Neuroscience Inspired Graph Operators Towards Edge-Deployable Virtual Sensing for Irregular Geometries

When Spike Sparsity Does Not Translate to Deployed Cost: VS-WNO on Jetson Orin Nano

EdgeSpike: Spiking Neural Networks for Low-Power Autonomous Sensing in Edge IoT Architectures