Efficient Neural Network Inference

Ask about this topic

Overview

The field studies how to run trained neural networks with less latency, memory, and energy, from phones and microcontrollers to GPUs, FPGAs, and large serving clusters. Common approaches include quantization, pruning, distillation, early exits and conditional computation, alongside compiler and kernel optimization, memory-aware scheduling, and specialized or neuromorphic hardware. Work spans vision, language, graphs, and streaming tasks, with evaluation increasingly measuring end-to-end latency, throughput, and energy on target devices rather than relying only on parameter counts or FLOPs. A central tension is that theoretical savings such as sparsity or fewer operations do not always yield faster real deployments, and efficiency gains must be balanced against accuracy, reliability, and hardware constraints.

Latest papers

CardsList

471 papers46 papers in the last four weeks, up 7% on the four weeks before. 0.3% of all new papers.

Apr 19, 2026

Fully Analog Resonant Recurrent Neural Network via Metacircuit

Physical neural networks offer a transformative route to edge intelligence, providing superior inference speed and energy efficiency compared to conventional digital architectures. However, realizing scalable, end-to-end, fully analog recurrent neural networks for temporal information processing remains challenging due to the difficulty of faithfully mapping trained network models onto physical hardware. Here we present a fully analog resonant recurrent neural network (R2^2NN) implemented via a metacircuit architecture composed of coupled electrical local resonators. A reformulated mechanical-electrical analogy establishes a direct mapping between the R2^2NN model and metacircuit elements, enabling accurate physical implementation of trained neural network parameters. By integrating jointly trainable global resistive coupling and local resonances, which generate effective frequency-dependent negative resistances, the architecture shapes an impedance landscape that steers currents along frequency-selective pathways. This mechanism enables direct extraction of discriminative spectral features, facilitating real-time temporal classification of raw analog inputs while bypassing analog-to-digital conversion. We demonstrate the cross-domain versatility of this framework using integrated hardware for tactile perception, speech recognition, and condition monitoring. This work establishes a scalable, fully analog paradigm for intelligent temporal processing and paves the way for low-latency, resource-efficient physical neural hardware for edge intelligence.

Zixin Zhou, Tianxi Jiang, Menglong Yang +3

Institute of Humanoid Robots, CAS Key Laboratory of Mechanical Behavior and Design of Materials, Department of Precision Machinery and Precision Instrumentation, University of Science and Technology of China, Hefei 230026, People’s Republic of China · State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai 200240, People’s Republic of China

Efficient Neural Network InferenceTime Series ClassificationRecurrent Neural NetworksNeuromorphic ComputingEdge MLPhysical Neural Networks
AskPDF
Apr 19, 2026

Revisiting Auxiliary Losses for Conditional Depth Routing: An Empirical Study

Conditional depth execution routes a subset of tokens through a lightweight cheap FFN while the remainder execute the standard full FFN at each controlled layer. The central difficulty is gate training: the gate decision must propagate through many layers before it influences the language modeling (LM) loss, so the resulting gradients are weak and noisy. Auxiliary losses are commonly stacked to stabilise training, yet the interactions among them -- particularly between a predictive auxiliary and explicit score supervision -- have not been systematically compared under controlled conditions. We evaluate two gate designs under a 157.5M-parameter decoder-only model with controller-only training, 50% full-path budget, and 3-seed runs on a fineweb-edu subset. The MLP gate (G1) maps the current hidden state to a utility score; the JEPA-guided gate (G3) adds an action-conditional predictor that forecasts, in a low-dimensional latent space, the outcome of executing full vs. cheap per token, aligned against a fixed target head. Under the standard recipe with oracle-style utility regression and pairwise rank supervision (util/rank), G3 improves early-to-mid optimisation over G1 in 3/3 seeds (lower avg LM, faster threshold hits, ~10.3x lower grad norms), with 20k-step endpoint LM within a 0.005 heuristic reference. A key finding (ablation A3): jointly removing util/rank improves best/avg LM and threshold-hit speed in 3/3 seeds for both gates, and the early-to-mid advantage of G3 over G1 disappears. We trace this to an off-policy oracle label that assumes all subsequent layers execute full, whereas gated execution routes only a fraction through full -- making util/rank net-negative under the current recipe. Removing util/rank also cuts the training FLOPs proxy from ~1.53x to ~1.07x full-only (2.87h to 1.75h on a V100-32GB, ~39%). Conclusions are scoped to the studied regime.

Qingwei Lin

Independent Researcher

Efficient Neural Network InferenceConditional Computation
AskPDF
Apr 18, 2026

When Spike Sparsity Does Not Translate to Deployed Cost: VS-WNO on Jetson Orin Nano

Spiking neural operators are appealing for neuromorphic edge computing because event-driven substrates can, in principle, translate sparse activity into lower latency and energy. Whether that advantage survives deployment on commodity edge-GPU software stacks, however, remains unclear. We study this question on a Jetson Orin Nano 8 GB using five pretrained variable-spiking wavelet neural operator (VS-WNO) checkpoints and five matched dense wavelet neural operator (WNO) checkpoints on the Darcy rectangular benchmark. On a reference-aligned path, VS-WNO exhibits substantial algorithmic sparsity, with mean spike rates decreasing from 54.26% at the first spiking layer to 18.15% at the fourth. On a deployment-style request path, however, this sparsity does not reduce deployed cost: VS-WNO reaches 59.6 ms latency and 228.0 mJ dynamic energy per inference, whereas dense WNO reaches 53.2 ms and 180.7 mJ, while also achieving slightly lower reference-path error (1.77% versus 1.81%). Nsight Systems indicates that the request path remains launch-dominated and dense rather than sparsity-aware: for VS-WNO, cudaLaunchKernel accounts for 81.6% of CUDA API time within the latency window, and dense convolution kernels account for 53.8% of GPU kernel time; dense WNO shows the same pattern. On this Jetson-class GPU stack, spike sparsity is measurable but does not reduce deployed cost because the runtime does not suppress dense work as spike activity decreases.

Jason Yoo, Shailesh Garg, Souvik Chakraborty +1

University of Illinois Urbana-Champaign Department of Nuclear, Plasma & Radiological Engineering Urbana, IL, USA · Indian Institute of Technology Delhi Department of Applied Mechanics New Delhi, India · National Center for Supercomputing Applications Urbana, IL, USA

Efficient Neural Network InferenceActivation SparsityEdge ComputingCost-Aware InferenceSpiking Neural NetworksNeural Operators
AskPDF
Apr 18, 2026

Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks

Privacy-preserving machine learning (PPML) has become increasingly important in applications where sensitive data must remain confidential. Homomorphic Encryption (HE) enables computation directly on encrypted data, allowing neural network inference without revealing raw inputs. While prior works have largely focused on inference over a single encrypted image, batch processing of encrypted inputs lags behind, despite being critical for high-throughput inference scenarios and training-oriented workloads. In this work, we address this gap by developing optimized algorithms for batched HE-friendly neural networks. We also introduced a pipeline architecture designed to maximize resource efficiency for different batch size execution. We implemented these algorithms and evaluated our work using HE-friendly ResNet-20 and ResNet-34 models on encrypted CIFAR-10 and CIFAR-100 datasets, respectively. For ResNet-20, our approach achieves an amortized inference time of 8.86 seconds per image when processing a batch of 512 encrypted images, with a peak memory usage of 98.96 GB. These results represent a 1.78x runtime improvement and a 3.74x reduction in memory usage compared to the state-of-the-art design. For the deeper ResNet-34 model, we achieve an amortized inference time of 28.14 on a batch of 256 encrypted images using 246.78GB of RAM

Nges Brian Njungle, Eric Jahns, Michel A. Kinsy

STAM Center, Ira A. Fulton Schools of Engineering · Arizona State University, Tempe, AZ 85281, USA

Efficient Neural Network InferenceMemory-Efficient InferenceHomomorphic EncryptionPrivacy-Preserving ML
AskPDF
Apr 18, 2026

EdgeVTP: Exploration of Latency-efficient Trajectory Prediction for Edge-based Embedded Vision Applications

Vehicle trajectory prediction is central to highway perception, but deployment on roadside edge devices necessitates bounded, deterministic end-to-end latency. We present EdgeVTP, an embedded-first trajectory predictor that combines interaction-aware graph modeling with a lightweight transformer backbone and a one-shot curve decoder. By predicting future motion as compact curve parameters (anchored at the last observed position) rather than horizon-scaled autoregressive waypoints, EdgeVTP reduces decoding overhead while producing smooth trajectories. To keep runtime predictable in crowded scenes, we explicitly bound interaction complexity via a locality graph with a hard neighbor cap. Across three highway benchmarks and two Jetson-class platforms, EdgeVTP achieves the lowest measured end-to-end latency under a protocol that includes graph construction and post-processing, while attaining state-of-the-art (SotA) prediction accuracy on two of the three datasets and competitive error on other benchmarks. Our code is available at https://github.com/SeungjinStevenKim/EdgeVTP.

Seungjin Kim, Reza Jafarpourmarzouni, Christopher Neff +2

University of Wyoming · University of North Carolina at Charlotte · North Carolina A&T State University

Efficient Neural Network InferenceEdge InferenceMulti-Agent Trajectory PredictionTrajectory PredictionVehicle Trajectory Prediction
AskPDF
Apr 17, 2026

Combining Convolution and Delay Learning in Recurrent Spiking Neural Networks

Spiking neural networks (SNNs) are rapidly gaining momentum as an alternative to conventional artificial neural networks in resource constrained edge systems. In this work, we continue a recent research line on recurrent SNNs where axonal delays are learned at runtime along with the other network parameters. The first proposed approach, dubbed DelRec, demonstrated the benefit of recurrent delay learning in SNNs. Here, we extend it by advocating the use of convolutional recurrent connections in conjunction with the DelRec delay learning mechanism. According to our tests on an audio classification task, this leads to a streamlined architecture with smaller memory footprint (around 99% savings in terms of number of recurrent parameters) and a much faster (52x) inference time, while retaining DelRec's accuracy. Our code is available at: https://github.com/luciozebendo/delrec_snn/tree/conv_delays

Lúcio Folly Sanches Zebendo, Eleonora Cicciarella, Michele Rossi

University of Padova, Department of Information Engineering · Department of Mathematics “Tullio Levi Civita” at the University of Padova, Italy

Convolutional RNNsEfficient Neural Network InferenceRecurrent Neural NetworksSpiking Neural Networks
AskPDF
Apr 17, 2026

MambaBack: Bridging Local Features and Global Contexts in Whole Slide Image Analysis

Whole Slide Image (WSI) analysis is pivotal in computational pathology, enabling cancer diagnosis by integrating morphological and architectural cues across magnifications. Multiple Instance Learning (MIL) serves as the standard framework for WSI analysis. Recently, Mamba has become a promising backbone for MIL, overtaking Transformers due to its efficiency and global context modeling capabilities originating from Natural Language Processing (NLP). However, existing Mamba-based MIL approaches face three critical challenges: (1) disruption of 2D spatial locality during 1D sequence flattening; (2) sub-optimal modeling of fine-grained local cellular structures; and (3) high memory peaks during inference on resource-constrained edge devices. Studies like MambaOut reveal that Mamba's SSM component is redundant for local feature extraction, where Gated CNNs suffice. Recognizing that WSI analysis demands both fine-grained local feature extraction akin to natural images, and global context modeling akin to NLP, we propose MambaBack, a novel hybrid architecture that harmonizes the strengths of Mamba and MambaOut. First, we propose the Hilbert sampling strategy to preserve the 2D spatial locality of tiles within 1D sequences, enhancing the model's spatial perception. Second, we design a hierarchical structure comprising a 1D Gated CNN block based on MambaOut to capture local cellular features, and a BiMamba2 block to aggregate global context, jointly enhancing multi-scale representation. Finally, we implement an asymmetric chunking design, allowing parallel processing during training and chunking-streaming accumulation during inference, minimizing peak memory usage for deployment. Experimental results on five datasets demonstrate that MambaBack outperforms seven state-of-the-art methods. Source code and datasets are publicly available.

Sicheng Chen, Chad Wong, Tianyi Zhang +3

Nhu Department of Electrical Engineering and Computer Science, University of California, Irvine · Department of Electrical & Computer Engineering, National University of Singapore · PuzzleLogic Pte Ltd, Singapore 229594, Singapore

Efficient Neural Network InferenceMultiple Instance LearningWhole-Slide Image AnalysisState Space Models
AskPDF
Apr 16, 2026

Towards Topology-Aware Very Large-Scale Photonic AI Accelerators

The rapid growth of deep neural networks (DNNs) has exposed fundamental limitations in electronic accelerators, where data movement dominates energy consumption, commonly referred to as the memory wall. Photonic accelerators offer a compelling alternative due to their inherent parallelism and high-speed matrix operations. However, existing research largely focuses on device-level innovations, leaving system-level scalability insufficiently explored. In this paper, we present a scalable photonic accelerator architecture based on a modular scale-out paradigm using 4 X 4 photonic tensor core units. We perform a systematic architectural analysis that incorporates the practical scaling limits of photonic hardware, including insertion loss, fanout penalties, and laser power limits, which restrict monolithic photonic scaling. Through evaluation on representative DNN workloads (GoogleNet, ResNet-18, MobileNet, and AlphaGo Zero) with up to 1024 processing elements, we identify a topology-dominated scaling bottleneck in the photonic domain, termed the Utilization Wall, where performance is governed by grid topology rather than hardware size. We further establish the Symmetric Grid Rule, demonstrating that symmetric topologies improve utilization by up to 6X while reducing memory access by over 40% compared to linear configurations, which reveal that topology-aware scaling is essential for achieving energy-efficient and high-performance photonic AI accelerators.

Belal Jahannia, Abdolah Amirany, Hamed Dalir

Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL, USA

Efficient Neural Network InferencePhotonic Computing
AskPDF
Apr 16, 2026

NeuroMesh: A Unified Neural Inference Framework for Decentralized Multi-Robot Collaboration

Deploying learned multi-robot models on heterogeneous robots remains challenging due to hardware heterogeneity, communication constraints, and the lack of a unified execution stack. This paper presents NeuroMesh, a multi-domain, cross-platform, and modular decentralized neural inference framework that standardizes observation encoding, message passing, aggregation, and task decoding in a unified pipeline. NeuroMesh combines a dual-aggregation paradigm for reduction- and broadcast-based information fusion with a parallelized architecture that decouples cycle time from end-to-end latency. Our high-performance C++ implementation leverages Zenoh for inter-robot communication and supports hybrid GPU/CPU inference. We validate NeuroMesh on a heterogeneous team of aerial and ground robots across collaborative perception, decentralized control, and task assignment, demonstrating robust operation across diverse task structures and payload sizes. We plan to release NeuroMesh as an open-source framework to the community.

Yang Zhou, Yash Shetye, Long Quang +8

New York University, New York, USA · U.S. Army Combat Capabilities Development Command, Army Research Laboratory, Adelphi, MD 20783, USA · Vanderbilt University, Tennessee, USA +1

Efficient Neural Network InferenceMulti-Robot SystemsMulti-Robot CoordinationMulti-Agent CommunicationDistributed Inference
AskPDF
Apr 16, 2026

KVNN: Learnable Multi-Kernel Volterra Neural Networks

Higher-order learning is fundamentally rooted in exploiting compositional features. It clearly hinges on enriching the representation by more elaborate interactions of the data which, in turn, tends to increase the model complexity of conventional large-scale deep learning models. In this paper, a kernelized Volterra Neural Network (kVNN) is proposed. The key to the achieved efficiency lies in using a learnable multi-kernel representation, where different interaction orders are modeled by distinct polynomial-kernel components with compact, learnable centers, yielding an order-adaptive parameterization. Features are learned by the composition of layers, each of which consists of parallel branches of different polynomial orders, enabling kVNN filters to directly replace standard convolutional kernels within existing architectures. The theoretical results are substantiated by experiments on two representative tasks: video action recognition and image denoising. The results demonstrate favorable performance-efficiency trade-offs: kVNN consistently yields reduced model (parameters) and computational (GFLOPs) complexity with competitive and often improved performance. These results are maintained even when trained from scratch without large-scale pretraining. In summary, we substantiate that structured kernelized higher-order layers offer a practical path to balancing expressivity and computational cost in modern deep networks.

Haoyu Yun, Hamid Krim, Yufang Bao

Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC, USA · Department of Mathematics and Computer Science, Fayetteville State University, Fayetteville, NC, USA

Efficient Neural Network InferenceKernel MethodsConvolutional Neural Networks
AskPDF
Apr 16, 2026

DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provide strong decoding performance but incur substantial overhead when retained as inference backbones, making knowledge distillation a natural route for optimizing compact students; and (ii) direct distillation from a fixed teacher representation underutilizes EFM knowledge, as task-discriminative information is distributed across intermediate layers rather than concentrated in the final layer. These observations motivate DLink (Distilling Layer-wise and Dominant Knowledge), a spectrally guided distillation framework with input-conditioned layer routing for transferring EFM knowledge into compact students. DLink uses a lightweight router to aggregate teacher layers for each input, and aligns magnitude and phase spectra to mitigate compression-induced spectral distortion in learned representations. The routed teacher knowledge is internalized by a project-then-compress student; the teacher and router are used only during training. Experiments on four EEG benchmarks show that DLink improves matched compact students and remains competitive with lightweight baselines, narrowing the gap to fine-tuned EFMs while substantially reducing parameters, FLOPs, and CPU-only inference latency.

Jingyuan Wang, Zhihao Jia, Chenyu Liu +7

1Xiamen University · 2Nanyang Technological University · Institute of Automation, Chinese Academy of Sciences +1

Efficient Neural Network InferenceEEG DecodingKnowledge DistillationEEG Foundation Models
AskPDF
Apr 16, 2026

Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference

Deploying high-quality automatic speech recognition (ASR) on edge devices requires models that jointly optimize accuracy, latency, and memory footprint while operating entirely on CPU without GPU acceleration. We conduct a systematic empirical study of state-of-the-art ASR architectures, encompassing encoder-decoder, transducer, and LLM-based paradigms, evaluated across batch, chunked, and streaming inference modes. Through a comprehensive benchmark of over 50 configurations spanning OpenAI Whisper, NVIDIA Nemotron, Parakeet TDT, Canary, Conformer Transducer, and Qwen3-ASR, we identify NVIDIA's Nemotron Speech Streaming as the strongest candidate for real-time English streaming on resource-constrained hardware. We then re-implement the complete streaming inference pipeline in ONNX Runtime and conduct a controlled evaluation of multiple post-training quantization strategies, including importance-weighted k-quant, mixed-precision schemes, and round-to-nearest quantization, combined with graph-level operator fusion. These optimizations reduce the model from 2.47 GB to as little as 0.67 GB while maintaining word error rate (WER) within 1% absolute of the full-precision PyTorch baseline. Our recommended configuration, the int4 k-quant variant, achieves 8.20% average streaming WER across eight standard benchmarks, running comfortably faster than real-time on CPU with 0.56 s algorithmic latency, establishing a new quality-efficiency Pareto point for on-device streaming ASR.

Nenad Banfic, David Fan, Kunal Vaishnavi +5

CoreAI, Microsoft

Efficient Neural Network InferenceStreaming ASRPost-Training QuantizationOn-Device Inference
AskPDF
Apr 15, 2026

SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces

Implicit neural representations are powerful for geometric modeling, but their practical use is often limited by the high computational cost of network evaluations. We observe that implicit representations require progressively lower accuracy as query points move farther from the target surface, and that even within the same iso-surface, representation difficulty varies spatially with local geometric complexity. However, conventional neural implicit models evaluate all query points with the same network depth and computational cost, ignoring this spatial variation and thereby incurring substantial computational waste. Motivated by this observation, we propose an efficient neural implicit geometry representation framework with spatially adaptive network depth (SAND). SAND leverages a volumetric network-depth map together with a tailed multi-layer perceptron (T-MLP) to model implicit representation. The volumetric depth map records, for each spatial region, the network depth required to achieve sufficient accuracy, while the T-MLP is a modified MLP designed to learn implicit functions such as signed distance functions, where an output branch, referred to as a tail, is attached to each hidden layer. This design allows network evaluation to terminate adaptively without traversing the full network and directs computational resources to geometrically important and complex regions, improving efficiency while preserving high-fidelity representations. Extensive experimental results demonstrate that our approach can significantly improve the inference-time query speed of implicit neural representations.

Chuanxiang Yang, Junhui Hou, Yuan Liu +5

Shandong University, China · City University of Hong Kong, China · Hong Kong University of Science and Technology, China +2

Implicit Neural RepresentationsEfficient Neural Network InferenceDynamic Neural Networks
AskPDF
Apr 15, 2026

Hardware-Efficient Neuro-Symbolic Networks with the Exp-Minus-Log Operator

Deep neural networks (DNNs) deliver state-of-the-art accuracy on regression and classification tasks, yet two structural deficits persistently obstruct their deployment in safety-critical, resource-constrained settings: (i) opacity of the learned function, which precludes formal verification, and (ii) reliance on heterogeneous, library-bound activation functions that inflate latency and silicon area on edge hardware. The recently introduced Exp-Minus-Log (EML) Sheffer operator, eml(x, y) = exp(x) - ln(y), was shown by Odrzywolek (2026) to be sufficient - together with the constant 1 - to express every standard elementary function as a binary tree of identical nodes. We propose to embed EML primitives inside conventional DNN architectures, yielding a hybrid DNN-EML model in which the trunk learns distributed representations and the head is a depth-bounded, weight-sparse EML tree whose snapped weights collapse to closed-form symbolic sub-expressions. We derive the forward equations, prove computational-cost bounds, analyse inference and training acceleration relative to multilayer perceptrons (MLPs) and physics-informed neural networks (PINNs), and quantify the trade-offs for FPGA/analog deployment. We argue that the DNN-EML pairing closes a literature gap: prior neuro-symbolic and equation-learner approaches (EQL, KAN, AI-Feynman) work with heterogeneous primitive sets and do not exploit a single hardware-realisable Sheffer element. A balanced assessment shows that EML is unlikely to accelerate training, and on commodity CPU/GPU it is also unlikely to accelerate inference; however, on a custom EML cell (FPGA logic block or analog circuit) the asymptotic latency advantage can reach an order of magnitude with simultaneous gain in interpretability and formal-verification tractability.

Eymen Ipek

Institute of Automotive Engineering, Graz University of Technology, Austria

Neurosymbolic LearningEfficient Neural Network InferenceNeural Network InterpretabilityFPGA-Based Neural Network AccelerationNeural Network Verification
AskPDF
Apr 15, 2026

A Resource-Efficient Hybrid CNN-LSTM network for image-based bean leaf disease classification

Accurate and resource-efficient automated diagnosis is a cornerstone of modern agricultural expert systems. While Convolutional Neural Networks (CNNs) have established benchmarks in plant pathology, their ability to capture long-range spatial dependencies is often limited by standard pooling layers, and their high memory footprint hinders deployment on portable devices. This paper proposes a lightweight hybrid CNN-LSTM system for bean leaf disease classification. By integrating an LSTM layer to model the spatial-sequential relationships within feature maps, our hybrid architecture achieves a 94.38% accuracy while maintaining an exceptionally small footprint of 1.86 MB; a 70% reduction in size compared to traditional CNN-based systems. Furthermore, we provide a systematic evaluation of image augmentation strategies, demonstrating that tailored transformations are superior to generic combinations for maintaining the integrity of diagnostic patterns. Results on the ibean\textit{ibean} dataset confirm that the proposed system achieves state-of-the-art F1 scores of 99.22% with EfficientNet-B7+LSTM, providing a robust and scalable framework for real-time agricultural decision support in resource-constrained environments. The code and augmented datasets used in this study are publicly available on this Github repo.

Hye Jin Rhee, Joseph Damilola Akinyemi

University of York, Heslington, York, YO10 5DD, United Kingdom

Plant Disease ClassificationLightweight CNNsEfficient Neural Network Inference
AskPDF
Apr 15, 2026

Cross-Layer Co-Optimized LSTM Accelerator for Real-Time Gait Analysis

Long Short-Term Memory (LSTM) neural networks have penetrated healthcare applications where real-time requirements and edge computing capabilities are essential. Gait analysis that detects abnormal steps to prevent patients from falling is a prominent problem for such applications. Given the extremely stringent design requirements in performance, power dissipation, and area, an Application-Specific Integrated Circuit (ASIC) enables an efficient real-time exploitation of LSTMs for gait analysis, achieving high accuracy. To the best of our knowledge, this work presents the first cross-layer co-optimized LSTM accelerator for real-time gait analysis, targeting an ASIC design. We conduct a comprehensive design space exploration from software down to layout design. We carry out a bit-width optimization at the software level with hardware-aware quantization to reduce the hardware complexity, explore various designs at the register-transfer level, and generate alternative layouts to find efficient realizations of the LSTM accelerator in terms of hardware complexity and accuracy. The physical synthesis results show that, using the 65 nm technology, the die size of the accelerator's layout optimized for the highest accuracy is 0.325 mm^2, while the alternative design optimized for hardware complexity with a slightly lower accuracy occupies 15.4% smaller area. Moreover, the designed accelerators achieve accurate gait abnormality detection 4.05x faster than the given application requirement.

Mohammad Hasan Ahmadilivani, Levent Aksoy, Mohammad Eslami +2

Tallinn University of Technology, Tallinn, Estonia

AI Accelerator InferenceEfficient Neural Network InferenceGait Analysis
AskPDF
Apr 14, 2026

Analog Optical Inference on Million-Record Mortgage Data

Analog optical computers promise large efficiency gains for machine learning inference, yet no demonstration has moved beyond small-scale image benchmarks. We benchmark the analog optical computer (AOC) digital twin on mortgage approval classification from 5.84 million U.S. HMDA records and separate three sources of accuracy loss. On the original 19 features, the AOC reaches 94.6% balanced accuracy with 5,126 parameters (1,024 optical), compared with 97.9% for XGBoost; the 3.3 percentage-point gap narrows by only 0.5pp when the optical core is widened from 16 to 48 channels, suggesting an architectural rather than hardware limitation. Restricting all models to a shared 127-bit binary encoding drops every model to 89.4--89.6%, with an encoding cost of 8pp for digital models and 5pp for the AOC. Seven calibrated hardware non-idealities impose no measurable penalty. The three resulting layers of limitation (encoding, architecture, hardware fidelity) locate where accuracy is lost and what to improve next.

Sofia Berloff, Pavel Koptev, Konstantin Malkov

Department of Mathematics, University of York, York, UK · ai1Technologies, 27 Beach Rd Suite 5A Monmouth Beach, NJ 07750, USA

Analog ComputingPhotonic ComputingEfficient Neural Network InferenceTabular MLCredit Scoring
AskPDF
Apr 12, 2026

Towards Green Wearable Computing: A Physics-Aware Spiking Neural Network for Energy-Efficient IMU-based Human Activity Recognition

Wearable IMU-based Human Activity Recognition (HAR) relies heavily on Deep Neural Networks (DNNs), which are burdened by immense computational and buffering demands. Their power-hungry floating-point operations and rigid requirement to process complete temporal windows severely cripple battery-constrained edge devices. While Spiking Neural Networks (SNNs) offer extreme event-driven energy efficiency, standard architectures struggle with complex biomechanical topologies and temporal gradient degradation. To bridge this gap, we propose the Physics-Aware Spiking Neural Network (PAS-Net), a fully multiplier-free architecture explicitly tailored for Green HAR. Spatially, an adaptive symmetric topology mixer enforces human-joint physical constraints. Temporally, an O(1)O(1)-memory causal neuromodulator yields context-aware dynamic threshold neurons, adapting actively to non-stationary movement rhythms. Furthermore, we leverage a temporal spike error objective to unlock a flexible early-exit mechanism for continuous IMU streams. Evaluated across seven diverse datasets, PAS-Net achieves state-of-the-art accuracy while replacing dense operations with sparse 0.1 pJ integer accumulations. Crucially, its confidence-driven early-exit capability drastically reduces dynamic energy consumption by up to 98%. PAS-Net establishes a robust, ultra-low-power neuromorphic standard for always-on wearable sensing. The source code and pre-trained models are publicly available at https://github.com/zhengnaichuan2022/PAS-Net.git.

Naichuan Zheng, Hailun Xia, Zepeng Sun +2

School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China · International School, Beijing University of Posts and Telecommunications, Beijing, China

Spiking Neural NetworksHuman Activity RecognitionEfficient Neural Network InferenceWearable SensingNeuromorphic Computing
AskPDF
Apr 11, 2026

Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge

Precise estimation of model inference latency is crucial for time-critical mobile edge applications, enabling devices to calculate latency margins against deadlines and trade them for enhanced model performance or resource savings. However, the ubiquity of Dynamic Voltage and Frequency Scaling (DVFS) renders traditional static profiling invalid in real-world deployments, as inference latency fluctuates with varying processor (CPU and GPU) frequencies. While extensive profiling across frequency combinations is theoretically possible, it is prohibitively expensive, particularly for emerging Small Language Models (SLMs), where variable context lengths explode the profiling up to days. We observe that simple analytic scaling fails to predict these fluctuations due to the complex asynchronous coupling between CPU (kernel launching) and GPU (execution). In this paper, we introduce FLAME to accurately estimate inference latency across frequency combinations. It features a novel layer-wise modeling that quantifies the overlapping parallelism and then aggregates dynamic pipeline bubbles caused by asynchronous processor interactions when extending to the full model. This bottom-up approach ensures generalizability across diverse models from DNNs to SLMs, and its precise modeling allows for profiling a sparse subset of samples, cutting DNN profiling from hours to minutes and SLM profiling from days to mere minutes, while maintaining small estimation errors across frequencies. We further showcase FLAME's utility in a deadline-aware DVFS, outperforming the state-of-the-art approach in both power efficiency and latency guarantees.

Jiesong Chen, Jun You, Zhidan Liu +1

Department of Computer Science, City University of Hong Kong, Hong Kong, China · INTR Thrust, System Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China

Edge InferenceEfficient Neural Network InferenceOn-Device Inference
AskPDF
Apr 11, 2026

MPNet: A Robust and Efficient Manifold Pooling Network for Multi-Rhythm EEG Signal Decoding

Deep Riemannian networks provide a powerful framework for Electroencephalography (EEG) decoding, but their practical applications are severely constrained. Accurately decoding EEG signals requires modeling complex temporal dynamics across multiple rhythms, which results in high-dimensional Riemannian inputs and significant computational costs. To address this, we propose the Manifold Pooling Network (MPNet). MPNet uses a rhythm-adaptive convolutional frontend to extract comprehensive time-frequency representations and generate multi-view Riemannian nodes. A novel manifold node pooling layer is then proposed to aggregate these nodes into a single fusion node with a fixed size, enabling the following deep Riemannian network to process it with greatly reduced costs. Experiments on two public EEG datasets show that MPNet achieves state-of-the-art accuracy, runs up to 10 times faster than the comparable Riemannian model, and maintains robust performance under limited-data conditions. These findings highlight MPNet's practicality and efficiency for real-world EEG applications.

Guoqing Cai, Kai Zeng, Shoulin Huang +1

Harbin Institute of Technology (Shenzhen), Shenzhen, China · Guangxi Normal University, Guilin, China · Peng Cheng Laboratory, Shenzhen, China

Riemannian Neural NetworksEEG DecodingEfficient Neural Network InferenceManifold Learning
AskPDF
Apr 11, 2026

Engineering Resource-constrained Software Systems with DNN Components: a Concept-based Pruning Approach

Deep Neural Networks (DNNs) are widely used by engineers to solve difficult problems that require predictive modeling from data. However, these models are often massive, with millions or billions of parameters, and require substantial computational power, RAM, and storage. This becomes a limitation in practical scenarios where strict size and resource constraints must be respected. In this paper, we present a novel concept-based pruning technique for DNNs that guides pruning decisions using human-interpretable concepts, such as features, colors, and classes. This is particularly important in a software engineering context, as DNNs are integrated into systems and must be pruned according to specific system requirements. Our concept-based pruning solution analyzes neuron activations to identify important neurons from a system requirements viewpoint and uses this information to guide the DNN pruning. We assess our solution using the VGG-19 network and a dataset of 26'384 RGB images, focusing on its ability to produce small, effective pruned DNNs and on the computational complexity and performance of these pruned DNNs. We also analyzed the pruning efficiency of our solution and compared alternative configurations. Our results show that concept-based pruning efficiently generates much smaller, effective pruned DNNs. Pruning greatly improves the computational efficiency and performance of DNNs, properties that are particularly useful for practical applications with stringent memory and computational time constraints. Finally, alternative configuration options enable engineers to identify trade-offs adapted to different practical situations.

Federico Formica, Andrea Rota, Aurora Francesca Zanenga +4

University of Bergamo, Bergamo, Italy

Neural Network PruningNeural Network CompressionEfficient Neural Network InferenceSoftware Engineering
AskPDF
Apr 10, 2026

TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inference

Implicit neural video representations encode entire video sequences within the parameters of a neural network and enable constant time frame reconstruction. Recent work on Neural Representations for Videos (NeRV) has demonstrated competitive reconstruction performance while avoiding the sequential decoding process of conventional video codecs. However, most existing studies focus on moderate or high capacity models, leaving the behavior of extremely compact configurations required for constrained environments insufficiently explored. This paper presents a systematic study of tiny NeRV architectures designed for efficient deployment. Two lightweight configurations, NeRV-T and NeRV-T+, are introduced and evaluated across multiple video datasets in order to analyze how aggressive capacity reduction affects reconstruction quality, computational complexity, and decoding throughput. Beyond architectural scaling, the work investigates strategies for improving the performance of compact models without increasing inference cost. Knowledge distillation with frequency-aware focal supervision is explored to enhance reconstruction fidelity in low-capacity networks. In addition, the impact of lowprecision inference is examined through both post training quantization and quantization aware training to study the robustness of tiny models under reduced numerical precision. Experimental results demonstrate that carefully designed tiny NeRV variants can achieve favorable quality efficiency trade offs while substantially reducing parameter count, computational cost, and memory requirements. These findings provide insight into the practical limits of compact neural video representations and offer guidance for deploying NeRV style models in resource constrained and real-time environments. The official implementation is available at https: //github.com/HannanAkhtar/TinyNeRV-Implementation.

Muhammad Hannan Akhtar, Ihab Amer, Tamer Shanableh

Department of Computer Science & Engineering, American University of Sharjah, Sharjah, United Arab Emirates

Neural Video CompressionNeural Network CompressionEfficient Neural Network Inference
AskPDF
Apr 10, 2026

MATCHA: Efficient Deployment of Deep Neural Networks on Multi-Accelerator Heterogeneous Edge SoCs

Deploying DNNs on System-on-Chips (SoC) with multiple heterogeneous acceleration engines is challenging, and the majority of deployment frameworks cannot fully exploit heterogeneity. We present MATCHA, a unified DNN deployment framework that generates highly concurrent schedules for parallel, heterogeneous accelerators and uses constraint programming to optimize L3/L2 memory allocation and scheduling. Using pattern matching, tiling, and mapping across individual HW units enables parallel execution and high accelerator utilization. On the MLPerf Tiny benchmark, using a SoC with two heterogeneous accelerators, MATCHA improves accelerator utilization and reduces inference latency by up to 35% with respect to the the state-of-the-art MATCH compiler.

Enrico Russo, Mohamed Amine Hamdi, Alessandro Ottaviano +6

University of Catania, Italy · Politecnico di Torino, Italy · ETH Zurich, Switzerland +1

Efficient Neural Network InferenceNeural Network Hardware AccelerationHeterogeneous ComputingEdge Computing
AskPDF
Apr 10, 2026

Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation

Memory-efficient transfer learning (METL) approaches have recently achieved promising performance in adapting pre-trained models to downstream tasks. They avoid applying gradient backpropagation in large backbones, thus significantly reducing the number of trainable parameters and high memory consumption during fine-tuning. However, since they typically employ a lightweight and learnable side network, these methods inevitably introduce additional memory and time overhead during inference, which contradicts the ultimate goal of efficient transfer learning. To address the above issue, we propose a novel approach dubbed Masked Dual Path Distillation (MDPD) to accelerate inference while retaining parameter and memory efficiency in fine-tuning with fading side networks. Specifically, MDPD develops a framework that enhances the performance by mutually distilling the frozen backbones and learnable side networks in fine-tuning, and discard the side network during inference without sacrificing accuracy. Moreover, we design a novel feature-based knowledge distillation method for the encoder structure with multiple layers. Extensive experiments on distinct backbones across vision/language-only and vision-and-language tasks demonstrate that our method not only accelerates inference by at least 25.2% while keeping parameter and memory consumption comparable, but also remarkably promotes the accuracy compared to SOTA approaches. The source code is available at https://github.com/Zhang-VKk/MDPD.

Yutong Zhang, Jiaxin Chen, Honglin Chen +5

State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China · School of Computer Science and Engineering, Beihang University, China · College of Information Technology, United Arab Emirates University, United Arab Emirates

Memory-Efficient Fine-TuningEfficient Neural Network InferenceKnowledge DistillationFeature-Based Knowledge Distillation
AskPDF
Apr 10, 2026

Thermal Anomaly Detection using Physics Aware Neuromorphic Networks: Comparison between Raw and L1C Sentinel-2 Data

Damage caused by bushfires and volcanic eruptions escalates rapidly when detection is delayed, making fast and reliable early warning capabilities essential. Recent Earth Observation (EO) approaches have shown that thermal anomaly detection can be performed directly on decompressed Level-0 (L0) sensor data, avoiding computationally expensive preprocessing chains. However, direct exploitation of raw data remains challenging due to domain shift, sensor drift, radiometric inconsistencies, and the scarcity of labelled training samples. To address these challenges, this work proposes a Physics-Aware Neuromorphic Network (PANN) framework for onboard thermal anomaly detection. The proposed lightweight architecture, inspired by physical neural network principles and neuromorphic computing paradigms, is evaluated using two Sentinel-2 datasets: decompressed L0 with additional metadata (i.e. raw) and Level-1C (L1C). The PANN achieves a Matthews Correlation Coefficient (MCC) of 0.8090.809 on raw measurements, compared to 0.8750.875 when using ground-processed L1C products. The mean processing latency per L0 granule is 2.44±0.09 s2.44 \pm 0.09~\mathrm{s}, which is below the Sentinel-2 acquisition time of 3.6 s3.6~\mathrm{s}, demonstrating the feasibility of real-time, onboard processing. Furthermore, the projected execution time for the corresponding neuromorphic hardware instantiation is substantially lower at 0.1290±0.0002 s0.1290 \pm 0.0002~\mathrm{s}. Memory usage, including all necessary programs and packages, remains within realistic onboard constraints, with requirements of 0.673±0.007 Gb0.673 \pm 0.007~\mathrm{Gb} for the software PANN and 0.393±0.004 Gb0.393 \pm 0.004~\mathrm{Gb} for the estimated hardware realisation. Overall, these results indicate that PANN offers a promising pathway toward low-latency and resource-efficient onboard EO processing for thermal event detection.

Stephen Smith, Cormac Purcell, Gabriele Meoni +2

School of Physics, University of Sydney, Sydney, Australia · BolgianTen Ltd., Liverpool, United Kingdom · School of Computer Science, University of New South Wales, Sydney, Australia +3

Anomaly DetectionPhysics-Informed MLNeuromorphic ComputingWildfire DetectionEfficient Neural Network Inference
AskPDF
Apr 9, 2026

EfficientSign: An Attention-Enhanced Lightweight Architecture for Indian Sign Language Recognition

How do you build a sign language recognizer that works on a phone? That question drove this work. We built EfficientSign, a lightweight model which takes EfficientNet-B0 and focuses on two attention modules (Squeeze-and-Excitation for channel focus, and a spatial attention layer that focuses on the hand gestures). We tested it against five other approaches on 12,637 images of Indian Sign Language alphabets, all 26 classes, using 5-fold cross-validation. EfficientSign achieves the accuracy of 99.94% (+/-0.05%), which matches the performance of ResNet18's 99.97% accuracy, but with 62% fewer parameters (4.2M vs 11.2M). We also experimented with feeding deep features (1,280-dimensional vectors pulled from EfficientNet-B0's pooling layer) into classical classifiers. SVM achieved the accuracy of 99.63%, Logistic Regression achieved the accuracy of 99.03% and KNN achieved accuracy of 96.33%. All of these blow past the 92% that SURF-based methods managed on a similar dataset back in 2015. Our results show that attention-enhanced learning model provides an efficient and deployable solution for ISL recognition without requiring a massive model or hand-tuned feature pipelines anymore.

Rishabh Gupta, Shravya R. Nalla

Sign Language RecognitionEfficient Neural Network Inference
AskPDF
Apr 9, 2026

MonoUNet: A Robust Tiny Neural Network for Automated Knee Cartilage Segmentation on Point-of-Care Ultrasound Devices

Objective: To develop a robust and compact deep learning model for automated knee cartilage segmentation on point-of-care ultrasound (POCUS) devices. Methods: We propose MonoUNet, a novel, highly compact segmentation model consisting of (i) an aggressively reduced U-Net backbone, (ii) a trainable monogenic block that extracts multi-scale local phase features from the input, and (iii) a gating mechanism that injects these features into the encoder stages to reduce sensitivity to variations in ultrasound image appearance. MonoUNet segmentation performance was evaluated on a multi-site, multi-device knee cartilage ultrasound dataset using Dice score and mean average surface distance (MASD). Agreement between MonoUNet and manual cartilage outcomes (thickness and echo intensity) was assessed using Bland-Altman analysis with 95% limits of agreement, and reliability was assessed using intraclass correlation coefficient (ICC2,k_{2,k}). Results: Overall, MonoUNet outperformed existing lightweight segmentation models, with average Dice scores ranging from 92.62% to 94.82% and MASD values between 0.133 mm and 0.254 mm. MonoUNet reduces the number of parameters by 10x--700x and computational cost by 14x--2000x relative to existing lightweight models. MonoUNet cartilage outcomes showed excellent reliability and agreement with the manual outcomes: intraclass correlation coefficients (ICC2,k)_{2,k})=0.96 and bias=2.00% (0.047 mm) for average thickness, and ICC2,k_{2,k}=0.99 and bias=0.80% (0.328 a.u.) for echo intensity. Conclusion: Incorporating trainable local phase features improves the robustness of highly compact neural networks for knee cartilage segmentation across varying acquisition settings and could support scalable ultrasound-based assessment and monitoring of knee osteoarthritis using POCUS devices. The code is publicly available at https://github.com/alvinkimbowa/monounet.

Alvin Kimbowa, Arjun Parmar, Ibrahim Mujtaba +5

School of Biomedical Engineering, The University of British Columbia, Canada · Department of Kinesiology, Michigan State University, USA · Department of Rheumatology, The University of British Columbia, Canada +2

Medical Image SegmentationUltrasound Image SegmentationU-NetUltrasound ImagingEfficient Neural Network Inference
AskPDF
Apr 6, 2026

StableTTA: Improving Vision Model Performance by Training-free Test-Time Adaptation Methods

Ensemble methods improve predictive performance but often incur high memory and computational costs. We identify an aggregation instability induced by nonlinear projection and voting operations. To address both efficiency challenges and this inconsistency, we propose StableTTA, a training-free test-time adaptation method with two variants. StableTTA-I targets coherent-batch inference settings, where temporally or semantically adjacent observations are likely to belong to the same class. Examples include burst photography, video streams, robotics perception, and industrial inspection. Under coherent-batch inference, StableTTA-I substantially improves prediction consistency and accuracy through variance-aware logit aggregation. StableTTA-II establishes feature-level cropping, enabling efficient logit aggregation with a single forward pass on a single model backbone. Experiments on ImageNet-1K across 71 models demonstrate that StableTTA-I consistently improves prediction accuracy under coherent-batch inference, while StableTTA-II provides lightweight and architecture-agnostic accuracy improvements with minimal computational overhead. These results suggest that inference-time semantic coherence and aggregation stability provide useful perspectives for improving practical test-time adaptation systems.

Zheng Li, Jerry Cheng, Huanying Helen Gu

Department of Computer Science New York Institute of Technology

Test-Time AdaptationEnsemble LearningEfficient Neural Network Inference
AskPDF
Apr 6, 2026

TinyNina: A Resource-Efficient Edge-AI Framework for Sustainable Air Quality Monitoring via Intra-Image Satellite Super-Resolution

Nitrogen dioxide (NO2_2) is a primary atmospheric pollutant and a significant contributor to respiratory morbidity and urban climate-related challenges. While satellite platforms like Sentinel-2 provide global coverage, their native spatial resolution often limits the precision required, fine-grained NO2_2 assessment. To address this, we propose TinyNina, a resource-efficient Edge-AI framework specifically engineered for sustainable environmental monitoring. TinyNina implements a novel intra-image learning paradigm that leverages the multi-spectral hierarchy of Sentinel-2 as internal training labels, effectively eliminating the dependency on costly and often unavailable external high-resolution reference datasets. The framework incorporates wavelength-specific attention gates and depthwise separable convolutions to preserve pollutant-sensitive spectral features while maintaining an ultra-lightweight footprint of only 51K parameters. Experimental results, validated against 3,276 matched satellite-ground station pairs, demonstrate that TinyNina achieves a state-of-the-art Mean Absolute Error (MAE) of 7.4 μμg/m3^3. This performance represents a 95% reduction in computational overhead and 47×\times faster inference compared to high-capacity models such as EDSR and RCAN. By prioritizing task-specific utility and architectural efficiency, TinyNina provides a scalable, low-latency solution for real-time air quality monitoring in smart city infrastructures.

Prasanjit Dey, Zachary Yahn, Bianca Schoen-Phelan +1

ADAPT Research Centre, Dublin, Ireland · School of Computer Science, Technological University Dublin, Ireland · School of Computer Science, University College Dublin, Ireland +1

Autoregressive Image Super-ResolutionAir Quality PredictionRemote Sensing Image Super-ResolutionImage Super-ResolutionEdge InferenceEfficient Neural Network Inference
AskPDF
Apr 6, 2026

Integer-Only Operations on Extreme Learning Machine Test Time Classification

We present a theoretical analysis and empirical evaluations of a novel set of techniques for computational cost reduction of test time operations of network classifiers based on extreme learning machine (ELM). By exploring some characteristics we derived from these models, we show that the classification at test time can be performed using solely integer operations without compromising the classification accuracy. Our contributions are as follows: (i) We show empirical evidence that the input weights values can be drawn from the ternary set with limited reduction of the classification accuracy. This has the computational advantage of dismissing multiplications; (ii) We prove the classification accuracy of normalized and non-normalized test signals are the same; (iii) We show how to create an integer version of the output weights that results in a limited reduction of the classification accuracy. We tested our techniques on 5 computer vision datasets commonly used in the literature and the results indicate that our techniques can allow the reduction of the computational cost of the operations necessary for the classification at test time in FPGAs. This is important in embedded applications, where power consumption is limited, and crucial in data centers of large corporations, where power consumption is expensive.

Emerson Lopes Machadoa, Cristiano Jacques Miosso, Ricardo Pezzuol Jacobi

Dept. of Computer Science, University of Brasilia, Brazil · University of Brasilia at Gama, Brazil

Efficient Neural Network InferenceNeural Network QuantizationFPGA-Based Neural Network Acceleration
AskPDF