Efficient Neural Network Inference

Follow this topic Ask about this topic

Momentum

39 papers in the last four weeks, up 225% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 314

May 13, 2026cs.LG

TabPFN-3: Technical Report

Tabular data underpins most high-value prediction problems in science and industry, and TabPFN has driven the foundation model revolution for this modality. Designed with feedback from our users, TabPFN-3 builds on this foundation to scale state-of-the-art performance to datasets with 1M training rows and substantially reduce training and inference time. Pretrained exclusively on synthetic data from our prior, TabPFN-3 dramatically pushes the frontier of tabular prediction and brings substantial gains on time series, relational, and tabular-text data. On the standard tabular benchmark TabArena, a forward pass of TabPFN-3 outperforms all other models, including tuned and ensembled baselines, by a significant margin, and pareto-dominates the speed/performance frontier. On more diverse datasets, TabPFN-3 ranks first on datasets with many classes, and beats 8-hour-tuned gradient-boosted-tree baselines on datasets up to 1M training rows and 200 features. TabPFN-3 introduces test-time compute scaling to tabular foundation models. Our API offering TabPFN-3-Plus (Thinking) exploits this to beat all non-TabPFN models by over 200 Elo on TabArena, rising to 420 Elo on the largest data subset, and outperforms AutoGluon 1.5 extreme while being 10x faster, without using LLMs, real data, internet search or any other model besides TabPFN. TabPFN-3 extends the capabilities of our models, enabling SOTA prediction on relational data (new SOTA foundation model on RelBenchV1) and tabular-text data (SOTA on TabSTAR via TabPFN-3-Plus); and improves existing integrations: a specialized checkpoint, TabPFN-TS-3, ranks 2nd on the time-series benchmark fev-bench, and SHAP-value computation is up to 120x faster. TabPFN-3 achieves this performance while being up to 20x faster than TabPFN-2.5. In addition, a reduced KV cache and row-chunking scale to 1M rows on one H100 with fast inference speed.

Léo Grinsztajn, Klemens Flöge, Oscar Key +38

Tabular Foundation Models Efficient Neural Network Inference Inference-Time Scaling Tabular ICL Tabular Prediction

May 13, 2026cs.NE

Dual-axis attribution of zebrafish tectal microcircuits for energy-efficient and robust neurocomputing

Biological neural circuits contain specialized substructures that support distinct computational functions, yet many bio-inspired neural networks borrow biological motifs without identifying their circuit-level origins. In this study, we investigate whether zebrafish tectal microcircuits can be attributed along two computational axes: energy-efficient information processing and robustness-preserving stabilization. We reconstruct a directed zebrafish-inspired retinotectal microcircuit graph and verify retinotectal signal propagation through dynamic simulation. A leaky integrate-and-fire spiking neural network is then used as a nonlinear perturbation testbed, where predefined subcircuits are selectively ablated and evaluated using the Energy Sensitivity Index and the Robustness Sensitivity Index.The results reveal a functional dissociation between two tectal subcircuits.The \textit{ns_TIN} subcircuit shows a low spike footprint but a measurable influence on prediction error, suggesting a role as a spike-efficient internal information gate.In contrast, the \textit{superficial_TIN} subcircuit produces the highest robustness sensitivity, suggesting a feedback-like role in maintaining system-level stability.We further transfer these attributed functions into ResNet18-based artificial neural networks and evaluate them on CIFAR-10 under inference-budget reduction and Gaussian noise corruption. The \textit{ns_TIN}-inspired module improves performance preservation under reduced computation, whereas the \textit{superficial_TIN}-inspired module improves robustness under input noise. These findings provide a subcircuit-level route for linking biological circuit organization with bio-inspired neural architecture design.

Ningping Li, Hao Zhang, Yi Zhou

University of Science and Technology of China, Hefei, China

Efficient Neural Network Inference Neural Processes Neural Network Robustness

May 11, 2026cs.LG

Neural Statistical Functions

Classical deep learning typically operates on individual cases. Despite its success, real-world usage often requires repeated inference to estimate statistical quantities for complex decision-making tasks involving uncertainty or extreme-value analysis, resulting in substantial latency. We introduce neural statistical functions, a new family of models learned from pre-trained single-sample predictors and scattered data samples, which can directly infer statistics over continuous operating condition ranges without explicit sampling. By introducing the notion of prefix statistics, we transform and unify diverse statistical functions (e.g., integrals, quantiles, and maxima) into an interval-conditional framework, in which a principled identity between the prefix statistics and the individual-case regression serves as the learning objective. Neural statistical functions achieve strong performance in estimating essential statistics of complex physical processes, including accumulated energy in dynamical systems, quantiles of aerodynamic responses, and maximum stress in crash processes, while achieving up to a 100

\times

reduction in model evaluations.

Daniel Xu, Yuxin Xie, Minghao Guo +2

Columbia University · MIT CSAIL

Neural Surrogate Modeling Efficient Neural Network Inference

May 11, 2026eess.IV

Brain Tumor Classification in MRI Images: A Computationally Efficient Convolutional Neural Network

Improving patient outcomes depends on the prompt and accurate diagnosis of brain tumors, but manual MRI scan analysis is still time-consuming and unreliable. Although deep learning has shown promise, many of the models that are now in use are computationally intensive and have difficulty handling the intrinsic complexity and variety of different types of brain tumors. In this work, we propose a lightweight yet high-performing Convolutional Neural Network (CNN) for multi-class brain tumor classification, employing MRI images to target gliomas, meningiomas, pituitary tumors, and healthy (no tumor) instances. The model was rigorously evaluated on two publicly accessible datasets from Figshare and Kaggle. Leveraging efficient feature extraction and optimized training strategies, our CNN achieved classification accuracies of 99.03% and 99.28%, along with ROC scores of 99.88% and 99.94% on Dataset 1 and Dataset 2, respectively-all while utilizing significantly fewer parameters than popular pre-trained architectures. In contrast to cutting-edge models like DenseNet201, MobileNetV2, VGG19, Xception, InceptionV3, and ResNet50, our approach consistently demonstrated superior performance with reduced computational overhead. These findings highlight the potential of the proposed model as a practical and reliable diagnostic aid in clinical environments.

Md Fahimul Kabir Chowdhury, Jannatul Ferdous

Department of Computer Science and Engineering, University of North Texas, USA · Department of Electrical and Electronic Engineering, International Islamic University Chittagong, Bangladesh

Efficient Neural Network Inference Medical Image Classification Convolutional Neural Networks Brain Tumor Classification

May 11, 2026cs.NE

Energy-Efficient Implementation of Spiking Recurrent Cells on FPGA

Spiking Neural Networks (SNNs) can reduce energy consumption compared to conventional Artificial Neural Networks (ANNs) when spiking activity is sparse and the neuron model is hardware-friendly. However, biologically faithful models are often too costly to implement on FPGAs, whereas very simple models (e.g., IR/LIF) sacrifice part of the neuronal dynamics. In this work, we present an FPGA accelerator for an SNN using Spiking Recurrent Cell (SRC) neurons, providing a trade-off between biological plausibility and hardware cost. We propose a set of mathematical simplifications that remove costly unary operators (\textit{tanh}, \textit{exp}) and avoid floating-point arithmetic through scaling and piecewise-defined approximations. The complete network is implemented in VHDL and validated using spiking traces derived from the MNIST dataset. The weight matrices computed off-line are stored directly in LUT-registers without any adaptation. This demonstrates the robustness of SRC cells. Experiments were conducted on an Artix-7 XC7A200T clocked at 100 MHz. The reference implementation achieves 96.31% accuracy with a 220-image spiking trace and a processing time of 1.7424 ms per digit. We then investigate accuracy/energy trade-offs by reducing the spiking trace length and quantizing synaptic weights down to 4 bits, achieving 93.32% accuracy at 0.55 mJ per digit (55 images, 5-bit weights) and 92.89% at 0.45 mJ (44 images, 4-bit weights). These results show that SRC-based SNNs can deliver competitive performance with reduced energy consumption, while preserving richer neuronal dynamics than standard LIF/IR models.

Pascal Harmeling, Florent De Geeter, Guillaume Drion

Efficient Neural Network Inference Spiking Neural Networks FPGA-Based Neural Network Acceleration

May 11, 2026cs.LG

PhysEDA: Physics-Aware Learning Framework for Efficient EDA With Manhattan Distance Decay

Electronic design automation (EDA) addresses placement, routing, timing analysis, and power-integrity verification for integrated circuits. Learning methods -- attention (Transformer) and reinforcement learning (RL) -- have recently emerged on EDA tasks, yet face two common bottlenecks: vanilla attention's quadratic complexity limits scaling, and data-scarce models overfit statistical noise and amplify weak long-range correlations against the underlying physics. We observe that EDA tasks share a physical prior -- pairwise electrical and routing interactions decay exponentially along Manhattan distance -- and integrate it as a unified inductive bias into both architecture and training. We propose PhysEDA, comprising two components Physics-Structured Linear Attention (PSLA) folds the separable Manhattan decay into the linear-attention kernel as a multiplicative bias, reducing complexity from quadratic to linear; Potential-Based Reward Shaping (PBRS) constructs a physical potential from the same kernel, providing dense reward signal under sparse RL while preserving the optimal policy via the policy-invariance theorem. Across three EDA scenarios -- decoupling-capacitor placement, macro placement, and IR-drop prediction -- PhysEDA improves zero-shot cross-scale transfer by 56.8% and achieves 14x inference speedup with 98.5% memory savings on 100x100 grids; PBRS adds another 10.8% in sparse-reward DPP.

Zetao Yang

School of Mathematics and Statistics Beijing Institute of Technology Beijing, China

Efficient Neural Network Inference Reward Shaping Electronic Design Automation Linear Attention Physics-Informed ML Sparse-Reward RL

May 11, 2026cs.LG

Efficient Neural Architectures for Real-Time ECG Interpretation on Limited Hardware

Electrocardiogram (ECG) interpretation is essential for diagnosing a wide range of cardiac abnormalities. While deep learning has shown strong potential for automating ECG classification, many existing models rely on large, computationally intensive architectures that hinder practical deployment. In this paper, we present an empirical study of convolutional neural network (CNN) architectures, exploring tradeoffs between diagnostic accuracy and computational efficiency. We benchmark two established baselines: AttiaNet, a compact model composed of sequential temporal and spatial blocks, and DeepResidualCNN, the winning architecture of the 2021 PhysioNet/Computing in Cardiology Challenge. Building on these, we propose three lightweight models: (i) ParallelCNN, which employs dual temporal and spatial branches for parallel pattern extraction; (ii) ParallelCNNew, a variant with symmetric weight initialization for balanced feature learning; and (iii) SimpleNet, a streamlined architecture that jointly processes temporal and spatial dimensions. Our experiments span three publicly available 12-lead ECG datasets from Germany, China, and the United States, covering binary, multiclass, and multilabel classification tasks across diverse patient populations. We further evaluate the impact of integrating low-cost demographic metadata (age and sex) to improve performance with minimal overhead. To ensure fair comparison, we introduce a unified Efficiency Score that integrates model size, inference speed, memory usage, and AUC performance. By balancing diagnostic performance and efficiency, our models offer a scalable and viable foundation for next-generation AI systems in cardiovascular care.

Ashery Mbilinyi, Callum O'Riley, Julia Handra +8

Department of Computer Science University of Victoria Victoria, Canada · Department of Electrical and Computer Engineering University of British Columbia Vancouver, Canada · Faculty of Medicine University of British Columbia Vancouver, Canada +4

Efficient Neural Network Inference Electrocardiogram Classification Convolutional Neural Networks Edge ML

May 10, 2026physics.geo-ph

Evaluating PhaseNet on Teleseismic Data with MsPASS

Numerous studies have shown that the machine-learning picker PhaseNet produces accurate P and S picks on local earthquake signals, but its performance can degrade sharply on teleseismic signals. To address this limitation, we present a reproducible MsPASS workflow that (i) enables scalable data preparation and management for large seismic archives and (ii) supports standardized PhaseNet training and inference. We assembled a control dataset of 1.6 million waveforms linked to teleseismic P-wave picks made by analysts at the USArray Array Network Facility (ANF). The control dataset confirms that the PhaseNet model trained on regional signals performs poorly on these data. We then trained PhaseNet from scratch on the training split of the ANF control dataset and evaluated it on a non-overlapping held-out test split, increasing P-pick recall by 741.5% and yielding 683.9% more picks within a 0.1s residual window. We also evaluated PhaseNet across different model sizes on both CPUs and GPUs. Increasing the model size by about 120 times improved precision and recall by 15.6% and 23.2%, respectively. However, the scaled model reduced inference throughput by 87.2% on an NVIDIA A100 GPU and by 97.3% on a 128-core high-performance CPU node. These results indicate that scaling PhaseNet is more practical on GPUs than on CPUs, and that simply enlarging the model is not an efficient way to achieve large accuracy gains.

Jinxin Ma, Yinzhi Wang, Gary L. Pavlis +1

Texas Advanced Computing Center, The University of Texas at Austin · Department of Earth and Atmospheric Sciences, Indiana University, Bloomington, IN 47405

Efficient Neural Network Inference Seismology

May 10, 2026cs.DC

Adaptive DNN Partitioning and Offloading in Heterogeneous Edge-Cloud Continuum

In recent years, the use of artificial intelligence on resource-constrained IoT devices has grown significantly. However, existing approaches to DNN partitioning and offloading across the edge-cloud continuum typically rely on static methods that ignore runtime dynamics. Furthermore, they are often evaluated in simulated environments rather than on real hardware. To address this gap, we propose a framework that dynamically splits neural network layers across the heterogeneous continuum. The framework profiles the model at startup, measures network link conditions between nodes, and periodically re-evaluates the partition to adapt to environmental changes. We created a physical testbed comprising a Raspberry Pi edge device, a laptop fog, and a high-performance desktop PC as the cloud. We evaluated the framework over three widely adopted convolutional neural networks: VGG16, AlexNet, and MobileNetV2. Our results show that the framework achieves reductions in energy and end-to-end latency of 27.09--35.82% and 6.34--22.92%, respectively, compared to a static partitioning baseline. These findings confirm the superiority of adaptive to static partitioning.

Akuen Akoi Deng, Eimantas Butkus, Alfreds Lapkovskis +1

Department of Computer and Systems Sciences, Stockholm University, Borgarfjordsgatan 12, Stockholm, Sweden

Efficient Neural Network Inference Energy-Efficient ML Edge-Cloud Computing

May 10, 2026cs.CV

AnyDepth-DETR/-YOLO: Any-depth object detection with a single network

Modern object detectors are static, fixed-depth networks optimized for a single operating point, requiring separate models for different deployment scenarios. We present an any-depth detection framework that enables a single network to span a continuous range of accuracy--efficiency trade-offs by controlling depth at inference time without retraining. Each backbone and neck stage is divided into an essential path, which always executes, and a skippable refinement path; this decomposition preserves the full multi-scale feature hierarchy at every depth configuration, unlike conventional early exiting that discards entire stages. To train such a network, jointly optimizing many sub-networks of varying depth introduces conflicting gradient signals. We address this via self-distillation between only the two extremes, with prediction-level and feature-level alignment losses that enforce stage-wise modularity, ensuring the outputs of each stage remain compatible regardless of the paths taken. Instantiated on RT-DETR and YOLOv12, our full-depth configurations match or surpass their respective SOTA baselines with negligible parameter overhead, while the most efficient configurations achieve up to

1.82\times

speedup at a cost of only 2.0 AP, all from a single set of weights.

Woochul Kang, Hyungseop Lee, Jiho Lee

Incheon Nat’l Univ.

Efficient Neural Network Inference Dynamic Neural Networks Efficient Object Detection Object Detection Real-Time Object Detection

May 10, 2026cs.DC

Split CNN Inference on Networked Microcontrollers

Running deep neural networks on microcontroller units (MCUs) is severely constrained by limited memory resources. While TinyML techniques reduce model size and computation, they often fail in practice due to excessive peak Random Access Memory (RAM) usage during inference, dominated by intermediate activations. As a result, many models remain infeasible on standalone MCUs. In this work, we present a fine-grained split inference system for networked MCUs that enables collaborative inference of Convolutional Neural Networks (CNN) models across multiple devices. Our key insight is that breaking the memory bottleneck requires splitting inference at sub-layer granularity rather than at layer boundaries. We reinterpret pre-trained models to enable kernel-wise and neuron-wise partitioning, and distribute both model parameters and intermediate activations across multiple MCUs. A lightweight, resource-aware coordinator orchestrates the inference across MCU devices with heterogeneous resources. We implement the proposed system on a real testbed and evaluate it on up to 8 MCUs using MobileNetV2, a representative CNN model. Our experimental results show that CNN models infeasible on a single MCU can be executed across networked MCUs, reducing the per-MCU peak RAM usage while maintaining the practical end-to-end inference latency. All the source code of this work can be found here: https://github.com/shashsuresh/split-inference-on-MCUs.

Junyu Lu, Shashwath Suresh, Hao Liu +2

Delft University of Technology, Delft, The Netherlands

Efficient Neural Network Inference Edge Inference Split Inference Convolutional Neural Networks On-Device Inference

May 9, 2026cs.LG

AESOP: Adversarial Execution-path Selection to Overload Deep Learning Pipelines

Modern machine learning deployments increasingly compose specialized models into dynamic inference pipelines, where upstream components produce intermediate predictions that determine the workload and inputs of downstream components. The cost of processing an input is therefore not determined by any single model, but by two coupled factors: the per-inference cost of each invoked component and its workload volume. Because these pipelines run under hard real-time constraints, efficiency is a fundamental requirement for system availability. We show that this structure creates an efficiency-attack surface that existing methods targeting single models cannot exploit: on identical inputs and budgets, path-aware targeting inflates FLOPs by

2,407\times

while the strongest single-model baseline achieves

117\times

-- a

20\times

gap attributable entirely to where the attack is directed. We formalize this as the adversarial path-selection problem and present AESOP, a framework combining vulnerability-guided path ranking with adaptive loss weighting. We evaluate AESOP on five pipelines plus a production-realistic deployment variant with batching, bounded buffering, and confidence-threshold defenses. AESOP achieves up to

2,407\times

FLOPs and

419\times

latency inflation in white-box setting and 58

\times

FLOPs / 17

\times

latency in gray-box settings. Under system-level defenses, the attack is not neutralized but redirected: pipelines are forced to choose between throughput collapse (

0.578 \to 0.006

input/s) and

96.7\%

data loss to sustain throughput.

Tingxi Li, Mingfang Ji, Ravishka Shemal Rathnasuriya +3

The University of Texas at Dallas · Tianjin University

Efficient Neural Network Inference Adversarial Attacks

May 9, 2026cs.AR

A Reconfigurable Multiplier Architecture for Error-Resilient Applications in RISC-V Core

Neural Networks (NNs) have been widely adopted due to their outstanding efficacy and adaptability across computer vision and deep learning applications. The optimization of NNs is necessary to enable their deployment on energy constrained embedded devices, where the limited available energy poses a significant challenge for efficient inference. This paper presents a runtime reconfigurable multiplier architecture integrated into the RISC-V core, targeting energy efficient neural network inference and edge AI applications. The proposed multiplier supports adaptability for exact and approximate computation with multiple configurable accuracy levels via a dedicated mulscr, enabling fine-grained energy accuracy control within a standard processor pipeline. The proposed design achieves 44%-52% and 62%-68% power reduction in exact and approximate modes respectively, while maintaining the computational performance of 1.89 DMIPS/MHz. Evaluations on error-tolerant workloads including 2d convolution and matrix multiplication demonstrate up to 63% reduction in energy consumption, with the proposed design achieving 1.21 pJ/instruction for matrix multiplication, confirming its effectiveness for energy-constrained edge AI deployments.

Pragun Jaswal, L. Hemanth Krishna, B. Srinivasu

Indian Institute of Technology Mandi, Mandi–175005, India

Efficient Neural Network Inference Energy-Efficient ML Edge ML Approximation Algorithms Approximate Computing

May 8, 2026cs.LG

Private Vertical Federated Inference for Time-Series

Institutions may benefit from collaborative inference on time-series data. In settings where privacy is necessary, multi-party computation (MPC) is a straightforward approach to providing strong guarantees, yet it remains prohibitively expensive and scales poorly with modern transformer architectures. Vertical Federated Learning (VFL) offers efficiency but suffers from privacy leakage at the embedding level, and securing the entire VFL model head via MPC remains prohibitively slow and communication-heavy for larger models. To enable practical, secure inference at scale, we propose "Public/Private Hybrid Head-VFL" (PPHH-VFL). This hybrid architecture splits the model head into an efficient plaintext public head and a secure, lightweight MPC private head. By applying adversarial training to the public embeddings, we mitigate privacy leakage; concurrently, the small private head securely preserves the flow of sensitive information needed for high downstream utility. Empirical evaluations on models ranging up to 86 million parameters demonstrate that PPHH-VFL accelerates inference by up to six orders of magnitude compared to end-to-end MPC. Compared to a standard VFL+MPC baseline, our approach scales significantly better, achieving a speedup of up to 44.4x in WAN and a 91.2x reduction in communication costs (dropping from 1.7 GB to 19 MB per batch), while simultaneously improving downstream classification accuracy by 2.50% and regression RMSE by 40.7%.

Lucas Fenaux, Larris Xie, Aditya Bang +3

University of Waterloo · Borealis AI

Efficient Neural Network Inference Privacy-Preserving ML Federated Learning

May 8, 2026cs.NE

Direct-to-Event Spiking Neural Network Transfer

Spiking Neural Networks (SNNs) have gained increasing attention due to their potential for low-power computation on neuromorphic hardware. A widely adopted training strategy for SNNs is direct coding, which enable backpropagation on neuron implementations using continuous-valued surrogate activations. However, recent studies have shown that direct-coded SNNs remain substantially less energy-efficient than their event-based counterparts, limiting their practical deployment in energy sensitive scenarios. Still, to promote the reusability of pretrained SNN database on direct code, this motivates an important yet underexplored question: How can a SNN pretrained with direct code be effectively converted into an event-based representation? In this research, we present the first systematic investigation into this transfer problem, analyze the key challenges that arise when transitioning from direct-coded to event-based computation and propose a set of methods to enable energy-efficient transfer while preserving model performance.

Nhan Trong Luu, Duong Trung Luu, Pham Ngoc Nam +1

College of ICT, Can Tho University, Vietnam · CECS, VinUniversity, Hanoi, Vietnam · Center of Digital Transformation and Communication, Can Tho University, Vietnam +1

Efficient Neural Network Inference Spiking Neural Networks Transfer Learning

May 7, 2026cs.AR

CARMEN: CORDIC-Accelerated Resource-Efficient Multi-Precision Inference Engine for Deep Learning

This paper presents CARMEN, a runtime-adaptive, CORDIC-accelerated multi-precision vector engine for resource-efficient deep learning inference. The key insight is that CORDIC iteration depth directly governs computational accuracy, enabling dynamic switching between approximate and accurate execution modes without hardware modification. The architecture integrates a low-resource iterative CORDIC-based MAC unit with a time-multiplexed multi-activation function block, supporting flexible 8/16-bit precision and high hardware utilization. ASIC implementation in 28 nm CMOS achieves up to 33% reduction in computation cycles and 21% power savings per MAC stage; a 256-PE configuration delivers 4.83 TOPS/mm2 compute density and 11.67 TOPS/W energy efficiency. FPGA deployment on PynqZ2 validates 154.6 ms latency at 0.43 W for real-time object detection.

Sonu Kumar, Mukul Lokhande, Santosh Kumar Vishvakarma +1

Dept. of Electrical Engineering Indian Institute of Technology Indore Simrol 453552, Indore, India · EnICS Labs Bar-Ilan University Ramat Gan, Israel

Efficient Neural Network Inference Energy-Efficient ML FPGA-Based Neural Network Acceleration Approximate Computing

May 7, 2026cs.CL

Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models

Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users. Standard inference demands expensive datacenter GPUs or cloud API access, leaving over one billion personal computers underutilized for AI workloads. Ternary models offer a path forward: their weights are constrained to {-1, 0, +1}, theoretically eliminating the need for floating-point multiplication. However, existing frameworks fail to exploit this structure, treating ternary models as dense floating-point networks. We address this gap with custom SIMD kernels that replace matrix multiplication with simple addition and subtraction operations, targeting the integer dot product instructions available on modern CPUs. Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 18.15x higher throughput, 7.15x faster time-to-first-token and 6.03x memory reduction compared to standard PyTorch inference on Apple Silicon, with comparable or higher throughput speedups up to 95.81x on Intel and AMD processors.

Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2

Mindbeam AI

LLM Quantization Efficient Neural Network Inference LLM Inference LLM Inference Acceleration Language Modeling

May 7, 2026cs.CV

On-Orbit Real-Time Wildfire Detection Under On-Board Constraints

We present a deployed system for on-orbit wildfire detection aboard a nine-satellite commercial thermal infrared constellation, operating under demanding joint constraints: sub-megabyte model footprint, sub-150 ms per-batch TensorRT FP16 inference on an NVIDIA Jetson Xavier NX, and an end-to-end alert pipeline targeting under 10 minutes from satellite overpass to fire event communication. The system operates on uncalibrated mid-wave infrared (MWIR) single-band imagery at 200 m ground sampling distance, where fires frequently appear as sub-pixel or single-pixel thermal anomalies under extreme class imbalance -- challenges not addressed by the contextual thermal-thresholding pipelines (MODIS, VIIRS) that currently dominate operational fire monitoring. We present an empirical study of lightweight dense representation learning for this regime using a proprietary nine-satellite MWIR dataset. We compare dense masked autoencoding (DenseMAE) and a hybrid DenseMAE+EMA (exponential moving average) distillation variant, and evaluate representations via linear probing and full-distribution pixel-level average precision (AP) under extreme class imbalance. DenseMAE pretraining enables compact downstream models on the latency-accuracy Pareto frontier: our fastest SSL-pretrained model achieves 0.640 test AP and 0.69 event-level Fire-F1 with 65.34 ms latency per batch and a 0.52 MB engine, without pruning or compression. The best configuration reaches 0.699 AP and 0.744 Fire-F1 below 1 MB, outperforming a supervised baseline (0.650 AP) under comparable constraints.

Matthias Rötzer, Veronika Pörtge, Martin Ickerott +6

OroraTech GmbH, St.-Martin-Str. 112, 81669 Munich, Germany · Ludwig-Maximilians-Universität München, Munich, Germany

Efficient Neural Network Inference Wildfire Detection Infrared Small Target Detection Remote Sensing On-Device Inference Masked Image Modeling

May 7, 2026cs.AR

PoTAcc: A Pipeline for End-to-End Acceleration of Power-of-Two Quantized DNNs

Power-of-two (PoT) quantization significantly reduces the size of deep neural networks (DNNs) and replaces multiplications with bit-shift operations for inference. Prior work has shown that PoT-quantized DNNs can preserve accuracy for tasks such as image classification; however, their performance on resource-constrained edge devices remains insufficiently understood. While general-purpose edge CPUs and GPUs do not provide optimized backends for bit-shift operations, custom hardware accelerators can better exploit PoT quantization by implementing dedicated shift-based processing elements. However, deploying PoT-quantized models on such accelerators is challenging due to limited support in existing inference frameworks. In addition, the impact of different PoT quantization strategies on hardware design, performance, and energy efficiency during full inference has not been systematically explored. To address these challenges, we propose PoTAcc, an open-source end-to-end pipeline for accelerating and evaluating PoT-quantized DNNs on resource-constrained edge devices. PoTAcc enables seamless preparation and deployment of PoT-quantized models via TensorFlow Lite (TFLite) across heterogeneous platforms, including CPU-only systems and hybrid CPU-FPGA systems with custom accelerators. We design shift-based processing element (shift-PE) accelerators for three PoT quantization methods and implement them on two FPGA platforms. We evaluate accuracy, performance, energy efficiency, and resource utilization across a range of models, including CNNs and Transformer-based architectures. Results show that our CPU-accelerator design achieves up to 3.6x speedup and 78% energy reduction compared to CPU-only execution for PoT-quantized DNNs on PYNQ-Z2 and Kria boards. The code will be publicly released at https://github.com/gicLAB/PoTAcc

Rappy Saha, Jude Haris, Nicolas Bohm Agostini +2

Efficient Neural Network Inference FPGA-Based Neural Network Acceleration Neural Network Quantization On-Device Inference

May 7, 2026cs.LG

DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression

In this paper, we propose DiBA (Diagonal and Binary Matrix Approximation), a compact matrix factorization for neural network weight compression. Many components of modern networks, including linear layers,

1\times1

convolutions, attention projections, and embedding layers, have dense matrix weights. DiBA approximates

A\in\mathbb{R}^{m\times n}

by

\widehat A=D_1B_1D_2B_2D_3

, where

D_1,D_2,D_3

are diagonal matrices and

B_1,B_2

are

0/1

binary matrices. The intermediate dimension

k

controls the trade-off between theoretical storage and approximation accuracy. For matrix-vector products, DiBA decomposes dense multiplication into three element-wise scaling operations and two binary mixing operations, reducing the floating-point multiplication count from

mn

to

m+k+n

. For optimization, we introduce DiBA-Greedy, an alternating solver that combines closed-form least-squares updates for the diagonal factors with exact one-bit improvement tests for the binary factors. We also introduce DiBARD (DiBA with Retuning only Diagonal factors), which replaces dense-matrix layers by DiBA factors, freezes the binary matrices, and retunes only the diagonal entries on downstream data. This preserves compact binary mixing without discrete search during adaptation. On 40 dense weight matrices extracted from public pretrained models, DiBA-Greedy yields consistent SNR improvements as the theoretical storage ratio increases. After DiBA replacement in two component-replacement studies, DiBARD improves DistilBERT/WikiText masked-token accuracy from 0.4447 to 0.5210 and Speech Commands test accuracy for an Audio Spectrogram Transformer from 0.7684 to 0.9781 without reoptimizing the binary factors.

Nobutaka Ono

Tokyo Metropolitan University, 6–6 Asahigaoka, Hino-shi, Tokyo 191–0065, Japan.

Efficient Neural Network Inference Matrix Optimization Neural Network Compression Matrix Factorization

May 7, 2026cs.CV

Edge Deep Learning in Computer Vision and Medical Diagnostics: A Comprehensive Survey

Edge deep learning, a paradigm change reconciling edge computing and deep learning, facilitates real-time decision making attuned to environmental factors through the close integration of computational resources and data sources. Here we provide a comprehensive review of the current state of the art in edge deep learning, focusing on computer vision applications, in particular medical diagnostics. An overview of the foundational principles and technical advantages of edge deep learning is presented, emphasising the capacity of this technology to revolutionise a wide range of domains. Furthermore, we present a novel categorisation of edge hardware platforms based on performance and usage scenarios, facilitating platform selection and operational effectiveness. Following this, we dive into approaches to effectively implement deep neural networks on edge devices, encompassing methods such as lightweight design and model compression. Reviewing practical applications in the fields of computer vision in general and medical diagnostics in particular, we demonstrate the profound impact edge-deployed deep learning models can have in real-life situations. Finally, we provide an analysis of potential future directions and obstacles to the adoption of edge deep learning, with the intention to stimulate further investigations and advancements of intelligent edge deep learning solutions. This survey provides researchers and practitioners with a comprehensive reference shedding light on the critical role deep learning plays in the advancement of edge computing applications.

Yiwen Xu, Tariq M. Khan, Yang Song +1

School of Computer Science and Engineering

Efficient Neural Network Inference Edge Inference Edge Computing

May 6, 2026cs.LG

AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures

Deep neural network (DNN) inference at the edge demands simultaneous improvements in accuracy, computational efficiency, and energy consumption. Approximate computing and Mixture-of-Experts (MoE) architectures have each been studied as independent routes towards efficient inference, the former by replacing exact arithmetic with low-power approximate multipliers, the latter by routing inputs through specialized expert sub-networks to enable conditional computation. However, their interaction remains entirely unexplored. This paper presents AxMoE, the first study of the impact of approximate multiplication on MoE DNN architectures. We evaluate three MoE variants: Hard MoE, Soft MoE, and Cluster MoE against dense baselines across three CNN architectures (ResNet-20, VGG11_bn, VGG19_bn) on CIFAR-100 and a Vision Transformer (ViT-Small) on Tiny ImageNet-200 dataset, using eight 8-bit signed multipliers (including one exact baseline) from the EvoApproxLib library. Results show that, without retraining, the Dense baseline is the most resilient topology across all CNN architectures, whereas on ViT-Small, all topologies degrade at comparable rates regardless of routing strategy. After approximate-aware retraining, recovery varies substantially across architectures, topologies, and multipliers. ResNet-20 achieves full recovery across the entire multiplier range, whereas VGG architectures recover at moderate multipliers but fail irreversibly at aggressive ones for all topologies except Cluster MoE on VGG11_bn; on ViT-Small, Hard MoE outperforms Dense under aggressive approximation at equal normalized inference cost. These results pave the way for future approximate MoE hardware-software co-design strategies.

Omkar B Shende, Marcello Traiola, Gayathri Ananthanarayanan

Dept. of Computer Science and Engineering · Univ Rennes, CNRS, Inria, IRISA

Efficient Neural Network Inference Edge Inference Mixture-of-Experts Models Approximate Computing

May 6, 2026cs.LG

FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation

Adapting pretrained models typically involves a trade-off between the high training costs of backpropagation and the heavy inference overhead of memory-based or in-context learning. We propose FAAST, a forward-only associative adaptation method that analytically compiles labeled examples into fast weights in a single pass. By eliminating memory or context dependence, FAAST achieves constant-time inference and decouples task adaptation from pretrained representation. Across image classification and language modeling benchmarks, FAAST matches or exceeds backprop-based adaptation while reducing adaptation time by over 90% and is competitive to memory/context-based adaptation while saving memory usage by up to 95%. These results demonstrate FAAST as a highly efficient, scalable solution for supervised task adaptation, particularly for resource-constrained models. We release the code and models at https://github.com/baoguangsheng/faast.

Guangsheng Bao, Hongbo Zhang, Han Cui +4

Zhejiang University · Westlake University · Shanghai Polytechnic University +1

Efficient Neural Network Inference Test-Time Adaptation

May 5, 2026cs.LG

Spatiotemporal Convolutions on EEG signal -- A Representation Learning Perspective on Efficient and Explainable EEG Classification with Convolutional Neural Nets

Classification of EEG signals using shallow Convolutional Neural Networks (CNNs) is a prevalent and successful approach across a variety of fields. Most of these models use independent one-dimensional (1D) convolutional layers along the spatial and temporal dimensions, which are concatenated without a non-linear activation layer between. In this paper, we investigate an alternative encoding that operates a bi-dimensional (2D) spatiotemporal convolution. While 2D convolutions are numerically identical to two concatenated 1D convolutions along the two dimensions, the impact on learning is still uncertain. We test 1D and 2D CNNs and a CNN+transformer hybrid model in a low-dimensional (3-channel) and a high-dimensional (22-channel) BCI motor imagery classification task. We observe that 2D convolutions significantly reduce training time in high-dimensional tasks while maintaining performance. We investigate the root of this improvement and find no difference in spectral feature importance. However, a clear pattern emerges in representational similarity across models: 1D and 2D models yield vastly different representational geometries. Overall, we suggest an improved model with a 2D convolutional layer for faster training and inference. We also highlight the importance of architecturally-driven encoding when processing complex multivariate signals, as reflected in internal representations rather than purely in performance metrics.

Laurits Dixen, Stefan Heinrich, Paolo Burelli

IT University of Copenhagen Denmark

Efficient Neural Network Inference MI Classification EEG Decoding Convolutional Neural Networks

May 5, 2026cs.CV

Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUs

While deep-learning-based image restoration has achieved unprecedented fidelity, deployment on mobile Neural Processing Units (NPUs) remains bottlenecked by operator incompatibility and memory-access overhead. We propose an NPU-aware hardware-algorithm co-design approach for real-world image denoising on mobile NPUs. Our approach employs a high-capacity teacher to supervise a lightweight student network specifically designed to leverage the tiled-memory architectures of modern mobile SoCs. By prioritizing NPU-native primitives -- standard 3x3 convolutions, ReLU activations, and nearest-neighbor upsampling -- and employing a progressive context expansion strategy (up to 1024x1024 crops), the model achieves 37.66 dB PSNR / 0.9278 SSIM on the validation benchmark and 37.58 dB PSNR / 0.9098 SSIM on the held-out test benchmark at full resolution (2432x3200) in the Mobile AI 2026 challenge. Following the official challenge rules, the inference runtime is measured under a standardized Full HD (1088x1920) protocol, where it runs in 34.0 ms on the MediaTek Dimensity 9500 and 46.1 ms on the Qualcomm Snapdragon 8 Elite NPU. We further reveal an "Inference Inversion" effect, where strict adherence to NPU-compatible operations enables dedicated NPU execution up to 3.88x faster than the integrated mobile GPU. The 1.96M-parameter student recovers 99.8% of the teacher's restoration quality via high-alpha knowledge distillation (alpha = 0.9), achieving a 21.2x parameter reduction while closing the PSNR gap from 1.63 dB to only 0.05 dB. These results establish hardware-aware distillation as an effective strategy for unifying high-fidelity denoising with practical deployment across diverse mobile NPU architectures. The proposed lightweight student model (LiteDenoiseNet) and its training statistics are provided in the NN Dataset, available at https://github.com/ABrain-One/NN-Dataset.

Faraz Kayani, Sarmad Kayani, Asad Ahmed +2

Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany

Efficient Neural Network Inference Knowledge Distillation On-Device Inference Image Denoising

May 5, 2026cs.SD

Smart Passive Acoustic Monitoring: Embedding a Classifier on AudioMoth Microcontroller

Passive Acoustic Monitoring (PAM) is an efficient and non-invasive method for surveying ecosystems at a reduced cost. Typically, autonomous recorders allow the acquisition of vast bioacoustic datasets which are then analyzed. However, power consumption and data storage are both scarce and limit the duration of acquisition campaigns. To address this issue, we propose a smart PAM system which allows the in-situ analysis of the soundscape by embedding a classifier directly onto an AudioMoth microcontroller. Specifically, we propose an optimized yet simple 1D Convolutional Neural Network (1D-CNN) to classify the raw audio. The model focuses on the specific call of Scopoli Shearwater seabirds (endangered species) and is trained on a real-world dataset with a classification accuracy of 91% (balanced accuracy of 89%). We also propose a process to optimize the model to fit the severe resource constraints of the AudioMoth, achieving a ~10kB RAM memory footprint and 20ms inference time. Finally, we present an open-source tutorial of our model optimization and export strategy which can be used for embedding models beyond the scope of our study. Our modified version of the AudioMoth firmware adds two functions: (F1) which selectively records data when the target species has been detected and (F2) which logs the continuous classification results in real time. This work intends to facilitate the conception of intelligent sensors, enhancing the efficiency and scalability of bioacoustic monitoring campaigns.

Louis Lerbourg, Paul Peyret, Juliette Linossier +1

Univ. Grenoble Alpes, CEA, List, Grenoble, France · Biophonia, France

Efficient Neural Network Inference Sound Event Detection Edge ML Bioacoustics Passive Acoustic Monitoring Audio Classification+1

May 4, 2026cs.AR

Resource Utilization of Differentiable Logic Gate Networks Deployed on FPGAs

On-edge machine learning (ML) often strives to maximize the intelligence of small models while miniaturizing the circuit size and power needed to perform inference. Meeting these needs, differentiable Logic Gate Networks (LGN) have demonstrated nanosecond-scale prediction speeds while reducing the required resources as compares to traditional binary neural networks. Despite these benefits, the trade-offs between LGN parameters and resulting hardware synthesis characteristics are not well characterized. This paper therefore studies the tradeoffs between power, resource utilization, inference speed, and model accuracy when varying the depth and width of LGNs synthesized for Field Programmable Gate Arrays (FPGA). Results reveal that the final layer of an LGN is critical to minimize timing and resource usage (i.e. 28% decrease), as this layer dictates the logic size of summing operations. Subject to timing and routing constraints, deeper and wider LGNs can be synthesized for FPGA when the final layer is narrow. Further tradeoffs are presented to help ML engineers select baseline LGN architectures for FPGAs with a set number of Look Up Tables (LUT).

Stephen Wormald, Gilon Kravatsky, Damon Woodard +1

Florida Institute of National Security University of Florida Gainesville, Florida, USA

Efficient Neural Network Inference Edge ML FPGA-Based Neural Network Acceleration

May 4, 2026eess.SP

Memory-Efficient EDA Denoising via Knowledge Distillation for Wearable IoT Under Severe Motion Artifacts and Underwater Conditions

Electrodermal activity (EDA) is widely used in wearable Internet of Medical Things (IoMT) systems for continuous health monitoring, including autonomic assessment. However, EDA signals are highly vulnerable to motion artifacts and environmental noise, limiting reliable deployment in harsh operating conditions such as underwater. This study proposes a robust, deployable EDA denoising framework that generalizes across multiple measurement locations and harsh environments. The framework integrates a hybrid CNN-Transformer teacher model with a lightweight depth-wise separable CNN student model via a knowledge distillation (KD) strategy. To further improve robustness, a realistic data augmentation scheme is introduced to simulate diverse motion artifacts and environmental distortions. The KD-based student model significantly reduces model size (7.87 MB to 0.51 MB) and computational cost (105.1M to 11.61M FLOPs) while maintaining denoising performance (MAE: 0.144, SNR improvement: 12.08 dB) using the public dataset validation. In real-world underwater conditions (UMAC dataset) testing, the proposed method substantially improves skin conductance response reconstruction, reducing mean absolute error from 2.809 to 0.215. Furthermore, on independent testing using the CNS-OT dataset, the denoised signals enhanced downstream CNS-OT prediction performance, achieving the highest AUROC (0.806) compared to prior denoising methods. The proposed method also improved the early prediction rate (sensitivity) from 0.550 to 0.767, enabling CNS-OT prediction up to a median of 6.9 minutes before symptom onset. These results demonstrate that the proposed framework not only improves EDA signal quality but also enhances clinically relevant prediction performance while remaining suitable for deployment in resource-constrained wearable Internet of Things systems operating in harsh environments.

Yongbin Lee, Andrew Peitzsch, Youngsun Kong +4

Department of Biomedical Engineering, University of Connecticut, Storrs, CT 06269, USA

Efficient Neural Network Inference Wearable Health Monitoring

May 4, 2026cs.AI

Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions

The efficacy of deep neural networks is heavily reliant on the design of non-linear activation functions, yet existing approaches often struggle to balance optimization stability with computational efficiency. While piecewise linear functions offer inference speed, they suffer from optimization instability due to non-differentiability at the origin, whereas smooth counterparts typically incur significant computational overhead through their reliance on transcendental operations. To address these limitations, this paper proposes a general smoothing framework based on constructive approximation theory and introduces the Bernstein Linear Unit (BerLU). This novel activation function utilizes Bernstein polynomials to construct a differentiable quadratic transition region that effectively eliminates singularities while maintaining a piecewise linear structure. Theoretical analysis demonstrates that the proposed method guarantees strictly continuous differentiability and a non-expansive Lipschitz constant of one, which ensures stable gradient propagation and prevents the gradient explosion problems common in deep architectures. Comprehensive empirical evaluations across representative Vision Transformer and Convolutional Neural Network architectures confirm that this approach consistently outperforms state-of-the-art baselines on standard image classification benchmarks while delivering superior computational and memory efficiency.

Wentao Zhang, Yutong Zhang, Yifan Zhu +1

Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · College of Computer Science, Sichuan University, Chengdu, China

Efficient Neural Network Inference Neural Network Optimization Neural Network Activation Functions

May 4, 2026cs.CV

Low-Latency Embedded Driver Monitoring System with a Multi-Task Neural Network

Road traffic accidents remain a significant global concern, with the majority attributed to human factors such as driver distraction and fatigue. This study proposes a camera-based approach to derive useful indicators to assess driver attentiveness and alertness. The proposed pipeline jointly satisfies the stringent real-time requirements imposed by the critical application and minimizes the computational requirements to allow for deployment on a tight computational budget. To this end, we develop a lightweight multi-task neural network that predicts multiple indicators for the face region in a single forward pass. The developed model is integrated into a complete execution workflow to produce a real-time estimate of attentiveness, fatigue, and engagement in distracting activities.

Carmelo Scribano, Giovanni Cappelletti, Elia Giacobazzi +3

University of Modena and Reggio Emilia, Italy

Efficient Neural Network Inference Driver Monitoring