Efficient Neural Network Inference

Follow this topic Ask about this topic

Momentum

39 papers in the last four weeks, up 225% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 314

Aug 3, 2026cs.LG

Gecko: Fast Private Inference via Secure Public Encoder Offloading

Private inference protects both user inputs and server models during neural network inference, but existing solutions remain too slow for practical deployment. This motivates recent efforts to run a public encoder, such as a pretrained backbone, outside the protection boundary and evaluate only a small private predictor cryptographically. While appealing for efficiency, this design is not inherently secure: naively offloading a public encoder may create a feature-space shortcut: an extraction adversary may learn the remaining private predictor's feature-to-output mapping more easily than the original model's input-to-output behavior. We present Gecko, designed to limit this additional risk while retaining a compact encrypted predictor. We leverage a frozen backbone that contributes hierarchical features, fixed Fastfood projections that compress them, and private feature gating that prepares them for prediction. We formalize ideal independence and information-preservation conditions as design guidance, then separately evaluate component-reuse extraction attacks. Across image and audio tasks, Gecko achieves 0.4-2.2 second inference with at most 10.8 MB communication and accuracy comparable to transfer-learning baselines. Under the evaluated attacks, reusing the offloaded public encoder provides no significant advantage to model-extraction adversaries. Source code and a demo are available at https://github.com/CassiniHuy/gecko-infer.

Cheng'an Wei, Kai Chen, Yue Zhao +2

Institute of Information Engineering, Chinese Academy of Sciences, China, and the School of Cyber Security, University of Chinese Academy of Sciences, China

Efficient Neural Network Inference Model Extraction Attacks

Aug 3, 2026cs.CV

SpikeRestormer: Towards Energy-Efficient All-in-One Image Restoration via Unified Event Reasoning

ANN-based All-in-One image restoration (AiOIR) unifies diverse degradation handling but incurs high computational costs, limiting its real-time deployment. While Spiking Neural Networks (SNNs) offer a low-power alternative, applying them to static images remains challenging. This difficulty arises because explicit event signals are absent, and degradation cues are heavily entangled with scene structures, hindering the learning of reliable restoration-oriented spike events. To address these issues, we propose SpikeRestormer, an energy-efficient SNN for AiOIR that performs event reasoning over internally generated spike cues. Specifically, we propose a degradation-event perception process to extract spike-based degradation events through Subtractive Degradation Event Attention (SDEA). Moreover, we introduce Hierarchical Bayesian Skip Masking (HBSM) and Additive Restoration Event Attention (AREA) processes for event-reliability inference and restoration-event construction, respectively. By integrating these complementary processes, SpikeRestormer formulates restoration as a unified process of degradation-event perception, degradation-event reliability inference, and restoration-event construction, liberating the potential of SNNs for energy-efficient AiOIR. Extensive experiments show that SpikeRestormer delivers competitive performance against ANN-based methods and establishes new state-of-the-art results among SNN-based methods with significantly lower energy consumption.

Shengkai Hu, Jie Shao, Jiaqi Ma +5

School of Information Engineering, Zhongnan University of Economics and Law, Wuhan, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates · School of Computer Science, Wuhan University, Wuhan, China +1

Efficient Neural Network Inference All-in-One Image Restoration Spiking Neural Networks Image Restoration

Aug 3, 2026cs.LG

CARE: A Cascaded Framework for Efficient and Reliable Time Series Anomaly Detection

While deep learning models have achieved state-of-the-art performance in time series anomaly detection, their complex architectures incur substantial inference overhead. Existing methods typically apply a uniform inference strategy across all data points, which is inefficient given that anomalies are inherently scarce and the vast majority of temporal data consists of predictable normal patterns. To mitigate this bottleneck, we propose CARE, a model-agnostic cascaded inference framework that integrates a Lightweight Pre-filter Model (LPM) with an existing high-capacity Complex Detection Model (CDM). The LPM rapidly filters high-confidence normal samples using a Residual MLP AutoEncoder and a Normality-Conditioned Gating mechanism. Crucially, we introduce a Structure Attention module to explicitly capture channel-wise anomaly contributions, and optimize the gating network via a confidence-guided selective routing objective that learns reliable routing decisions to reduce unnecessary CDM invocations. Extensive experiments across eight real-world benchmarks demonstrate that CARE effectively isolates high-confidence normal samples. By routing only uncertain samples to the CDM, our framework achieves

2.7\times

to

4.8\times

inference speedup compared to the most accurate SOTA approaches, while still maintaining competitive detection quality.

Zemin Chao, Qianhui Xu, Jianhe Cen +3

Massive Data Computing Lab, Harbin Institute of Technology

Efficient Neural Network Inference Cost-Aware Inference Time-Series Anomaly Detection

Aug 3, 2026cs.NE

Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

Handwritten Text Recognition (HTR) is computationally imbalanced in two ways: most image pixels are background, and many width-axis sequence positions are blank-dominated. This creates a mismatch for Spiking Neural Networks (SNNs): handwriting is observed as a static image, whereas spiking computation unfolds over timesteps. We propose Spike-HTR, a hybrid spiking recognizer that controls both the number of spiking steps and the number of width positions processed by the deep sequence mixer. To make a static image suitable for short-horizon spiking inference, InkCoder converts it into a coarse-to-fine input stream, where early steps cover broad stroke regions and later steps emphasize sharper stroke details. To reduce sequence computation, a CTC-guided length reducer keeps likely character or uncertain positions and compresses long blank-dominated stretches before deep mixing. With

T{=}2

, Spike-HTR trains only on target data, decodes without language models or lexicons, and reaches validation/test CERs of 3.5/5.4, 2.3/2.5, and 4.2/3.9 on IAM, LAM, and READ2016. Codes are available at https://github.com/QomolangmaH/SpikeHTR.

Xiubo Liang, Jinxing Han, Yuke Li +3

School of Software Technology, Zhejiang University, Ningbo, China

Efficient Neural Network Inference Spiking Transformers Spiking Neural Networks

Aug 3, 2026cs.LG

Meganeura: Portable GPU Training and Inference through Vulkan and Metal

Training and deployed inference often cross export, conversion, and platform-specific runtime boundaries. Meganeura asks whether one compact native compiler can span both phases on consumer GPUs. Its typed static graph, automatic differentiation, optimizer, checkpoint, memory planner, and runtime lower specialized programs through Vulkan and Metal. We compare five matched workloads with PyTorch on NVIDIA and AMD discrete GPUs, an AMD APU, Apple silicon, and an Intel iGPU. The protocol separates strict f32 from validated fast paths and gates forward and backward independently. Forty-eight of 50 device-workload-mode cells pass both gates; the other two share one unresolved backward-reference disagreement on a newly supported APU. In strict f32, Meganeura wins 12 of 20 GPU-referenced minimal-latency cells and has a median valid training gap of 1.8x. On the discrete AMD GPU, four of five inference workloads are within 1.10x of compiled ROCm PyTorch and three training workloads are faster. Under accelerated contracts, the worst training gap is 4.6x. Compilation takes 0.1-2.4 seconds versus 6-96 seconds for torch.compile on supported GPU paths; the stripped binary is 13 MiB. Dispatch profiles localize the largest gaps to convolution derivatives and attention backward. A physical Android XR case study transfers a Meganeura-trained decoder into an Adreno/OpenXR application sharing the graphics queue. The results show that general consumer graphics APIs can support a compact shared train-to-deploy stack at useful, sometimes vendor-competitive performance. The measured gaps point to kernel coverage, scheduling, and arithmetic policy rather than an identified API limitation.

Dzmitry Malyshau

Independent Researcher

Efficient Neural Network Inference GPU Kernel Optimization GPU Acceleration GPU Kernel Generation

Aug 2, 2026cs.LG

Breaking Diversity Collapse in Spiking Pseudo-Ensembles for Efficient OOD Detection in Remote Sensing

Spiking Neural Networks (SNNs) are attractive for resource-constrained remote-sensing systems, but reliable out-of-distribution (OOD) detection remains challenging. Deep ensembles provide strong predictive uncertainty, yet require multiple complete models and backbone evaluations. We propose an efficient spiking pseudo-ensemble that attaches multiple lightweight classification heads to a frozen SNN backbone. Naively training these heads with cross-entropy can lead to diversity collapse, where independently parameterized heads may produce correlated predictions. To address this, we introduce an agree--disagree objective that preserves correct predictions on clean in-distribution samples while encouraging diversity on structured, uncertainty-inducing transformations of the same inputs. This provides a diversity-promoting training signal without requiring external OOD data. Experiments with Spikformer and ResNet19-SNN on EuroSAT demonstrate consistent improvements over conventionally trained pseudo-ensembles. Using three backbones with five heads each matches or improves upon a five-model deep ensemble on UCM and AID, while requiring approximately 38% fewer parameters and 40% fewer backbone evaluations. These results show that explicit diversity promotion can recover useful ensemble-style uncertainty at substantially lower deployment cost.

Srinivas Anumasa, Rushi Shah, Qiran Zou +1

National University of Singapore

Efficient Neural Network Inference Ensemble Learning Remote Sensing Spiking Neural Networks OOD Detection Ensemble Diversity

Aug 1, 2026cs.LG

An Embedded RISC-V Evaluation of Kolmogorov--Arnold Networks in Hard-Constrained Recurrent Physics-Informed Models

Hard-constrained recurrent physics-informed networks (HRPINNs) embed known dynamics inside a recurrent numerical integrator and restrict a neural branch to learning only the residual dynamics that the first-principles model does not capture. Kolmogorov--Arnold Networks (KANs) have been proposed as parameter-efficient replacements for multilayer perceptrons (MLPs) in such residual branches, but their learnable B-spline activations follow a markedly different execution profile. Building on prior work that characterized when a vanilla B-spline KAN matches or underperforms an MLP as an HRPINN residual branch in discovery accuracy, this paper asks whether that parameter efficiency survives deployment. Using identical trained weights, we measured execution latency, energy per integration step, and dependability under post-training quantization in the closed recurrent loop on a RISC-V RV64GC platform without vector extensions (StarFive VisionFive~2, SiFive U74). For the two accuracy-comparable pairs, the KAN residual branch executed

13.5\times

and

8.0\times

slower and consumed

11.3\times

and

5.6\times

more energy per integration step (3.7,

μ

J against 0.33,

μ

J for the smallest pair); across all four parameter-matched size tiers the ranges are

4.7\times

--

14.5\times

and

4.7\times

--

18.7\times

. Under INT8 quantization, KAN trajectories diverged up to

43\times

earlier than matched MLPs; the damage traces to weight quantization, not to input-side knot-interval misassignment. These results indicate that the parameter efficiency reported for KANs does not transfer to deployment cost on scalar embedded cores, and that an MLP residual branch is the more dependable default for embedded HRPINN deployment unless specific quantization co-design is used.

Enzo Nicolas Spotorno, Josafat Leal Filho

Department of Informatics and Statistics - Graduate Program in Computer Science (PPGCC)

Efficient Neural Network Inference Kolmogorov-Arnold Networks Physics-Informed ML On-Device Inference

Aug 1, 2026cs.AR

CascadeLUT: Information-Ordered Streaming Inference for Bandwidth-Constrained FPGAs

Mapping neural networks to FPGAs enables low-latency, energy-efficient inference, particularly for lookup table (LUT)-based models that eliminate multipliers and map directly to reconfigurable fabric. While prior work achieves high compute efficiency, it typically assumes full-sample availability, causing pipeline stalls in bandwidth-limited streaming scenarios. Here, the bottleneck shifts from computation to data movement, as large input transfers limit throughput and energy efficiency. We present CascadeLUT, an information-structured inference framework organized around bandwidth constraints. Instead of buffering the full input, features are partitioned into ordered subsets and predictions are progressively refined as subsets arrive. The cascade statically controls which layers consume incoming features, enabling deterministic streaming inference without runtime branching. By co-designing feature scheduling with hardware dataflow, CascadeLUT reduces data movement while maintaining accuracy. Across datasets, it achieves 4.0 to 12.5 times lower latency, 3.0 to 5.0 times higher throughput and up to 13.8 times lower energy/sample than prior LUT baselines, using 1.2 to 4.4 times the LUTs of the smallest DWN baseline per task. We also demonstrate on-device input quantization integrated with LUT-based inference and present end-to-end FPGA results on real-world workloads, with 5 times reductions in quantization overhead.

Oliver Cassidy, Marta Andronic, George A. Constantinides

Department of Electrical and Electronic Engineering, Imperial College London, UK

Efficient Neural Network Inference FPGA-Based Neural Network Acceleration

Jul 31, 2026cs.CV

Lightweight Neural Networks for Affordance Segmentation: Enhancement of the Decoder Module

The deployment of deep neural networks for visual affordance segmentation on wearable robots poses may prove critical, due to some conflicting aspects of the problem. On one hand, affordance segmentation requires high-level abstraction capabilities, that typically involve large-size models. On the other hand, computing resources hosted on wearable robots prevent to run large-size models in real-time. The paper presents an analysis of the role of the segmentation head in the trade-off between generalization performance and compute cost. The obtained models outperform modern baseline solutions in well-known, real-world datasets while meeting low computing requirements.

Simone Lugani, Edoardo Ragusa, Rodolfo Zunino +1

Department of Electrical, Electronic, Telecommunications Engineering and Naval Architecture (DITEN), University of Genoa, Genoa, Italy

Efficient Neural Network Inference Robotic Perception Semantic Segmentation

Jul 30, 2026cs.LG

SEDR-Seq2P: A Lightweight Dilated Residual Sequence-to-Point Network for Multi-Task Industrial NILM

Industrial NILM remains challenging because measurement noise and widespread concurrent machine operation reduce the generalization of models tuned on residential data. This work adopts a one-to-many, multi-task disaggregation setting, in which a single network estimates multiple industrial machine loads from aggregate power. Under a unified evaluation protocol on IMDELD, we benchmark Seq2Seq, Seq2SubSeq, Seq2Point, GRU, and WaveNet using energy-estimation metrics and the accuracy-delay criterion. While Seq2Point offers a stronger accuracy-delay balance than Seq2Seq/Seq2SubSeq, GRU and WaveNet achieve higher accuracy at markedly higher computational cost. To close this gap, we propose SEDR-Seq2P, a lightweight Seq2Point extension with dilated residual blocks and squeeze-and-excitation attention. Relative to the Seq2Point baseline, SEDR-Seq2P reduces MAE by approximately 7%, improves the coefficient of determination by approximately 1%, and increases the match rate by approximately 0.8%. In addition, compared to WaveNet, SEDR-Seq2P reduces inference latency by approximately 58%, yielding a favorable accuracy-delay trade-off for scalable industrial deployment.

Hatem Haddad, Feres Jerbi, Issam Smaali

Wattnow Tunis, Tunisia

Non-Intrusive Load Monitoring Efficient Neural Network Inference Multi-Task Learning Sequence Modeling

Jul 29, 2026cs.NE

The Sparsity Ceiling: Where Spiking Networks Can and Cannot Trade Activity for Energy

Spiking neural networks (SNNs) are promoted as an energy-efficient substrate because sparse, event-driven activity replaces dense multiply-accumulates with cheap accumulates. We argue the energy dividend of sparsity is not a property of SNNs but of the task. Holding architecture fixed and swapping only the hidden unit (continuous vs. leaky-integrate-and-fire), plus a two-sided target-firing-rate probe, we measure how far activity can be pushed down before quality breaks. Low-load feed-forward perception sparsifies to 5% firing at no accuracy cost; a recurrent language model cannot go below ~50% -- the recurrent state must stay active to carry information. A spiking Transformer, by contrast, sparsifies freely to 2% (3 seeds) -- so the ceiling is a property of recurrent compression, not sequence modeling. Attention escapes the floor only by storing the full key-value cache, trading a firing floor for a memory wall: on neuromorphic hardware, recurrence and attention pay on different axes, neither escapes. We formalize the ceiling with an information-theoretic bound rho >= H_b^{-1}(log2 M / H) and confirm its predictions: the floor rises with memory load, falls with state width, and (refuting a naive memory-only reading) rises with task difficulty. A layer-wise input floor further caps op reduction under dense input, isolating event-driven perception as where neuromorphic hardware wins.

Zeyu Wang

Georgia Institute of Technology

Efficient Neural Network Inference Spiking Transformers Activation Sparsity Recurrent Neural Networks Spiking Neural Networks Neuromorphic Computing

Jul 29, 2026cs.DC

ServerlessT2I: Efficient Text-to-Image Workflow Serving on a Serverless Platform

Text-to-image (T2I) workflows are increasingly deployed on serverless platforms because users often compose customized workflows and invoke them intermittently. Existing platforms typically deploy each workflow as an opaque GPU function, provisioning, placing, and scaling all constituent models in the workflow together. This monolithic design obscures workflow structure, inflates scaling overhead, forces users to manage low-level GPU coordination, and limits fine-grained fairness in multi-tenant clusters. In this paper, we present ServerlessT2I, a serverless-native system that decomposes a T2I workflow into loosely coupled model functions that can be independently managed and scheduled. By explicitly managing individual model execution, ServerlessT2I enables per-model scaling, declarative workflow composition, transparent GPU-resident communication, and fairness-aware scheduling. To make this decomposition efficient, ServerlessT2I harvests slack GPU memory left idle by compute-bound T2I inference to build a data plane that reduces model loading and data communication overheads. \sys{} further introduces a fair scheduler for multi-tenant serving. Using production traces, ServerlessT2I sustains up to 2

\times

higher request rates than existing T2I workflow serving systems with the same GPU budget; for a fixed request rate, it saves up to 3

\times

GPU resources while satisfying service level objectives (SLOs).

Xiaoxiao Jiang, Suyi Li, Sheng Yao +5

Hong Kong University of Science and Technology · Alibaba Group

Efficient Neural Network Inference T2I Generation

Jul 28, 2026cs.AR

At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

Fine-grained weight pruning and activation sparsification have emerged as effective approaches for reducing the compute and memory cost of inference for Transformer models. In the moderate-sparsity regime, Gustavson's dataflow provides a natural execution model for exploiting both activation and weight sparsity on vector processors through metadata-driven indexed accumulation. However, existing RVV architectures lack native support for this pattern, forcing kernels to rely on software index decoding and L1-backed indexed memory operations that keep sparse tensor contractions far below their roofline performance bound. We present Ventaglio, a runtime-configurable sparse execution unit coupled with RVV ISA extensions that drives sparse tensor contractions toward their roofline through indexed gather-accumulate-scatter support. Integrated into an open-source vector processing cluster and implemented in 12nm FinFET, Ventaglio accelerates sparse tensor contraction kernels by

6.9\text{--}7.4\times

over optimized RVV baselines, with only

3.1\%

area overhead for a cluster of tightly-L1 coupled vector processing elements. We build a performance-accurate instruction-level model of the Ventaglio extension, calibrate it against RTL implementation, and leverage it for scale-out performance analysis on a large

4\times4

multi-cluster system. Using a DuoGPT-pruned LLaMA-3-8B model with practical

40\text{--}60\%

dual sparsity, Ventaglio achieves

2.40\text{--}5.25\times

and

2.06\text{--}3.16\times

speedup over dense baselines during prefill and autoregressive decoding, respectively.

Bowen Wang, Chi Zhang, Diyou Shen +3

*IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy

Efficient Neural Network Inference Transformer Inference Activation Sparsity LLM Inference Acceleration

Jul 28, 2026eess.AS

faster-enhancer.c: A Dependency-Free int8 Runtime for Streaming Speech Enhancement on Commodity CPUs

This is an implementation and measurement study of what it costs to run a streaming speech enhancer on a CPU. We port FastEnhancer-Medium at 48 kHz to faster-enhancer.c, a C runtime with six int8 GEMM tiers selected at initialization, leaving architecture and weights untouched. One Apple M2 core reaches 0.069 real-time factor, against 0.230 for the fp32 ONNX Runtime graph on the same machine, a 3.3x speedup. A Galaxy S23+ (Snapdragon 8 Gen 2) reaches 0.096. The speedup comes from specializing every layer of the runtime around one fixed model. Activation ranges are recomputed per frame, so no calibration set is needed; the k=3 convolutions use Winograd F(2,3); cross-stage state is fp16; the GRU and the dequantization epilogues are fused; and nothing is allocated after startup. Over 824 VoiceBank-DEMAND utterances the engine tracks fp32 to within -0.006 PESQ and -0.08 dB SNR. Speed alone does not settle deployment cost. The enhancer holds a fraction of a core for as long as the microphone is open, so its real-time factor is a duty cycle. A benchmark races through a file; an audio callback does not. Pacing to the 6.67 ms deadline costs 4.2x per frame, saves 49% of the energy, and leaves the cheapest core placement missing 96% of its deadlines. All SIMD tiers within an architecture family emit byte-identical output. The runtime is released as a dependency-free library.

Gyeongmin Kim

Department of Computer Science, Hanyang University, Seoul, Korea

Efficient Neural Network Inference Speech Enhancement On-Device Inference

Jul 24, 2026cs.DC

Unified Static-Dynamic Pruning for Efficient LLM Inference

The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP), which permanently removes redundant weights but lacks adaptivity, or dynamic pruning (DP), which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs. SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format-kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24x-1.37x average speedup (up to 2.51x) over state-of-the-art sparse frameworks such as SpInfer, while matching perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency-quality Pareto frontier, showing that unified static-dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving.

Jinhyeok Kim, Yejoon Lee, Jaeyoung Do

Seoul National University Seoul, South Korea

Efficient Neural Network Inference LLM Pruning LLM Inference Acceleration Neural Network Pruning

Jul 24, 2026cs.CE

Generalized Neural Operator for Parametric and Boundary-Value Problems

Developing foundational neural simulators for Partial Differential Equations (PDEs) requires robust generalization across diverse physical parameters and boundary conditions. However, current deep learning approaches largely face a structural trade-off between condition-agnostic deployment and physical fidelity. Purely data-driven operators infer the underlying physics implicitly and thus lack the explicit constraints needed to ensure physically valid solutions across varying domains, rendering the learning problem ill-posed. On the other hand, Physics-Informed Neural Networks (PINNs) enforce rigorous physical constraints but necessitate costly, instance-specific optimization. Furthermore, the massive scale of emerging foundational operators has severely degraded their inference speeds, making them computationally uncompetitive with traditional numerical solvers. To address this bottleneck between condition-agnostic deployment, physical rigor, and inference efficiency, we propose a \textit{Generalized Neural Operator}. By formalizing the classical conditions for well-posedness within neural operators, our framework demonstrates the theoretical benefits of explicitly conditioning on PDE parameters and boundary conditions. To implement this synthesis without compromising computational speed, we introduce three novel architectural components: a parameter-gated mixture of kernels for efficient parameter generalization, a generalized boundary transfer operator that projects arbitrary boundary constraints into a unified latent Dirichlet representation, and a specialized training objective to ensure stability. Extensive experiments demonstrate that our theoretically grounded approach achieves superior generalization across heterogeneous physical regimes while maintaining strict inference efficiency comparable to conventional numerical baselines.

Ruoyan Li, Yizhou Sun, Wei Wang

University of California, Los Angeles

Efficient Neural Network Inference PDE Solving PDE Operator Learning Physics-Informed Neural Operators Neural Operators

Jul 23, 2026cs.CV

Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform

Equivariant networks embed geometric symmetries as structural priors through weight sharing, achieving remarkable parameter efficiency across vision tasks. However, this parameter efficiency does not translate into compute efficiency: most existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels, so an equivariant layer costs no fewer MACs than its non-equivariant counterpart. In this paper, we observe that the equivariant linear (EQ-Linear) layer---the most fundamental and frequently used module in modern equivariant architectures---is essentially a circular convolution along the group dimension composed with a linear transform along the channel dimension. Building on this observation, we propose Flash EQ-Linear, an exact acceleration algorithm that reduces the cost to

2(T-1)/T^2

of the original dense formulation (

T

is the equivariant group size) by combining the Fourier convolution theorem along the group dimension with the conjugate symmetry of the real DFT. To translate these computational savings into wall-clock speedups, we further develop dedicated CUDA kernels for the

\mathrm{p}4

group. At the operator level, Flash EQ-Linear achieves up to

2.1\times

forward speedup over PyTorch's highly optimized F.linear; at the network level, Flash EQ-ViT achieves up to

{1.7\times}

end-to-end speedup over both equivariant and non-equivariant baselines. As an operator-level acceleration algorithm, Flash EQ-Linear provides plug-and-play acceleration for diverse pretrained equivariant models, including EQ-ViT, EQ-Swin, EQ-VMamba, and EQ-INR, without retraining or architectural changes. Code is available at https://github.com/zhongchenzhao/FlashEQLinear.

Zhongchen Zhao, Jixin Wang, Qi Xie +4

Xi’an Jiaotong University · The Hong Kong Polytechnic University · OPPO Research

Efficient Neural Network Inference Equivariant Neural Networks

Jul 23, 2026cs.SD

VibeVoice-ASR-BitNet Technical Report

We present VibeVoice-ASR-BitNet, a compressed variant of VibeVoice-ASR optimized for real-time inference on edge CPUs. We apply heterogeneous quantization tailored to the computational characteristics of each stage: the VAE acoustic tokenizer uses full-pipeline INT8 quantization (I8_S) with kernel fusion and SIMD optimization, while the autoregressive language model adopts BitNet-style ternary weights (I2_S). To preserve accuracy under aggressive compression, we employ a progressive quantization-aware training strategy. For inference, we implement custom SIMD kernels and fused operators within the ggml framework targeting both ARM and x86 platforms, achieving real-time recognition (RTF < 1) on low-thread-count CPUs. VibeVoice-ASR-BitNet is 1.6--2.3x faster than Whisper.cpp at comparable model sizes (~1.6 GB), with only modest accuracy degradation compared to the FP16 baseline.

Songchen Xu, Ting Song, Shaohan Huang +10

Shanghai Jiao Tong University · Microsoft Research · University of Chinese Academy of Sciences +1

Efficient Neural Network Inference Automatic Speech Recognition On-Device Inference

Jul 23, 2026cs.AI

Efficient and Interpretable Body-Based Emotion Recognition with Lightweight Temporal Convolutional Networks

Body-based emotion recognition is important for real-time affective systems, but graph-based skeleton models can be computationally expensive. This paper studies whether lightweight temporal convolutional networks (TCNs) can provide an efficient and interpretable alternative for body-based emotion classification. We evaluate a family of TCN models on DIEM-A and compare them with a graph-based time-series graph (G-TSG) baseline using accuracy, macro-F1, parameter count, and inference latency. Although G-TSG achieves the highest mean performance, TCN-Base remains within

1.58

accuracy points and

1.25

macro-F1 points while using

79.18\%

fewer parameters and reducing classifier latency by approximately

12.5\times

. We also analyze body-region contributions using region-specific TCN models, zero-based occlusion, and G-TSG gradient saliency. The results show that upper-body motion provides the strongest standalone regional cue, that the usefulness of body regions varies across emotions, and that different interpretability methods capture distinct aspects of model behavior. These findings suggest that lightweight TCNs can support efficient body-based emotion recognition while also providing practical insight into how motion cues contribute to classification.

Christian Arzate Cruz, Stefanos Gkikas, Houshyar Asadi

Honda Research Institute Japan Wako City, Japan · Institute for Intelligent Systems Research and Innovations (IISRI) Deakin University Geelong, Australia

Efficient Neural Network Inference Explainable Artificial Intelligence Temporal Convolutional Networks Lightweight CNNs Emotion Recognition

Jul 22, 2026cs.SD

Scalable Keyword Spotting via Modular Network Expansion

Keyword spotting (KWS) models on embedded devices often need to add new keywords after deployment, but updates are difficult when original training data are unavailable and regressions on existing triggers are unacceptable. At a fixed operating point, our method reduces average new-keyword false reject rate (FRR) from 6.46 to 4.37 versus a parameter-matched separate-model baseline and outperforms parameter-efficient tuning baselines (adapters, LoRA), while using fewer multiply-accumulate operations (MACs) under the same added-parameter budget (

\leq

10k): 16.34M vs 18.45M/20.52M. We achieve this via parameter-capped modular expansion: the base network, including batch-normalization statistics and the core classifier, is frozen, and only a lightweight expansion branch with a separate new-keyword head is trained, preserving core logits, shipped outputs, and thresholds for existing keywords.

Viktor Khaymonenko, Dzmitry Saladukha, Aliaksei Rak +1

Embedded Voice Input Team, Yandex, Russia · Embedded Voice Input Team, Yandex, Belarus

Class-Incremental Learning Efficient Neural Network Inference Keyword Spotting Tiny ML Modular Neural Networks

Jul 22, 2026cs.CV

StrokeSeg2: Stroke Lesion Segmentation in Clinical Research Workflows

Deep learning frameworks like nnU-Net achieve state-of-theart brain lesion segmentation performance but remain difficult to deploy in clinical research environments due to, among other reasons, software dependencies and computational requirements. We introduce StrokeSeg2, a lightweight, modular, cross-platform C++/Qt framework designed to adapt resource-intensive 3D stroke segmentation pipelines into portable and reproducible applications. To improve compatibility with standard clinical workstations, we investigate the combined effect of architectural compression through knowledge distillation and inference optimisation using ONNX Runtime with Float16 quantisation. Across heterogeneous hardware configurations (CPU, integrated GPU, and dedicated GPU) architectural distillation emerged as the primary contributor to efficiency gains, contributing to over 90% reduction in energy consumption and an average 84% reduction in inference time. Specifically, we identify a 0.84M-parameter student model as the most favourable trade-off, reducing the original 102.3M-parameter teacher architecture to a 2.1 MB disk footprint while preserving robust lesion localisation and competitive segmentation performance. This small footprint supports the development of a self-contained installer for clinical workstation targets. Finally, StrokeSeg2 packages these optimisations into standalone installers for Windows, macOS, and Linux. By providing both graphical and commandline interfaces without Docker or external environment dependencies, StrokeSeg2 facilitates deployment of high-performance segmentation workflows for routine clinical research pipelines.

Youwan Mahé, Axel Plessis, Stéphanie Leplaideur +3

EMPENN, MALT · Univ Rennes, Inria, CNRS, Inserm, IRISA UMR 6074, Empenn, Rennes, France · Siemens Healthineers, Courbevoie, France +6

Efficient Neural Network Inference Medical Image Segmentation 3D Medical Image Segmentation

Jul 22, 2026cs.CV

Current Injection Spiking Neural Network for Infrared and Visible Image Fusion

Infrared and visible image fusion (IVIF) integrates the complementary information of two modalities into a single image with richer scene content. While existing methods are largely built on artificial neural networks (ANNs), which densely compute over all activations, spiking neural networks (SNNs) communicate through sparse binary spikes and compute only where and when a spike occurs, offering a route to more energy-efficient fusion. However, directly applying SNNs to IVIF creates a fundamental tension: cross-modal fusion relies on fine-grained responses from both modalities, whereas binary spikes can discard complementary cues that remain below the firing threshold. The membrane potential retains these subthreshold responses before firing, letting both modalities jointly shape the output when integrated at this stage. Building on this, we propose CIS-Fuse, a spiking network that performs cross-modal fusion directly at the membrane-potential level. At its core is the current injection spiking (CIS) operator, which injects one modality as a gated auxiliary current into the driving neuron of the other, so the two integrate before spike firing, with a per-channel learnable injection strength that adaptively regulates the modulation magnitude. Building on CIS, we construct a bidirectional cross-modal fusion (BCMF) module and deploy it on a dual-branch architecture with asymmetric stacking depths, where the two branches develop a clear functional specialization. Extensive experiments on four IVIF benchmarks and on downstream detection and segmentation show that CIS-Fuse achieves fusion quality on par with state-of-the-art ANN-based methods while inheriting the energy efficiency of spike-based computation, with roughly an order of magnitude lower inference energy than the similarly-sized ANN-based DCEvo. Code will be released upon publication.

Rui Zhao, Zhuoyuan Li, Wenrui Li +4

Efficient Neural Network Inference Multi-Source Image Fusion Infrared-Visible Image Fusion Spiking Neural Networks Multimodal Fusion

Jul 22, 2026cs.LG

How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF

In RLHF pipelines, reward scoring blocks policy updates. Slow scoring bottlenecks the entire loop, since no update runs until every rollout gets a score. And yet most setups just default to PyTorch eager mode or torch.compile, no one checks if that's actually fastest. Scoring itself is small. Rollout generation eats far more of a typical RLHF step. But scoring and generation fight over the same CPU and GPU resources, so a faster scoring engine doesn't shrink step time on its own. It mainly frees up capacity generation can use instead. We built a native C++ inference engine on ONNX Runtime. First step: confirm correctness. Output matched the PyTorch reference to 5.7 x 10^-6 on CPU and 4.2 x 10^-3 on GPU, close enough to trust. Then we tested it against PyTorch eager mode, torch.compile, and FastAPI, on both CPU and GPU. CPU was decisive. Our engine beat every baseline, confidence intervals didn't even overlap. GPU gave a different view: we beat PyTorch and FastAPI, but torch.compile came out ahead. Further testing traced the speedup to ONNX Runtime itself, not C++ as a language. And batching strategy mattered more than either the language or the runtime choice, more than we expected. The results are from repeated, independent runs, since single runs just aren't reliable enough to trust.

Venkata Naga Sai Vishnu Rohit Pulipaka, Anish Katta, Deva Rohit Reddy Peddireddy

Efficient Neural Network Inference Reward Modeling Inference Acceleration LLM Inference Acceleration

Jul 21, 2026cs.CV

Fluid-SDF: Ultra-Lightweight and Editable Implicit Shape Representation via Differentiable Primitives

Implicit Neural Representations (INRs) have become the standard for continuous 2D shape modeling, but they suffer from black-box uneditability, vulnerability to noise, and high parameter counts that severely hinder deployment on edge devices. We introduce Fluid-SDF, a highly compressed, differentiable Constructive Solid Geometry (CSG) framework that models shapes using explicit geometric primitives blended via a smooth minimum function. By replacing traditional multi-layer perceptrons (MLPs) with a parameterized primitive engine, Fluid-SDF reconstructs complex, non-convex topologies using strictly under 100 parameters, achieving comparable or superior intersection-over-union (mIoU) to standard neural baselines. Furthermore, we demonstrate that Fluid-SDF acts as a powerful geometric prior, inherently resisting high-frequency dataset noise where capacity-matched neural networks catastrophically overfit. Finally, unlike standard INRs, Fluid-SDF's explicit parameter space allows for direct, zero-shot user editing of local and global shape features without retraining. By bypassing expensive on-device gradient updates entirely, Fluid-SDF is uniquely suited for mobile AI, augmented reality, and resource-constrained embedded environments

Pradyumna Sripada, Chinmay Nadgir, Ksheer Agrawal +1

Efficient Neural Network Inference Implicit Neural Representations Parametric CAD Modeling

Jul 20, 2026cs.AR

BRIM: Workload-Balanced Dual-Sided Bit-Serial Sparse Inference Accelerator

Bit-serial accelerators exploit bit-level sparsity to reduce DNN inference cost, but existing designs exploit sparsity on only one operand, bounding the speedup. Extending sparsity exploitation to both operands simultaneously yields compounding reductions in partial products but introduces a critical new bottleneck: workload imbalance. Because each concurrent weight - activation pair's execution cost depends on the product of two independently varying operand non-zero bit counts, pairs that must complete together finish at vastly different times, leaving faster computations idle. We show this limits PE utilization to 56 - 64% in existing dual-sided designs. We present BRIM, a hardware - software co-designed dual-sided bit-serial sparse accelerator that directly targets this bottleneck. BRIM combines two integrated mechanisms: 1) Cyclic-Balanced Pruning (CBP), a post-training weight optimization that reshapes weight representations based on profiled activation statistics to equalize expected workloads across concurrently processed pairs offline; and 2) Pairwise Slot Donation, a lightweight hardware mechanism that absorbs residual runtime imbalance with negligible area overhead. Evaluated across CNNs, ViTs, and LLMs under iso-area constraints, BRIM achieves over 90% PE utilization, up to 2.37x speedup, and up to 1.63x energy efficiency improvement over prior dual-sided designs.

Varun Manjunath, Ruokai Yin, Donghyun Lee +2

University of Southern California Los Angeles, USA · Yale University New Haven, USA

Efficient Neural Network Inference AI Accelerator Inference Structured Sparsity

Jul 20, 2026cs.CR

PRISM: Sensitivity-Aware PolynoMial PRuning for EffIcient Neural Network Encryption

Structured pruning is essential for making neural network inference feasible under homomorphic encryption (HE), yet its impact on model reliability has remained unexplored. This paper presents a systematic reliability characterization of pruned CKKS-encrypted neural networks and introduces Polynomial-Sensitivity-Aware Pruning (PSAP), a structured pruning method that is inherently reliability-aware. PSAP scores filters jointly by weight magnitude, polynomial activation sensitivity, and rotation cost, which concentrates pruning in fault-tolerant regions. Across two architectures, two datasets, two numerical representations, and five bit-error rates (40 full-model and 108 per-layer experiments), PSAP-pruned models limit catastrophic (>10 pp accuracy drop) layers to at most two versus 5--14 for magnitude-pruned baselines, reducing worst-case vulnerability by up to 29 times under int32 bit-flip injection. Direct CKKS encrypted fault injection indicates a safe operating boundary near BER~ 10^{-5}, supporting int32 injection as a conservative reliability proxy. The fault-critical structural layers account for only 1.1% of parameters, enabling selective hardening at minimal overhead. These reliability gains are obtained alongside competitive efficiency: PSAP reduces Halevi--Shoup rotations by up to 45.2% on ResNet-32, and an adaptive mixed-degree allocation scheme lowers multiplicative depth from 66 to 56 levels, enabling leveled inference without bootstrapping.

Sahaj Majavdia, Mahdi Taheri

Brandenburgische Technische Universität Cottbus-Senftenberg, Germany · Humboldt University of Berlin, Germany · Tallinn University of Technology, Tallinn, Estonia

Efficient Neural Network Inference Homomorphic Encryption Neural Network Pruning

Jul 19, 2026cs.DC

Taurus: Accelerating Out-of-Core Graph Neural Network Inference on Billion-Scale Graphs

Graph Neural Network (GNN) inference on billion-scale graphs is challenging due to the large memory footprint of features and embeddings and high disk I/O costs in out-of-core settings. Existing distributed GNN systems incur high communication times and infrastructure costs, while disk-based GNN systems are primarily tailored to training and experience massive wasted reads during inference on the entire graph. We present Taurus, a single-machine system for GNN inference on graphs that do not fit in RAM, supporting both \textit{exact} full-graph inference and fanout-sampled inference. To avoid random and repeated feature gathers, Taurus reformulates layer-wise inference as source-centric broadcasts over sequential SSD scans, backed by a pipelined GPU-CPU-SSD hierarchy, topology-aware reordering, pending-message eviction, and a GPU-resident store for high-degree vertices. It further uses non-buffered sequential reads and GPU-backed writes to reduce page-cache pollution, host-memory pressure, and write overheads. On out-of-core graphs with up to

269M

vertices,

4B

edges, and

514

GiB of features, Taurus outperforms the strongest layer-wise baseline, DGI, by

7

-

25\times

, and vertex-wise baselines by

40

-

140\times

.

Pranjal Naman, Yogesh Simmhan

Department of Computational and Data Sciences (CDS), Indian Institute of Science (IISc), Bangalore 560012 India

Efficient Neural Network Inference Graph Neural Networks Memory-Efficient Inference

Jul 19, 2026cs.AR

ThRIve: Thermally Robust CNN Inference via Low-Rank Adaptation in Heterogeneous PIM Architectures

Processing-In-Memory (PIM) has emerged as a promising technology for accelerating machine learning (ML) workloads. Specifically, non-volatile memory-based PIM architectures have enabled effective ML acceleration due to their ability to perform energy-efficient matrix-vector multiplication operations. However, these devices suffer from non-idealities such as thermal noise. This noise alters the stored values in the memory cells which correspond to actual model weights, compromising the inference accuracy. In this work, we introduce ThRIve, a noise-aware training methodology that leverages low-rank adaptation to enable thermally robust inference on heterogeneous PIM architectures. ThRIve selectively stores these low-rank noise-aware parameters on a hardware that is less susceptible to thermal noise, enabling robustness against temperature-induced noise variations. ThRIve mitigates the effects of thermal-noise and prevent the drop in inference accuracy across the entire operating temperature range. Experimental results demonstrate that ThRIve-enabled architectures maintain consistent inference accuracy, with the mean accuracy staying within 2% of the ideal (i.e., noise-free) accuracy, and the variation in accuracy across the entire operating temperature range remaining within 2% of the mean. The proposed methodology achieves accuracy and robustness comparable to thermally-resilient Static Random-Access Memory (SRAM)-based PIM systems, while delivering up to 5.4x reduction in energy-delay product (EDP) during CNN model inferencing.

Vibhanshu Sharma, Pratyush Dhingra, Janardhan Rao Doppa +1

Washington State University

Efficient Neural Network Inference Compute-in-Memory Noise Robustness

Jul 17, 2026cs.LG

Compact convolutional neural networks for AI-based drone detection system

The increasing use of first-person-view drones in modern conflicts has created a demand for compact and reliable detection systems capable of operating in complex electromagnetic environments. These drones continuously transmit video signals through onboard video transmitters, generating radio-frequency emissions that can be exploited for early detection. This study investigates the use of lightweight convolutional neural networks for automated detection of drone signals captured by a software-defined radio-based electronic warfare framework. Samples are converted into rasterized time-domain images, providing a computationally efficient input representation suitable for embedded systems. Several custom model architectures were designed and benchmarked in terms of accuracy, model size, and inference performance using a dataset containing approximately 40,000 labeled images. In addition to offline testing, the models were integrated into a GNU Radio signal processing chain for real-time evaluation. The results show that compact models can achieve high detection accuracy while maintaining low computational requirements, making them suitable for embedded radio-frequency monitoring applications. Compared with existing spectrogram-based RF detection methods, the proposed approach eliminates frequency-domain preprocessing and achieves comparable accuracy with significantly reduced computational cost.

Gábor Farkas, Gábor Fazekas, Karakai Patrik +2

Department of Electronic Warfare, Ludovika University of Public Service, Budapest, H-1083, Hungary · Faculty of Informatics, Eötvös Lor´and University, Budapest, H-1117, Hungary

Efficient Neural Network Inference Convolutional Neural Networks Aerial Object Detection

Jul 17, 2026cs.CV

BCG-Former: Toward Pareto-Efficient Hyperspectral Image Classification via Band-Contextual Gating

Hyperspectral image (HSI) classification systems are increasingly deployed on platforms with strict computational budgets, such as UAVs and small spaceborne sensors. In these settings, accuracy alone is not enough; the model must also run within tight latency and memory constraints. Most recent HSI classifiers, however, focus on accuracy and pay relatively little attention to these constraints. We propose BCG-Former, a lightweight CNN-Transformer hybrid that targets this trade-off. The model introduces three innovations: (1) Band-Contextual Gating (BCG) for adaptive spectral recalibration using local inter-band context and learnable temperature sharpening, (2) a spectral summary token that bridges spectral and spatial features, and (3) single-pass Band-RoPE combined with linear attention for efficient joint representation learning. Evaluated on classical airborne (Pavia University, Salinas, Indian Pines, Houston 2013/2018) and UAV-borne benchmark datasets (WHU-Hi-LongKou, HongHu, and HanChuan), BCG-Former achieves over-all accuracy ranging from 91.51% on Houston 2018 to 99.49% on Houston 2013, while maintaining sub-millisecond inference latency (0.91-0.95ms) and using only 0.10-0.23M parameters. Across all eight benchmarks, BCG-Former consistently resides on or near the Pareto frontier of accuracy versus latency, outperforming or matching recent CNN-, Transformer-, and Mamba-based methods at a fraction of their computational cost. Ablation studies confirm that all three components are complementary, with BCG providing the largest individual contribution. These results establish BCG-Former as a strong accuracy-efficiency Pareto candidate for real-time and large-scale remote sensing applications.

Gaurav Sharma, Eungjoo Lee

College of Information Science, University of Arizona, Tucson, AZ, USA · Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, USA

Efficient Neural Network Inference Hybrid CNN-Transformer Architectures Hyperspectral Image Classification