Efficient Neural Network Inference

Ask about this topic

Overview

The field studies how to run trained neural networks with less latency, memory, and energy, from phones and microcontrollers to GPUs, FPGAs, and large serving clusters. Common approaches include quantization, pruning, distillation, early exits and conditional computation, alongside compiler and kernel optimization, memory-aware scheduling, and specialized or neuromorphic hardware. Work spans vision, language, graphs, and streaming tasks, with evaluation increasingly measuring end-to-end latency, throughput, and energy on target devices rather than relying only on parameter counts or FLOPs. A central tension is that theoretical savings such as sparsity or fewer operations do not always yield faster real deployments, and efficiency gains must be balanced against accuracy, reliability, and hardware constraints.

Latest papers

CardsList

471 papers46 papers in the last four weeks, up 7% on the four weeks before. 0.3% of all new papers.

  1. An FPGA-Based SoC Architecture with a RISC-V Controller for Energy-Efficient Temporal-Coding Spiking Neural Networks

    Mar 17, 2026Mohammad Javad Sekonji, Ali Mahani, Maryam Mirsadeghi +1Spiking Neural NetworksFPGA-Based Neural Network Acceleration

    Shahid Bahonar University of Kerman, Kerman, Iran · Brandenburg Technical University, Cottbus, Germany · Tallinn University of Technology, Tallinn, Estonia

  2. vAccSOL: Efficient and Transparent AI Vision Offloading for Mobile Robots

    Mar 17, 2026Adam Zahir, Michele Gucciardom Falk Selker, Anastasios Nanos +4Efficient Neural Network InferenceEdge Inference

    University Carlos III of Madrid (UC3M), Madrid, Spain. · NEC Laboratories Europe, Heidelberg, Germany. · Nubificus Ltd, United Kingdom.

  3. Implementation of tangent linear and adjoint models for neural networks based on a compiler library tool

    Mar 17, 2026Sa Xiao, Hao Jing, Honglu Sun +1Weather ForecastingData Assimilation

    State Key Laboratory of Severe Weather Meteorological Science and Technology (LaSW), Beijing, China · CMA Earth System Modeling and Prediction Centre (CEMC), Beijing, China

  4. Fast-HaMeR: Boosting Hand Mesh Reconstruction using Knowledge Distillation

    Mar 17, 2026Hunain Ahmed Jillani, Ahmed Tawfik Aboukhadra, Ahmed Elhayek +33D Hand Reconstruction3D Reconstruction

    RPTU, Kaiserslautern, Germany · DFKI-AV, Kaiserslautern, Germany · UPM, Saudi Arabia +1

  5. DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

    Mar 17, 2026Yong Il ChoiDynamic Neural NetworksConditional Computation

    Sorynorydotcom Co., Ltd./AI Open Research Lab

  6. S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight

    Mar 17, 2026Haodong Yan, Zhide Zhong, Jiaguan Zhu +10RoboticsRobotic Manipulation

    The Hong Kong University of Science and Technology (Guangzhou) · Huawei Foundation Model Department

  7. Fast SAM 3D Body: Accelerating SAM 3D Body for Real-Time Full-Body Human Mesh Recovery

    Mar 16, 2026Timing Yang, Sicheng He, Hongyi Jing +4Human Mesh Recovery3D Human Motion Reconstruction

    USC Physical Superintelligence (PSI) Lab · University of California, San Diego · NVIDIA +1

  8. Federated Learning of Binary Neural Networks: Enabling Low-Cost Inference

    Mar 16, 2026Nitin Priyadarshini Shankar, Soham Lahiri, Sheetal Kalyani +1Binary Neural NetworksFederated Learning

    Department of Electrical Engineering, Indian Institute of Technology Madras · Department of Computer Science and Engineering, Jadavpur University

  9. Spiking Layer-Adaptive Magnitude-based Pruning

    Mar 16, 2026Junqiao Wang, Zhehang Ye, Yuqi OuyangSpiking Neural NetworksNeural Network Pruning

    Sichuan University-Pittsburgh Institute, Sichuan University, Chengdu, 610207, China · College of Computer Science, Sichuan University, Chengdu, 610065, China

  10. Evaluating Four FPGA-accelerated Space Use Cases based on Neural Network Algorithms for On-board Inference

    Mar 14, 2026Pedro Antunes, Muhammad Ihsan Al Hafiz, Jonah Ekelund +4FPGA-Based Neural Network AccelerationEfficient Neural Network Inference

    Department of Computer Science, Kungliga Tekniska högskolan (KTH), Stockholm, Sweden · Plasma-astrophysics, KU Leuven, Leuven, Belgium

  11. Exploiting temporal parallelism for LSTM Autoencoder acceleration on FPGA

    Mar 14, 2026Aimilios Leftheriotis, Dimosthenis Masouros, Dimitrios Soudris +1FPGA-Based Neural Network AccelerationEfficient Neural Network Inference

    University of Patras, Patras, Greece · National Technical University of Athens, Athens, Greece

  12. MO-SAE:Multi-Objective Stacked Autoencoders Optimization for Edge Anomaly Detection

    Mar 14, 2026Lizhao Zhang, Shengsong Kong, Tao Guo +2AutoencodersEfficient Neural Network Inference

    Faculty of computing, Harbin Institute of Technology, Harbin 150000, China.

  13. Collapse or Preserve: Data-Dependent Temporal Aggregation for Spiking Neural Network Acceleration

    Mar 14, 2026Jiahao QinSpiking Neural NetworksEfficient Neural Network Inference

  14. SRAM-Based Compute-in-Memory Accelerator for Linear-decay Spiking Neural Networks

    Mar 13, 2026Hongyang Shang, Shuai Dong, Yahan Yang +3LIF NeuronsSpiking Neural Networks

    Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China

  15. SNAP-V: A RISC-V SoC with Configurable Neuromorphic Acceleration for Small-Scale Spiking Neural Networks

    Mar 12, 2026Kanishka Gunawardana, Sanka Peeris, Kavishka Rambukwella +4Neuromorphic ComputingEfficient Neural Network Inference

    Department of Computer Engineering, Faculty of Engineering, University of Peradeniya, Peradeniya 20400, Sri Lanka

  16. MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?

    Mar 12, 2026Xingze Zou, Jing Wang, Yuhua Zheng +8LLM Code GenerationEfficient Neural Network Inference

    Zhejiang University · Westlake University · HKUST +2

  17. ECoLAD: Selecting Anomaly Detectors for Automotive Deployment via Compute-Reduction Evaluation

    Mar 11, 2026Kadir-Kaan Özer, René Ebeling, Markus EnzweilerEfficient Neural Network InferenceAutomated Evaluation

    Mercedes-Benz AG, Germany. · Esslingen University of Applied Sciences, Germany.

  18. An Event-Driven E-Skin System with Dynamic Binary Scanning and real time SNN Classification

    Mar 11, 2026Gaishan Li, Zhengnan Fu, Anubhab Tripathi +2Robotic Tactile SensingTactile Sensing

    Department of Electrical Engineering, City University of Hongkong, Hongkong SAR, China

  19. COHORT: Hybrid RL for Collaborative Large DNN Inference on Multi-Robot Systems Under Real-Time Constraints

    Mar 11, 2026Mohammad Saeid Anwar, Anuradha Ravi, Indrajeet Ghosh +3Efficient Neural Network InferenceMulti-Robot Task Allocation

    University of Maryland Baltimore County · DEVCOM Army Research Laboratory

  20. Hardware Efficient Approximate Convolution with Tunable Error Tolerance for CNNs

    Mar 10, 2026Vishal Shashidhar, Anupam Kumari, Roy P PailyEfficient Neural Network InferenceNeural Network Hardware Acceleration

    Department of Electronics and Electrical Engineering, Indian Institute of Technology Guwahati

  21. TinyNav: End-to-End TinyML for Real-Time Autonomous Navigation on Microcontrollers

    Mar 10, 2026Pooria Roy, Nourhan Jadallah. Tomer Lapid, Shahzaib Ahmad +2Robot NavigationOn-Device Inference

    Queen’s University

  22. Multi-DNN Inference of Sparse Models on Edge SoCs

    Mar 10, 2026Jiawei Luo, Di Wu, Simon Dobson +1Efficient Neural Network InferenceEdge Inference

    School of Computer Science, University of St Andrews, Scotland UK

  23. DendroNN: Dendrocentric Neural Networks for Energy-Efficient Classification of Event-Based Data

    Mar 10, 2026Jann Krausse, Zhe Su, Kyrus Mama +4Efficient Neural Network InferenceSpiking Neural Networks

    Infineon Technologies, Dresden, Germany · Karlsruhe Institute of Technology, Karlsruhe, Germany · Institute of Neuroinformatics, University of Zurich and ETH Zurich, Zurich, Switzerland +2

  24. Scalable Neural Vocoder from Range-Null Space Decomposition

    Mar 9, 2026Andong Li, Tong Lei, Zhihang Sun +4Neural VocodersSpeech Processing

    the Key Laboratory of Noise and Vibration Research, Institute of Acoustics, Chinese Academy of Sciences, Beijing, 100190, China, and also with University of Chinese Academy of Sciences, Beijing, 100049, China. · Tencent AI Lab. · School of Communications and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China.

  25. ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

    Mar 8, 2026Yuzhuang Xu, Xu Han, Yuxuan Li +1Efficient Neural Network InferenceLLM Inference Acceleration

    Harbin Institute of Technology, Harbin, China · Tsinghua University, Beijing, China

  26. An LLM-Guided Query-Aware Inference System for GNN Models on Large Knowledge Graphs

    Mar 4, 2026Waleed Afandi, Hussein Abdallah, Ashraf Aboulnaga +1Efficient Neural Network InferenceGraph Neural Networks

    Concordia University · University of Texas at Arlington

  27. InstMeter: An Instruction-Level Method to Predict Energy and Latency of DL Model Inference on MCUs

    Mar 4, 2026Hao Liu, Qing Wang, Marco ZunigaEfficient Neural Network InferenceTiny ML

    Delft University of Technology, Delft, The Netherlands

  28. When Small Variations Become Big Failures: Reliability Challenges in Compute-in-Memory Neural Accelerators

    Mar 3, 2026Yifan Qin, Jiahao Zheng, Zheyu Yan +3Compute-in-MemoryNeural Network Robustness

    University of Notre Dame · NC State University

  29. Toward Generalizable Deep Learning Based Peatland Fire Detection via Walsh Hadamard Transform and Domain Adaptation

    Mar 2, 2026Emadeldeen Hamdan, Ahmad Faiz Tharima, Mohd Zahirasri Mohd Tohir +4Efficient Neural Network InferenceWalsh-Hadamard Transform

    Department of Electrical and Computer Engineering, University of Illinois Chicago, Chicago, IL, USA · Fire and Rescue Department of Malaysia, Malaysia · Dept. of Chemical and Environmental Engineering, Faculty of Engineering, Universiti Putra Malaysia, Malaysia. +2

  30. Search Multilayer Perceptron-Based Fusion for Efficient and Accurate Siamese Tracking

    Mar 2, 2026Tianqi Shen, Huakao Lin, Ning AnVisual Object TrackingMultilayer Perceptrons

    Institute of Mining Artificial Intelligence, Chinese Institute of Coal Science Department of Computer Science, City University of Hong Kong Beijing 100013, China / Hong Kong SAR · Image Processing Center, School of Astronautics, Beihang University Beijing 100191, China · Institute of Mining Artificial Intelligence, Chinese Institute of Coal Science State Key Laboratory of Intelligent Coal Mining and Strata Control Beijing 100013, China