Efficient Neural Network Inference

Ask about this topic

Overview

The field studies how to run trained neural networks with less latency, memory, and energy, from phones and microcontrollers to GPUs, FPGAs, and large serving clusters. Common approaches include quantization, pruning, distillation, early exits and conditional computation, alongside compiler and kernel optimization, memory-aware scheduling, and specialized or neuromorphic hardware. Work spans vision, language, graphs, and streaming tasks, with evaluation increasingly measuring end-to-end latency, throughput, and energy on target devices rather than relying only on parameter counts or FLOPs. A central tension is that theoretical savings such as sparsity or fewer operations do not always yield faster real deployments, and efficiency gains must be balanced against accuracy, reliability, and hardware constraints.

Latest papers

CardsList

471 papers46 papers in the last four weeks, up 7% on the four weeks before. 0.3% of all new papers.

  1. REVBRAIN: Real-time Probabilistic EEG Forecasting Using Residual Vector Quantization

    Oct 28, 2026Hanna Pankka, Jaakko Lehtinen, Timo RoineTime Series ForecastingProbabilistic Time Series Forecasting

    Department of Neuroscience and Biomedical Engineering, Aalto University School of Science, Finland · Department of Computer Science, Aalto University School of Science, Finland · NVIDIA, Finland +1

  2. SLICE-CiM: Selective Layer-wise Inference Control Engine with Confidence-Based Early Exit for SRAM based Digital Compute-in-Memory

    Oct 27, 2026Pratham Sharma, Mohd Faisal Khan, Santosh Kumar VishvakarmaEfficient Neural Network InferenceEnergy-Efficient ML

    Department of Electrical Engineering, Indian Institute of Technology Indore, Indore, India

  3. Hardware-Matched Spiking Neural Networks for Real-Time Purkinje Cell Spike Classification

    Oct 27, 2026Arooba Farhan, Afsah Syed, Muhammad Fahd Ibrahim Bhatti +4Spiking Neural NetworksEfficient Neural Network Inference

    Electrical Engineering Department, Lahore University of Management Sciences, Pakistan · Department of Neuroscience, Erasmus University Medical Center, Rotterdam, The Netherlands

  4. One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

    Oct 26, 2026Aditya Chatterjee, Anand Joseph X M, Sanjay Singh ChauhanEfficient Neural Network InferenceInference-Time Optimization

    NVIDIA, Bangalore, India · NVIDIA, Hyderabad, India · NVIDIA, Pune, India

  5. Pacifo: An Efficient Fully Homomorphic Encryption-based Lightweight ResNet for Trustworthy Machine Learning as a Service

    Oct 24, 2026Yang Li, Gang He, Ruichen Zhang +2Privacy-Preserving MLEfficient Neural Network Inference

    School of Communication and Information Engineering, Shanghai University, Shanghai, China · College of Computing and Data Science, Nanyang Technological University, Singapore

  6. Optimizing Physical Neural Networks through Device Variable Selection

    Oct 24, 2026Sjoerd van den Belt, Manuel Escudero, Mohamadreza Zolfagharinejad +2Physical Neural NetworksNeural Network Optimization

    BRAINS Center for Brain-Inspired Computing, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands · MESA+ Institute, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands · Institute of Physics, University of Münster, 48149 Münster, Germany

  7. FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference

    Oct 17, 2026Liang Su, David Chen, Jason Li +2Efficient Neural Network InferenceEfficient VLA Model Inference

    AnyDevice AI & Alps Lab

  8. TorchTPU - A Hybrid Eager/Compiled Architecture for Native PyTorch Execution on TPU

    Oct 16, 2026Claudio Basile, Michael Voznesensky, Gunhyun Park +50Neural Network Hardware AccelerationEfficient Neural Network Inference

  9. Physics-Aware Distillation of a Grid Foundation Model for Real-Time Edge Inference

    Oct 15, 2026Vlad Alexandru Sănduleac, Shahariya Rasheed Rayamarakkarveettil Puthanpurayil, Noman Khan +1Knowledge DistillationPower Systems

    Faculty of Power Engineering, National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania · AAU Energy, Aalborg University, Aalborg, Denmark

  10. CoViG: Coarsened Vision GNN for Image Classification

    Oct 15, 2026Martial Guidez, Stefan Duffner, Christophe GarciaGraph Neural NetworksImage Classification

    INSA Lyon, CNRS, Universit´e Claude Bernard Lyon 1, LIRIS, UMR5205, 69621 Villeurbanne, France

  11. The Physics Is the Network: Fast, Training-Free Neural Emulation

    Oct 14, 2026Luka Todorovic, Kaushal Gianchandani, Sai RavelaNeural PDE SolversNeural Surrogate Modeling

    Department of Mathematics and Department of Statistics, University of Michigan, Ann Arbor, MI · NASA Postdoctoral Program Fellow, NASA Goddard Institute for Space Studies, New York, NY · Department of Earth, Atmospheric, and Planetary Sciences, Massachusetts Institute of Technology, Cambridge, MA

  12. Towards Training and Inference of Graph Neural Networks in Graph Databases

    Oct 13, 2026Victor Pekkari, Harry Wik, Alfred Clemedtson +2Graph Neural NetworksEfficient Neural Network Inference

    Lund University, Faculty of Engineering, Lund, Sweden · Neo4j, London, England

  13. Sea-Net: A Small and Inherently Interpretable Neural Network for Time Series Classification

    Oct 13, 2026Ubaid Ur Rehman, Tanmoy MondalInterpretable Time-Series ClassificationTime Series Classification

    FOX Team, Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) University of Lille, CNRS, Centrale Lille, UMR 9189 F-59000 Lille, France

  14. Neural Bayesian Networks for Modular and Scalable Decision-Making Under Uncertainty

    Oct 12, 2026Giovanni Briglia, Marco Lippi, Stefano Mariani +1Bayesian NetworksEfficient Neural Network Inference

    University of Modena and Reggio Emilia, Reggio Emilia, Italy · University of Florence, Florence, Italy

  15. Removing the Compute Bottleneck for Event-based Visual Place Recognition with Spiking Neural Networks

    Oct 12, 2026Somayeh Hussaini, Tobias Fischer, Michael MilfordVisual Place RecognitionEfficient Neural Network Inference

  16. BrainAid: Efficient Brain Tumor Detection Using Lightweight CNNs and Explainable AI for Resource-Constrained Environments

    Oct 10, 2026Sai Sriram Gonthina, Ross Gore, Bikash Chandra Singh +4Lightweight Neural NetworksBrain Tumor Classification

    Department of Computer Science and Engineering, IIIT - Naya Raipur, Raipur, Chhattisgarh, India · Center for Secure & Intelligent Critical Systems, Old Dominion University, Virginia, USA · Department of Computer Science, California State University, Fresno, USA +2

  17. Linear-Time Non-Local Codec Placement for Machine Learning Model Compression

    Oct 10, 2026Matteo SpallanzaniEfficient Neural Network InferenceNeural Network Compression

    Axelera AI Zürich, Switzerland

  18. SpikCTU: A Compact Spiking Network with Adaptive Recurrent Computation for Speech Command Recognition

    Oct 10, 2026Li Jin, Jiang Wenbin, Chen Tao +1Spiking Neural NetworksEfficient Neural Network Inference

    Hangzhou Dianzi University · Olympus Hedge Fund Investment Co., Ltd. · Shanghai Jiao Tong University

  19. Rationally Linearized MLPs

    Oct 9, 2026Arijit Sehanobish, Byeongchan Kim, Avinava Dubey +2Multilayer PerceptronsEfficient Neural Network Inference

    Kensho Technologies · Seoul National University · Google Research +2

  20. Lightweight Time-Frequency Vibration Sensing for Consumer Device Condition Monitoring

    Oct 9, 2026Po-Heng Chou, Jen-Yu Chiu, Chun-Yu Yeh +2Time-Frequency AnalysisFault Diagnosis

    Research Center for Information Technology Innovation (CITI), Academia Sinica (AS), Taipei 11529, Taiwan · Bradley Department of Electrical and Computer Engineering (ECE), Virginia Tech (VT), Alexandria, VA 22305, USA · Department of Electrical Engineering, National Yunlin University of Science and Technology (NYUST), Yunlin 64002, Taiwan

  21. When Has a Bayesian Neural Network Sampled Enough? Adaptive Inference Time with Statistical Guarantees

    Oct 8, 2026Fabian Denoodt, Sibylle HessAdaptive SamplingBayesian Neural Networks

    Eindhoven University of Technology

  22. DEX: Digit-Level Early Exit for Energy-Efficient MSDF Neural Network Inference

    Oct 8, 2026Yousef Sadegheih, Dorit Merhof, Muhammad UsmanEfficient Neural Network InferenceEarly-Exit Neural Networks

    Faculty of Informatics and Data Science, University of Regensburg, 93053 Regensburg, Germany

  23. SDPAD: A Fully Spike-Driven Pipeline for End-to-End Autonomous Driving

    Oct 8, 2026Chengjun Zhang, Yuhao Zhang, Jie Yang +1Autonomous Driving PlanningEnd-to-End Autonomous Driving

    Zhejiang Key Laboratory of 3D Micro/Nano Fabrication and Characterization, Westlake Institute for Optoelectronics, Fuyang, Hangzhou, China · Integrated-On-Chips Brain-Computer Interfaces Zhejiang Engineering Research Center, Hangzhou, Zhejiang, China · CenBRAIN Neurotech, School of Engineering, Westlake University, Hangzhou, Zhejiang, China

  24. SpikingVLA: Asynchronous Spiking Vision-Language-Action Models

    Oct 7, 2026Jingya Wang, Dehao Zhang, Shuai Wang +3Efficient VLA Model InferenceSpiking Neural Networks

    University of Electronic Science and Technology of China · Shenzhen Loop Area Institute · The Chinese University of Hong Kong (Shenzhen)

  25. EM-SNN: Efficiently Modulated Spiking Neural Network for Remote Sensing Image Dehazing

    Oct 7, 2026Jie Shao, Jiaqi Ma, Wenwen Min +4Image DehazingSpiking Neural Networks

    Zhongnan University of Economics and Law, Wuhan, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Yunnan University, Kunming, China +2

  26. Hardware-aware Calibrated Clustered Attention for Efficient Visual Geometric Transformers

    Oct 7, 2026Weitian Wang, Shubham Rai, Cecilia De La Parra +1Neural Network Hardware AccelerationEfficient ViTs

    Robert Bosch GmbH, Germany · Ruhr University Bochum, Germany

  27. AnyBottle: A Recipe to Only Keep the Concepts You Really Need

    Oct 6, 2026Wolfgang Stammer, Sukrut Rao, Hevra Petekkaya +2Efficient Neural Network InferenceConcept Bottleneck Models

    Max Planck Institute for Informatics, Saarland Informatics Campus · Technical University of Darmstadt

  28. Energy-Efficient Spiking Neural Networks for Sensing-Aided Beam Prediction

    Oct 6, 2026Nhan Thanh Nguyen, Phuong Nam Tran, Mengyuan Ma +3Spiking Neural NetworksEfficient Neural Network Inference

    Centre for Wireless Communications, University of Oulu, P.O.Box 4500, FI-90014, Finland · Faculty of Information Technology, Van Lang School of Technology, Van Lang University, Ho Chi Minh City, Vietnam · School of Electrical, Computer, and Energy Engineering, Arizona State University, AZ, USA

  29. Measuring and Reducing Cross-Vendor Mismatch in Language Models

    Oct 4, 2026Erland Hilman Fuadi, Chong Tian, Xiaosong Ma +1Efficient Neural Network InferenceML Reproducibility

    Mohamed bin Zayed University of Artificial Intelligence

  30. TIRMamba: A Thermal-Prior-Modulated State-Space Network for Sub-Million-Parameter Infrared Image Super-Resolution

    Oct 4, 2026Chun-An Lin, Tsung-Jung Liu, Yen-Chieh OuyangThermal ImagingEfficient Neural Network Inference

    Department of Electrical Engineering, National Chung Hsing University, Taichung 402202, Taiwan · College of Computing and Informatics, Providence University, Taichung 433303, Taiwan