Neural Network Inference

Neural network inference focuses on efficiently executing trained neural networks to make predictions, a crucial step in deploying machine learning models for various applications. Current research emphasizes optimizing inference speed and resource usage through techniques like network compression (pruning, quantization, binarization), algorithmic improvements (speculative decoding, efficient dataflow management), and hardware acceleration (FPGAs, specialized processors, Processing-in-DRAM). These advancements are vital for enabling the deployment of increasingly complex models, such as large language models and deep convolutional networks, in resource-constrained environments and high-throughput applications across diverse scientific domains and industries.

Papers

November 6, 2024

Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision
Dinithi Jayasuriya, Nastaran Darabi, Maeesha Binte Hashem, Amit Ranjan Trivedi
Low Precision Neural Network Inference DNN Inference High Precision Accuracy Loss Universal Polarization Neural Polarization

October 28, 2024

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference
Changwoo Lee, Soo Min Kwon, Qing Qu, Hun-Seok Kim
Foundation Model Efficient Deep Neural Network Inference Structured Matrix Multiple Sequence Alignment Dense Matrix Multiplication

September 2, 2024

Erasure Coded Neural Network Inference via Fisher Averaging
Divyansh Jhunjhunwala, Neharika Jali, Gauri Joshi, Shiqiang Wang
Practical Algorithm Neural Network Model Accurate Decoding Neural Network Inference Linear Time Algorithm Empirical Fisher

May 13, 2024

Neural Network Compression for Reinforcement Learning Tasks
Dmitry A. Ivanov, Denis A. Larionov, Oleg V. Maslennikov, Vladimir V. Voevodin
Reinforcement Learning Reinforcement Learning Algorithm Sparsity Increase Efficient Inference Neural Network Compression Neural Network Inference Reinforcement Learning Task

April 29, 2024

Scalable Bayesian Inference in the Era of Deep Learning: From Gaussian Processes to Deep Neural Networks
Javier Antoran
Neural Network Deep Learning Deep Neural Network Deep Learning Model Gaussian Process New Era Model Uncertainty Posterior Sampling Neural Network Inference Scalable Bayesian

April 18, 2024

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, Beidi Chen
Long Sequence Speculative Decoding Key Value Cache Neural Network Inference Lossless Acceleration

March 13, 2024

Architectural Implications of Neural Network Inference for High Data-Rate, Low-Latency Scientific Applications
Olivia Weng, Alexander Redding, Nhan Tran, Javier Mauricio Duarte, Ryan Kastner
Low Latency Deep Learning Application Neural Network Inference Reconfigurable Intelligent Chip Memory Neural Network Parameter Better on Chip Data Data Rate

February 19, 2024

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, Beidi Chen
Large Language Model Potential Scalability Efficient Inference Speculative Decoding Neural Network Inference

February 6, 2024

LightHGNN: Distilling Hypergraph Neural Networks into MLPs for $100\times$ Faster Inference
Yifan Feng, Yihe Luo, Shihui Ying, Yue Gao
Fast Inference Hypergraph Neural Network Synthetic Graph Neural Network Inference Hypergraph Datasets

January 29, 2024

Emergent Explainability: Adding a causal chain to neural network inference
Adam Perrett
Explainable AI Emergent Communication Neural Network Inference Causal Interpretation Transparent AI

January 9, 2024

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, Yuxiong He
Large Language Model Text Generation Neural Network Inference DeepSpeed Data Efficiency

October 1, 2023

YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs
Cyrus Zhou, Zack Hassman, Ruize Xu, Dhirpal Shah, Vaugnn Richard, Yanjing Li
Code Generation Binary Neural Network Neural Network Inference Inference Workload Modern Neural Network Dataflow Analysis

September 29, 2023

ONNXExplainer: an ONNX Based Generic Framework to Explain Neural Networks Using Shapley Values
Yong Zhao, Runxin He, Nicholas Kersting, Can Liu, Shubham Agrawal, Chiranjeet Chetia, Yu Gu
Neural Network Neural Network Model Shapley Value Pytorch Model Neural Network Inference Generic Model Open Neural Network Exchange

September 6, 2023

Bandwidth-efficient Inference for Neural Image Compression
Shanzhi Yin, Tongda Xu, Yongsheng Liang, Yuanyuan Wang, Yanghao Li, Yan Wang, Jingjing Liu
Image Compression Neural Compression Neural Image Compression Model Quantization Neural Network Inference Activation Compression

August 31, 2023

Fault Injection on Embedded Neural Networks: Impact of a Single Instruction Skip
Clement Gaine, Pierre-Alain Moellic, Olivier Potin, Jean-Max Dutertre
Global Impact 32 Bit Microcontrollers Neural Network Inference Fault Injection Adversarial Objective Embedded Neural Network

August 9, 2023

SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inference
Edouard Yvinec, Arnaud Dapogny, Kevin Bailly, Xavier Fischer
Deep Neural Network High Efficiency Neural Network Inference DNN Pruning SAfER Instruct

July 30, 2023

RoseNNa: A performant, portable library for neural network inference with application to computational fluid dynamics
Ajay Bati, Spencer H. Bryngelson
Neural Network Application Proficiency Computational Fluid Dynamic Neural Network Inference Modular Library

July 11, 2023

Fast Neural Network Inference on FPGAs for Triggering on Long-Lived Particles at Colliders
Andrea Coccaro, Francesco Armando Di Bello, Stefano Giagu, Lucrezia Rambelli, Nicola Stocchetti
Field Programmable Gate Array Neural Network Inference Collider Physic Particle Collision Particle Physic

July 4, 2023

Learning Discrete Weights and Activations Using the Local Reparameterization Trick
Guy Berger, Aviv Navon, Ethan Fetaya
Targeted Activation Penalty Weight Freezing Reparameterization Method Neural Network Inference Discrete Activation Binary Activation

June 29, 2023

NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes
Hao-Lun Sun, Lei Hsiung, Nandhini Chandramoorthy, Pin-Yu Chen, Tsung-Yi Ho
Prompt Recovery Neural Network Inference DNN Accuracy DRAM Access Energy Low Voltage

Neural Network Inference

Papers

Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

Erasure Coded Neural Network Inference via Fisher Averaging

Neural Network Compression for Reinforcement Learning Tasks

Scalable Bayesian Inference in the Era of Deep Learning: From Gaussian Processes to Deep Neural Networks

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

Architectural Implications of Neural Network Inference for High Data-Rate, Low-Latency Scientific Applications

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

LightHGNN: Distilling Hypergraph Neural Networks into MLPs for $100\times$ Faster Inference

Emergent Explainability: Adding a causal chain to neural network inference

DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs

ONNXExplainer: an ONNX Based Generic Framework to Explain Neural Networks Using Shapley Values

Bandwidth-efficient Inference for Neural Image Compression

Fault Injection on Embedded Neural Networks: Impact of a Single Instruction Skip

SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inference

RoseNNa: A performant, portable library for neural network inference with application to computational fluid dynamics

Fast Neural Network Inference on FPGAs for Triggering on Long-Lived Particles at Colliders

Learning Discrete Weights and Activations Using the Local Reparameterization Trick

NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference in Low-Voltage Regimes