Efficient Inference

Efficient inference for large language models (LLMs) aims to reduce the substantial computational cost and memory demands of LLM deployment, enabling wider accessibility and practical applications. Current research focuses on techniques like model compression (quantization, pruning, knowledge distillation), optimized decoding strategies (speculative decoding, early exiting), and novel architectures (e.g., linear attention mechanisms, recurrent networks) to improve speed and resource efficiency. These advancements are crucial for deploying LLMs on resource-constrained devices and reducing the environmental impact of their operation, impacting both scientific research and various industries.

Papers

November 14, 2022

Pruning Very Deep Neural Network Channels for Efficient Inference
Yihui He
Deep Convolutional Neural Network Efficient Inference Channel Pruning Iterative Algorithm Optimal Reconstruction

September 26, 2022

Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tile
Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, Zhe Wang
Quantization Operator Efficient Inference Convolution Operation Winograd Convolution Square Tile Computer Vision Benchmark

August 21, 2022

Memristive Computing for Efficient Inference on Resource Constrained Devices
Venkatesh Rammamoorthy, Geng Zhao, Bharathi Reddy, Ming-Yang Lin
Deep Learning Efficient Inference Memristive Device Resource Constrained Device Computing in Memory Non Volatile Memory Resistive Memory

August 19, 2022

FP8 Quantization: The Power of the Exponent
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, Tijmen Blankevoort
Real Power Post Training Quantization Efficient Inference Exponential Mechanism P Bit Neural Network Inference Low Bit Floating Point Quantization

August 5, 2022

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
Large Language Model Efficient Inference Parallel Training Transformer Based LLM Expert Language Model

July 12, 2022

Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors
Sairam Sri Vatsavai, Ishan G Thakkar
Convolutional Neural Network Hardware Accelerator Efficient Inference Photonic Accelerator Sparse Tensor

June 30, 2022

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
Transformer Model Transformer Based Model Efficient Inference Transformer Inference Neural Network Inference Global Scale GPU Inference

June 22, 2022

Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices
Kartik Gupta, Marios Fournarakis, Matthias Reisser, Christos Louizos, Markus Nagel
Efficient Inference Federated Averaging Device Learning Federated Optimization Heterogeneous Device

June 20, 2022

LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, Dongsoo Lee
Efficient Inference Matrix Multiplication Low Bit Quantization Weight Only Quantization Efficient Dequantization LUT Based

May 24, 2022

Learning Interacting Dynamical Systems with Latent Gaussian Process ODEs
Çağatay Yıldız, Melih Kandemir, Barbara Rakitsch
Gaussian Process Dynamical System Efficient Inference Sparse Variational Gaussian Process Continuous Time Dynamic Uncertainty Aware Model

May 22, 2022

All Birds with One Stone: Multi-task Text Classification for Efficient Inference with One Forward Pass
Jiaxin Huang, Tianqi Liu, Jialu Liu, Adam D. Lelkes, Cong Yu, Jiawei Han
Multi Task Learning Efficient Inference Bird Specie Forward Pas STONE Deterioration Pattern Multi Task Transformer N Task

May 20, 2022

SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System
Liang Shen, Zhihua Wu, WeiBao Gong, Hongxiang Hao, Yangfan Bai, HuaChao Wu, Xinxuan Wu, Jiang Bian, Haoyi Xiong, Dianhai Yu, Yanjun Ma
Training Data Mixture of Expert Efficient Inference Inference System Scalable Inference DeepSpeed Data Efficiency

April 29, 2022

PIE: a Parameter and Inference Efficient Solution for Large Scale Knowledge Graph Embedding Reasoning
Linlin Chao, Xiexiong Lin, Taifeng Wang, Wei Chu
Knowledge Graph Complex Reasoning Many Parameter Efficient Inference Large Scale Knowledge Graph Fine Grained Entity Typing Information Dense Sparse Symbolic Concept

April 7, 2022

Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch
Lukas Hedegaard, Alexandros Iosifidis
Deep Neural Network Continual Learning Pytorch Model Efficient Inference Easy to Use Library Online Inference

March 3, 2022

Multi-Tailed Vision Transformer for Efficient Inference
Yunke Wang, Bo Du, Wenyuan Wang, Chang Xu
Vision Transformer Efficient Inference Self Attention Layer Transformer Encoder Transformer Encoders

February 15, 2022

Post-Training Quantization for Cross-Platform Learned Image Compression
Dailan He, Ziming Yang, Yuan Chen, Qi Zhang, Hongwei Qin, Yan Wang
Image Compression Post Training Quantization Efficient Inference Multiplier Free Quantization Learned Image Compression Rate Distortion Performance

January 25, 2022

Explanatory Learning: Beyond Empiricism in Neural Networks
Antonio Norelli, Giorgio Mariani, Luca Moschella, Andrea Santilli, Giambattista Parascandolo, Simone Melzi, Emanuele Rodolà
Neural Network Machine Learning Line by Line Explanation Efficient Inference Deliberation Model Exploratory Learning

December 8, 2021

Neural Network Quantization for Efficient Inference: A Survey
Olivia Weng
Neural Network Timely Survey Efficient Inference Neural Network Quantization

December 6, 2021

Traversing Time with Multi-Resolution Gaussian Process State-Space Models
Krista Longi, Jakob Lindinger, Olaf Duennbier, Melih Kandemir, Arto Klami, Barbara Rakitsch
Gaussian Process Efficient Inference Gaussian Process State Space Model

November 19, 2021

A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits
Doyeon Kim, Jeonghwan Lee, Hye Won Chung
Efficient Inference Crowdsourcing Context Data Labeling Fundamental Limit Task Specialization Crowdsourcing System Noisy Explanation

Efficient Inference

Papers

Pruning Very Deep Neural Network Channels for Efficient Inference

Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tile

Memristive Computing for Efficient Inference on Resource Constrained Devices

FP8 Quantization: The Power of the Exponent

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models

Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices

LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models

Learning Interacting Dynamical Systems with Latent Gaussian Process ODEs

All Birds with One Stone: Multi-task Text Classification for Efficient Inference with One Forward Pass

SE-MoE: A Scalable and Efficient Mixture-of-Experts Distributed Training and Inference System

PIE: a Parameter and Inference Efficient Solution for Large Scale Knowledge Graph Embedding Reasoning

Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch

Multi-Tailed Vision Transformer for Efficient Inference

Post-Training Quantization for Cross-Platform Learned Image Compression

Explanatory Learning: Beyond Empiricism in Neural Networks

Neural Network Quantization for Efficient Inference: A Survey

Traversing Time with Multi-Resolution Gaussian Process State-Space Models

A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits