Efficient Inference

Efficient inference for large language models (LLMs) aims to reduce the substantial computational cost and memory demands of LLM deployment, enabling wider accessibility and practical applications. Current research focuses on techniques like model compression (quantization, pruning, knowledge distillation), optimized decoding strategies (speculative decoding, early exiting), and novel architectures (e.g., linear attention mechanisms, recurrent networks) to improve speed and resource efficiency. These advancements are crucial for deploying LLMs on resource-constrained devices and reducing the environmental impact of their operation, impacting both scientific research and various industries.

Papers

March 7, 2023

Gradient-Free Structured Pruning with Unlabeled Data
Azade Nova, Hanjun Dai, Dale Schuurmans
Model Compression Unlabeled Data Efficient Inference Structured Pruning Inference Cost Gradient Pruning

February 20, 2023

Progressive Ensemble Distillation: Building Ensembles for Efficient Inference
Don Kurian Dennis, Abhishek Shetty, Anish Sevekari, Kazuhito Koishida, Virginia Smith
Strong Generalization Diverse Ensemble Efficient Inference Inference Cost Student Model Device Inference

February 3, 2023

An Optical XNOR-Bitcount Based Accelerator for Efficient Inference of Binary Neural Networks
Sairam Sri Vatsavai, Venkata Sai Praneeth Karempudi, Ishan Thakkar
Efficient Inference Binary Neural Network Phantom 2D Accelerator Optical System BNN Layer XOR Gate

January 30, 2023

SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms
Yaodan Xu, Jingzhou Sun, Sheng Zhou, Zhisheng Niu
Efficient Inference Large Batch Multi GPU Batch Size Automatic Batching Public Benchmark

January 19, 2023

Batch Prompting: Efficient Inference with Large Language Model APIs
Zhoujun Cheng, Jungo Kasai, Tao Yu
Large Language Model Efficient Inference Inference Cost Shot in Context Learning

January 11, 2023

Pruning Compact ConvNets for Efficient Inference
Sayan Ghosh, Karthik Prasad, Xiaoliang Dai, Peizhao Zhang, Bichen Wu, Graham Cormode, Peter Vajda
Generalization Performance Efficient Inference Neural Network Pruning ImageNet Benchmark

December 22, 2022

EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Models
Xinlin Li, Mariana Parazeres, Adam Oberman, Alireza Ghaffari, Masoud Asgharian, Vahid Partovi Nia
Deep Learning Model Convolutional Layer Efficient Inference Deep Learning Application Adaptive Operation

December 16, 2022

SplitGP: Achieving Both Generalization and Personalization in Federated Learning
Dong-Jun Han, Do-Yeon Kim, Minseok Choi, Christopher G. Brinton, Jaekyun Moon
Strong Generalization Split Learning Efficient Inference Generalization Capability

November 29, 2022

Differentiable User Models
Alex Hämäläinen, Mustafa Mert Çelikok, Samuel Kaski
Efficient Inference Human Behavior Cognitive Model User Modeling

November 28, 2022

November 25, 2022

Signed Binary Weight Networks
Sachit Kuhar, Alexey Tumanov, Judy Hoffman
Deep Neural Network Supervised ImageNet Efficient Inference Unstructured Sparsity Binary Weight Binary Network

November 18, 2022

Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
Young Jin Kim, Rawn Henry, Raffy Fahim, Hany Hassan Awadalla
Machine Translation Efficient Inference Training Model Sparse Model Pink Elephant Large Scale Cloud

November 14, 2022

Pruning Very Deep Neural Network Channels for Efficient Inference
Yihui He
Deep Convolutional Neural Network Efficient Inference Channel Pruning Iterative Algorithm Optimal Reconstruction

September 26, 2022

Going Further With Winograd Convolutions: Tap-Wise Quantization for Efficient Inference on 4x4 Tile
Renzo Andri, Beatrice Bussolino, Antonio Cipolletta, Lukas Cavigelli, Zhe Wang
Quantization Operator Efficient Inference Convolution Operation Winograd Convolution Square Tile Computer Vision Benchmark

August 21, 2022

Memristive Computing for Efficient Inference on Resource Constrained Devices
Venkatesh Rammamoorthy, Geng Zhao, Bharathi Reddy, Ming-Yang Lin
Deep Learning Efficient Inference Memristive Device Resource Constrained Device Computing in Memory Non Volatile Memory Resistive Memory

August 19, 2022

FP8 Quantization: The Power of the Exponent
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, Tijmen Blankevoort
Real Power Post Training Quantization Efficient Inference Exponential Mechanism P Bit Neural Network Inference Low Bit Floating Point Quantization

August 5, 2022

Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, Luke Zettlemoyer
Large Language Model Efficient Inference Parallel Training Transformer Based LLM Expert Language Model

July 12, 2022

Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors
Sairam Sri Vatsavai, Ishan G Thakkar
Convolutional Neural Network Hardware Accelerator Efficient Inference Photonic Accelerator Sparse Tensor

June 30, 2022

DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, Yuxiong He
Transformer Model Transformer Based Model Efficient Inference Transformer Inference Neural Network Inference Global Scale GPU Inference