Efficient Inference

Efficient inference for large language models (LLMs) aims to reduce the substantial computational cost and memory demands of LLM deployment, enabling wider accessibility and practical applications. Current research focuses on techniques like model compression (quantization, pruning, knowledge distillation), optimized decoding strategies (speculative decoding, early exiting), and novel architectures (e.g., linear attention mechanisms, recurrent networks) to improve speed and resource efficiency. These advancements are crucial for deploying LLMs on resource-constrained devices and reducing the environmental impact of their operation, impacting both scientific research and various industries.

Papers

February 13, 2024

Gaussian Ensemble Belief Propagation for Efficient Inference in High-Dimensional Systems
Dan MacKinlay, Russell Tsuchida, Dan Pagendam, Petra Kuhnert
High Dimensional Data Assimilation Efficient Inference Belief Propagation Ensemble Kalman Filter

February 7, 2024

Online Cascade Learning for Efficient Inference over Streams
Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, Swarat Chaudhuri
LLM Inference Efficient Inference Inference Cost Link Stream Cascade Learning

February 6, 2024

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, Xiaojuan Qi
Large Language Model Continuum Limit Post Training Quantization Efficient Inference Language Processing

January 17, 2024

Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices
Lei Xun, Jonathon Hare, Geoff V. Merrett
Neural Network Deep Neural Network Efficient Inference Embedded System Mobile User Runtime Monitoring Dynamic Neural Network Deep Neural Network Performance

January 12, 2024

Transitional Grid Maps: Efficient Analytical Inference of Dynamic Environments under Limited Sensing
José Manuel Gaspar Sánchez, Leonard Bruns, Jana Tumova, Patric Jensfelt, Martin Törngren
Autonomous Agent Dynamic Environment Efficient Inference Particle Filter Limited Sensing Grid Map

December 19, 2023

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference
Ziqian Zeng, Yihuai Hong, Hongliang Dai, Huiping Zhuang, Cen Chen
Scientific Inference Strong Consistency Efficient Inference Early Exit Reinforcement Learning Problem Language Model Inference

December 15, 2023

Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference
Bartosz Wójcik, Alessio Devoto, Karol Pustelnik, Pasquale Minervini, Simone Scardapane
Speech Recognition Pre Trained Model Transformer Model Efficient Inference Linear Attention Adaptive Computation

December 5, 2023

Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit
Fanfei Meng, Lele Zhang, Yu Chen, Yuxin Wang
Transformer Based Multi Layer Thompson Sampling Efficient Inference Hierarchical Transformer Head Motion Dynamic Transformer

December 4, 2023

Signed Binarization: Unlocking Efficiency Through Repetition-Sparsity Trade-Off
Sachit Kuhar, Yash Jain, Alexey Tumanov
Deep Neural Network DNN Framework Efficient Inference Binarization Method Enhancing Efficiency Partial Binarization

November 30, 2023

Semiparametric Efficient Inference in Adaptive Experiments
Thomas Cook, Alan Mishler, Aaditya Ramdas
Efficient Inference Semi Parametric Adaptive Experiment Policy Estimation Sequential Experiment Sequential Inference

October 29, 2023

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models
Zhixu Du, Shiyu Li, Yuhao Wu, Xiangyu Jiang, Jingwei Sun, Qilin Zheng, Yongkai Wu, Ang Li, Hai "Helen" Li, Yiran Chen
High Efficiency Mixture of Expert Potential Scalability Efficient Inference Efficient Model Model Capacity Sparsity Aware

October 18, 2023

Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models
Pascal Kündig, Fabio Sigrist
Gaussian Process Efficient Inference Iterative Method

October 14, 2023

Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices
Zhepeng Wang, Isaacshubhanand Putla, Weiwen Jiang, Youzuo Lin
Edge Device Efficient Inference Full Waveform Inversion

October 4, 2023

xVal: A Continuous Number Encoding for Large Language Models
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho
Efficient Inference Data Encoding Adaptive Token Various Number Specific Tokenization Scheme

October 2, 2023

Do Compressed LLMs Forget Knowledge? An Experimental Study with Practical Implications
Duc N. M Hoang, Minsik Cho, Thomas Merth, Mohammad Rastegari, Zhangyang Wang
Large Language Model Knowledge Based Efficient Inference Future Implication Experimental Study Inference Latency LLM Compression Dynamic Prompting

September 23, 2023

Early-Exit with Class Exclusion for Efficient Inference of Neural Networks
Jingcun Wang, Bing Li, Grace Li Zhang
Neural Network Deep Neural Network Efficient Inference Early eXit Early Exit Dynamic Inference Class Interference

September 22, 2023

Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences
Hugo Malard, Salah Zaiem, Robin Algayres
Automatic Speech Recognition Large Model Efficient Inference Automatic Speech Recognition Model Inference Cost Whisper Encoder

July 27, 2023

Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs
Or Sharir, Anima Anandkumar
Deep Learning Transformer Architecture Incremental Learning Efficient Inference Iterative Inference Incremental Algorithm Computable Learning Partial Input

July 26, 2023

DPBERT: Efficient Inference for BERT based on Dynamic Planning
Weixin Wu, Hankz Hankui Zhuo
Ticket BERT Efficient Inference Scale Pre Trained Language Model Adaptive Inference Original BERT Dynamic Planning

July 1, 2023

Q-YOLO: Efficient Inference for Real-time Object Detection
Mingze Wang, Huixin Sun, Jun Shi, Xuhui Liu, Baochang Zhang, Xianbin Cao
Object Detection Model Efficient Inference Real Time Object Activation Quantization

Efficient Inference

Papers

Gaussian Ensemble Belief Propagation for Efficient Inference in High-Dimensional Systems

Online Cascade Learning for Efficient Inference over Streams

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

Dynamic DNNs and Runtime Management for Efficient Inference on Mobile/Embedded Devices

Transitional Grid Maps: Efficient Analytical Inference of Dynamic Environments under Limited Sensing

ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference

Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference

Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit

Signed Binarization: Unlocking Efficiency Through Repetition-Sparsity Trade-Off

Semiparametric Efficient Inference in Adaptive Experiments

SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models

Iterative Methods for Vecchia-Laplace Approximations for Latent Gaussian Process Models

Edge-InversionNet: Enabling Efficient Inference of InversionNet on Edge Devices

xVal: A Continuous Number Encoding for Large Language Models

Do Compressed LLMs Forget Knowledge? An Experimental Study with Practical Implications

Early-Exit with Class Exclusion for Efficient Inference of Neural Networks

Big model only for hard audios: Sample dependent Whisper model selection for efficient inferences

Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs

DPBERT: Efficient Inference for BERT based on Dynamic Planning

Q-YOLO: Efficient Inference for Real-time Object Detection