LLM Inference

February 15, 2024

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
Taesu Kim, Jongho Lee, Daehyun Ahn, Sarang Kim, Jiwoong Choi, Minkyu Kim, Hyungjun Kim
LLM Inference Efficient Inference Small Kernel Interleaving Method CUDA Kernel Matrix Multiplication Kernel

February 14, 2024

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, Beidi Chen
Large Language Model LLM Inference Key Value Cache Cache Context Temporal Difference Type Recurrence Key Value Cache Compression Memory Bottleneck
HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference
Yashas Samaga B L, Varun Yerram, Chong You, Srinadh Bhojanapalli, Sanjiv Kumar, Prateek Jain, Praneeth Netrapalli
LLM Inference Generative Large Language Model Inference Latency High Bandwidth Memory
SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, Jae-Joon Kim
Large Language Model Natural Language Processing Task LLM Inference Redundancy Aware Transformer Block Elimination Voting

February 7, 2024

February 5, 2024

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu
LLM Inference Key Value Cache Kinship Verification Cache Quantization

February 3, 2024

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
Yichao Fu, Peter Bailis, Ion Stoica, Hao Zhang
Large Language Model LLM Inference Speculative Decoding Autoregressive Decoding Sequential Dependency Lookahead Decoding

February 2, 2024

February 1, 2024

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
Xuchen Pan, Yanxi Chen, Yaliang Li, Bolin Ding, Jingren Zhou
LLM Inference Scalable Solution

January 25, 2024

ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models
Yao Fu, Leyang Xue, Yeqi Huang, Andrei-Octavian Brabete, Dmitrii Ustiugov, Yuvraj Patel, Luo Mai
LLM Inference Model Checkpoint

January 19, 2024

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao
LLM Inference Speculative Decoding Inference Acceleration Auto Regressive Decoding

January 15, 2024

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, Zhifang Sui
Comprehensive Survey LLM Inference Speculative Decoding Inference Latency Large Language Model Inference Enhancing Efficiency Autoregressive Decoding

December 31, 2023

Fairness in Serving Large Language Models
Ying Sheng, Shiyi Cao, Dacheng Li, Banghua Zhu, Zhuohan Li, Danyang Zhuo, Joseph E. Gonzalez, Ion Stoica
Procedural Fairness LLM Inference Fair Scheduling

December 23, 2023

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference
Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, Zhiru Zhang
Full Potential LLM Inference Large Language Model Inference High Level Synthesis Inference Workload Spatial Accelerator

December 19, 2023

Efficient LLM inference solution on Intel GPU
Hui Wu, Yi Gan, Feng Yuan, Jing Ma, Wei Zhu, Yutao Xu, Hong Zhu, Yuhua Zhu, Xiaoli Liu, Jinghui Gu, Peng Zhao
LLM Inference Single GPU Decoder Only LLM

December 13, 2023

Reconciling Shared versus Context-Specific Information in a Neural Network Model of Latent Causes
Qihong Lu, Tan T. Nguyen, Qiong Zhang, Uri Hasson, Thomas L. Griffiths, Jeffrey M. Zacks, Samuel J. Gershman, Kenneth A. Norman
Context Learning Context Information Neural Network Model LLM Inference Latent Causal Sharing Matter Contextual Structure Common Cause

December 12, 2023

ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexity
Henry Bae, Aghyad Deeb, Alex Fleury, Kehang Zhu
Language Model Complexity Matter LLM Inference Model Complexity Task Complexity

December 8, 2023

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, Jingren Zhou
Medical LLM Scientific Inference LLM Inference Inference Speed Early Exit Large Scale Training 3D Parallelism

Papers

QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference

Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference

HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference

SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks

Hydragen: High-Throughput LLM Inference with Shared Prefixes

Online Cascade Learning for Efficient Inference over Streams

KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

InferCept: Efficient Intercept Support for Augmented Large Language Model Inference

Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models

ServerlessLLM: Locality-Enhanced Serverless Inference for Large Language Models

Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

Fairness in Serving Large Language Models

Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference

Efficient LLM inference solution on Intel GPU

Reconciling Shared versus Context-Specific Information in a Neural Network Model of Latent Causes

ComplexityNet: Increasing LLM Inference Efficiency by Learning Task Complexity

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism