Large Language Model Inference

Large language model (LLM) inference research focuses on optimizing the speed and efficiency of generating text from LLMs, aiming to reduce computational costs and latency without sacrificing accuracy. Current efforts concentrate on techniques like quantization, model compression (including pruning and knowledge distillation), improved caching strategies (especially for key-value stores), and novel decoding methods such as speculative decoding and multi-token generation. These advancements are crucial for deploying LLMs on resource-constrained devices and for making large-scale LLM applications more economically and environmentally sustainable.

Papers

July 17, 2024

LLM Inference Serving: Survey of Recent Advances and Opportunities
Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari
Large Language Model Machine Learning Timely Survey Medical LLM Emerging Opportunity Recent Advance Large Language Model Inference

July 12, 2024

Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference
Zongyue Qin, Ziniu Hu, Zifan He, Neha Prakriya, Jason Cong, Yizhou Sun
Large Language Model Speculative Decoding Large Language Model Inference Perplexity Analysis Greedy Decoding

July 10, 2024

Interpretable Differential Diagnosis with Dual-Inference Large Language Models
Shuang Zhou, Mingquan Lin, Sirui Ding, Jiashuo Wang, Genevieve B. Melton, James Zou, Rui Zhang
Large Language Model Inference Differential Diagnosis Diagnostic Reasoning

July 4, 2024

The Price of Prompting: Profiling Energy Use in Large Language Models Inference
Erik Johannes Husom, Arda Goknil, Lwin Khin Shar, Sagar Sen
Large Language Model LLM Inference Power Consumption Large Language Model Inference Curious Price Driven Inference

June 25, 2024

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training
Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che
Large Language Model Industrial Disturbing Noise Supervised Fine Tuning Large Language Model Inference Inference Speed Token Generation Noisy Student Training

June 16, 2024

Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp
Longhao Chen, Yina Zhao, Qiangjun Xie, Qinghua Sheng
Optimization Purpose Large Language Model Inference Inference Performance Compiler Optimization Virtual ARM

June 12, 2024

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference
Christopher Wolters, Xiaoxuan Yang, Ulf Schlichtmann, Toyotaro Suzumura
Large Language Model Transformer Based Model AI System Memory Trace LLM Inference Large Language Model Inference Representation Bottleneck

June 10, 2024

PowerInfer-2: Fast Large Language Model Inference on a Smartphone
Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, Haibo Chen
Large Language Model Inference Inference Speed Smartphone Device Based Inference

May 24, 2024

RAEE: A Robust Retrieval-Augmented Early Exiting Framework for Efficient Inference
Lianming Huang, Shangyu Wu, Yufei Cui, Ying Xiong, Xue Liu, Tei-Wei Kuo, Nan Guan, Chun Jason Xue
Training Free Efficient Inference Large Language Model Inference Model Inference Early Exiting

May 12, 2024

Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization
Xinyuan Zhang, Jiang Liu, Zehui Xiong, Yudong Huang, Gaochang Xie, Ran Zhang
Generative Artificial Intelligence Quantization Operator Large Language Model Inference Edge Intelligence Batch Data Edge Resource

May 7, 2024

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization
Tianyi Zhang, Jonah Yi, Zhaozhuo Xu, Anshumali Shrivastava
Quantization Operator Large Language Model Inference P Bit Key Value Cache Channel Quality Key Value Cache Compression

May 3, 2024

Efficient and Economic Large Language Model Inference with Attention Offloading
Shaoyuan Chen, Yutong Lin, Mingxing Zhang, Yongwei Wu
Large Language Model High Efficiency Hardware Accelerator Large Language Model Inference Dual Attention Attention Computation Attention Operator

April 24, 2024

CORM: Cache Optimization with Recent Message for Large Language Model Inference
Jincheng Dai, Zhuowei Huang, Haiyun Jiang, Chen Chen, Deng Cai, Wei Bi, Shuming Shi
Fine Tuning Transformer Model Large Language Model Inference Co Occurrence KV Cache System Message Inference Memory Usage Cache Overloading

April 14, 2024

Self-Selected Attention Span for Accelerating Large Language Model Inference
Tian Jin, Wanzin Yazar, Zifei Xu, Sayeh Sharify, Xin Wang
Large Language Model Autoregressive Model LLM Inference Large Language Model Inference Inference Framework

April 9, 2024

FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Zhengran Zeng, Wei Ye, Jindong Wang, Yue Zhang, Shikun Zhang
Large Language Model Automatic Evaluation Large Language Model Inference Modular Framework Efficient Evaluation Meta Evaluation

April 7, 2024

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
Zihao Wang, Bin Cui, Shaoduo Gan
Large Language Model 2 Dimensional Large Language Model Inference Key Value Cache Key Value Cache Compression Multi Session Budget Optimization Token Sparsification

March 29, 2024

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs
Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang, Qin Xie
Large Language Model Large Language Model Inference Inference Speed Transformer XL Uncertainty Aware Deployment Mobile GPUs Mobile Inference

March 26, 2024

ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching
Youpeng Zhao, Di Wu, Jun Wang
Large Language Model Natural Language Processing Transformer Architecture LLM Inference Large Language Model Inference KV Cache Compression Memory Access

March 19, 2024

Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference
Baolin Li, Yankai Jiang, Vijay Gadepally, Devesh Tiwari
Generative Artificial Intelligence Generative Large Language Model Large Language Model Inference GenAI Integration AI Technology High Quality Generation Generation Framework

February 29, 2024

FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning
Xupeng Miao, Gabriele Oliaro, Xinhao Cheng, Mengdi Wu, Colin Unger, Zhihao Jia
System Description Large Language Model Inference Parameter Efficient Finetuning Artificial Intelligence Inference

Large Language Model Inference

Papers

LLM Inference Serving: Survey of Recent Advances and Opportunities

Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inference

Interpretable Differential Diagnosis with Dual-Inference Large Language Models

The Price of Prompting: Profiling Energy Use in Large Language Models Inference

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training

Optimization of Armv9 architecture general large language model inference performance based on Llama.cpp

Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inference

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

RAEE: A Robust Retrieval-Augmented Early Exiting Framework for Efficient Inference

Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization

KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization

Efficient and Economic Large Language Model Inference with Attention Offloading

CORM: Cache Optimization with Recent Message for Large Language Model Inference

Self-Selected Attention Span for Accelerating Large Language Model Inference

FreeEval: A Modular Framework for Trustworthy and Efficient Evaluation of Large Language Models

SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching

Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference

FlexLLM: A System for Co-Serving Large Language Model Inference and Parameter-Efficient Finetuning