Batch Inference

Batch inference optimizes the processing of multiple inputs simultaneously in machine learning models, aiming to improve efficiency and throughput. Current research focuses on enhancing batch inference for large language models (LLMs) through techniques like early exiting, efficient key-value cache management, and adaptive retrieval and composition of multiple models (e.g., using LoRA). These advancements are crucial for deploying large models in resource-constrained environments and for scaling up applications across various domains, including natural language processing and computer vision.

Papers

November 18, 2024

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica
Large Language Model Inference Task Dense Model GPU Memory Batch Inference

October 24, 2024

BATON: Enhancing Batch-wise Inference Efficiency for Large Language Models via Dynamic Re-batching
Peizhuang Cong, Qizhi Chen, Haochen Zhao, Tong Yang
Full Model LLM Inference Large Language Inference Service Automatic Batching Batch Inference

July 25, 2024

An Efficient Inference Framework for Early-exit Large Language Models
Ruijie Miao, Yihan Yan, Xinshuo Yao, Tong Yang
LLM Inference Efficient Inference Inference Efficiency Batch Inference

April 15, 2024

AntBatchInfer: Elastic Batch Inference in the Kubernetes Cluster
Siyuan Li, Youshao Xiao, Fanzhuang Meng, Lin Ju, Lei Liang, Lin Wang, Jun Zhou
Real Time Inference Kubernetes Cluster Batch Inference

April 8, 2024

Transductive Zero-Shot and Few-Shot CLIP
Ségolène Martin, Yunshi Huang, Fereshteh Shakeri, Jean-Christophe Pesquet, Ismail Ben Ayed
Zero Shot Single CLIP Shot Image Classification Transductive Inference Batch Inference Shot Video Classification

February 15, 2024

LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, Fei Wu
Large Language Model Wild Challenge Open Domain Compositional Ability Mixed Task Batch Inference

October 11, 2023

MatFormer: Nested Transformer for Elastic Inference
Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham Kakade, Ali Farhadi, Prateek Jain
ViT Architecture Batch Inference

August 8, 2023

Accelerating LLM Inference with Staged Speculative Decoding
Benjamin Spector, Chris Re
Large Language Model Speculative Decoding Large Language Model Inference Large Batch Batch Inference

May 26, 2023

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, Anshumali Shrivastava
Large Language Model Test Time Memory Management Inference Workload KV Cache Compression Batch Inference Inference Memory Usage

January 31, 2023

Differentiable Entailment for Parameter Efficient Few Shot Learning
Ethan Kim, Jerry Yang
Pre Trained Language Model Shot Learning Parameter Efficient Shot Keyword Batch Inference Differentiable Reasoning

December 10, 2021

Batch Label Inference and Replacement Attacks in Black-Boxed Vertical Federated Learning
Yang Liu, Tianyuan Zou, Yan Kang, Wenhan Liu, Yuanqin He, Zhihao Yi, Qiang Yang
Homomorphic Encryption Vertical Federated Learning Clean Label Backdoor Attack Label Inference Attack Synonym Substitution Attack Batch Inference