Virtual Memory

Virtual memory management is crucial for efficiently serving large language models (LLMs), which demand substantial GPU memory for inference. Current research focuses on optimizing memory usage through techniques like paged attention mechanisms and novel tensor structures, aiming to minimize fragmentation and maximize throughput by dynamically allocating and sharing memory resources. These advancements significantly improve the speed and scalability of LLM deployment, enabling more efficient processing of large-scale language tasks and reducing computational costs. The resulting performance gains are particularly impactful for applications requiring high throughput and low latency, such as interactive chatbots and real-time language translation.

Papers

July 22, 2024

vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving
Jiale Xu, Rui Zhang, Cong Guo, Weiming Hu, Zihan Liu, Feiyang Wu, Yu Feng, Shixuan Sun, Changxu Shao, Yuhong Guo, Junping Zhao, Ke Zhang, Minyi Guo, Jingwen Leng
Key Value Cache Dynamic Memory Computation Kernel Efficient LLM Memory Aware Scheduling Virtual Memory

May 7, 2024

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar
Medical LLM Web Page KV Cache Dynamic Memory Virtual Memory

September 12, 2023

Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica
Large Language Model Key Value Cache Web Page KV Cache Memory Management Virtual Memory

Virtual Memory

Papers

vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Efficient Memory Management for Large Language Model Serving with PagedAttention