Memory-Efficient Inference

Overview

Memory-efficient inference studies how to run models within limited GPU, device, or server memory, especially as model weights and attention caches grow. Common approaches include weight and KV-cache quantization, pruning and eviction, sparse attention, and recurrent models; systems also use paging, CPU or SSD offloading, streaming, and expert caching for mixture-of-experts models. Researchers evaluate these methods on long-context retrieval and reasoning, model-serving workloads, and edge devices, measuring not only memory but also accuracy, latency, and throughput. The central tension is that aggressive compression or offloading can hurt quality or add data-movement costs, so current work increasingly adapts memory allocation to the model, request, and hardware.

Latest papers

CardsList

179 papers11 papers in the last four weeks, down 69% on the four weeks before. 0.1% of all new papers.

  1. SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

    Nov 10, 2026Haolin Jiang, Weihang Li, Zhongzhu Zhou +3LLM Inference AccelerationMemory-Efficient Inference

  2. MINE: Entropy-coded Memory-efficient Inference for Large Language Models

    Nov 6, 2026Emil Schätzle, Markus PüschelStructured Neural Network CompressionLLM Inference Acceleration

  3. LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

    Nov 4, 2026Teresa Delgado, Hugo Hazard, Martin Benfeghoul +2KV-Cache CompressionMemory-Efficient Inference

  4. Rimuru: Unified Memory Management for Modern LLM Serving via Elastic Expert Offloading

    Oct 29, 2026Zhiyuan Fang, Dingyang Li, Jianfeng Zhong +6LLM ServingMemory-Efficient Inference

  5. CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

    Oct 27, 2026Congwei Zhang, Youming Tao, Yifei Zou +6Mixture-of-Experts InferenceMemory-Efficient Inference

  6. BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

    Oct 22, 2026Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick +1KV-Cache ReuseMemory-Efficient Inference

  7. TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

    Oct 17, 2026Yu Wang, Zhu Wang, Ning Hu +4Memory-Efficient InferenceLong-Context Language Model Inference

  8. MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

    Oct 13, 2026Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5KV-Cache QuantizationMemory-Efficient Inference

  9. SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

    Oct 13, 2026Jieyun Deng, Keming Hu, Yanghe Liu +1LLM Inference AccelerationOn-Device Language Model Inference

  10. FORGE: Procedural Weight Generation for Sparse MoE Inference

    Oct 12, 2026Chanyoung Moon, Travis Pence, Daisuke Yamada +4Mixture-of-Experts InferenceLLM Inference Acceleration

  11. PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion

    Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient InferenceLong-Context Language Model Inference

  12. Preserving Updates in Low-Bit Recurrent State Quantization

    Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State QuantizationLow-Bit Quantization

  13. Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention

    Oct 2, 2026Inbasekaran SKV-Cache EvictionMemory-Efficient Inference

  14. AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning

    Oct 2, 2026Han Yu, Wenhui Zhu, Xiwen Chen +9KV-Cache EvictionCache Replacement

  15. The Surprising Effectiveness of Shared Memory in Looped Transformers

    Oct 1, 2026Giovanni Monea, Keshav Ramji, Yousef El-Kurdi +4Memory-Efficient InferenceRecurrent Transformers

  16. Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

    Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient InferenceConvolutional Neural Networks