Memory-Efficient Inference

Ask about this topic

Overview

Memory-efficient inference studies how to run models within limited GPU, device, or server memory, especially as model weights and attention caches grow. Common approaches include weight and KV-cache quantization, pruning and eviction, sparse attention, and recurrent models; systems also use paging, CPU or SSD offloading, streaming, and expert caching for mixture-of-experts models. Researchers evaluate these methods on long-context retrieval and reasoning, model-serving workloads, and edge devices, measuring not only memory but also accuracy, latency, and throughput. The central tension is that aggressive compression or offloading can hurt quality or add data-movement costs, so current work increasingly adapts memory allocation to the model, request, and hardware.

Latest papers

CardsList

168 papers27 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

    Oct 27, 2026Congwei Zhang, Youming Tao, Yifei Zou +6Mixture-of-Experts InferenceMemory-Efficient Inference

    School of Electrical Engineering and Computer Science, TU Berlin, Berlin, Germany · School of Computer Science and Technology, Shandong University, Qingdao, China · Department of Data Science, City University of Hong Kong, Hong Kong, China

  2. BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing

    Oct 22, 2026Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick +1KV-Cache ReuseMemory-Efficient Inference

    University of California, San Diego · Together AI

  3. MemT-1.0: Towards Memory-Native Evolution

    Oct 19, 2026Zhiyu Li, Zeyu Zhang, Linyun Liu +71Memory-Augmented Language ModelsPersistent Memory for Language Models

  4. TideKV: Delayed KV Eviction via Layer-Wise Chunked Prefill for Memory-Efficient Long-Context LLM Inference

    Oct 17, 2026Yu Wang, Zhu Wang, Ning Hu +4Memory-Efficient InferenceLong-Context Language Model Inference

    Northwestern Polytechnical University, No. 127 West Youyi Road, Xian, 710129, Shaanxi, China · Xi’an Aeronautics Computing Technique Research Institute, AVIC, No. 15 Jinye 2nd Road, Xi’an, 710065, Shaanxi, China · College of Computer Science and Technology, Harbin Engineering University, No. 145 Nantong Street, Harbin, 150001, Heilongjiang, China

  5. Peak-Memory Schedules You Can Trust: Exact Realization, Certified Optimality, and the Limits of Recomputation for Inference

    Oct 16, 2026Ivan Fomin, Ivan Koryabkin, Boris SheludkoInference-Time OptimizationMemory-Efficient Inference

    Yandex, Moscow, Russia.

  6. From Long Context to Selective Memory:A Survey of Memory-Centric Compression for Large Language Models

    Oct 15, 2026Chaogui GouLLM Context ManagementMemory-Efficient Inference

    USTB

  7. MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

    Oct 13, 2026Yangchengyu Zhou, Zhiping Wu, Runchang Jiang +5KV-Cache QuantizationMemory-Efficient Inference

    Nanjing University · The Hong Kong University of Science and Technology · Geely +1

  8. SparPR: Runtime KV-Cache Recovery for Resource-Constrained LLM Inference on Consumer Hardware

    Oct 13, 2026Jieyun Deng, Keming Hu, Yanghe Liu +1LLM Inference AccelerationOn-Device Language Model Inference

    Southwest Jiaotong University, Chengdu, China · Wuhan Institute of Technology, Wuhan, China · Southwest Petroleum University, Chengdu, China

  9. FORGE: Procedural Weight Generation for Sparse MoE Inference

    Oct 12, 2026Chanyoung Moon, Travis Pence, Daisuke Yamada +4Mixture-of-Experts InferenceLLM Inference Acceleration

    University of Wisconsin-Madison Madison, Wisconsin, USA · Korea University Seoul, Republic of Korea · NVIDIA Santa Clara, California, USA

  10. GpFlash: Maximizing Speed and Minimizing Memory for Gaussian Process Training and Prediction

    Oct 12, 2026Andre MarchildonGaussian ProcessesGaussian Process Regression

    Applied Scientist at Amazon Prime Air, Seattle, USA

  11. PAIR: Memory-Efficient and Fast Inference with Reading--Reasoning Fusion

    Oct 12, 2026Sejun Park, Haocheng Xia, Yongjoo ParkMemory-Efficient InferenceLong-Context Language Model Inference

    Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign, Urbana, IL 61801, USA

  12. Preserving Updates in Low-Bit Recurrent State Quantization

    Oct 11, 2026Haotian Li, Liran Dong, Kaicheng Xiao +1Recurrent-State QuantizationLow-Bit Quantization

    The Chinese University of Hong Kong

  13. Compile the Table: Query-Calibrated Operator Compression for Tabular In-Context Learning

    Oct 8, 2026Xu Zhao, Jiaming Zhao, Bin Zhao +1Context CompressionTabular ICL

  14. PageWeaver: KV-Guided Query Unions for Sparse Attention

    Oct 8, 2026Zhiyuan Li, Zihan Li, Zefang Yuan +2Sparse AttentionGPU Kernel Optimization

    SenseTime

  15. Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

    Oct 5, 2026Runguo LiCoT ReasoningKV-Cache Eviction

    University of Illinois Urbana-Champaign

  16. Page-EntroKV: Hardware-Aligned, Entropy-Weighted KV-Cache Eviction under Grouped-Query Attention

    Oct 2, 2026Inbasekaran SKV-Cache EvictionMemory-Efficient Inference

    Department of Computer Science and Engineering SRM Institute of Science and Technology, India

  17. AvoKV-E: Payload-Aware KV Cache Eviction for Long Reasoning

    Oct 2, 2026Han Yu, Wenhui Zhu, Xiwen Chen +9KV-Cache EvictionCache Replacement

    LinkedIn Corporation · Clemson University · Rice University +1

  18. Exact Memory-Time Optimization for Prefix-Cached Language Model Serving

    Oct 2, 2026Shivam GuptaLLM ServingMemory-Efficient Inference

    Independent Researcher, Delhi, India

  19. The Surprising Effectiveness of Shared Memory in Looped Transformers

    Oct 1, 2026Giovanni Monea, Keshav Ramji, Yousef El-Kurdi +4Memory-Efficient InferenceRecurrent Transformers

    Cornell University · IBM Research

  20. Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

    Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient InferenceConvolutional Neural Networks

    PowerLabs Technologies, Lagos, Nigeria

  21. Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

    Sep 29, 2026Sanjali Yadav, Bahar AsgariMixture-of-Experts InferenceMemory-Efficient Inference

    University of Maryland, College Park

  22. UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching

    Sep 29, 2026Jiajun Le, Yifan Lu, Zizhuo Li +3Image MatchingMemory-Efficient Inference

    Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China +1

  23. SANTA++: Sampling Attention through Representative Keys

    Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient InferenceLLM Inference Acceleration

    Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA

  24. DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines

    Sep 28, 2026Haixiao Gao, Yimin Zheng, Linyou Xiao +1Memory-Efficient InferenceFull-Duplex Spoken Dialogue Systems

    Jinan University, China · Hong Kong University of Science and Technology (Guangzhou), China

  25. One Distribution, Two Channels: Coverage-Calibrated KV Retention and Compensation

    Sep 28, 2026Shuxin Liu, Qing Liu, Yi Du +1KV-Cache EvictionKV-Cache Management

    Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences

  26. DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

    Sep 28, 2026Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu +2LLM ServingLLM Inference

    Department of Next Generation Semiconductor Convergence and Open Sharing System (COSS), Seoul National University, Seoul 08826, South Korea · Van-Lang Institute of Semiconductor Technology (VIST), Hanoi, Vietnam · Efficient Computation Research Group, Moreh Vietnam +1

  27. MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

    Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert OffloadingMixture-of-Experts Language Models

    Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research

  28. PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

    Sep 28, 2026Hyesung Jeon, Hyeongju Ha, Jae-Joon KimMulti-Agent LLM SystemsMemory-Efficient Inference

    Department of Electrical and Computer Engineering Seoul National University

  29. When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression

    Sep 27, 2026Haeyong Kang, Chang D. YooKV-Cache EvictionMemory-Efficient Inference

    Duksung Women’s University · KAIST

  30. Budgeted Cache Repair for Cross-Context KV-Cache Reuse

    Sep 27, 2026Haeyong Kang, Chang D. YooKV-Cache ManagementKV-Cache Reuse

    Duksung Women’s University · KAIST