Memory-Efficient Inference

Ask about this topic

Overview

Memory-efficient inference studies how to run models within limited GPU, device, or server memory, especially as model weights and attention caches grow. Common approaches include weight and KV-cache quantization, pruning and eviction, sparse attention, and recurrent models; systems also use paging, CPU or SSD offloading, streaming, and expert caching for mixture-of-experts models. Researchers evaluate these methods on long-context retrieval and reasoning, model-serving workloads, and edge devices, measuring not only memory but also accuracy, latency, and throughput. The central tension is that aggressive compression or offloading can hurt quality or add data-movement costs, so current work increasingly adapts memory allocation to the model, request, and hardware.

Latest papers

CardsList

168 papers27 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

  1. Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference

    Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTransformer InferenceTransformer

    Zyphra

  2. WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition

    Apr 28, 2026Erfan Ramezani, Mohammad Mahdi Giahi, Mohammad Erfan Zarabadipour +2Streaming ASRSpeech Processing

    Mechatronics Research Laboratory, Dept. of Computer and Electrical Engineering, Qazvin Islamic Azad University, Qazvin, Iran · Department of Electrical Engineering, Qa. C., Islamic Azad University, Qazvin, Iran.

  3. DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference

    Apr 27, 2026Zahra Dehghanighobadi, Asja FischerLLM PruningKV Caching

    1Ruhr University Bochum · 2UAR Research Center for Trustworthy Data Science and Security

  4. MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

    Apr 22, 2026Anurita DasMemory-Efficient OptimizationMemory-Efficient Inference

    Genovation Technological Solutions Pvt Ltd

  5. Stream-CQSA: Exact Out-of-Memory Recovery for Attention

    Apr 22, 2026Yiming Bian, Joshua M. AkeySelf-AttentionMemory-Efficient Inference

    Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540

  6. Temporally Extended Mixture-of-Experts Models

    Apr 22, 2026Zeyu Shen, Peter HendersonMixture-of-Experts InferenceMemory-Efficient Inference

    1Princeton University

  7. Streaming Structured Inference with Flash-SemiCRF

    Apr 20, 2026Benjamin K. Johnson, Thomas Goralski, Ayush Semwal +2Efficient InferenceConditional Random Fields

    Department of Epigenetics, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA. · Department of Cell Biology, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA.

  8. Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models

    Apr 20, 2026Tobias Grantner, Emanuel Sallinger, Martin FlechlRecurrent Language ModelsMemory-Efficient Inference

    Dynatrace Research · TU Wien · University of Oxford

  9. Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks

    Apr 18, 2026Nges Brian Njungle, Eric Jahns, Michel A. KinsyEfficient Neural Network InferenceMemory-Efficient Inference

    STAM Center, Ira A. Fulton Schools of Engineering · Arizona State University, Tempe, AZ 85281, USA

  10. MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration

    Apr 16, 2026Xinyu Liu, Xin Liu, Bo Jin +8Multi-Token PredictionKV Caching

    School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China

  11. CASK: Core-Aware Selective KV Compression for Reasoning Traces

    Apr 13, 2026Buseong Kim, Heejun GwonKV-Cache CompressionMemory-Efficient Inference

    d’strict Korea

  12. ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrieval

    Apr 13, 2026David H. Yang, Yuxuan Zhu, Mohammad Mohammadi Amiri +4Memory-Efficient InferenceEfficient Language Model Reasoning

    Rensselaer Polytechnic Institute · IBM Research

  13. SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

    Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts InferenceLLM Inference Acceleration

    KAIST · UNIST

  14. A-IO: Adaptive Inference Orchestration for Memory-Bound NPUs

    Apr 10, 2026Chen Zhang, Yan Ding, Haotian Wang +3LLM Inference AccelerationLLM Inference Scheduling

    Hunan University Changsha, China

  15. SHIELD: A Segmented Hierarchical Memory Architecture for Energy-Efficient LLM Inference on Edge NPUs

    Apr 8, 2026Jintao Zhang, Xuanyao FongOn-Device Language Model InferenceEfficient Language Model Inference

    Department of Electrical and Computer Engineering, National University of Singapore, Singapore

  16. Learning to Unscramble Feynman Loop Integrals with SAILIR

    Apr 6, 2026David ShihScientific MLMemory-Efficient Inference

    NHETC, Dept. of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA

  17. Stochastic KV Routing: Enabling Adaptive Depth-Wise Cache Sharing

    Apr 3, 2026Anastasiia Filippova, David Grangier, Marco Cuturi +1KV-Cache ManagementMemory-Efficient Inference

    Apple

  18. FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

    Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert OffloadingGPU Acceleration

    CUHK · Unaffiliated · SCITIX +1

  19. GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference

    Mar 30, 2026Soutrik Mukherjee, Sangwhan ChaEfficient Transformer InferenceEfficient Neural Network Inference

    Department of Computer and Information Sciences, Harrisburg University of Science and Technology, Harrisburg, PA 17101, USA

  20. Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling

    Mar 29, 2026Songchen Ma, Hongyi Li, Weihao Zhang +8Mixture-of-Experts InferenceEfficient Neural Network Inference

    AI Chip Center for Emerging Smart Systems, Hong Kong SAR, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Shanghai UniVista Industrial Software Group Co., Ltd., Shanghai, China

  21. MotiMem: Motion-Aware Approximate Memory for Energy-Efficient Neural Perception in Autonomous Vehicles

    Mar 28, 2026Haohua Que, Mingkai Liu, Jiayue Xie +5Energy-Efficient MLMemory-Efficient Inference

    University of Georgia · Peking University · Tsinghua University +1

  22. PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection

    Mar 23, 2026Hyoseok Park, Yeonsang ParkLLM Inference AccelerationLong-Context LLM Inference

    Department of Physics, Chungnam National University, Daejeon 34134, Republic of Korea

  23. The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference

    Mar 20, 2026Kaleem Ullah Qasim, Jiashu Zhang, Muhammad Kafeel Shaheen +2Transformer InferenceMemory-Efficient Inference

    School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, 611756, China

  24. ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

    Mar 18, 2026Ruibo Fan, Xiangrui Yu, Xinglin Pan +5Model CompressionLLM Compression

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · The Hong Kong University of Science and Technology, Hong Kong, Hong Kong SAR

  25. Efficient Reasoning on the Edge

    Mar 17, 2026Yelysei Bondarenko, Thomas Hehn, Rob Hesselink +15Efficient Language Model ReasoningOn-Device Language Model Inference

    Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.

  26. SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging

    Mar 15, 2026Shunlong Wu, Hai Lin, Shaoshen Chen +5KV-Cache CompressionMemory-Efficient Inference

    Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China · Seoul National University, South Korea

  27. LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing

    Mar 13, 2026Jiawei Hao, Zhiwei Hao, Jianyuan Guo +4Mixture-of-Experts Language ModelsLLM Compression

    School of Information and Electronics, Beijing Institute of Technology, Beijing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China · Shenzhen Campus of Sun Yat-sen University, Shenzhen, China +2

  28. Accelerating Suffix Jailbreak attacks with Prefix-Shared KV-cache

    Mar 12, 2026Xinhai Wang, Shaopeng Fu, Shu Yang +3Jailbreak AttacksLLM Jailbreak Attacks

    King Abdullah University of Science and Technology (KAUST) · Zhejiang University

  29. Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

    Mar 12, 2026Xingyu Xie, Zhaochen Yu, Yue Liao +3LLM Inference AccelerationLong-Context Language Model Inference

    National University of Singapore · ByteDance

  30. LongFlow: Efficient KV Cache Compression for Reasoning Models

    Mar 12, 2026Yi Su, Zhenxu Tian, Dan Qiao +3KV-Cache CompressionLLM Inference Acceleration

    School of Computer Science and Technology, Soochow University, China · ByteDance