Memory-Efficient Inference

Follow this topic Ask about this topic

Memory-efficient inference studies how to run models within limited GPU, device, or server memory, especially as model weights and attention caches grow. Common approaches include weight and KV-cache quantization, pruning and eviction, sparse attention, and recurrent models; systems also use paging, CPU or SSD offloading, streaming, and expert caching for mixture-of-experts models. Researchers evaluate these methods on long-context retrieval and reasoning, model-serving workloads, and edge devices, measuring not only memory but also accuracy, latency, and throughput. The central tension is that aggressive compression or offloading can hurt quality or add data-movement costs, so current work increasingly adapts memory allocation to the model, request, and hardware.

Latest papers

168 papers27 papers in the last four weeks, up 13% on the four weeks before. 0.2% of all new papers.

The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference
Mar 9, 2026Vignesh Adhinarayanan, Nuwan JayasenaMixture-of-Experts Inference Long-Context Language Model Inference
AMD Research and Advanced Development (RAD).
Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
Mar 1, 2026Mengqi Liao, Lu Wang, Chaoyun Zhang +6LLM Inference Acceleration LLM Inference Scheduling
School of Computer Science and Technology, Beijing Jiaotong University · Microsoft · Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence
KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Merging
Mar 1, 2026Lianjun Liu, Hongli An, Weiqi Yan +4KV-Cache Compression LLM Inference Acceleration
School of Information and Communication Engineering, Hainan University, Haikou, China · School of Cyberspace Security, Hainan University, Haikou, China · MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China +1
RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse
Feb 28, 2026Yingsheng Geng, Yuchong Gao, Weihong Wu +2LLM Inference Acceleration KV-Cache Reuse
Beijing University of Posts and Telecommunications, Beijing, China · Tsinghua University, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +1
ICaRus: Identical Cache Reuse for Efficient Multi Model Inference
Feb 27, 2026Sunghyeon Woo, Jaeeun Kil, Hoseung Kim +9LLM Inference Acceleration Memory-Efficient Inference
NAVER Cloud
CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
Feb 24, 2026Chao Fei, Guozhong Li, Chenxi Liu +1Long-Context Language Model Inference LLM Inference Acceleration
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences, Hong Kong, China
Jolt Atlas: Verifiable Inference via Lookup Arguments in Zero Knowledge
Feb 19, 2026Wyatt Benno, Alberto Centelles, Antoine Douchet +1Zero-Knowledge ML Privacy-Preserving ML
ICME Labs
DRetHTR: Linear-Time Decoder-Only Retentive Network for Handwritten Text Recognition
Feb 19, 2026Changhun Kim, Martin Mayr, Thomas Gorges +4Memory-Efficient Inference Efficient Transformer Inference
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations
Feb 17, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Hangyu Zheng +4Efficient Neural Network Inference Memory-Efficient Inference
University of Georgia Athens, GA, USA · University of Texas at Arlington Arlington, TX, USA
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
Feb 12, 2026Shuhuai Li, Jianghao Lin, Dongdong Ge +1Mixture-of-Experts Inference Edge Inference
Shanghai University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Standford University, California, United States
VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
Feb 11, 2026Muyan Hu, Ahan Gupta, Jiachen Yuan +7Efficient Neural Network Inference Memory-Efficient Inference
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Microsoft
FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge Inference Mixture-of-Experts Inference
Cerium: A Multi-GPU Framework for Terabyte-Scale Encrypted Inference
Dec 12, 2025Siddharth Jayashankar, Joshua Kim, Michael B. Sullivan +2Homomorphic Encryption GPU Kernel Optimization
Carnegie Mellon University · UT Austin · NVIDIA
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts Inference LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuText Embeddings Word Embeddings
Independent Researcher Beijing, China
ReFreeKV: Towards Threshold-Free KV Cache Compression
Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV Caching Memory-Efficient Optimization
Nanjing University of Aeronautics and Astronautics · WeChat AI, Tencent · Independent Researcher +1
FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM Quantization On-Device Language Model Inference
Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA

Mar 9, 2026J/K move · Enter open · S save

The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference

Vignesh Adhinarayanan, Nuwan Jayasena

AMD Research and Advanced Development (RAD).

Mixture-of-Experts Inference Long-Context Language Model Inference Memory-Efficient Inference

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models deliver high quality at low training FLOPs, but this efficiency often vanishes at inference. We identify a double penalty that structurally disadvantages MoE architectures during decoding: first, expert routing fragments microbatches and reduces weight reuse; second, massive resident expert pools reduce high-bandwidth memory (HBM) headroom for the KV cache. This phenomenon, formalized as reuse fragmentation, pushes feed-forward networks (FFNs) into a bandwidth-bound regime, especially at long context lengths. We introduce the

qs

inequality, a predictive criterion that identifies when MoE is structurally disadvantaged relative to a quality-matched dense model. This criterion unifies sparsity (

s

), the fraction of parameters activated per token, and the quality-equivalence factor (

q

), the size multiplier required for a dense model to match MoE performance. Our evaluation across frontier models including DeepSeek-V3, Qwen3-235B, Grok-1, and Switch-C demonstrates that this fragmentation is a general architectural phenomenon. For DeepSeek-V3 at 128k context, this results in a 4.5x throughput advantage for a quality-matched dense baseline. Crucially, massive architectures like Switch-C can become infeasible on cluster sizes where a quality-matched dense model remains viable. Our results suggest that training-time FLOP efficiency is an incomplete proxy for inference-time performance in long-context serving. They also indicate that MoE may be best viewed as a training-time optimization, with distillation into dense models as a possible path toward inference-efficient deployment.