Memory-Efficient Inference

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 157% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 101

All topics

Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning
Oct 5, 2026Runguo LiCoT Reasoning KV-Cache Eviction
University of Illinois Urbana-Champaign
Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference
Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient Inference Convolutional Neural Networks
PowerLabs Technologies, Lagos, Nigeria
UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching
Sep 29, 2026Jiajun Le, Yifan Lu, Zizhuo Li +3Image Matching Memory-Efficient Inference
Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China +1
SANTA++: Sampling Attention through Representative Keys
Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient Inference LLM Inference Acceleration
Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA
DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines
Sep 28, 2026Haixiao Gao, Yimin Zheng, Linyou Xiao +1Memory-Efficient Inference Full-Duplex Spoken Dialogue Systems
Jinan University, China · Hong Kong University of Science and Technology (Guangzhou), China
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Sep 28, 2026Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu +2LLM Serving LLM Inference
Department of Next Generation Semiconductor Convergence and Open Sharing System (COSS), Seoul National University, Seoul 08826, South Korea · Van-Lang Institute of Semiconductor Technology (VIST), Hanoi, Vietnam · Efficient Computation Research Group, Moreh Vietnam +1
MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference
Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert Offloading Mixture-of-Experts Language Models
Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research
PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction
Sep 28, 2026Hyesung Jeon, Hyeongju Ha, Jae-Joon KimMulti-Agent LLM Systems Memory-Efficient Inference
Department of Electrical and Computer Engineering Seoul National University
When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression
Sep 27, 2026Haeyong Kang, Chang D. YooKV-Cache Eviction Memory-Efficient Inference
Duksung Women’s University · KAIST
MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression
Sep 24, 2026Youpeng Zhao, Tian Tan, Liqian Peng +2Memory-Efficient Inference LLM Inference Acceleration
Google · University of Central Florida
Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
Sep 24, 2026Musa ShamsExpert Offloading On-Device Language Model Inference
Independent Researcher
Support-Compiled Feature Folding: More Evidence at Lower Memory Across Tabular Foundation Models
Sep 23, 2026Tian Zhou, Beverly Jin, Xue Wang +6Tabular Foundation Models Memory-Efficient Inference
Ant Group · Independent Researcher
Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets
Sep 23, 2026Beomgu Kang, SoJin Yun, Hojoon Kim +1KV-Cache Eviction Memory-Efficient Inference
Department of Artificial Intelligence, Korea University, Republic of Korea
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference Efficiency Long-Context Language Model Inference
Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA
ARM: Attention with Routed-Memory for Learnable Sparse Control
Sep 21, 2026Qiuhao Zeng, Jerry Huang, Peng Lu +7Memory-Augmented Language Models Long-Context Language Model Inference
University of Toronto, Canada · Vector Institute, Canada · RIKEN AIP, Japan +3
TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
Sep 18, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu +4On-Device Language Model Inference Long-Context Language Model Inference
University of Georgia Athens, GA, USA · Peking University Beijing, China · Western Digital Research San Jose, CA, USA
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert Offloading Mixture-of-Experts Inference
AutoArk
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation
Sep 8, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +2KV Caching KV-Cache Eviction
College of Computer Science, Sichuan University, Chengdu, China · 2Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference Efficiency On-Device Language Model Inference
Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1
GaLe: memory-efficient Global Approximate and Local Exact features
Sep 2, 2026Alberto Ancilotto, Elisabetta FarellaMemory-Efficient Inference Edge AI
Fondazione Bruno Kessler (FBK), Trento, Italy
mzCache: On-Device LLM Memory Management under Multitasking
Sep 1, 2026Hongseung Yu, Minsung Kim, Jongseok Park +1On-Device Language Model Inference Memory-Efficient Inference
Seoul National University · UC Berkeley
DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV Caching Memory-Efficient Inference
East China Normal University · Meituan · South China University of Technology
DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization
Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Mixed-Precision Quantization LLM Quantization
South China University of Technology · Meituan · East China Normal University
APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference
Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge Inference Mixture-of-Experts Inference
University of Wisconsin–Madison
MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices
Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model Inference LLM Inference Scheduling
Kyungpook National University Daegu, Republic of Korea
DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference
Aug 9, 2026Asaad AlthoubiKV Caching Long-Context Language Model Inference
Oklahoma State University Stillwater, OK USA
CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents
Aug 8, 2026Weizhong Huang, Jinchao Zhang, Xiawu ZhengKV Caching LLM Agent Memory
Xiamen University · WeChat AI, Tencent Inc., China
EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding
Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator Inference On-Device Language Model Inference
KAIST Daejeon, South Korea
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology

Oct 5, 2026, cs.CLJ/K move · Enter open · S save

Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

Runguo Li

University of Illinois Urbana-Champaign

CoT Reasoning KV-Cache Eviction Memory-Efficient Inference KV-Cache Compression KV-Cache Quantization

Open paper AskPDF arXiv

Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.

Momentum

Related topics

Latest papers 101