Memory-Efficient Inference

Ask about this topic

Momentum

18 papers in the last four weeks, up 200% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 100

All topics
CardsList

  1. Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

    Oct 5, 2026Runguo LiCoT ReasoningKV-Cache Eviction

    University of Illinois Urbana-Champaign

  2. Right In-Place (RiP) Convolution: A Simple, General, and Near-Optimal Strategy for Memory-Efficient CNN Inference

    Sep 30, 2026Opegbemi Matthias Busoye, Tolulope Matthew Busoye, Eghonghon-aye EigbeMemory-Efficient InferenceConvolutional Neural Networks

    PowerLabs Technologies, Lagos, Nigeria

  3. UltraMatch: Transport Path Routing for Ultra-Fast and Memory-Efficient Image Matching

    Sep 29, 2026Jiajun Le, Yifan Lu, Zizhuo Li +3Image MatchingMemory-Efficient Inference

    Electronic Information School, Wuhan University, China · Xiaomi Corporation, China · School of Computer Science and Technology, Harbin Institute of Technology, China +1

  4. SANTA++: Sampling Attention through Representative Keys

    Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient InferenceLLM Inference Acceleration

    Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA

  5. DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines

    Sep 28, 2026Haixiao Gao, Yimin Zheng, Linyou Xiao +1Memory-Efficient InferenceFull-Duplex Spoken Dialogue Systems

    Jinan University, China · Hong Kong University of Science and Technology (Guangzhou), China

  6. DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

    Sep 28, 2026Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu +2LLM ServingLLM Inference

    Department of Next Generation Semiconductor Convergence and Open Sharing System (COSS), Seoul National University, Seoul 08826, South Korea · Van-Lang Institute of Semiconductor Technology (VIST), Hanoi, Vietnam · Efficient Computation Research Group, Moreh Vietnam +1

  7. MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference

    Sep 28, 2026Junfeng Wu, Zehao Fan, Hadjer Benmeziane +3Expert OffloadingMixture-of-Experts Language Models

    Department of Industrial and Systems Engineering, Rensselaer Polytechnic Institute · Department of Electrical, Computer, and Systems Engineering, Rensselaer Polytechnic Institute · IBM Research

  8. PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

    Sep 28, 2026Hyesung Jeon, Hyeongju Ha, Jae-Joon KimMulti-Agent LLM SystemsMemory-Efficient Inference

    Department of Electrical and Computer Engineering Seoul National University

  9. When to Evict, Not What to Keep: Draft-Guided Eviction for Training-Free KV-Cache Compression

    Sep 27, 2026Haeyong Kang, Chang D. YooKV-Cache EvictionMemory-Efficient Inference

    Duksung Women’s University · KAIST

  10. MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compression

    Sep 24, 2026Youpeng Zhao, Tian Tan, Liqian Peng +2Memory-Efficient InferenceLLM Inference Acceleration

    Google · University of Central Florida

  11. Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone

    Sep 24, 2026Musa ShamsExpert OffloadingOn-Device Language Model Inference

    Independent Researcher

  12. Support-Compiled Feature Folding: More Evidence at Lower Memory Across Tabular Foundation Models

    Sep 23, 2026Tian Zhou, Beverly Jin, Xue Wang +6Tabular Foundation ModelsMemory-Efficient Inference

    Ant Group · Independent Researcher

  13. Risk-Controlled KV-Cache Eviction: From Memory Budgets to Risk Targets

    Sep 23, 2026Beomgu Kang, SoJin Yun, Hojoon Kim +1KV-Cache EvictionMemory-Efficient Inference

    Department of Artificial Intelligence, Korea University, Republic of Korea

  14. Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

    Sep 22, 2026Md Mostafizer Rahman, Md Faizul Ibne Amin, Md Shahajada Mia +2LLM Inference EfficiencyLong-Context Language Model Inference

    Lucy Family Institute for Data & Society, University of Notre Dame, IN, USA · The University of Aizu, Aizuwakamatsu, Japan · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame, IN, USA

  15. ARM: Attention with Routed-Memory for Learnable Sparse Control

    Sep 21, 2026Qiuhao Zeng, Jerry Huang, Peng Lu +7Memory-Augmented Language ModelsLong-Context Language Model Inference

    University of Toronto, Canada · Vector Institute, Canada · RIKEN AIP, Japan +3

  16. TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching

    Sep 18, 2026Zhihao Shu, Md Musfiqur Rahman Sanim, Jie Hu +4On-Device Language Model InferenceLong-Context Language Model Inference

    University of Georgia Athens, GA, USA · Peking University Beijing, China · Western Digital Research San Jose, CA, USA

  17. The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

    Sep 16, 2026Yu Lin, Yiming Wang, Runyuan Cai +2Expert OffloadingMixture-of-Experts Inference

    AutoArk

  18. JustFit: Just-in-Time State Management for Local LLM Serving

    Sep 15, 2026Yuhua ChenLLM ServingOn-Device Language Model Inference

  19. Jacap: Robust KV Cache Eviction via Jacobian-Based Nonlinear Information Capacity Preservation

    Sep 8, 2026Jiaming Yang, Chenwei Tang, Liangli Zhen +2KV CachingKV-Cache Eviction

    College of Computer Science, Sichuan University, Chengdu, China · 2Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu, China · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore

  20. LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference

    Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference EfficiencyOn-Device Language Model Inference

    Richard · George Mason University · Global Technology Applied Research, JPMorganChase +1

  21. mzCache: On-Device LLM Memory Management under Multitasking

    Sep 1, 2026Hongseung Yu, Minsung Kim, Jongseok Park +1On-Device Language Model InferenceMemory-Efficient Inference

    Seoul National University · UC Berkeley

  22. DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

    Aug 31, 2026Yanqi Yu, Pingwei Sun, Jianchao Tan +4KV CachingMemory-Efficient Inference

    East China Normal University · Meituan · South China University of Technology

  23. DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

    Aug 27, 2026Tao Zhang, Jianchao Tan, Pingwei Sun +6Mixed-Precision QuantizationLLM Quantization

    South China University of Technology · Meituan · East China Normal University

  24. APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

    Aug 12, 2026Alish Kanani, Layan Badawi, Umit Y. OgrasEdge InferenceMixture-of-Experts Inference

    University of Wisconsin–Madison

  25. MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices

    Aug 11, 2026Eunjeong Kim, Yeong Jun Jeon, Myeonggyun HanOn-Device Language Model InferenceLLM Inference Scheduling

    Kyungpook National University Daegu, Republic of Korea

  26. DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inference

    Aug 9, 2026Asaad AlthoubiKV CachingLong-Context Language Model Inference

    Oklahoma State University Stillwater, OK USA

  27. CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents

    Aug 8, 2026Weizhong Huang, Jinchao Zhang, Xiawu ZhengKV CachingLLM Agent Memory

    Xiamen University · WeChat AI, Tencent Inc., China

  28. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    Aug 5, 2026Sangwoo Ha, Hyunwoo Seo, Yurim Jo +2AI Accelerator InferenceOn-Device Language Model Inference

    KAIST Daejeon, South Korea

  29. AnchorKV: Anchor-Residual KV Cache Compression

    Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV CachingLong-Context Language Model Inference

    Department of Computer Science, Technion - Israel Institute of Technology

  30. Gram-Space: Structure-Preserving Codebook Compression for Memory-Efficient Neuro-Symbolic AI

    Aug 2, 2026Weilun Wang, Wantong LiMemory-Efficient InferenceVector Symbolic Architectures

    University of California, Riverside

  31. SemPIC: Learning Semantic Position-Independent KV Caches

    Jul 30, 2026Hui Xie, Peng Xiao, Yutong Deng +3KV CachingMemory-Efficient Inference

    Institute of Artificial Intelligence, Beihang University · State Key Laboratory of Complex & Critical Software Environment · Shen Yuan Honors College, Beihang University +1

  32. WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

    Jul 30, 2026Fanzhe Wei, Li Liu, Ziyang Wang +1KV CachingMemory-Efficient Inference

    Metask Lab

  33. Memory Efficient Tabular Foundation Models

    Jul 30, 2026Shuting Luo, Monika Mikhail Kanaan, Cameron Gordon +2Model CompressionTabular Foundation Models

    Commonwealth Bank of Australia, Australia · Australian Institute for Machine Learning, University of Adelaide, Adelaide, Australia

  34. LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

    Jul 29, 2026Ming-Yen Lee, Hanchen Yang, Faaiq Waqar +4LLM Inference EfficiencyLLM Serving

    Georgia Institute of Technology, Atlanta, GA, USA · Samsung Semiconductor Inc., San Jose, CA, USA

  35. Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating

    Jul 27, 2026Maruthi Vemula, Neeraj Praneeth GajulaKV CachingKV-Cache Eviction

    University of North Carolina at Chapel Hill · New York University

  36. RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

    Jul 24, 2026Anderson R. SantosLong-Context Language Model InferenceMemory-Efficient Inference

    Faculty of Computing, Federal University of Uberlândia (UFU), Brazil.

  37. Lean-SAM2: Target-Anchored Memory and Encoder Acceleration for SAM2

    Jul 22, 2026Xudong Ouyang, Wenlun Zhang, Yimin Xu +2Memory-Augmented Neural NetworksVideo Object Segmentation

    School of Biomedical Engineering, Hainan University, Hainan 570228, China · Department of Electronics and Electrical Engineering, Keio University, Kanagawa 223-8522, Japan · College of Computer Science, Shenyang Aerospace University, Shenyang 110135, China +1

  38. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1On-Device Language Model InferenceMemory-Efficient Inference

    Department of Computer Networks, Paderborn University, Paderborn, Germany

  39. Taurus: Accelerating Out-of-Core Graph Neural Network Inference on Billion-Scale Graphs

    Jul 19, 2026Pranjal Naman, Yogesh SimmhanEfficient Neural Network InferenceGraph Neural Networks

    Department of Computational and Data Sciences (CDS), Indian Institute of Science (IISc), Bangalore 560012 India

  40. SlimPer: Make Personalization Model Slim and Smart

    Jul 14, 2026Siqi Wang, Xianjie Chen, Shaofeng Deng +42Learning to RankMemory-Efficient Inference

  41. On Exploring Input Resolution Scaling For Anytime LiDAR Object Detection

    Jul 9, 2026Ahmet Soyyigit, Shuochao Yao, Heechul YunAdaptive InferenceLidar-Based 3D Object Detection

    The National Defense University, Istanbul, Turkiye · George Mason University, Fairfax, VA, USA · The University of Kansas, Lawrence, KS, USA

  42. Akashic: A Low-Overhead LLM Inference Service with MemAttention

    Jul 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +7LLM ServingMemory-Augmented Language Models

    Xiaohongshu Inc., China · ShangHai JiaoTong University · Peking University

  43. MiLSD: A Micro Line-Segment Detector for Resource-Constrained Devices

    Jul 7, 2026Parsa Hassani Shariat Panahi, Amir Hossein Jalilvand, M. Hassan NajafiTiny MLMemory-Efficient Inference

  44. AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

    Jul 4, 2026Sadra SaremiOn-Device Language Model InferenceMemory-Efficient Inference

    Department of Physics, Iran University of Science and Technology, Tehran 16846-13114, Iran.

  45. The risk of KV cache compression

    Jul 1, 2026Lukas Haverbeck, Carmen Amo Alonso, Andres Felipe Posada-Moreno +2Transformer InferenceKV Caching

    RWTH Aachen University · Stanford University

  46. Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

    Jun 28, 2026Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi +1Long-Context Language Model InferenceKV-Cache Eviction

    RBC Borealis

  47. Affix Cache for Diffusion Large Language Models

    Jun 26, 2026Kaihua Liang, An Zhong, Xin Tan +4KV CachingMemory-Efficient Inference

    KAUST · CUHK · LUMS

  48. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1LLM ServingLLM Inference Scheduling

    Gaoling School of Artificial Intelligence Renmin University of China Beijing, China · Department of Management Science and Engineering Stanford University Stanford, CA 94305-4026 · Department of Industrial Engineering and Decision Analytics HKUST Hongkong, China

  49. WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

    Jun 20, 2026Jiamu Zhang, Liang Wu, Mayank Darbari +1Expert OffloadingMixture-of-Experts Inference

    Nokia, Sunnyvale, CA 94085, USA

  50. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    Jun 20, 2026Xin GaoTransformer InferenceKV Caching

    Department of Mathematics and Statistics York University Toronto, M3J 1P3

  51. Memory Is No Longer a Bottleneck: Memory-Efficient Graph Filtering for Scalable Collaborative Filtering

    Jun 19, 2026Jin-Duk Park, Won-Yong ShinMemory-Efficient InferenceKrylov Subspace Methods

    School of Mathematics and Computing (Computational Science and Engineering), Yonsei University, Seoul 03722, Republic of Korea

  52. MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference

    Jun 17, 2026Craig OpieExpert OffloadingOn-Device Language Model Inference

    Holocron Security, Inc.

  53. Sticky Routing: Training MoE Models for Memory-Efficient Inference

    Jun 12, 2026Ali KayyamMixture-of-Experts InferenceMemory-Efficient Inference

    BrainChip Inc., Laguna Hills, CA

  54. VQ4SNN: Vector Quantization for Memory-Efficient FPGA Spiking Neural Networks

    Jun 12, 2026Dimitrios Sekertzis, Giorgos DimitrakopoulosMemory-Efficient InferenceVector Quantization

    Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece

  55. Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories

    Jun 9, 2026Travis Pence, Daisuke Yamada, Vikas SinghMemory-Efficient InferenceVector Symbolic Architectures

    Department of Computer Science, University of Wisconsin-Madison, United States · Department of Biostatistics and Med. Info., University of Wisconsin-Madison, United States

  56. FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

    Jun 8, 2026Yan Wang, Qifan Zhang, Jiachen Yu +12KV CachingLong-Context Language Model Inference

    Independent Researchers · Tencent · The Hong Kong University of Science and Technology (Guangzhou) +1

  57. Operator Fusion for LLM Inference on the Tensix Architecture

    Jun 3, 2026Qingbo Wu, Ke Li, Wenzhu Wang +3Efficient Transformer InferenceAI Accelerator Inference

    KylinSoft Co., Ltd, Tianjin, China · openKylin Community, Tianjin, China · College of Computer Science and Technology, National University of Defense Technology, Changsha, China

  58. PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers

    Jun 2, 2026Rachmad Vidya Wicaksana Putra, Achyuta Muthuvelan, Alberto Marchisio +1Memory-Efficient InferenceSpiking Neural Networks

    eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)

  59. AURA: Action-Gated Memory for Robot Policies at Constant VRAM

    Jun 1, 2026Josef ChenVisuomotor Policy LearningMemory-Efficient Inference

    KAIKAKU

  60. dMoE: dLLMs with Learnable Block Experts

    May 29, 2026Sicheng Feng, Zigeng Chen, Gongfan Fang +2Mixture-of-Experts Language ModelsMemory-Efficient Inference

    National University of Singapore