Memory-Efficient Inference

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 157% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 101

All topics

dMoE: dLLMs with Learnable Block Experts
May 29, 2026Sicheng Feng, Zigeng Chen, Gongfan Fang +2Mixture-of-Experts Language Models Memory-Efficient Inference
National University of Singapore
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
May 29, 2026Liang He, Jingbo Wen, Qishi Zhan +4KV Caching Long-Context Language Model Inference
Shanghai Institute of Optics and Fine Mechanics · The University of Sydney · Marquette University +4
Adaptive Mass-Segmented KV Compression for Long-Context Reasoning
May 22, 2026Junzhe Yang, Xiaoyu ShenKV Caching Long-Context Language Model Inference
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenKV Caching Memory-Efficient Optimization
Codepawl Ho Chi Minh City, Vietnam
ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning
May 21, 2026Yeqiu Chen, Ziyan Liu, Zhenxin Huang +3Inference-Time Search Tree of Thoughts
University of Science and Technology of China
Memory-Efficient Partitioned DNN Inference on Resource-Constrained Android Crowds
May 20, 2026Lakshani Manamperi, Disumi Pathirana, Thiwanka Pathirana +2Efficient Neural Network Inference Edge Computing
Department of Computer Science and Engineering, University of Moratuwa, Moratuwa, Sri Lanka.
Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
May 20, 2026Reese Levine, Rithik Sharma, Nikhil Jain +5GPU Acceleration LLM Inference
UC Santa Cruz · Microsoft Research
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +11KV Caching Rotation-Based Quantization
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3
TinySAM 2: Extreme Memory Compression for Efficient Track Anything Model
May 18, 2026Zhaoyuan Ding, Yijing Yang, Han Shu +1Video Object Segmentation Memory-Efficient Inference
Huawei Noahs Ark Lab
KV-Fold: One-Step KV-Cache Recurrence for Long-Context Inference
May 12, 2026Alireza Nadali, Patrick Cooper, Ashutosh Trivedi +1Transformer Inference KV Caching
Department of Computer Science University of Colorado Boulder Boulder, Colorado, 80301
OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models
May 12, 2026Seungwoo Roh, Huiyeong Kim, Jong-Chan KimVLMs for Autonomous Driving Efficient VLA Model Inference
Graduate School of Automobile and Mobility, Kookmin University, Korea
ChunkFlow: Communication-Aware Chunked Prefetching for Layerwise Offloading in Distributed Diffusion Transformer Inference
May 11, 2026Han Meng, Danny Willow Liu, Dong LiDiffusion Transformer GPU Acceleration
University of California, Merced · University of Chicago · University of California, Merced and Yotta Labs +1
CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
May 11, 2026Yuning Han, Yangchenchen Jin, Dylan Zhao +1On-Device Language Model Inference Memory-Efficient Inference
University of Florida Gainesville, FL 32611
Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
May 10, 2026Aojie Yuan, Tianqi Shen, Dajun ZhangKV-Cache Offloading KV Caching
University of Southern California · †University of Wisconsin–Madison
Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models
May 8, 2026Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo +3KV-Cache Management Memory-Efficient Inference
Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators
May 7, 2026Anupama Sridhar, Alexander JohansenKoopman Operator Learning Memory-Efficient Inference
California, USA · Stanford University Computer Science California, USA
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Compression Memory-Efficient Inference
Tsinghua University · Huazhong University of Science and Technology
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
May 6, 2026Chengyi Nie, Nian Si, Zijie ZhouLLM Serving LLM Inference
Department of Electrical and Computer Engineering, Stony Brook University · Department of Industrial Engineering and Decision Analytics, HKUST
StreamIndex: Memory-Bounded Compressed Sparse Attention via Streaming Top-k
May 4, 2026Jaber Jaber, Osama JaberGPU Kernel Optimization Self-Attention
RightNow AI
SURGE: SuperBatch Unified Resource-efficient GPU Encoding for Heterogeneous Partitioned Data
May 1, 2026Shashank Kapadia, Deep Narayan Mishra, Sujal Reddy Alugubelli +3Efficient Neural Network Inference GPU Acceleration
Walmart Inc. · USA
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1KV Caching Memory-Efficient Inference
Carnegie Mellon University
DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
Apr 29, 2026Bodon Jeong, Hongsu Byun, Youngjae Kim +4On-Device Language Model Inference KV-Cache Offloading
1Sogang University, Seoul, Republic of Korea · 2Florida State University, FL, USA · 3Samsung Electronics Co.
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
Folding Tensor and Sequence Parallelism for Memory-Efficient Transformer Training & Inference
Apr 29, 2026Vasu Shyam, Anna Golubeva, Quentin AnthonyTransformer Inference Transformer
Zyphra
WhisperPipe: A Resource-Efficient Streaming Architecture for Real-Time Automatic Speech Recognition
Apr 28, 2026Erfan Ramezani, Mohammad Mahdi Giahi, Mohammad Erfan Zarabadipour +2Streaming ASR Speech Processing
Mechatronics Research Laboratory, Dept. of Computer and Electrical Engineering, Qazvin Islamic Azad University, Qazvin, Iran · Department of Electrical Engineering, Qa. C., Islamic Azad University, Qazvin, Iran.
DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inference
Apr 27, 2026Zahra Dehghanighobadi, Asja FischerLLM Pruning KV Caching
1Ruhr University Bochum · 2UAR Research Center for Trustworthy Data Science and Security
MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference
Apr 22, 2026Anurita DasMemory-Efficient Optimization Memory-Efficient Inference
Genovation Technological Solutions Pvt Ltd
Stream-CQSA: Exact Out-of-Memory Recovery for Attention
Apr 22, 2026Yiming Bian, Joshua M. AkeySelf-Attention Memory-Efficient Inference
Lewis-Sigler Institute of Integrative Genomics, Princeton University, Princeton, NJ 08540
Temporally Extended Mixture-of-Experts Models
Apr 22, 2026Zeyu Shen, Peter HendersonMixture-of-Experts Inference Memory-Efficient Inference
1Princeton University
Streaming Structured Inference with Flash-SemiCRF
Apr 20, 2026Benjamin K. Johnson, Thomas Goralski, Ayush Semwal +2Efficient Inference Conditional Random Fields
Department of Epigenetics, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA. · Department of Cell Biology, Van Andel Institute, 333 Bostwick Ave. NE, Grand Rapids, 49503, MI, USA.
Linear-Time and Constant-Memory Text Embeddings Based on Recurrent Language Models
Apr 20, 2026Tobias Grantner, Emanuel Sallinger, Martin FlechlMemory-Efficient Inference Text Embedding Models
Dynatrace Research · TU Wien · University of Oxford
Towards Deep Encrypted Training: Low-Latency, Memory-Efficient, and High-Throughput Inference for Privacy-Preserving Neural Networks
Apr 18, 2026Nges Brian Njungle, Eric Jahns, Michel A. KinsyEfficient Neural Network Inference Memory-Efficient Inference
STAM Center, Ira A. Fulton Schools of Engineering · Arizona State University, Tempe, AZ 85281, USA
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
Apr 16, 2026Xinyu Liu, Xin Liu, Bo Jin +8Multi-Token Prediction KV Caching
School of Computer Science and Engineering, Northeastern University, China · Meituan Inc. · NiuTrans Research, Shenyang, China
VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
Feb 11, 2026Muyan Hu, Ahan Gupta, Jiachen Yuan +7Efficient Neural Network Inference Memory-Efficient Inference
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Microsoft
FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge Inference Mixture-of-Experts Inference
NOSA: Native and Offloadable Sparse Attention
Oct 15, 2025Yuxiang Huang, Pengjie Wang, Jicheng Han +9KV-Cache Offloading GPU Acceleration
NLP Group, DCST, IAI, BNRIST, Tsinghua University, Beijing, China. · OpenBMB, China. · BUPT, Beijing, China. +1
Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management
Jun 2, 2025Yimin DuText Embeddings Word Embeddings
Independent Researcher Beijing, China
ReFreeKV: Towards Threshold-Free KV Cache Compression
Feb 24, 2025Xuanfan Ni, Liyan Xu, Chenyang Lyu +6KV Caching Memory-Efficient Optimization
Nanjing University of Aeronautics and Astronautics · WeChat AI, Tencent · Independent Researcher +1
FlexQuant: Elastic Quantization Framework for Locally Hosted LLM on Edge Devices
Jan 13, 2025Yuji Chai, Mujin Kwun, David Brooks +1LLM Quantization On-Device Language Model Inference
Harvard John A. Paulson School Of Engineering And Applied Sciences Cambridge, Massachusetts, USA
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Date pendingQingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Date pendingZhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of Experts Memory-Efficient Optimization
Huawei Technologies · Beihang University · Tianjin University +2

May 29, 2026, cs.CLJ/K move · Enter open · S save

dMoE: dLLMs with Learnable Block Experts

Sicheng Feng, Zigeng Chen, Gongfan Fang, Xinyin Ma+1

National University of Singapore

Mixture-of-Experts Language Models Memory-Efficient Inference Diffusion Language Model Inference Diffusion Language Models

Open paper AskPDF arXiv

Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide expert routing in a more coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% of the original performance. Meanwhile, it reduces memory usage by 76.64% to 79.84% and achieves 1.14

\times

to 1.66

\times

end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE

Momentum

Related topics

Latest papers 101