LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving
May 10, 2026Zhiqing Zhong, Zhijing Ye, Jian Zhang +3KV Caching KV-Cache Management
Stevens Institute of Technology Hoboken, New Jersey, USA · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Argonne National Laboratory Lemont, Illinois, USA +1
Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasTest-Time Adaptation Test-Time Optimization
The University of Texas at Austin
Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Mixture-of-Experts Language Models Recurrent Transformers
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
ProxyKV: Cross-Model Proxy Pruning for Efficient Long-Context LLM Inference
May 9, 2026Junjie Li, Jiong Lou, Jie LiLLM Pruning KV Caching
Shanghai Jiao Tong University
BubbleSpec: Turning Long-Tail Bubbles into Speculative Rollout Drafts for Synchronous Reinforcement Learning
May 9, 2026Yuhang Xu, Kaibin Tian, Yang Tian +6Reinforcement Learning Speculative Decoding
1Shanghai Jiao Tong University · 2Bytedance
ReST-KV: Robust KV Cache Eviction with Layer-wise Output Reconstruction and Spatial-Temporal Smoothing
May 9, 2026Yongqi An, Chang Lu, Kuan Zhu +5KV Caching Long-Context Language Model Inference
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · University of Electronic Science and Technology of China, Chengdu, China +2
LAQuant: A Simple Overhead-free Large Reasoning Model Quantization by Layer-wise Lookahead Loss
May 9, 2026Euntae Choi, Sumin Song, Sungjoo YooLLM Quantization Quantization-Aware Training
Seoul National University
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
May 9, 2026Benjamin L. BadgerRecurrent Neural Networks Autoregressive Language Modeling
IBM
PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding
May 9, 2026Zihao An, Taichi Liu, Ziqiong Liu +3Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc. · Rutgers University · Michigan State University
PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
May 9, 2026Xingyu Qu, Tianhao Lin, Yiqi Li +2LLM Serving KV Caching
School of Computer Science, Wuhan University · Amazon
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression
May 9, 2026Hengyi Zhu, Zhendong Mi, Grace Li Zhang +1LLM Compression Low-Rank Matrix Decomposition
1Stevens Institute of Technology, Hoboken, USA · 2Technical University of Darmstadt, Darmstadt, Germany
Fast Byte Latent Transformer
May 8, 2026Julie Kallini, Artidoro Pagnoni, Tomasz Limisiewicz +5Speculative Decoding LLM Inference Acceleration
FAIR at Meta · Stanford University · University of Washington
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
May 8, 2026Wenhao Wu, Zishan Shao, Kangning Cui +5Efficient Transformer Inference LLM Compression
Department of Electrical & Computer Engineering, Duke University · Department of Statistical Science, Duke University · Department of Computer Science, Wake Forest University +1
An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inference
May 8, 2026Feiyu Yao, Zhixiong Niu, Xiaqing Li +3GPU Acceleration Self-Attention
Beijing University of Technology Beijing, China · Microsoft Research Beijing, China · Beijing Jiaotong University Beijing, China
Priming: Hybrid State Space Models From Pre-trained Transformers
May 8, 2026Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul +6Long-Context Language Model Inference LLM Inference Acceleration
AWS Agentic AI
MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
May 8, 2026Ruijie Zhou, Fanxu Meng, Yufei Xu +4Self-Attention Long-Context Language Model Inference
CSR: Infinite-Horizon Real-Time Policies with Massive Cached State Representations
May 8, 2026Robin Karlsson, Go SuzuiRobotic Control Language Model-Based Control
1GODOT Inc, Hyogo, Japan · 2Graduate School of Informatics, Nagoya University, Aichi, Japan.
LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification
May 8, 2026Xuan Li, Yining Wang, Yuchen Liu +7CoT Reasoning Efficient Language Model Reasoning
University of Science and Technology of China · Unisound AI Technology Co., Ltd · MAIS, Institute of Automation, Chinese Academy of Sciences
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
May 8, 2026Weijie Shi, Qiang Xu, Fan Deng +9Speculative Decoding LLM Inference Acceleration
Hong Kong University of Science and Technology · MetaX · Zhejiang Normal University +1
Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
May 7, 2026Mohsen Dehghankar, Abolfazl AsudehSelf-Attention KV Caching
Department of Computer Science University of Illinois Chicago
Grokers: Bottom-Up Inductive Comprehension and Write-Time Intelligence over Typed Knowledge Graphs
May 7, 2026Gregory MagarshakKG Construction Memory-Augmented Language Models
Qbix, Inc. & Intercoin, Inc. · IE University NYC · New York, NY, USA
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
May 7, 2026Nii Osae Osae Dade, Tony Morri, Moinul Hossain Rahat +2LLM Quantization Efficient Neural Network Inference
Mindbeam AI
LatentRAG: Latent Reasoning and Retrieval for Efficient Agentic RAG
May 7, 2026Yijia Zheng, Marcel WorringLLM Inference Acceleration Agentic RAG
University of Amsterdam, Amsterdam, the Netherlands
UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
May 7, 2026Qihang Fan, Huaibo Huang, Zhiying Wu +2Long-Context Language Model Inference LLM Inference Acceleration
1MAIS&NLPR, CASIA · 2UCAS · 3WeChat, Tencent
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Expert Parallelism Mixture-of-Experts Inference
Stanford University
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
May 7, 2026Jungsuk Oh, Hyeseo Jeon, Hyunjune Ji +2Self-Attention Long-Context Language Model Inference
Graduate School of Data Science Seoul National University Seoul, Republic of Korea
Requests of a Feather Must Flock Together: Batch Size vs. Prefix Homogeneity in LLM Inference
May 7, 2026Saksham Rathi, Preeti, Mythili VutukuruLLM Inference Scheduling KV-Cache Management
Indian Institute of Technology Bombay
HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devices
May 7, 2026Shen Xu, Xiangwen Zhuge, Zhe Xu +3LLM Compression Memory-Efficient Inference
Tsinghua University · Huazhong University of Science and Technology
When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
May 7, 2026Mohamed Amine BergachGPU Kernel Optimization KV Caching
Illumina, San Diego, CA, USA

May 10, 2026, cs.ARJ/K move · Enter open · S save

KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving

Zhiqing Zhong, Zhijing Ye, Jian Zhang, Weijian Zheng+2

Stevens Institute of Technology Hoboken, New Jersey, USA · The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, Guangdong, China · Argonne National Laboratory Lemont, Illinois, USA+1

KV Caching KV-Cache Management LLM Inference Serving LLM Inference Acceleration

Open paper AskPDF arXiv

Static-graph LLM decoders provide predictable launches, fixed tensor shapes, and low submission overhead, but online decoding exposes highly irregular KV-cache behavior: request lengths differ, EOS events arrive asynchronously, and logical histories fragment over time. Dynamic runtimes recover flexibility through paged KV management and step-level scheduling, while static-graph executors often over-reserve memory and suffer burst-time latency outliers. This paper studies whether much of this variability can be absorbed below a fixed decode interface. We present KV-RM, a runtime design that regularizes KV-cache movement beneath a static-graph LLM decoder. KV-RM decouples logical KV histories from physical storage, tracks active KV state through a block pager, and materializes each decode step through a single committed descriptor. A merge-staged transport path coalesces non-contiguous KV mappings into a small number of large transfer groups before a fixed-shape attention kernel consumes them. Optional bounded far-history summaries can be enabled under the same interface, but the core design does not depend on them. On a 2-GPU NVIDIA A100 node, KV-RM improves mixed-length decoding throughput and tail latency relative to a static-graph baseline, reduces reserved KV memory across workload families, and removes severe burst-time latency spikes under production-trace replay. These results suggest that KV-cache movement, rather than kernel shape, can be an effective boundary for recovering runtime flexibility in static-graph LLM serving.

Momentum

Related topics

Latest papers 677