LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving
May 7, 2026Bole Ma, Jan Eitzinger, Harald KöstlerKV Caching LLM Inference Serving
Erlangen National High Performance Computing Center · Erlangen, Germany
UniVer: A Unified Perspective for Multi-step and Multi-draft Speculative Decoding
May 6, 2026Yepeng Weng, Qiao Hu, Takehisa YairiSpeculative Decoding LLM Inference Acceleration
The University of Tokyo · 2Lenovo AI Technology Center · National Center for Mathematics and Interdisciplinary Sciences (NCMIS), AMSS, CAS
Budgeted LoRA: Distillation as Structured Compute Allocation for Efficient Inference
May 5, 2026Mohammed Sabry, Anya BelzLLM Compression Low-Rank Adaptation
ADAPT Centre, Dublin City University, Ireland
Parallel Prefix Verification for Speculative Generation
May 5, 2026Yuncheng Yao, Yuxuan Xia, Shengjie Wang +1Speculative Decoding LLM Inference Acceleration
Duke University · New York University
AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reuse
May 5, 2026Jie Ou, Jinyu Guo, Shiyao Guo +5Long-Context Language Model Inference KV-Cache Management
School of Information and Software Engineering, University of Electronic Science and Technology of China
Performance Optimization and Comparative Analysis of Generative AI Models on Advanced Accelerators
May 5, 2026Amitash Nanda, Javier Hernandez Nicolau, Madhusudan Gujral +3Mixed-Precision Quantization AI Accelerator Inference
University of California San Diego, San Diego, CA, USA · San Diego Supercomputer Center, San Diego, CA, USA
SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction
May 4, 2026Sin Yu Bonnie Ho, Arlie Coles, Erik Larsson +3Retrieval-Augmented Generation LLM Inference Acceleration
Microsoft
Gated Subspace Inference for Transformer Acceleration
May 4, 2026Stephen J. ThomasTransformer Inference LLM Inference Acceleration
Department of Computer Science and Engineering, Lehigh University, Bethlehem, PA 18015
SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
May 4, 2026Shikhar ShuklaSpeculative Decoding LLM Inference Acceleration
University of Kentucky, Lexington, KY, USA
The Model Knows, the Decoder Finds: Future Value Guided Particle Power Sampling
May 4, 2026Tu Nguyen, Matthieu Zimmer, Rasul Tutunov +2Language Model Decoding LLM Inference Acceleration
1Huawei Heisenberg Research Center · 2Huawei Noah’s Ark Lab · 3UCL Centre for Artificial Intelligence
Statistically-Lossless Quantization of Large Language Models
May 4, 2026Michael Helcig, Eldar Kurtic, Dan AlistarhLLM Quantization LLM Inference Acceleration
ETH Z̈urich · ISTA
HALO: Hybrid Adaptive Latent Reasoning for Language Models
May 3, 2026Micah ZhangLLM Inference Acceleration Adaptive Computation
Lockheed Martin
Stochastic Sparse Attention for Memory-Bound Inference
May 3, 2026Kyle Lee, Corentin Delacour, Kevin Callahan-Coray +5GPU Acceleration Self-Attention
Department of Electrical and Computer Engineering, University of California, Santa Barbara, CA, USA · Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, MI, USA · Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA, USA
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
May 3, 2026Long Zhao, Qinghe Wang, Jiaan Zhu +5Tensor Parallelism LLM Inference Acceleration
Anhui University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center · University of Science and Technology of China +2
SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
May 3, 2026Yipin Guo, Siddharth JoshiLLM Serving KV Caching
University of Notre Dame IN, USA
KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
May 1, 2026Shen Han, Yuyang Wu, Junpu Yu +1KV Caching Memory-Efficient Inference
Carnegie Mellon University
Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edge
May 1, 2026M. Grailoo, J. Núñez-YáñezLLM Inference Acceleration FPGA Acceleration
∗Linköping University, Sweden · †Universidad Politecnica de Madrid, Spain
BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs
May 1, 2026Zhixiong Zhao, Zukang Xu, Dawei YangLLM Quantization LLM Inference Acceleration
1Houmo AI
Rethinking LLM Ensembling from the Perspective of Mixture Models
May 1, 2026Jiale Fu, Yuchu Jiang, Peijun Wu +3LLM Inference Acceleration LLM Routing
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2
Caracal: Causal Architecture via Spectral Mixing
Apr 30, 2026Bingzheng Gan, Tianyi Zhang, Yusu Li +4Long-Context Language Modeling Autoregressive Language Modeling
Huawei Technologies Co., Ltd., China.
Position-Aware Drafting for Inference Acceleration in LLM-Based Generative List-Wise Recommendation
Apr 30, 2026Jiaju Chen, Chongming Gao, Chenxiao Fan +4Speculative Decoding LLM Inference Acceleration
University of Science and Technology of China, Hefei, China · Independent Researcher
To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing
Apr 30, 2026Wei Cheng, Yongchang Cao, Chen Shen +4Code Generation LLM Inference Acceleration
State Key Laboratory for Novel Software Technology, Nanjing University, China · Tongyi Lab, Alibaba Group, China
Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding
Apr 29, 2026Hayate Iso, Tiyasa Mitra, Sudipta Mondal +15Reinforcement Learning Speculative Decoding
NVIDIA
When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?
Apr 29, 2026Tianyu Liu, Yuhao Shen, Xinyi Hu +8Speculative Decoding LLM Inference Acceleration
1Qwen Applications Business Group of Alibaba · University of Science and Technology of China · 3Zhejiang University
Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens
Apr 29, 2026Zhenyu Zhao, Sander Land, Daniel M. Bikel +1LLM Compression LLM Interpretability
Writer, Inc.
Efficient, VRAM-Constrained xLM Inference on Clients
Apr 29, 2026Aditya Ukarande, Deep Shekhar, Marc Blackstein +1GPU Acceleration Efficient VLM Inference
NVIDIA Corporation, Santa Clara, CA, USA · NVIDIA Graphics Pvt Ltd., Bangalore, India
Breaking the Autoregressive Chain: Hyper-Parallel Decoding for Efficient LLM-Based Attribute Value Extraction
Apr 29, 2026Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani +2LLM Inference Acceleration Parallel Decoding
Amazon.com, Inc. · McGill University · Mila - Quebec AI Institute +1
AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Serving
Apr 28, 2026Zhongkai Yu, Haotian Ye, Chenyang Zhou +9Compute-in-Memory LLM Inference Acceleration
University of California, San Diego · Columbia University · Yonsei University +2
RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
Apr 28, 2026Vyom Sharma, Debajyoti DattaGPU Kernel Optimization GPU Acceleration
Hippocratic AI
Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models
Apr 28, 2026Ajmain Inqiad Alam, Palash Roy, Chanchal K. Roy +2Software Engineering LLM Compression
University of Saskatchewan, Canada

May 7, 2026, cs.DCJ/K move · Enter open · S save

Irminsul: MLA-Native Position-Independent Caching for Agentic LLM Serving

Bole Ma, Jan Eitzinger, Harald Köstler

Erlangen National High Performance Computing Center · Erlangen, Germany

KV Caching LLM Inference Serving LLM Inference Acceleration LLM Agent Serving Multi-Head Latent Attention

Open paper AskPDF arXiv

Agentic LLM workloads put bit-identical tokens at shifted positions every turn, voiding prefix caches at the first byte of divergence. Operators report cache-hit regressions ranging from moderate slowdowns to severe TTFT spikes of 10-16s on unchanged content. Prior position-independent caching systems correct RoPE on the full

d_K

-dimensional key, an architectural cost imposed by GQA, not by caching itself. Multi-Head Latent Attention, deployed at scale in DeepSeek-V2/V3/R1, Kimi-K2/Moonlight, GLM-5, and Mistral Large 3, factors each KV row into a position-free

c_{KV}

and a 64-dim

k_r

correctable in closed form; this structure motivates content-addressed caching as a natural fit rather than a GQA workaround. We present Irminsul, which extends SGLang's radix cache with content-hash keying over CDC-chunked segments and a

δ

-rotation rule for

k_r

. We evaluate three native MLA-MoE deployments - DeepSeek-V2-Lite (16B/2.4B), Kimi Moonlight-16B-A3B, and JoyAI-Flash (48B/3B) - with output-consistency on all three and recovery measured on the two endpoints; Irminsul recovers up to ~83% of prompt tokens above exact-prefix on agentic traffic while delivering 63% prefill energy savings per cache hit. We argue that content-addressed caching belongs in the serving stack as a first-class primitive, not a retrofit over prefix matching.

Momentum

Related topics

Latest papers 677