LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 683

All topics

Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching
Jun 5, 2026Qianli Ma, Zhiqing Tang, Hanshuai Cui +2LLM Serving KV Caching
Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China +1
Skip a Layer or Loop It? Learning Program-of-Layers in LLMs
Jun 4, 2026Ziyue Li, Yang Li, Tianyi ZhouLLM Inference LLM Inference Acceleration
University of Maryland, College Park, MD, USA · 2MBZUAI, Abu Dhabi, UAE.
You Only Index Once: Cross-Layer Sparse Attention with Shared Routing
Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Long-Context Language Model Inference KV-Cache Management
Microsoft Research · Tsinghua University
Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents
Jun 4, 2026Zhuoming Chen, Xinrui Zhong, Qilong Feng +5LLM Serving LLM Inference Acceleration
1Carnegie Mellon University · 2Rice University · 3National University of Singapore
NTILC: Neural Tool Invocation via Learned Compression
Jun 4, 2026Andrew Krikorian, Yayuan Li, Jason J. CorsoLLM Inference Acceleration LLM Tool Use
Department of Robotics University of Michigan Ann Arbor, MI 48104 · Department of ECE University of Michigan Ann Arbor, MI 48104
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
Jun 4, 2026Hyungmin Kim, Minsoo Kim, Hongseok Kim +1LLM Serving KV Caching
When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet
Jun 4, 2026Luoming Zhang, Yuwei Ren, Kui Zhang +7Efficient Transformer Inference LLM Inference Acceleration
Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc.
QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving
Jun 4, 2026Jianxin Yan, Wangze Ni, Zhenxin Li +8Retrieval-Augmented Generation KV Caching
Zhejiang University, Hangzhou, China · Ant Group, Shanghai, China · The Hong Kong Polytechnic University, Hong Kong, China +5
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
Jun 4, 2026PSBC LLM Team, Huawei LLM Team, Ruihan Long +56LLM Compression LLM Inference Acceleration
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
Jun 4, 2026Runheng Liu, Jincheng Xie, Wen Hu +2Speculative Decoding LLM Inference Acceleration
School of Computer Science and Technology, Beijing Institute of Technology · Department of Mathematical Sciences, Tsinghua University · JDT AI Infra
AgentCompile: An LLM-Guided Compiler for Direct CUDA Inference
Jun 4, 2026Xuanzhe Li, Ziyan Weng, Zhiyu Zhu +1Efficient Transformer Inference GPU Acceleration
City University of Hong Kong (Dongguan) · City University of Hong Kong
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Jun 4, 2026Haoze He, Xinkai Zou, Xuan Jiang +4Mixture-of-Experts Pruning LLM Pruning
Carnegie Mellon University · UCSD · MIT
Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models
Jun 3, 2026Rayyan Abdalla, Amir Hussein, Min Wu +1LLM Quantization LLM Inference Acceleration
Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge
Jun 3, 2026Haotian Zheng, Zhanwei Wang, Mingyao Cui +3On-Device Language Model Inference Edge Computing
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China
Cartridges at Scale: Training Modular KV Caches over Large Document Collections
Jun 3, 2026Momchil Hardalov, Gonzalo Iglesias, Adrià de GispertMemory-Augmented Language Models KV Caching
Amazon AGI
SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference
Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2KV-Cache Offloading Long-Context Language Model Inference
1NVIDIA · 2Thinking Machines Lab · 3ByteDance Seed +1
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
TreeFlash: Parallel AR-Approximation for Faster Speculative Decoding
Jun 2, 2026Peer Rheinboldt, Frédéric Berdoz, Roger WattenhoferSpeculative Decoding LLM Inference Acceleration
ETH Zurich
E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments
Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3LLM Serving Edge Computing
Department of Informatics University of Oslo Oslo, Norway · Department of Computer Science UiT The Arctic University of Norway Tromsø, Norway
Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
Jun 2, 2026Mehmet Utku ColakAI Coding Agents LLM Prompting
Department of Computer Engineering, Istanbul Technical University · Istanbul, Turkey
KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
Jun 2, 2026Lorenz K. Muller, Philippe Bich, Chiara Boretti +3LLM Quantization KV Caching
Huawei
Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
Jun 2, 2026Yanyu Zhu, Hoilam Pao, Niu Hu +6RL for Language Models Speculative Decoding
1Tsinghua Shenzhen International Graduate School · 3Huawei Noah’s Ark Lab · 4Northwestern Polytechnical University +1
Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
Jun 2, 2026Runpeng Dai, Tong Zheng, Rui Liu +2Reinforcement Learning Test-Time Scaling
University of North Carolina at Chapel Hill · University of Maryland, College Park · Washington University in St. Louis
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
Jun 1, 2026Chunan Shi, Yilei Chen, Yilin Chen +2KV-Cache Offloading LLM Inference
Peking University Beijing, China
From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression
Jun 1, 2026Elia Cunegatti, Marcus Vukojevic, Erik Nielsen +1LLM Pruning LLM Compression
University of Trento
DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding
Jun 1, 2026Jiebin Zhang, Zhenghan Yu, Song Liu +9Speculative Decoding LLM Inference Acceleration
School of Computer Science, Peking University · Tencent
Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning
Jun 1, 2026Ting Xu, Xu He, Yupu Lu +4CoT Reasoning Test-Time Scaling
The Chinese University of Hong Kong · Huawei Technologies Ltd · School of Computing and Data Science, The University of Hong Kong
Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery
Jun 1, 2026Ekaterina Alimaskina, Darya Rudas, Denis Shveykin +3LLM Quantization Efficient Language Model Reasoning
BRAIn Lab, Moscow, Russia
HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
Jun 1, 2026Minghui Zheng, Hongxu Chen, Huimin Ren +8LLM Compression RL for Language Model Reasoning
Li Auto Inc.

Jun 5, 2026, cs.LGJ/K move · Enter open · S save

Semantic Cache Distillation: Efficient State Transfer via Reuse and Selective Patching

Qianli Ma, Zhiqing Tang, Hanshuai Cui, Zhi Yao+1

Faculty of Arts and Science, Beijing Normal University, Zhuhai 519087, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai 519087, Guangdong, China · School of Artificial Intelligence, Beijing Normal University, Beijing 100875, China+1

LLM Serving KV Caching Disaggregated LLM Serving KV-Cache Management LLM Inference Acceleration KV-Cache Compression

Open paper AskPDF arXiv

Disaggregated serving alleviates memory bottlenecks in Large Language Model (LLM) inference but creates a severe communication bottleneck: transmitting high-dimensional Key-Value (KV) caches often dominates time-to-first-token (TTFT). Moreover, reusing caches across heterogeneous models (e.g., base and fine-tuned variants) causes semantic misalignment that accumulates over layers, degrading generation quality. We propose Semantic Cache Distillation (SCD), a loss-constrained framework that replaces raw KV transmission with compact semantic codes. SCD addresses these challenges via two mechanisms: (1) Reuse, which reconstructs most layers from low-rank subspaces to minimize transfer cost, and (2) Patch, which predicts normalized inputs at sparse transition layers to truncate error propagation. Empirically, SCD delivers up to 2.65

\times

TTFT speedup over the oracle consumer prefill and dominates quantization and selective recomputation baselines on the quality--latency Pareto frontier in bandwidth-constrained regimes, while keeping generation quality within 5% F1 of the oracle.

Momentum

Related topics

Latest papers 683