LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
Oct 8, 2026Sreetama Sarkar, Saptarshi Mitra, Sitao Huang +2KV-Cache Reuse LLM Inference Acceleration
University of Southern California · University of California, Irvine · Intel Labs
Training Parallel Speculative Draft Models by Directly Minimizing Expected Decoding Rounds
Oct 7, 2026Yunxiao Zhao, Changxiao CaiSpeculative Decoding LLM Inference Acceleration
Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, USA.
Cache the Encoder Within:Compact, Reusable Memory across LLM Queries
Oct 7, 2026Hanzuo Liu, Chunyu Liu, Chaofan Lin +2Persistent Memory for Language Models Memory-Augmented Language Models
Tsinghua University
Evaluating Trajectory Features for Routing Final-Layer Attention
Oct 7, 2026Yupeng YaoLLM Routing LLM Inference Acceleration
Breaking the Space Barrier and its Application to Language Model Inference
Oct 6, 2026Arip AsadulaevConstrained Decoding Efficient Language Model Inference
MBZUAI
TAP: Efficient Long-Horizon Agent Pruning via Trajectory-Anchored Recovery
Oct 6, 2026Yuanzhe Li, Pengxin Wang, Yuxin Ren +5LLM Pruning LLM Inference Acceleration
University of Arizona · University of Pittsburgh · University of Central Florida
SPIN: Shadow Predictive Indexer for Sparse Attention
Oct 6, 2026Yao Fu, Jiahan Chang, Ritchie Zhao +11LLM Inference Acceleration Sparse Attention
Enabling Dynamic Computation in Looped LMs
Oct 6, 2026Aayush Mishra, Arnau Padrés Masdemont, Victor Conchello Vendrell +3LLM Inference Acceleration Adaptive Computation
Qualcomm AI Research · Johns Hopkins University
UNREAL: Unifying Retrieval and Long-Context with a Single Model
Oct 6, 2026Edan Kinderman, Elad Hoffer, Yochai Blau +4Retrieval-Augmented Generation Knowledge Augmentation for Language Models
NVIDIA · Technion
SSR: Sparse Segment Reduction for Ternary GEMM Acceleration
Oct 6, 2026Adeline Pittet, Shien Zhu, Valérie Verdan +1Ternary Quantization LLM Inference Acceleration
Systems Group, ETH Zurich, Switzerland
Hybrid Latent Attention for Looped Language Models
Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model Inference LLM Inference Acceleration
Virginia Tech · Independent Researcher
Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
Oct 6, 2026Shuhao Li, Fanghua Ye, Wanyu Lin +2Speculative Decoding LLM Inference Acceleration
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher
APEX: Speculate smarter, not deeper
Oct 6, 2026Manvi Jha, Zach Zhang, Zhichao Xu +3Speculative Decoding LLM Inference Acceleration
University of Illinois Urbana-Champaign AWS AI · AWS AI
TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models
Oct 6, 2026Xin Wang, Hao Yu, Zhengyang Zhuge +9LLM Quantization Quantization-Aware Training
Alibaba Token Hub, Alibaba Group · Ohio State University
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation
Oct 5, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4Mixed-Precision Quantization LLM Quantization
LLaVi Lab, Computer Science & Engineering, University of North Texas
Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model Training Recurrent Transformers
Institute of Foundation Models · CMU · USC
OVAL: Output-Aware Local Page Bases for KV Cache Retrieval
Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriLong-Context Language Model Inference LLM Inference Acceleration
College of Connected Computing Vanderbilt University Nashville, TN, USA
Behavior-Preserving KV Cache Compression
Oct 5, 2026Doo Hwan Hwang, Junyoung Jang, Junho Na +2KV-Cache Eviction LLM Inference Acceleration
Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea
SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding
Oct 5, 2026Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan +1Speculative Decoding LLM Inference Acceleration
School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China
Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents
Oct 5, 2026Tiffany Gu, Annie Guan, Manshu Huang +3KV-Cache Management LLM Inference Acceleration
Capital One · University of Washington
SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling
Oct 4, 2026Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1Long-Context Language Model Inference Speculative Decoding
UNIST
More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
Oct 3, 2026Noam Elata, Itay Lamprecht, Mikey Shechter +3LLM Inference Acceleration Sparse Attention
Technion – Haifa, Israel · Intel, Israel · Corma +1
TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference
Oct 1, 2026Mukund Agarwalla, Chih-Jen LinActivation Sparsity LLM Inference Acceleration
MBZUAI · National Taiwan University
DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair
Oct 1, 2026Zhuoyu Wang, Junnan Huang, Xinyu ChenSpeculative Decoding LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou)
HHR: Hierarchical Hash Retrieval for Efficient LLM Generation
Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5Long-Context Language Model Inference LLM Inference Acceleration
Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China
Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
Oct 1, 2026Prachi Badarayani, Aidan Jay, Chenghui Zhou +8Multi-Token Prediction Speculative Decoding
Microsoft
AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing
Sep 30, 2026Zhihui Gao, Tingjun Chen, Dirk EnglundOn-Device Language Model Inference LLM Inference Acceleration
Massachusetts Institute of Technology Cambridge, MA, USA · Duke University Durham, NC, USA
XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization
Sep 30, 2026Xiaofan Que, Nir Elkayam, Spandan Pyakurel +2LLM Quantization LLM Inference Acceleration
Arm, Inc. · Rochester Institute of Technology
Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs
Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert Parallelism Mixture-of-Experts Inference
Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation

Sreetama Sarkar, Saptarshi Mitra, Sitao Huang, Souvik Kundu+1

University of Southern California · University of California, Irvine · Intel Labs

KV-Cache Reuse LLM Inference Acceleration

Open paper AskPDF arXiv

Cross-model KV-cache reuse remains a key challenge in modern LLM serving. Coding agents and multi-model systems increasingly route a shared context across models: a user may switch models mid-session, or a cascade may escalate a difficult query. Because KV caches contain model-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch. Recent work shows that closed-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model-size gap widens. In this paper, we establish that these transfer failures are concentrated in a small subset of information-dense tokens. To bridge this gap, we introduce RaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation. RaReCache identifies these critical positions using a novel rank disagreement metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data. Across two model families and five benchmarks, on a 23x parameter gap (Qwen3-0.6B to 14B) recomputing just 30% of positions retains 95-99% of the target accuracy, whereas on a 8.8x gap (Llama3-8B to 70B), recomputing 40% retains 96.5% of the target accuracy. RaReCache largely removes sensitivity to source-model size, and achieves up to a 3.04x prefill speedup. For online serving, it handles 1.8x the request throughput of target prefill on a single GPU, and at the target's saturation load, reduces median and 99th-percentile time-to-first-token (TTFT) by 5.0x and 6.4x respectively, with a 30% recompute budget. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency.

Momentum

Related topics

Latest papers 677