LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference
Jun 18, 2026Kaizhen Tan, Rong Gu, Mingyuan LiLLM Inference Efficiency Retrieval-Augmented Generation
Heinz College of Information Systems and Public Policy, Carnegie Mellon University, USA
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Jun 17, 2026Minseo Kim, Minjae Lee, Seunghyuk Oh +7RL for Language Models Reinforcement Learning
FuriosaAI · University of California, Berkeley
Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applications
Jun 16, 2026Paresh Dashore, Shreyas Kulkarni, Uttam Gurram +5Multi-Agent LLM Systems Multi-Agent System Optimization
AI Foundations, Capital One
JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Jun 16, 2026Lanxiang Hu, Zhaoxiang Feng, Yulun Wu +9Speculative Decoding LLM Inference Acceleration
UC San Diego · Zhejiang University · UIUC +2
ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation
Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6LLM Inference Acceleration Sparse Attention
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · Baidu Inc.
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1
MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
Jun 16, 2026Kosmas Alexandridis, Giorgos DimitrakopoulosAI Accelerator Inference Inference Acceleration
Integrated Circuits Lab, Electrical and Computer Engineering, Democritus University of Thrace (DUTH), Greece
Communication-Efficient Verifiable Attention for LLM Inference
Jun 15, 2026Ziqun Chen, Ming Wu, Michael Heinrich +4LLM Inference Trusted Execution Environments
Nanyang Technological University · Singapore · Zero Gravity Labs +1
SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions
Jun 15, 2026Feiyang Chen, Haibo ChenLLM Inference Efficiency LLM Inference Acceleration
IPADS, Shanghai Jiao Tong University
QK-Normed MLA: QK normalization without full key caching
Jun 15, 2026Yizhou Han, Yao Zhao, Jun Zhou +2KV Caching Transformer Attention
School of Data Science, The Chinese University of Hong Kong, Shenzhen.
Latent Thought Flow: Efficient Latent Reasoning in Large Language Models
Jun 15, 2026Xiandong Zou, Jing Huang, Jianshu Li +1RL for Language Model Reasoning Efficient Language Model Reasoning
Singapore Management University · Ant Group
Models Take Notes at Prefill: KV Cache Can Be Editable and Composable
Jun 14, 2026Bojie LiKV Caching KV-Cache Management
Pine AI
MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization
Jun 14, 2026Yangjia Hu, Haodong Wang, Zicong Hong +8LLM Quantization 4-Bit Quantization
HKUST · EPFL · MetaX Integrated Circuits Co., Ltd
A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference
Jun 13, 2026Yingnan Zhao, Razvan Bunescu, Ahmed Louri +2AI Accelerator Inference Mixture-of-Experts Inference
George Washington University Washington, DC, USA · University of North Carolina at Charlotte Charlotte, NC, USA · Ohio University Athens, OH, USA
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
Jun 13, 2026Tao Lu, Haoyu Wang, Zonghui Wang +3LLM Pruning GPU Kernel Optimization
National University of Singapore · Zhejiang University
CONCORD: Asynchronous Sparse Aggregation for Device-Cloud RAG under Document Isolation
Jun 13, 2026Xuedong Hu, Zhiqing Tang, Zhi Yao +2Retrieval-Augmented Generation Privacy-Preserving Language Models
Faculty of Arts and Sciences, Beijing Normal University, Zhuhai, China · Institute of Artificial Intelligence and Future Networks, Beijing Normal University, Zhuhai, China · School of Artificial Intelligence, Beijing Normal University, Beijing, China +3
Director: Accelerating Distributed MoE Serving via Online Proactive Expert Placement
Jun 13, 2026Qianli Liu, Kaibin Guo, Zicong Hong +5Expert Parallelism LLM Serving
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · School of Software Engineering, Sun Yat-Sen University, China · School of Cyber Science and Engineering, Xi’an Jiaotong University, China +1
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Jun 12, 2026NVIDIA, :, Aaron Blakeman +571Mixture-of-Experts Language Models Long-Context Language Model Inference
Allan
Towards Direct Latent-Space Synthesis for Parallel Branches in LLM-Agent Workflows
Jun 12, 2026Shikun Liu, Mufei Li, Dongqi Fu +5Multi-Agent LLM Systems KV-Cache Management
1Georgia Institute of Technology · 2Meta
SuperThoughts: Reasoning Tokens in Superposition
Jun 11, 2026Zheyang Xiong, Shivam Garg, Max Yu +4Multi-Token Prediction LLM Inference Acceleration
mMicrosoft Research · iIndependent · pPrinceton University +2
Beyond Uniform Tokens: Adaptive Compression for Time Series Language Models
Jun 11, 2026Jialin Gan, Xin Qiu, Guangzhe Chen +1LLM Inference Acceleration Multimodal Token Compression
Zhejiang University · Harbin Institute of Technology · Shandong University
MiniMax Sparse Attention
Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14GPU Kernel Optimization Long-Context Language Modeling
MiniMax · Peking University · NVIDIA +4
ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling
Jun 11, 2026Sihwa Lee, Janghwan Lee, Donghoon Yoo +4LLM Quantization Language Model Decoding
Hanyang University, Seoul, Republic of Korea · Xenoscube Korean Inc., Seoul, Republic of Korea
MiniPIC: Flexible Position-Independent Caching in <100LOC
Jun 11, 2026Nathan Ordonez, Thomas ParnellLLM Inference Efficiency KV Caching
IBM Research · IBM Research Zurich, Switzerland
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
Jun 11, 2026Saehun Chun, Wonje Choi, Sera Choi +2LLM-Based Program Synthesis Agentic Code Generation
Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea.
MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling
Jun 11, 2026Wenbo Chen, Puheng Li, Mengyang Liu +2LLM Inference Efficiency Self-Consistency Decoding
Amazon · Stanford University · University of Pennsylvania
Doc-to-Atom: Learning to Compile and Compose Memory Atoms
Jun 10, 2026Xingjian Diao, Wenbo Li, Yashas Malur Saidutta +3Memory-Augmented Language Models Long-Context Language Model Inference
AI Center-Mountain View, Samsung Electronics · Dartmouth College
Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling
Jun 10, 2026Yucheng Li, Huiqiang Jiang, Yang Xu +14Multi-Token Prediction Speculative Decoding
Qwen Team, Alibaba Inc.
VIA-SD: Verification via Intra-Model Routing for Speculative Decoding
Jun 10, 2026Yuchen Xian, Yang He, Yunqiu Xu +1Speculative Decoding LLM Inference Acceleration
ReLER, The State Key Lab of Brain Machine Intelligence, Zhejiang University · College of Artificial Intelligence, Zhejiang University · CFAR, Agency for Science, Technology and Research, Singapore +2
Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning
Jun 10, 2026Kai Liu, Peijie Dong, Xinchen Xie +5RL for Language Model Reasoning LLM Inference Acceleration
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Hong Kong University of Science and Technology (Guangzhou) · Shanghai AI Laboratory

Jun 18, 2026, cs.CLJ/K move · Enter open · S save

CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference

Kaizhen Tan, Rong Gu, Mingyuan Li

Heinz College of Information Systems and Public Policy, Carnegie Mellon University, USA

LLM Inference Efficiency Retrieval-Augmented Generation LLM Inference Acceleration Prefix Caching Evidence-Grounded Generation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method keeps a prefix tree over recently served evidence sequences and uses a greedy walk to place the most reusable prefix first, while leaving the serving engine and retrieved evidence set unchanged. Across three vLLM configurations, the method lowers median time-to-first-token (TTFT) by about 20-33 percent relative to retrieval-order prefix caching, without hurting answer quality in our QA tests. The greedy policy reaches 97.5 percent of the median TTFT gain from oracle ordering, indicating that most reusable prefix locality can be recovered by a simple scheduling layer between retrieval and inference.

Momentum

Related topics

Latest papers 677