LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 683

All topics
CardsList

  1. Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents

    Jun 11, 2026Saehun Chun, Wonje Choi, Sera Choi +2LLM-Based Program SynthesisAgentic Code Generation

    Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea.

  2. MARS: Margin-Adversarial Risk-controlled Stopping for Parallel LLM Test-time Scaling

    Jun 11, 2026Wenbo Chen, Puheng Li, Mengyang Liu +2LLM Inference EfficiencySelf-Consistency Decoding

    Amazon · Stanford University · University of Pennsylvania

  3. Doc-to-Atom: Learning to Compile and Compose Memory Atoms

    Jun 10, 2026Xingjian Diao, Wenbo Li, Yashas Malur Saidutta +3Memory-Augmented Language ModelsLong-Context Language Model Inference

    AI Center-Mountain View, Samsung Electronics · Dartmouth College

  4. Breaking Entropy Bounds: Accelerating RL Training via MTP with Rejection Sampling

    Jun 10, 2026Yucheng Li, Huiqiang Jiang, Yang Xu +14Multi-Token PredictionSpeculative Decoding

    Qwen Team, Alibaba Inc.

  5. VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

    Jun 10, 2026Yuchen Xian, Yang He, Yunqiu Xu +1Speculative DecodingLLM Inference Acceleration

    ReLER, The State Key Lab of Brain Machine Intelligence, Zhejiang University · College of Artificial Intelligence, Zhejiang University · CFAR, Agency for Science, Technology and Research, Singapore +2

  6. Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoning

    Jun 10, 2026Kai Liu, Peijie Dong, Xinchen Xie +5RL for Language Model ReasoningLLM Inference Acceleration

    Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Hong Kong University of Science and Technology (Guangzhou) · Shanghai AI Laboratory

  7. Teaching Diffusion to Speculate Left-to-Right

    Jun 10, 2026Lexington Whalen, Yuki Ito, Ryo SakamotoSpeculative Decoding for Diffusion Language ModelsBlockwise Diffusion Language Models

    SB Intuitions, Tokyo, Japan

  8. ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models

    Jun 9, 2026Wenhao Liu, Hao Shi, Yunhe Li +7KV CachingMemory-Efficient Optimization

    1Tsinghua University · 2City University of Hong Kong · 3Peking University +2

  9. CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

    Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouMulti-Token PredictionLanguage Model Decoding

  10. K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling

    Jun 9, 2026Zhiwei Tang, Yuanyu He, Yizheng Han +4LLM ServingLanguage Model Decoding

    1DAMO Academy, Alibaba Group · 2Hupan Lab · 3Zhejiang University +1

  11. SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

    Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariLLM PruningHigh-Performance Computing

    Snowflake AI Research* · Seoul National University†

  12. Atomic Intent Reasoning: Bringing LLM Semantics to Industrial Cross-Domain Recommendations

    Jun 9, 2026Zhuohang Jiang, Yuxin Chen, Shijie Wang +6LLM Inference AccelerationUser Modeling

    The Hong Kong Polytechnic University Hong Kong SAR, China · Kuaishou Technology Beijing, China

  13. Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite

    Jun 9, 2026Zhiyuan Cheng, Longying LaiRetrieval-Augmented GenerationAI Accelerator Inference

    School of Engineering Stanford University Stanford, CA, USA · Simon Business School University of Rochester Rochester, NY, USA

  14. AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis

    Jun 8, 2026Jaber Jaber, Osama JaberGPU Kernel OptimizationHigh-Performance Computing

    RightNow AI

  15. FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing

    Jun 8, 2026Yuhan Ma, Yong Li, Stefan SchmidSecure Multi-Party ComputationPrivacy-Preserving Language Models

    Huawei Heisenberg Research Center, Huawei Technologies Düsseldorf GmbH, Düsseldorf, Germany · Technische Univer-sität Berlin, Berlin, Germany

  16. BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference

    Jun 8, 2026Yuhua Zhou, Shaoqi Yu, Shichao Weng +4LLM PruningCost-Aware Inference

    College of Computer Science and Technology, Zhejiang University · Zhejiang Lab · Ant Group +1

  17. From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

    Jun 8, 2026Zhanchao Xu, Haoyang Li, Qingfa Xiao +4Long-Context Language Model InferenceLLM Inference Acceleration

    Department of Computing, PolyU · DSA, HKUST(GZ) · CSE, HKUST

  18. SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

    Jun 8, 2026Rya Sanovar, Srikant Bharadwaj, Hritvik Taneja +1Retrieval-Augmented GenerationLLM Inference Acceleration

    Georgia Institute of Technology Atlanta, GA, USA · Microsoft Redmond, WA, USA

  19. Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

    Jun 8, 2026Haozhe Hu, Hao Wu, Anhao Zhao +4LLM PruningLLM Inference Acceleration

    Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University · Munich Center for Machine Learning, LMU Munich

  20. RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference

    Jun 7, 2026Anirudh SekarLLM Inference EfficiencyKV Caching

    Accepted to the ICML 2026 Workshop on Statistical Frameworks for Uncertainty in Agentic Systems, Seoul, South Korea, 2026.

  21. APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

    Jun 7, 2026Hong Guo, Nianhui Guo, Weixing Wang +3LLM QuantizationGPU Kernel Optimization

    Hasso Plattner Institute · German University of Digital Science

  22. Sample Where You Struggle: Sharpening Base Model Reasoning via Entropy-Guided Power Sampling

    Jun 7, 2026Hong Guo, Nianhui Guo, Christoph Meinel +1Markov Chain Monte CarloInference-Time Scaling

    Hasso Plattner Institut · German University of Digital Science

  23. SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving

    Jun 7, 2026Yang PengjuMixed-Precision QuantizationKV Caching

  24. STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control

    Jun 7, 2026Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang +3Mixed-Precision QuantizationKV Caching

    University of California San Diego, San Diego, CA, USA · Dnotitia Inc., Seoul, Republic of Korea · Hanyang University, Seoul, Republic of Korea

  25. IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

    Jun 6, 2026Junjie Li, Jiong Lou, Jie LiKV CachingKV-Cache Management

    Shanghai Jiao Tong University

  26. Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method

    Jun 6, 2026Kyoungmin Kim, Martin Catheland, Anastasia AilamakiSemantic SearchLLM Inference Acceleration

    EPFL Lausanne, Switzerland

  27. Still: Amortized KV Cache Compaction in a Single Forward Pass

    Jun 5, 2026Charles O'Neill, Alex Sandomirsky, Harry Partridge +2KV CachingLong-Context Language Model Inference

    Baseten San Francisco · Baseten London

  28. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

    Jun 5, 2026Young D. Kwon, Miles Williams, Rui Li +2Speculative DecodingLLM Inference Acceleration

    Samsung AI Center, Cambridge, UK

  29. SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

    Jun 5, 2026Ernests Lavrinovics, Marco Letizia, Roy Janco +3Model CompressionLLM Compression

    Department of Computer Science, Aalborg University Copenhagen, Denmark · MaLGa-DIBRIS, University of Genoa, Genoa, Italy · INFN, Sezione di Genova, Genoa, Italy +2

  30. How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models

    Jun 5, 2026Hongxing Wang, Harenome Razanajato, Zhen Zhang +2Grouped-Query AttentionLong-Context Language Modeling