LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 677

All topics
CardsList

  1. LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization

    Aug 9, 2026Zexun Lin, Yuan Feng, Junlin Lv +2Inference-Time OptimizationSpeculative Decoding for Diffusion Language Models

    Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, Jiangsu, China

  2. Efficient Test-Time Scaling for LLM-based Time Series Forecasting

    Aug 9, 2026Xuan-May Le, Minh-Tuan Tran, Ling Luo +3Long-Term Time Series ForecastingTime Series Forecasting

    The University of Melbourne Melbourne, Victoria, Australia · Monash University Melbourne, Victoria, Australia

  3. VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference

    Aug 9, 2026Wenxu Jia, Dongjie Fu, Xize Cheng +6Audio Token CompressionLLM Inference Acceleration

    1Zhejiang University · 2Meituan

  4. RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

    Aug 8, 2026Anthony. Lui, Mohamed. Elsaied, N. P. SavaniExpert OffloadingMixture-of-Experts Language Models

    Cognizant AI & Analytics, London, UB8 3PH, UK · Southern Methodist University (SMU), Engineering Management, Lyle School of Engineering, Dallas, TX, USA · Cognizant AI Lab, Responsible AI Office, San Francisco, USA +1

  5. SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding

    Aug 8, 2026Jiamu Zhang, Liang Wu, Kelly Wan +2Mixed-Precision QuantizationKV Caching

    Nokia · Rice University

  6. Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

    Aug 7, 2026Yehan Yang, Junyuan Shang, Yang Li +3Attention Head AnalysisLLM Inference Acceleration

    Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Baidu Inc. +1

  7. CubicQuant: Parametric Non-Uniform Codebooks for High-Throughput LLM Inference with 1-8-Bit Weights

    Aug 7, 2026Xuetian GaoLLM QuantizationLLM Inference Acceleration

  8. The Sparsity Whisperer

    Aug 6, 2026Linghao Kong, Inimai Subramanian, Micah Adler +3LLM PruningLLM Inference Acceleration

    MIT · IST Austria · IBM +1

  9. PaDoc: Layout-Grounded Parallel Decoding for Document Parsing

    Aug 6, 2026Hao Yu, Jiabo Zhan, Kang Liu +8LLM Inference AccelerationDocument Parsing

    1Tsinghua University · 2Wechat Vision, Tencent

  10. BALANCE: Hybrid Autoregressive-Speculative LLM Inference in Wireless Edge Networks

    Aug 6, 2026Guanqiao Qu, Shuo Chen, Qian Chen +2Edge ComputingLLM Inference Scheduling

    Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China · Electrical and Electronic Engineering Department, Imperial College, London SW7 2BT, U.K. · Electrical and Electronic Engineering, and Computing Departments, Imperial College, London SW7 2BT, U.K.

  11. MACRO: Markov Chain Routing of Transformer Layers

    Aug 6, 2026Paweł Batorski, Abtin Pourhadi, Akylgali Aitaza +2Markov ModelsLLM Inference Acceleration

    Heinrich Heine University Düsseldorf · Jagiellonian University · IDEAS Research Institute

  12. DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding

    Aug 5, 2026Amirmohammad Karimi, Chao Gao, Negar HassanpourSpeculative DecodingLLM Inference Acceleration

    Huawei Technologies Canada Co., Ltd.

  13. SpecRoll: Fast-Slow Verifier-Feedback Adaptation for Speculative Reinforcement Learning Rollouts

    Aug 5, 2026Nhat Minh Pham, Duy Tung Doan, Thi Duyen Ngo +2RL for Language Model ReasoningSpeculative Decoding

    VNU University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam · Viettel AI, Viettel Group, Hanoi, Vietnam

  14. Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

    Aug 5, 2026Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8Overthinking in Language ModelsKV Caching

    1HKUST · 2XJTU · 3TSE +1

  15. Training-Free Hashing-Based Attention via Binary Principal Components

    Aug 5, 2026Daohai Yu, Zhanpeng Zeng, Keyu Chen +6Long-Context Language ModelingLLM Inference Acceleration

    Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China. · Tencent YouTu Lab, Shenzhen, China · Sino-Russian Research Center for Digital Economy.

  16. Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection

    Aug 5, 2026Robin Pan, Raymond Liu, Daniel Fang +2Mixture-of-Experts InferenceLLM Inference Acceleration

    John A. Paulson School of Engineering and Applied Sciences, Harvard University · Department of Mathematics, Harvard University

  17. Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

    Aug 4, 2026Taekyung Heo, Rasoul Shafipour, Ritchie Zhao +6KV CachingLLM Inference Acceleration

    NVIDIA

  18. Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computes

    Aug 4, 2026Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan +2Speculative DecodingLLM Inference Acceleration

    1Japan Advanced Institute of Science and Technology (JAIST) · 2Tohoku University

  19. Approximate Speculative Decoding

    Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative DecodingLLM Inference Acceleration

    School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

  20. PI-Mem: Pushing Long-Context Reasoning to 3.6M Tokens with Parallel-Iterative Memory

    Aug 4, 2026Dawei Liu, Haixu Song, Shuang Cheng +9Memory-Augmented Language ModelsLong-Context QA

    Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Tsinghua University +3

  21. LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

    Aug 4, 2026Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh +1LLM ServingLLM Inference Acceleration

    SWAT Lab., Polytechnique Montr´eal, Montr´eal, Canada

  22. ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference

    Aug 3, 2026Shun-ichiro Hayashi, Daichi Mukunoki, Tetsuya Hoshino +1Long-Context Language Model InferenceLLM Inference Acceleration

    Graduate School of Informatics, Nagoya University · Information Technology Center, Nagoya University

  23. Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

    Aug 3, 2026Anusha Madan Gopal, Aras Pirbadian, Kristofor D. Carlson +2Retrieval-Augmented GenerationOn-Device Language Model Inference

    BrainChip Inc. 23041 Avenida de la Carlota, Laguna Hills, CA

  24. From Chains to Trees: Parent-Conditioned Drafting for Semi-Autoregressive Speculative Decoding

    Aug 3, 2026Zixian Li, Tong Li, Chi Xie +2Speculative DecodingLLM Inference Acceleration

    OPPO AI Center

  25. LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing

    Aug 3, 2026Wen Zan, Jiaqi Zhang, Jianchao Tan +11Long-Context ModelingLLM Inference Acceleration

    Meituan LongCat Team

  26. Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

    Aug 3, 2026Li Wang, Yi Su, Xiabao Wu +9Speculative DecodingLLM Inference Acceleration

    State Key Laboratory for Novel Software Technology, Nanjing University · Ant Group · Shanghai Jiao Tong University +1

  27. Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference

    Aug 2, 2026Ruokai Yin, Priyadarshini PandaGPU Kernel OptimizationLLM Inference

    Yale University New Haven, USA · University of Southern California Los Angeles, USA

  28. AOSpec: Action and Observation Co-Speculation for Low-Latency Agent Serving

    Aug 1, 2026Hao Mark Chen, Jinnan Guo, Wayne Luk +1LLM Inference AccelerationLong-Horizon Agent Tasks

    Imperial College London, UK

  29. OoO-Spec: Out-of-Order Semantic Speculation for Fast Tool Calling

    Aug 1, 2026Zhiheng Zhang, Mujie Xu, Feiyu Sun +1Speculative DecodingLLM Inference Acceleration

    The University of Tokyo · Peking University · Nanjing University

  30. TrimMoE A communication aware and adaptive depth framework for distributed edge inference

    Aug 1, 2026Ning Li, Shuting Bai, Xin Yuan +3Adaptive InferenceEdge Inference

    School of Artificial Intelligence, University of Science and Technology Beijing, China · School of Computer Science and Technology, Harbin Institute of Technology, Heilongjiang, China · School of Ocean Engineering, Harbin Institute of Technology, Heilongjiang, China +2