LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 677

All topics
CardsList

  1. CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding

    Aug 1, 2026Aofan Liu, Jingxiang Meng, Fangxin Liu +1Speculative DecodingLLM Inference Acceleration

  2. AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

    Aug 1, 2026Ziqiang Cui, Han Shi, Bowei He +8Multi-Token PredictionLLM Inference Acceleration

    City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2

  3. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    Jul 31, 2026Zhenyu Zhang, Zhichao CaoLLM ServingLLM Inference Acceleration

    Arizona State University · Tempe, Arizona, USA

  4. Decode-Branch Transformers: Decoupling the Primary Prefill Path from Additional Decode Computation

    Jul 31, 2026Liming Liu, Mingze Wang, Tuo ZhaoEfficient Transformer InferenceInference-Time Scaling

    Georgia Institute of Technology · Peking University

  5. Studying quantization trade-offs for efficient inference deployment in machine translation

    Jul 31, 2026Jim Zhao, Sohir Maskey, Koen Oostermeijer +2LLM QuantizationLLM Inference Acceleration

    University of Basel · Aleph Alpha Research · Graphcore

  6. ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition

    Jul 31, 2026Qingjian Lin, Yuxin Li, Haoyang Zhang +14Multi-Token PredictionAutomatic Speech Recognition

    StepFun · NTU · PKU +3

  7. Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

    Jul 31, 2026Jin-woo Lee, Minkyung Song, Junghyun Oh +4Memory-Augmented Language ModelsKV Caching

    Chungnam National University · KISTI

  8. LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs

    Jul 31, 2026Yirui Liu, Ruoling Qi, Longwen Wang +5KV CachingLLM Inference Acceleration

    Institute of Artificial Intelligence, China Telecom (TeleAI) · Shanghai Jiao Tong University · Xi’an Jiaotong University +1

  9. WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

    Jul 30, 2026Haozhe Hu, Hao Wu, Peiran Yin +3LLM PruningToken Pruning

    Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Munich Center for Machine Learning, LMU Munich

  10. CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residuals

    Jul 30, 2026Hanzuo Liu, Xuan Qi, Chunyu Liu +6Memory-Augmented Language ModelsLong-Context Language Model Inference

    Tsinghua University · Tencent

  11. A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

    Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model InferenceSpeculative Decoding

    University of Science and Technology of China

  12. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Jungjun Oh +2LLM QuantizationAI Accelerator Inference

    PIM Semiconductor Design Research Center (AI-PIM), Daejeon 34141, South Korea

  13. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Byeongcheol Kim +2AI Accelerator InferenceRotation-Based Quantization

    School of Electrical Engineering, Korea Advanced Institue of Science and Technology (KAIST)

  14. Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

    Jul 30, 2026Stephen Gould, Anton van den HengelKV CachingKV-Cache Eviction

    Metacognition AI · Australian National University · Adelaide University

  15. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

    Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Activation SparsityStructured Sparsity

    Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Computer Science, Nanjing University of Posts and Telecommunications

  16. DeepResearch Agent System

    Jul 30, 2026Yong Huang, Yulu Huang, for the team CollaborationDeep Research AgentsLLM Inference Acceleration

  17. InferScale: GPU-Native KV Injection for Personalized LLM Serving

    Jul 29, 2026Peter Li, Prashant PandeyLLM ServingMemory-Augmented Language Models

    Northeastern University Boston, Massachusetts, USA

  18. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative DecodingLLM Inference Acceleration

    Institute for Artificial Intelligence, Peking University

  19. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    Jul 29, 2026Tianyu Wang, Yuxuan Zhou, Wenbin Wang +3Speculative DecodingLLM Inference Acceleration

    1Independent Researcher · 2Baidu Inc. · 3Zhejiang University

  20. OneLatent: Latent Reasoning for Efficient Foundation Recommendation Models

    Jul 29, 2026Hao Jiang, Peiru Du, Pengfei Yao +10LLM Inference AccelerationLarge Language Model-Based Recommendation

    Kuaishou Technology, Beijing, China

  21. Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA

    Jul 28, 2026Tom Saliencro, Rohan Desai, Priya Nair +2Parameter-Free Mixture-of-Experts RoutingLLM Inference Acceleration

    University of California, Irvine · University of Washington

  22. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10LLM Inference SchedulingSpeculative Decoding for Diffusion Language Models

    Tencent Inc.

  23. At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

    Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Efficient Neural Network InferenceTransformer Inference

    *IIS, ETH Zürich, Switzerland · ‡Computing Systems Lab, Huawei, Switzerland · †DEI, Universit`a di Bologna, Italy

  24. CoSA: Accelerating Long-Context Inference via Proxy-Kernel Co-Designed Sparse Attention

    Jul 28, 2026Yufei Xue, Lin Niu, Hong Liu +6Long-Context Language Model InferenceLLM Inference Acceleration

    1Tencent · 2The Hong Kong University of Science and Technology

  25. PIVOT: Efficient Query-Group Indexing for Token-Level Sparse Attention

    Jul 27, 2026Hong Liu, Yuan Cheng, Lin Niu +5LLM Inference AccelerationEfficient Attention

    Tencent

  26. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    Jul 27, 2026Junsung HwangLow-Rank AttentionKV Caching

  27. DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference

    Jul 27, 2026Dengke HanExpert OffloadingOn-Device Language Model Inference

  28. KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems

    Jul 27, 2026Shuo Wang, Fang Xi, Wenyuan Huang +2LLM ServingKV-Cache Management

    QiYuanLab Beijing, China

  29. A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever

    Jul 26, 2026Sietse SchelpePersistent Memory for Language ModelsLLM Memory

    Corbenic AI

  30. HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding

    Jul 24, 2026Chao Fang, Jun Yin, Man Shi +1KV CachingMemory-Efficient Optimization

    ESAT-MICAS, KU Leuven, Leuven, Belgium · UC Berkley, CA, USA