LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 677

All topics
CardsList

  1. RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

    Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3KV CachingLong-Context Language Model Inference

    School of Information, Renmin University of China, Beijing, China · Peking University, Beijing, China

  2. Hierarchical Global Attention (HGA)

    Jun 29, 2026Woernle Frank, Fedosov Vladimir, Grinenko ArtemiyKV-Cache OffloadingLong-Context Modeling

    BMW Group

  3. Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

    Jun 29, 2026Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4Long-Context Language Model InferenceLLM Inference Acceleration

    University of Pittsburgh, Pittsburgh, PA, USA · HPE Labs, Milpitas, CA, USA

  4. When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

    Jun 29, 2026Aaryam SharmaSpeculative DecodingLLM Inference Acceleration

    Independent Researcher Waterloo, Ontario, Canada

  5. MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers

    Jun 29, 2026Linrui Ma, Chun Hei Lo, Xinyu Wang +12In-Context RetrievalLong-Context Language Modeling

    Huawei Canada · McGill University · Université de Montréal +1

  6. Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path

    Jun 28, 2026Victor NorgrenLLM Inference ServingLLM Inference Acceleration

    LayerScale, Inc.

  7. Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

    Jun 28, 2026Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi +1Long-Context Language Model InferenceKV-Cache Eviction

    RBC Borealis

  8. Depth Exploration for LLM Decoding

    Jun 28, 2026Weisi Yang, Zipeng Sun, Stephen XiaLanguage Model DecodingLLM Inference Acceleration

    Northwestern University · MILA – Quebec AI Institute · McGill University

  9. HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

    Jun 27, 2026Yuxuan Yang, Feiyang Ren, Bowen Zeng +4KV CachingLong-Context Language Model Inference

    The State Key Laboratory of Blockchain and Data Security, Zhejiang University

  10. FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models

    Jun 26, 2026Fan Mo, Yuxuan Han, Geng Zhang +2Mixture-of-Experts PruningLLM Pruning

    National University of Singapore

  11. End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

    Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM InferenceStructured Sparsity

    University of North Carolina at Chapel Hill · Meta AI

  12. EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

    Jun 25, 2026Carrie ChenMulti-Token PredictionLLM Inference Scheduling

    Cornell University

  13. HyperDFlash: Hyper-Connection-Aligned Block Speculative Decoding with Gated Residual Reduction

    Jun 25, 2026Luxi Lin, Shuang Peng, Rui Ma +7Hyper-ConnectionsSpeculative Decoding

    ByteDance

  14. PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs

    Jun 25, 2026Muhammad AhmedLLM ServingGPU Kernel Optimization

  15. CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

    Jun 25, 2026Shigeng Wang, Chao Li, Yangyuxuan Kang +2LLM QuantizationTernary Quantization

    1Intel Labs China

  16. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    Jun 25, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +44-Bit QuantizationLLM Inference Acceleration

    School of Computer Science and Technology, Tianjin University

  17. ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Driving

    Jun 24, 2026Sining Ang, Yuan Chen, Liu Haiyan +5LLM Inference AccelerationAutonomous Driving Planning

    Institute for AI Industry Research (AIR), Tsinghua University · Department of Automation, University of Science and Technology of China · Beijing University of Aeronautics and Astronautics +1

  18. Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

    Jun 23, 2026WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6Long-Context Language Model InferenceKV-Cache Management

    National Yang Ming Chiao Tung University · Technische Universität Darmstadt · Cornell University

  19. RoPE-Aware Bit Allocation for KV-Cache Quantization

    Jun 23, 2026Fengfeng Liang, Yuechen Zhang, Jiaya JiaKV CachingLLM Inference Acceleration

    1Hong Kong University of Science and Technology · 2The Chinese University of Hong Kong · 3MiMo, Xiaomi Corporation

  20. GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

    Jun 22, 2026Jette Oberländer, Jan Finkbeiner, Catherine M. Schöfmann +1Mixed-Precision QuantizationLLM Quantization

    Fakultät für Informatik, RWTH Aachen, Aachen, 52074, Germany · Peter Grünberg Institut, Forschungszentrum Jülich GmbH, Jülich, 52425, Germany · Fakultät für Elektrotechnik und Informationstechnik, RWTH Aachen, Aachen, 52062, Germany

  21. FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

    Jun 22, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM ServingOn-Device Language Model Inference

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

  22. EnerInfer: Energy-Aware On-Device LLM Inference

    Jun 22, 2026Bohua Zou, Nian Liu, Binqi Sun +6On-Device Language Model InferenceLLM Inference Acceleration

    Technical University of Munich · Shanghai Jiao Tong University · Huawei Hilbert Research Center (Dresden) +1

  23. SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

    Jun 22, 2026Huzama Ahmad, Se-Young YunLong-Context Language Model InferenceLLM Inference Acceleration

    KAIST

  24. SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

    Jun 19, 2026Yeqi Huang, Yanwei Ye, Guomin Chen +8Agentic WorkflowsLLM Inference Acceleration

    University of Edinburgh United Kingdom · Tencent China

  25. Recency/Frequency Adaptive KV Caching for Large Language Model Serving

    Jun 19, 2026Yang Shen, Meghana Madhyastha, Robert Underwood +2LLM ServingKV Caching

    Department of Computer Science, Johns Hopkins University, Baltimore, USA

  26. Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models

    Jun 18, 2026Durgesh Kalwar, Vardhan Palod, Jaya Adithya Pavuluri +1CoT DistillationEfficient Language Model Reasoning

    SCAI, Arizona State University

  27. Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

    Jun 18, 2026Liang SuLLM Inference ServingLLM Inference Acceleration

  28. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM ServingInference-Time Optimization

    Unicom Data Intelligence, China Unicom · Data Science and Artificial Intelligence Research Institute, China Unicom · Tsinghua University +7

  29. Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

    Jun 18, 2026Nico Harder, Daniel Becking, Karsten Mueller +1LLM CompressionLow-Rank Matrix Decomposition

    Fraunhofer HHI, Berlin, Germany

  30. SafeSpec: Fast and Safe LLM via Dynamic Reflective Sampling

    Jun 18, 2026Haotian Xu, Zeyang Zhang, Linbao Li +3Speculative DecodingLLM Inference Acceleration

    Zhejiang University, Hangzhou, China · Harbin Institute of Technology, Shenzhen, China · Huawei