LLM Inference Acceleration

LLM: Large Language Model

Ask about this topic

Momentum

108 papers in the last four weeks, up 125% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 667

All topics
CardsList

  1. UNREAL: Unifying Retrieval and Long-Context with a Single Model

    Oct 6, 2026Edan Kinderman, Elad Hoffer, Yochai Blau +4Retrieval-Augmented GenerationKnowledge Augmentation for Language Models

    NVIDIA · Technion

  2. SSR: Sparse Segment Reduction for Ternary GEMM Acceleration

    Oct 6, 2026Adeline Pittet, Shien Zhu, Valérie Verdan +1Ternary QuantizationLLM Inference Acceleration

    Systems Group, ETH Zurich, Switzerland

  3. Hybrid Latent Attention for Looped Language Models

    Oct 6, 2026Yuhan Chen, Siyuan Zhang, Nan Wang +2Long-Context Language Model InferenceLLM Inference Acceleration

    Virginia Tech · Independent Researcher

  4. Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

    Oct 6, 2026Shuhao Li, Fanghua Ye, Wanyu Lin +2Speculative DecodingLLM Inference Acceleration

    EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher

  5. APEX: Speculate smarter, not deeper

    Oct 6, 2026Manvi Jha, Zach Zhang, Zhichao Xu +3Speculative DecodingLLM Inference Acceleration

    University of Illinois Urbana-Champaign AWS AI · AWS AI

  6. TRACE: Rollout-Guided Quantization-Aware Training for FP4 Reinforcement Learning of MoE Language Models

    Oct 6, 2026Xin Wang, Hao Yu, Zhengyang Zhuge +9LLM QuantizationQuantization-Aware Training

    Alibaba Token Hub, Alibaba Group · Ohio State University

  7. DLoop: Looped Speculative Decoding

    Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative DecodingLLM Inference Acceleration

    NAVER AI Lab · Korea University · NAVER AI Search Platform

  8. AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

    Oct 5, 2026Hanzhi Zhang, Qiao Zhang, Qinglei Cao +4Mixed-Precision QuantizationLLM Quantization

    LLaVi Lab, Computer Science & Engineering, University of North Texas

  9. Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

    Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model TrainingRecurrent Transformers

    Institute of Foundation Models · CMU · USC

  10. OVAL: Output-Aware Local Page Bases for KV Cache Retrieval

    Oct 5, 2026Ashkan Shahbazi, Chayne Thrash, Soheil KolouriLong-Context Language Model InferenceLLM Inference Acceleration

    College of Connected Computing Vanderbilt University Nashville, TN, USA

  11. Behavior-Preserving KV Cache Compression

    Oct 5, 2026Doo Hwan Hwang, Junyoung Jang, Junho Na +2KV-Cache EvictionLLM Inference Acceleration

    Kim Jaechul Graduate School of AI, KAIST, Daejeon, Republic of Korea · KT Corporation, Seoul, Republic of Korea

  12. SchemaFill: Efficient LLM Tool Calling via Slot-Parallel Speculative Decoding

    Oct 5, 2026Zhi-Kai Chen, Song-Yan Li, De-Chuan Zhan +1Speculative DecodingLLM Inference Acceleration

    School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China

  13. Request Order Matters: Cache-History Sensitivity in Selective KV-Cache Reuse for Rolling Agents

    Oct 5, 2026Tiffany Gu, Annie Guan, Manshu Huang +3KV-Cache ManagementLLM Inference Acceleration

    Capital One · University of Washington

  14. SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

    Oct 4, 2026Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1Long-Context Language Model InferenceSpeculative Decoding

    UNIST

  15. More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding

    Oct 3, 2026Noam Elata, Itay Lamprecht, Mikey Shechter +3LLM Inference AccelerationSparse Attention

    Technion – Haifa, Israel · Intel, Israel · Corma +1

  16. TopK-Guided: Adaptive, Budget-Aware Activation Sparsity for Efficient LLM Inference

    Oct 1, 2026Mukund Agarwalla, Chih-Jen LinActivation SparsityLLM Inference Acceleration

    MBZUAI · National Taiwan University

  17. DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair

    Oct 1, 2026Zhuoyu Wang, Junnan Huang, Xinyu ChenSpeculative DecodingLLM Inference Acceleration

    The Hong Kong University of Science and Technology (Guangzhou)

  18. HHR: Hierarchical Hash Retrieval for Efficient LLM Generation

    Oct 1, 2026Lianjun Liu, Tiantian Zheng, You Huang +5Long-Context Language Model InferenceLLM Inference Acceleration

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  19. Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads

    Oct 1, 2026Prachi Badarayani, Aidan Jay, Chenghui Zhou +8Multi-Token PredictionSpeculative Decoding

    Microsoft

  20. AIR-LLM: Broadcasting AI Weights over Radio for Memory-Free Edge LLM Inference via RF Computing

    Sep 30, 2026Zhihui Gao, Tingjun Chen, Dirk EnglundOn-Device Language Model InferenceLLM Inference Acceleration

    Massachusetts Institute of Technology Cambridge, MA, USA · Duke University Durham, NC, USA

  21. XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization

    Sep 30, 2026Xiaofan Que, Nir Elkayam, Spandan Pyakurel +2LLM QuantizationLLM Inference Acceleration

    Arm, Inc. · Rochester Institute of Technology

  22. Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

    Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert ParallelismMixture-of-Experts Inference

    Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University

  23. IrekoGPT: Turning Structured Pruning into Post-Hoc Slimmable LLMs

    Sep 30, 2026Pietro Moriello, Pietro Buzzega, Angelo Porrello +1LLM CompressionLLM Inference Acceleration

    AImageLab, University of Modena and Reggio Emilia, Italy

  24. CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

    Sep 30, 2026Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1Long-Context Language Model InferenceKV-Cache Management

    Amazon AGI

  25. Beyond Accuracy: Prefix-Invariant Realizations of Low-Precision Fast Matrix Multiplication

    Sep 30, 2026Shuxiao Xie, Shuyang Xie, Yuan Cao +3LLM Inference AccelerationLow-Bit Quantization

  26. EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstruction

    Sep 30, 2026Jiawei Lin, Saibo Geng, Thomas BourgeatKV-Cache ManagementLLM Inference Acceleration

    EPFL

  27. LampAttention: Look-Ahead Mixed-Precision FlashAttention for Dedicated Accelerators

    Sep 30, 2026Stanislav Budzinskiy, Marian Gloser, Tolunay Yilmaz +5AI Accelerator InferenceLLM Inference Acceleration

    Faculty of Mathematics University of Vienna, Austria · Huawei Heisenberg Research Center, Munich, Germany · Huawei Technologies Co. Ltd

  28. Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

    Sep 30, 2026Sietse SchelpeLLM ServingLong-Context Language Model Inference

    Corbenic AI

  29. PatchKV: Weight-Space Compensation of KV Cache

    Sep 30, 2026Chanryeol Lee, Chanhyuk Lee, Yeonwoo Choi +2Long-Context Language Model InferenceLLM Inference Acceleration

    KAIST

  30. SparseEngine: Sparse-First Inference Engine

    Sep 30, 2026Jitai Hao, Quansheng Gu, Qiang Huang +1LLM InferenceLong-Context Language Model Inference

  31. Audio Token Attention Is Predictable Before the Language Model Runs

    Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token CompressionAudio Understanding

    The University of Texas at Austin

  32. STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

    Sep 29, 2026Bingchen Yao, Haobo Xu, Haokun Lin +6LLM QuantizationLLM Inference Acceleration

    Zhejiang University · Tsinghua University · NLPR & MAIS, Institute of Automation, CAS +3

  33. LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization

    Sep 29, 2026Yi Pan, Haocheng Xi, Kan Zhu +10LLM QuantizationLLM Inference Acceleration

    UC Berkeley · University of Washington · MIT +2

  34. KV-Kaizen: Learning Context-Adaptive Cache Compression Choices

    Sep 29, 2026Joao Monteiro, Louis Béthune, Anastasiia Filippova +3LLM Inference AccelerationLow-Rank Compression

    Apple

  35. Can a Cacheable Decision Model Follow Rules?

    Sep 29, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLLM Inference AccelerationLLM Decision-Making

    AltSlate Labs LLP

  36. DScale: Scaling Block-Diffusion Speculative Decoding with Adaptive Verification

    Sep 29, 2026Rongjian Chen, Minxian Xu, Zhengxin Fang +2Speculative DecodingLLM Inference Acceleration

    Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518052, China · University of Chinese Academy of Sciences, Beijing 100049, China · Victoria University of Wellington, Wellington, New Zealand +1

  37. LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

    Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Si Shen +1Long-Context Language Model InferenceBlockwise Diffusion Language Models

  38. LatCom: Cross-Agent Latent Compression for Efficient Multi-Agent Collaboration

    Sep 29, 2026Shinan Zhang, Tao Zhang, Qihui Zhu +7Multi-Agent LLM SystemsMulti-Agent Collaboration

    University of Science and Technology of China · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

  39. ARC-KV: Amortizing Anchor Search for Reconstruction-Based KV Cache Compaction

    Sep 29, 2026Zheyu Shen, Guanhua Wang, Dezhan Tu +5Long-Context Language ModelingKV-Cache Management

    University of Maryland, College Park · Meta

  40. QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching

    Sep 29, 2026Zunhai Su, Yuxuan Sun, Jianchao Tan +5LLM Inference AccelerationAttention Quantization

    The University of Hong Kong · Meituan LongCat Team · South China University of Technology +1

  41. MoRE: Scaling mixture of experts with hardware-aware low-rank routing

    Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts RoutingLow-Rank Matrix Decomposition

    University of Pennsylvania · The Wharton School, University of Pennsylvania

  42. Draft in Parallel, Condition Through Depth: Adjacent Causal Injection for Speculative Decoding

    Sep 28, 2026Haohui Zhang, Keyu Chen, Haocheng Sun +4Speculative DecodingLLM Inference Acceleration

    Shanghai Jiao Tong University · Tencent YouTu Lab · Xiamen University

  43. Improving Test-Time Scaling with Adaptive Looped Transformers

    Sep 28, 2026Yichen You, Tianyu Fu, Aosong Feng +4Test-Time ScalingRecurrent Transformers

    Tsinghua University · Yale University

  44. SANTA++: Sampling Attention through Representative Keys

    Sep 28, 2026Kyle Lee, Christian Z. Pratt, Ruoyu Fang +4Memory-Efficient InferenceLLM Inference Acceleration

    Department of Electrical and Computer Engineering University of California, Santa Barbara, CA, USA · Flucta, San Francisco, CA, USA

  45. Tetra: Serving Leech-Lattice Quantized LLMs at 2.7 Bits per Parameter

    Sep 28, 2026Pier-Jean MalandrinoLLM QuantizationLLM Inference Acceleration

    Scub, Bordeaux, France

  46. Beneath the Tokens: A Performance Engineering Study of Multi-Token Prediction in GPU-Accelerated LLM Inference

    Sep 28, 2026Suwesh Prasad SahLLM Inference EfficiencyMulti-Token Prediction

  47. CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion

    Sep 28, 2026Genglin Wang, Wangsong Yin, Yeerzhati Abudunuer +3Retrieval-Augmented GenerationKV-Cache Management

    The Chinese University of Hong Kong · Peking University · The Hong Kong University of Science and Technology

  48. TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash

    Sep 28, 2026Jay H. Park, Hyungjun Kim, Dong KimKV-Cache OffloadingKV-Cache Management

    Samsung Semiconductor

  49. Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

    Sep 28, 2026Zhengxiang Huang, Shengheng Chen, Chaoyue Niu +6AI Accelerator InferenceOn-Device Language Model Inference

    Shanghai Jiao Tong University Shanghai, China · Alibaba Group Hangzhou, China · Alibaba Group Hangzhou, Zhejiang, China

  50. The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

    Sep 28, 2026Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail +3Early StoppingLong-Context Language Model Evaluation

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · Department of Computer Science, Edge Hill University, Ormskirk, England

  51. PulseInfer: I/O-Centric Sparse KV Cache Offloading for Efficient Long-Context LLM Decoding

    Sep 28, 2026Qiuyang Zhang, Kai Zhou, Kai Lu +6KV-Cache OffloadingLong-Context Language Model Inference

    Huazhong University of Science and Technology Wuhan, China · UCloud Technology Co., Ltd. Shanghai, China

  52. Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

    Sep 28, 2026Guanghao Li, Zihan Su, Hao Yu +6Speculative DecodingRecurrent Transformers

    Tsinghua University · Southern University of Science and Technology · The University of Hong Kong +2

  53. Routing Without Embeddings: Fast And Interpretable Routing With Regular Expressions

    Sep 28, 2026Yifan Lu, Qiyue Zhang, Haotian Shan +2LLM Inference AccelerationLLM Routing

    Rice University

  54. SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

    Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-of-Experts PruningDisaggregated LLM Serving

    a2sys · KAIST

  55. PReCache: Efficient KV Cache Sharing for Multi-LoRA Agents via Low-Rank Precomputation and Neutral Reconstruction

    Sep 28, 2026Hyesung Jeon, Hyeongju Ha, Jae-Joon KimMulti-Agent LLM SystemsMemory-Efficient Inference

    Department of Electrical and Computer Engineering Seoul National University

  56. Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

    Sep 27, 2026Jungseob Lee, Seungyoon Lee, Seongtae Hong +2Language Model DecodingActivation Sparsity

    Korea University · Yonsei University Mirae Campus

  57. Faster Block-Diffusion Serving with Distribution-Free Risk Guarantees

    Sep 27, 2026Jungseob Lee, Dongyub Jude Lee, Chanjun Park +2Diffusion Language Model InferenceLLM Inference Acceleration

    Korea University · Zoom Communications · Soongsil University +1

  58. JET: Justification Evaluation in Transformer

    Sep 27, 2026Shenghao DingEfficient VLM InferenceLLM Inference Acceleration

    Yet Another AI

  59. PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

    Sep 27, 2026Kunming Shao, Jierun Chen, Yanli Wang +4Self-AttentionLLM Inference Acceleration

    The Hong Kong University of Science and Technology · Huawei Technologies Ltd. · Sun Yat-sen University +1

  60. Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration

    Sep 27, 2026Shangzhen Zhu, Muyan Hu, Tomasz KozlowskiEfficient Transformer InferenceSoftmax Attention

    University of Illinois Urbana-Champaign