LLM Inference Efficiency

Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

541 papers

Latest in LLM Inference Efficiency

Browse all topics →
CardsList

  1. Akashic: A Low-Overhead LLM Inference Service with MemAttention

    Jul 7, 2026Yang Liu, Zhaokai Luo, Huayi Jin +7Large Language Model MemoryLLM Inference Efficiency

  2. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

    Jul 6, 2026Guanyu Cai, Ruiming Tian, Lang Yang +4Neural Processing UnitsLLM Inference Efficiency

  3. Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

    Jul 6, 2026Daeyeon SonLLM Inference Efficiency

  4. HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

    Jul 5, 2026Hui Dong, Yanzhao Li, Jie Gao +5W4A4 QuantizationQuadratic Attention

  5. Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B

    Jul 5, 2026Jaeyeon Kim, Jewon Lee, Bo-Kyeong KimLLM Inference EfficiencyQwen3

  6. AdaptiveSD A Stability-Aware, Runtime-Adaptive Speculative Decoding Framework with Multi-Policy Orchestration for CPU-Constrained LLM Inference

    Jul 4, 2026Sadra SaremiSpeculative DecodingLLM Inference Efficiency

  7. ELiTeFormer: An Efficient Transformer for FPGAs

    Jul 4, 2026Victor Agostinelli, Nicolas Bohm Agostini, Antonino TumeoField-Programmable Gate ArraysLLM Inference Efficiency

  8. WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs

    Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezLLM Inference EfficiencyInference Workloads

  9. An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation

    Jul 2, 2026Haoran Wang, Jinchuan Tian, Siddhant Arora +1Audio RepresentationAudio Tokens

  10. Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference

    Jul 2, 2026Wenchen Han, Gingfung Matthew Yeung, Marco Barletta +3Key-Value Cache QuantizationKey-Value Cache

  11. AGI Maze as a Benchmark Framework for World-Modeling Agents

    Jul 1, 2026Alexey PotapovPlanar MazesWorld Models

  12. BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

    Jul 1, 2026Prabod Rathnayaka, Fabian Waschkowski, Lukas WesemannLLM Inference EfficiencyInference Engine

  13. Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

    Jun 30, 2026Zhaoyang Luo, Runmin Dong, Miao Yang +4Visual TokensVideo Multimodal Large Language Models

  14. RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

    Jun 30, 2026Wenhao Li, Jinhao Dong, Hailin Zhang +3LLM Inference EfficiencyDynamic Sparse Attention

  15. SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

    Jun 30, 2026Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1Key-Value Cache CompressionKey-Value Cache

  16. Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

    Jun 29, 2026Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4Dynamic Sparse AttentionEfficient Long-Context Inference

  17. When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

    Jun 29, 2026Aaryam SharmaSpeculative DecodingAcceptance

  18. RiverONE: Generating Knowledge-Intensive VLM by Simulated Quantum Machines

    Jun 29, 2026Xindian Ma, Xinyu Long, Yefei Zhang +11Large Language Model QuantizationLarge Vision Language Models

  19. MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers

    Jun 29, 2026Linrui Ma, Chun Hei Lo, Xinyu Wang +12Efficient Long-Context InferenceDynamic Sparse Attention

  20. Speculative Pre-Positioning: Decoding Stateful Sessions to the Next Decision Point Off the Critical Path

    Jun 28, 2026Victor NorgrenPrefillLLM Inference Efficiency

  21. Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

    Jun 28, 2026Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi +1Key-Value Cache EvictionLLM Inference Efficiency

  22. HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compression

    Jun 27, 2026Yuxuan Yang, Feiyang Ren, Bowen Zeng +4Key-Value Cache CompressionLLM Inference Efficiency

  23. KernelSight-LM: A Kernel-Level LLM Inference Simulator

    Jun 26, 2026Xiteng Yao, Taeho Kim, Hengzhi Pei +7LLM Inference EfficiencyEfficient Inference

  24. End-to-End Dynamic Sparsity for Resource-Adaptive LLM Inference

    Jun 26, 2026Yuhang Chen, Jinhao Duan, Ruichen Zhang +11LLM Inference EfficiencySparsity

  25. Bifocal Diffusion Language Models: Asymmetric Bidirectional Context for Parallel Generation

    Jun 26, 2026Yuhang Chen, Xianfeng Wu, Jinhao Duan +11Diffusion Language ModelsLLM Inference Efficiency

  26. EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

    Jun 25, 2026Carrie ChenNext-Token PredictionLLM Inference Efficiency

  27. TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference

    Jun 25, 2026Tinghao Wang, Yichen Guo, Rui Huang +11Visual Token PruningMulti-Modal Large Language Models

  28. Information-Aware KV Cache Compression for Long Reasoning

    Jun 25, 2026Jushi Kai, Zhuiri Xiao, Alexandra Birch +1Key-Value Cache CompressionKey-Value

  29. Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT

    Jun 25, 2026Jinghan Wang, Yanjun Chen, Wei Zhang +3Heterogeneous Pruning ScoresLLM Inference Efficiency

  30. SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

    Jun 25, 2026Haoqian Meng, Yilun Luo, Yafei Zhao +4W4A4 QuantizationSparse Expert Activation

  31. ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

    Jun 23, 2026Taicheng Guo, Haomin Zhuang, Kehan Guo +4Hyperparameter OptimizationAgentic Optimization

  32. CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference

    Jun 23, 2026Xiaolin Lin, Jingcun Wang, Olga Kondrateva +3Key-Value Cache CompressionLLM Inference Efficiency

  33. Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

    Jun 23, 2026WenHung Lee, Jian-Jia Chen, Xiaolin Lin +6Speculative DecodingEfficient Long-Context Inference

  34. Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

    Jun 22, 2026Yuhang Gan, Yiwei Yang, Yuyi Li +6LLM Inference EfficiencyKey-Value Cache

  35. GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

    Jun 22, 2026Jette Oberländer, Jan Finkbeiner, Catherine M. Schöfmann +1Large Language Model QuantizationExtreme Low-Bit Quantization

  36. FlexServe: A Fast and Secure LLM Serving System for Mobile Devices with Flexible Resource Isolation

    Jun 22, 2026Yinpeng Wu, Yitong Chen, Lixiang Wang +3LLM Inference EfficiencyDevice

  37. EnerInfer: Energy-Aware On-Device LLM Inference

    Jun 22, 2026Bohua Zou, Nian Liu, Binqi Sun +6LLM Inference EfficiencyExplicit Memory

  38. SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

    Jun 22, 2026Huzama Ahmad, Se-Young YunDynamic Sparse AttentionEfficient Long-Context Inference

  39. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1Large Language Model ServingOptimal Scheduling

  40. The Language-Energy Divide: Measuring Energy Costs of Multilingual LLM Inference

    Jun 20, 2026Naihao Deng, Alissa Shen, Yiming Feng +5Multilingual Large Language ModelsLLM Inference Efficiency

  41. HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval

    Jun 19, 2026Omin Kwon, Doyeon Kim, Jongseok Park +3LLM Inference EfficiencyGraphics Processing Unit Memory

  42. Recency/Frequency Adaptive KV Caching for Large Language Model Serving

    Jun 19, 2026Yang Shen, Meghana Madhyastha, Robert Underwood +2Key-Value CacheLLM Inference Efficiency

  43. Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL

    Jun 19, 2026Zhenting Zhu, Lucas Thai, Shan Yu +5Full-Precision PerformanceLLM Inference Efficiency

  44. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    Jun 18, 2026Shiguo Lian, Kai Wang, Zhaoxiang Liu +22LLM Inference EfficiencyLarge Models

  45. Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference

    Jun 18, 2026Huang Peng, Jiuyang Tang, Weixin Zeng +2Knowledge ConflictsLLM Inference Efficiency

  46. SafeSpec: Fast and Safe LLM via Dynamic Reflective Sampling

    Jun 18, 2026Haotian Xu, Zeyang Zhang, Linbao Li +3Inference-Time DefenseSpeculative Decoding

  47. Closing the Operational Gap in Semantic Caching

    Jun 18, 2026Aditeya Baral, Radoslav Ralev, Iliya Sotirov Zhechev +2CacheSemantic Gap

  48. JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting

    Jun 16, 2026Lanxiang Hu, Zhaoxiang Feng, Yulun Wu +9Speculative DecodingLLM Inference Efficiency

  49. ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

    Jun 16, 2026Yao Chen, Yinqi Yang, Junyuan Shang +6Sliding-Window AttentionSparsity

  50. DeepInsight: A Unified Evaluation Infrastructure Across the Physical AI Stack

    Jun 16, 2026Siyi Li, Chunyu Sun, Jiahao Zhang +6Unified EvaluationOnline Inference

  51. Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference

    Jun 15, 2026Alexander Yukhimchuk, Andrey Shulga, Mladen Kolar +1ObfuscationPrivacy-Preserving Framework

  52. Communication-Efficient Verifiable Attention for LLM Inference

    Jun 15, 2026Ziqun Chen, Ming Wu, Michael Heinrich +4LLM Inference EfficiencyNvidia Nemotron

  53. SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

    Jun 15, 2026Feiyang Chen, Haibo ChenLLM Inference EfficiencyMatrix Multiplication

  54. Your "Pro" LLM Subscription May Actually Be "Free": Exposing Fingerprint Spoofing Risks in LLM Inference Services

    Jun 15, 2026Jiahao Zhang, Xiuyu Li, Suhang WangFingerprintLanguage Model Evasion Attacks

  55. Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators

    Jun 14, 2026Shun Usami, Venkatram Vishwanath, E. Wes BethelLLM Inference EfficiencyAccelerator

  56. MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization

    Jun 14, 2026Yangjia Hu, Haodong Wang, Zicong Hong +8Large Language Model QuantizationLow-Bit

  57. Service-Induced Congestion in Memory-Constrained LLM Serving

    Jun 14, 2026Ruicheng Ao, Jing Dong, Gan Luo +1Large Language Model ServingLLM Inference Efficiency

  58. A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference

    Jun 13, 2026Yingnan Zhao, Razvan Bunescu, Ahmed Louri +2Mixture-Of-Expert InferenceLLM Inference Efficiency

  59. Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices

    Jun 13, 2026Tao Lu, Haoyu Wang, Zonghui Wang +3LLM Inference EfficiencyMatrix Multiplication

  60. OSDAG: Online Scheduling for Efficient Multi-Robot Collaboration

    Jun 13, 2026Thanh Nguyen Canh, Thang Tran Viet, Phuc Van Dinh +2Multi-Robot SystemsMulti-Agent Coordination