LLM Inference Efficiency

Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

541 papers

Latest in LLM Inference Efficiency

Browse all topics →
CardsList

  1. Exploring a Layer-Wise Design Space for KV Cache Eviction

    Jun 13, 2026Chao Fei, Kaihua Liang, Hanzhi Hu +4Key-Value Cache CompressionCache

  2. STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming

    Jun 11, 2026Anas Nassar, Steve Mohr, Leonard Apanasevich +1LLM Inference EfficiencyLarge Language Model Routing

  3. SuperThoughts: Reasoning Tokens in Superposition

    Jun 11, 2026Zheyang Xiong, Shivam Garg, Max Yu +4ThinkingLLM Inference Efficiency

  4. Accelerating Speculative Diffusions via Block Verification

    Jun 11, 2026Alexander Soen, Hisham Husain, Valentin De Bortoli +1Diffusion Language ModelsSpeculative Decoding

  5. MiniMax Sparse Attention

    Jun 11, 2026Xunhao Lai, Weiqi Xu, Yufeng Yang +14Dynamic Sparse AttentionLLM Inference Efficiency

  6. Efficient On-Device Diffusion LLM Inference with Mobile NPU

    Jun 11, 2026Tuowei Wang, Yanfan Sun, Ju RenNeural Processing UnitsLLM Inference Efficiency

  7. TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization

    Jun 11, 2026Zhixiong Zhao, Zukang Xu, Zhixuan Chen +3Large Language Model QuantizationLLM Inference Efficiency

  8. Multi-Bitwidth Quantization for LLMs Using Additive Codebooks

    Jun 11, 2026Liza Babaoglu, Shuangyi Chen, Ashish KhistiLarge Language Model QuantizationWeight-Only Quantization

  9. Arbor: Tree Search as a Cognition Layer for Autonomous Agents

    Jun 10, 2026Neha Prakriya, Chaojun Hou, Zheng Gong +5Tree SearchLLM Inference Efficiency

  10. VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

    Jun 10, 2026Yuchen Xian, Yang He, Yunqiu Xu +1Speculative DecodingLLM Inference Efficiency

  11. Adaptive Multi-Resolution Procedural Knowledge Compression for Large Language Models

    Jun 10, 2026Changyue Wang, Weihang Su, Qingyao Ai +5Large Language Model CompressionContext Compression

  12. DynamicPTQ: Mitigating Activation Quantization Collapse via Residual-Stream Dynamics

    Jun 10, 2026Zimo Zhao, Maolin Wang, Bowen Yu +3Weight-Only QuantizationExtreme Low-Bit Quantization

  13. Closing the Social-Semantic Gap: SPSD for Edge-Based Prompt Compression in Cloud LLM Inference

    Jun 10, 2026Abhinit Sen, Ajeet Kumar, Manaranjan PradhanLLM Inference EfficiencyLarge Language

  14. TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs

    Jun 9, 2026Wesley Pang, Gregory Hyegang Jun, Feiyang Liu +1Neural Processing UnitsLLM Inference Efficiency

  15. CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

    Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouNext-Token PredictionLLM Inference Efficiency

  16. Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

    Jun 9, 2026Jing Xiong, Qi Han, Shansan Gong +5Diffusion Language ModelsPrefill

  17. UniSVQ: 2-bit Unified Scalar-Vector Quantization

    Jun 9, 2026Haoyu Wang, Haiyan Zhao, Xingyu Yu +4QuantizationLow-Bit

  18. SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

    Jun 9, 2026Jaeseong Lee, Seung-won Hwang, Samyam RajbhandariSparsityLLM Inference Efficiency

  19. Atomic Intent Reasoning: Bringing LLM Semantics to Industrial Cross-Domain Recommendations

    Jun 9, 2026Zhuohang Jiang, Yuxin Chen, Shijie Wang +6LLM Inference EfficiencySemantic Gap

  20. Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite

    Jun 9, 2026Zhiyuan Cheng, Longying LaiRetrieval-Augmented Generation SystemsNeural Processing Units

  21. AutoMegaKernel: A Statically-Checked Agent Harness for Self-Retargeting Megakernel Synthesis

    Jun 8, 2026Jaber Jaber, Osama JaberGraphics Processing Unit KernelsLLM Inference Efficiency

  22. End-to-End Context Compression at Scale

    Jun 8, 2026Ang Li, Sean McLeish, Haozhe Chen +12Large Language Model CompressionEfficient Long-Context Inference

  23. Gradient-Guided Reward Optimization for Inference-time Alignment

    Jun 8, 2026Hankun Lin, Ruqi ZhangInference-Time DefenseLLM Inference Efficiency

  24. FuseFSS: Efficient Secure LLM Inference with Function Secret Sharing

    Jun 8, 2026Yuhan Ma, Yong Li, Stefan SchmidLLM Inference EfficiencySystematic Fp16 Divergence

  25. BUDDY: BUdget-Driven DYnamic Depth Routing for Adaptive Large Language Model Inference

    Jun 8, 2026Yuhua Zhou, Shaoqi Yu, Shichao Weng +4LLM Inference Efficiency

  26. From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs

    Jun 8, 2026Zhanchao Xu, Haoyang Li, Qingfa Xiao +4LLM Inference EfficiencyDynamic Sparse Attention

  27. SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

    Jun 8, 2026Rya Sanovar, Srikant Bharadwaj, Hritvik Taneja +1Retrieval-Augmented Generation SystemsLLM Inference Efficiency

  28. Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

    Jun 8, 2026Haozhe Hu, Hao Wu, Anhao Zhao +4Heterogeneous Pruning ScoresLLM Inference Efficiency

  29. FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

    Jun 8, 2026Yan Wang, Qifan Zhang, Jiachen Yu +12Efficient Long-Context InferenceDynamic Sparse Attention

  30. RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference

    Jun 7, 2026Anirudh SekarKv-Cache BlocksLLM Inference Efficiency

  31. APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing

    Jun 7, 2026Hong Guo, Nianhui Guo, Weixing Wang +3W4A4 QuantizationTensor Programs

  32. AsyncLane: Decoupling Refinement from Advancement in Diffusion Language Model Decoding

    Jun 7, 2026Yingxuan Ren, Yuxuan Lou, Yong Liu +4Diffusion Language ModelsAutoregressive Decoding

  33. IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference

    Jun 6, 2026Junjie Li, Jiong Lou, Jie LiKv-Cache BlocksCache

  34. When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference

    Jun 6, 2026Yasushi Sakai, Allen Song, Kent LarsonConsensusDelegation

  35. WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

    Jun 5, 2026Young D. Kwon, Miles Williams, Rui Li +2Speculative DecodingLLM Inference Efficiency

  36. OffQ: Taming Structured Outliers in LLM Quantization by Offsetting

    Jun 5, 2026Haoqi Wang, Lorenz K. Mueller, Jiawei Zhuang +2Large Language Model QuantizationLow-Bit

  37. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs

    Jun 4, 2026Ziyue Li, Yang Li, Tianyi ZhouLLM Inference EfficiencyLarge Language Model Reasoning

  38. You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

    Jun 4, 2026Yutao Sun, Yanqi Zhang, Li Dong +2Dynamic Sparse AttentionEfficient Long-Context Inference

  39. LLM Explainability with Counterfactual Chains and Causal Graphs

    Jun 4, 2026Nirit Nussbaum-Hoffer, Nitay Calderon, Liat Ein-Dor +1Causal GraphExplainability

  40. SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

    Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3LLM Inference Efficiency

  41. SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inference

    Jun 3, 2026Yaosheng Fu, Guangxuan Xiao, Xin Dong +2Dynamic Sparse AttentionLLM Inference Efficiency

  42. D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

    Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative DecodingDiffusion Language Models

  43. Operator Fusion for LLM Inference on the Tensix Architecture

    Jun 3, 2026Qingbo Wu, Ke Li, Wenzhu Wang +3Tensor ProgramsSparse Deep Neural Network Accelerators

  44. LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding

    Jun 3, 2026Haocheng Xia, Mihir Pamnani, Hanxi Fang +2LLM Inference EfficiencyPositional Encoding

  45. Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data

    Jun 2, 2026Devleena Das, Rajeev Patwari, Elliott Delaye +1Large Language Model QuantizationLLM Inference Efficiency

  46. NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference

    Jun 2, 2026Mubarak Adetunji OjewalePer-Token LatencyKey-Value Cache

  47. E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

    Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3Large Language Model ServingLLM Inference Efficiency

  48. LLM Compression with Jointly Optimizing Architectural and Quantization choices

    Jun 2, 2026Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1Large Language Model CompressionLLM Inference Efficiency

  49. The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

    Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertAlgorithm AuditingValidity

  50. Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning

    Jun 2, 2026Yanyu Zhu, Hoilam Pao, Niu Hu +6Speculative DecodingLLM Inference Efficiency

  51. Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs

    Jun 2, 2026Ting LiuLLM Inference EfficiencySparse Expert Activation

  52. Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving

    Jun 1, 2026Chunan Shi, Yilei Chen, Yilin Chen +2CacheLLM Inference Efficiency

  53. KForge: LLM-Driven Cross-Platform Kernel Generation for AI Accelerators

    Jun 1, 2026Taras Sereda, Burak Bartan, Ankita Nayak +3Graphics Processing Unit KernelsAccelerator

  54. Fast-dLLM++: Fréchet Profile Decoding for Faster Diffusion LLM Inference

    Jun 1, 2026Siva Rajesh Kasa, Yasong Dai, Sumit Negi +1Diffusion Language ModelsLLM Inference Efficiency

  55. Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference

    Jun 1, 2026Chi-Wei Huang, Chia-Chi TsaiWeight-Only QuantizationQuantization

  56. SimSD: Simple Speculative Decoding in Diffusion Language Models

    Jun 1, 2026Junxia Cui, Haotian Ye, Runchu Tian +9Diffusion Language ModelsSpeculative Decoding

  57. Not All Errors Are Equal: A Systematic Study of Error Propagation in Large Language Model Inference

    Jun 1, 2026Yafan Huang, Sheng Di, Guanpeng LiLLM Inference EfficiencyFault Injection Experiments

  58. Evolutionary Discovery of Bivariate Bicycle Codes with LLM-Guided Search

    Jun 1, 2026Juan Cruz-Benito, Andrew W. Cross, David Kremer +1Quantum Error CorrectionCodebooks

  59. DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decoding

    Jun 1, 2026Jiebin Zhang, Zhenghan Yu, Song Liu +9Speculative DecodingLLM Inference Efficiency

  60. Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time

    Jun 1, 2026Mingkuan Zhao, Yide Gao, Wentao Hu +6LLM Inference EfficiencyMemory-Augmented Large Language Models