LLM Inference Efficiency

Ask about this topic

Recent momentum

-5%

61 papers in the last 28 days · 1.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

21 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-14

12 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

Period ending 2026-09-07

16 new papers

A weekly snapshot of new work published in LLM Inference Efficiency.

540 papers

Latest in LLM Inference Efficiency

Browse all topics →
CardsList

  1. LightRot: A Light-Weighted Rotation Scheme and Architecture for Accurate Low-Bit Large Language Model Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Jungjun Oh +2LLM Inference EfficiencyAccelerator

  2. GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inference

    Jul 30, 2026Sangjin Kim, Yuseon Choi, Byeongcheol Kim +2LLM Inference EfficiencySynergy

  3. Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMs

    Jul 30, 2026Jinyi Liu, Wei Chen, Pengyu Chen +4Sparse Expert ActivationLLM Inference Efficiency

  4. OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

    Jul 29, 2026Jingbo Zhou, Yusai Zhao, Qi Bao +12Large Language Model AgentsLLM Inference Efficiency

  5. Beyond Accuracy: Auditing Spatial Provenance in Visual Token Pruning for OCR-Critical MLLM Inference

    Jul 29, 2026Feixiang Liu, Qiang Qiu, Hao Zhang +1Visual Token PruningTextvqa

  6. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    Jul 29, 2026Weiye Shi, Fanxu Meng, Muhan ZhangSpeculative DecodingLLM Inference Efficiency

  7. Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes

    Jul 29, 2026Tianyu Wang, Yuxuan Zhou, Wenbin Wang +3Speculative DecodingLLM Inference Efficiency

  8. From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs

    Jul 29, 2026Tina Vartziotis, Rodopi Kosteli, Elli Vartziotis +5LLM Inference EfficiencyNvidia Nemotron

  9. AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

    Jul 28, 2026Hong Liu, Rui Cen, Junhan Shi +10Speculative DecodingLLM Inference Efficiency

  10. OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs

    Jul 28, 2026Haoyang Huang, Wenjie Huang, Tianqi Xu +14Token CompressionAudio-Visual Reasoning

  11. IRIS: Reusable Identity Representations from Frozen LLMs for Entity Alignment

    Jul 28, 2026Xinran Liu, Shengtao Li, Shouqian Shi +2EntitiesKnowledge Graphs

  12. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

    Jul 27, 2026Yifan Dou, Shikan Lian, Shibo LiLLM Inference EfficiencyConformal Calibration

  13. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    Jul 27, 2026Junsung HwangEfficient Long-Context InferenceKey-Value

  14. GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models

    Jul 27, 2026Jun Ling, Tao Huang, Junzhuo Liu +2Large Vision Language ModelsLLM Inference Efficiency

  15. Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising

    Jul 24, 2026Dengyu Wu, Clement Ruah, Jiechen Chen +2Diffusion Language ModelsLLM Inference Efficiency

  16. Unified Static-Dynamic Pruning for Efficient LLM Inference

    Jul 24, 2026Jinhyeok Kim, Yejoon Lee, Jaeyoung DoLLM Inference EfficiencyHeterogeneous Pruning Scores

  17. RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention

    Jul 24, 2026Anderson R. SantosDynamic Sparse AttentionLLM Inference Efficiency

  18. Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

    Jul 23, 2026Alagappan ValliappanSpeculative DecodingPer-Token Latency

  19. Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs

    Jul 23, 2026Yidu Wu, Xiang Wang, Kejie Zhao +3LLM Inference EfficiencyLarge Language Model Pretraining

  20. Profiling Lightweight Large Language Models

    Jul 23, 2026Tomohiro Harada, Enrique Alba, Gabriel LuqueLLM Inference EfficiencyProfiling

  21. Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing

    Jul 22, 2026Sadegh Majidi, Niloofar Mireshghallah, Kazem TaramLanguage Model Evasion AttacksLLM Inference Efficiency

  22. PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

    Jul 22, 2026Niqi Lyu, Pengtao Shi, Wei Qiu +4LLM Inference EfficiencyInference Cost

  23. Efficient Clustering with Provable Guardrails for LLM Inference at Scale

    Jul 22, 2026Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara +3Image ClusteringLLM Inference Efficiency

  24. CausalGate: Causal Importance Distillation for Transformer Module Pruning

    Jul 21, 2026Kiran Nair, Smriti Regmi, Rodrigue RizkLLM Inference EfficiencyImportance

  25. Integrity of peer-to-peer distributed LLM inference under malicious nodes

    Jul 21, 2026Mert Cihangiroglu, Antonino NoceraLLM Inference EfficiencyLanguage Model Evasion Attacks

  26. AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters

    Jul 21, 2026Yu-Yang Qian, Hao-Cong Wu, Chen Chen +4Speculative DecodingDiffusion Language Models

  27. BaseRT: Advancing Best-in-Class LLM Inference with Apple M5 Neural Accelerators

    Jul 21, 2026Fabian Waschkowski, Prabod Rathnayaka, Lukas WesemannLLM Inference EfficiencyAccelerator

  28. SelectInfer: Selective Neuron Loading and Computation for On-Device LLMs

    Jul 20, 2026Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow +1LLM Inference Efficiency

  29. MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

    Jul 20, 2026Simla Burcu Harma, Danila Mishin, Zhengyuan Su +7Large Language Model QuantizationMixed-Precision Quantization

  30. C2^2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference

    Jul 20, 2026Chuheng Du, Junyi Chen, Hanlin Tang +7Key-Value CacheEfficient Long-Context Inference

  31. Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

    Jul 19, 2026Param ChordiyaSpeculative DecodingAutoregressive Decoding

  32. ThAME: 3D Memory-Enabled Heterogeneous Accelerator for LLM Mixture of Experts

    Jul 19, 2026Pratyush Dhingra, Pramit Kumar Pal, Janardhan Rao Doppa +1Mixture-Of-Expert InferenceLLM Inference Efficiency

  33. SpecLA: Efficient Speculative Decoding for Linear-Attention Models

    Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative DecodingQuadratic Attention

  34. LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

    Jul 16, 2026Changhai Zhou, Kieran Liu, Yuhua Zhou +17Efficient Long-Context InferenceLLM Inference Efficiency

  35. D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

    Jul 16, 2026Tianyu Liu, Yuhao Shen, Rui Cen +7Speculative DecodingLLM Inference Efficiency

  36. PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

    Jul 16, 2026Hyunwoo Oh, Suyeon Jang, Hanning Chen +4Weight-Only QuantizationLLM Inference Efficiency

  37. Seeing the End at Step Zero: Accelerating Diffusion MLLMs via MLP Sparsity-Aware Truncation

    Jul 16, 2026Qicheng Zhao, Qi Sun, Zheyu YanDiffusion Language ModelsLLM Inference Efficiency

  38. Smarter and Cheaper at Once: Byte-Exact KV-Cache Grafting Turns a Frozen Small Model into a Verified-Knowledge Flywheel

    Jul 15, 2026Sietse SchelpeFrozen ModelsLLM Inference Efficiency

  39. Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

    Jul 14, 2026Soumil MandalKey-Value Cache EvictionKey-Value Cache

  40. Automatic Model-Hardware Co-Adaptation for Heterogeneous AI Accelerators

    Jul 14, 2026Tian Chen, Mingheng Mi, Pu Wang +2LLM Inference EfficiencyInference Engine

  41. Accepted Prefixes Are Not All You Need: A Negative Result on PEFT-Based Block-Diffusion Drafting

    Jul 14, 2026Abdurrahman Javat, Allan KazakovSpeculative DecodingLLM Inference Efficiency

  42. Efficient and Privacy Aware Edge Cloud Collaborative Inference for Large Language Models

    Jul 14, 2026Yi Li, Chen Li, Jiexiong LiuLLM Inference EfficiencyHeterogeneous Edge Devices

  43. TreeThink: A Modular Tree Search Library for Mathematical Reasoning with LLMs

    Jul 13, 2026Burak S. Akbudak, Zeynel A. Uluşan, Can S. Erer +1TheoremTree Search

  44. FastTPS: An Optimized Method for LLM Token Phase for AI accelerators

    Jul 13, 2026Wenzong Yang, Danyang Zhang, Kun Cao +17LLM Inference EfficiencyAccelerator

  45. Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results

    Jul 13, 2026Sheng Xu, Junhua Wang, Boyuan Huang +3LLM Inference EfficiencyReal-Time Inference

  46. PromptGraph: Graph-Guided Prompt Sanitization for Balancing Privacy and Utility in LLM Inference

    Jul 12, 2026Chen Gu, Hui Wan, Donghui Hu +2Prompt-Level MitigationsAI Privacy Risks and Protection

  47. Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

    Jul 12, 2026Zipeng Gao, Zhi Zheng, Qingrong Xia +5Speculative DecodingAutoregressive Language Models

  48. MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

    Jul 12, 2026Venkatesha Matam, Keon KimKey-Value Cache EvictionCache

  49. Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices

    Jul 11, 2026Yangyijian Liu, Hongyi Ye, Mingyang Li +1Multi-Gpu SystemsLLM Inference Efficiency

  50. Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

    Jul 10, 2026Nada Zine, Tristan Coignion, Vincenzo Stoico +3LLM Inference EfficiencyInference Engine

  51. GATS: Graph-Augmented Tree Search with Layered World Models for Efficient Agent Planning

    Jul 9, 2026Maureese Williams, Dymitr NowickiLarge Language Model PlanningClassical Planning

  52. DominoTree: Conditional Tree-Structured Drafting with Domino for Speculative Decoding

    Jul 9, 2026Saw S. Lin, Jyh-Shing Roger JangSpeculative DecodingLLM Inference Efficiency

  53. Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls

    Jul 9, 2026Peter HollowsPenaltiesLogit Lens

  54. Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

    Jul 9, 2026Jiantong Jiang, Peiyu Yang, Rui Zhang +1Key-Value CacheLarge Language Model Serving

  55. Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment

    Jul 9, 2026Hyeju Shin, Chorwon Kim, Ryangsoo Kim +2Large Language Model QuantizationNvidia Jetson Orin Nano

  56. DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting

    Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative DecodingDrafter

  57. TF-Engram: A Train-Free Engram with SSD-Backed Memory for Large Language Models

    Jul 8, 2026Yutang Ma, Kecheng Huang, Xikun Jiang +1Large Language Model MemoryLLM Inference Efficiency

  58. Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference

    Jul 8, 2026Vladimir GusevKey-Value CacheLLM Inference Efficiency

  59. DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

    Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4Key-Value Cache CompressionEfficient Long-Context Inference

  60. FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

    Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4Key-Value Cache CompressionLLM Inference Efficiency