LLM Inference Efficiency

LLM: Large Language Model

Overview

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

CardsList

368 papers17 papers in the last four weeks, down 67% on the four weeks before. 0.1% of all new papers.

  1. TableRoute: Capability-Aware Routing for Scalable Table Question Answering

    Nov 19, 2026Yuxiang Wang, Junhao Gan, Jianzhong QiTable QALLM Routing

  2. Call-Efficient ICD-10 Code Prediction via Supervised Candidate Generation and Selective LLM Invocation

    Nov 16, 2026Ilia Kopanichuk, Daniil Kartashev, Yuri Bolkonsky +4Automated Medical CodingLLM Routing

  3. RoShift: Lazy Shift and Weight-aware Rotation for Activation Sparsity in LLM Inference

    Nov 16, 2026Geunsoo Song, Jiwon Seo, Hoeseok Yang +1LLM Inference EfficiencyActivation Sparsity

  4. LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent

    Nov 13, 2026Ibrahim Memon, Armin Karamzade, M. Hanif Jhaveri +1Text-to-SQLNatural Language Interfaces to Databases

  5. Beyond Query Similarity: Safety-Conditioned Cache Routing for LLM Response Reuse

    Nov 7, 2026Shilong Pan, Zhiliang Tian, Weiguo Chen +2Semantic CachingSafety-Utility Trade-Off in Language Models

  6. Intent-Aware Caching for Efficient LLM Serving

    Nov 6, 2026Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos +3LLM Inference EfficiencyLLM Serving

  7. StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

    Nov 5, 2026Jake Hyun, Ziyu Zhao, Jae-sun Seo +1LLM InferenceLLM Inference Efficiency

  8. When Strong Reasoners Join the Pool: Small Union Premiums and Costly Runaway Generations in LLM Routing

    Nov 3, 2026Teng-Ruei ChenLLM RoutingLanguage Model Reasoning Evaluation

  9. Semantic Caches for LLM Web Services Leave Their Error Budget Unspent

    Nov 2, 2026Hanshuai Cui, Qianli Ma, Zhiqing TangSemantic CachingLLM Serving

  10. As Expected? Efficient LLM Planning via Expectation-Observation Consistency Verification

    Nov 2, 2026Qingfu Zhu, Huanyang Zheng, Ting Liu +1LLM PlanningLLM Agent Efficiency

  11. TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents

    Oct 30, 2026Jihao Zhao, Shuaishuai Zu, Pengfei Li +4Speculative DecodingLLM Inference Efficiency

  12. Compute Has Shape: Self-Morphing the Internal Computation of Frozen LLMs

    Oct 28, 2026Junxi Wu, Yiwen Guo, Shu-Tao XiaInference-Time Adaptation of Language ModelsTest-Time Scaling

  13. Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

    Oct 26, 2026Qiaomu Li, Ying XieLLM RerankingRAG Evaluation

  14. SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management

    Oct 22, 2026Zihe Song, Wenting Zhao, Ye Liu +4Tool-Using Language AgentsLLM Inference Efficiency

  15. Amplifying Latency and Cost of LLM Inference via Rowhammer Attack

    Oct 22, 2026Mrityunjay Shukla, Shubham Roy, Biswabandan PandaLLM SecurityLLM Inference Efficiency

  16. When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding

    Oct 18, 2026Yongchao Long, Chunrui Zheng, Yuxuan Zhou +10Speculative DecodingLLM Inference Efficiency

  17. CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests

    Oct 15, 2026Lo Gullstrand Heander, Markus Borg, Emma Söderberg +1Software EngineeringAutomated Code Review

  18. JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

    Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent HarnessesLLM Agents

  19. Jev-LiteMAS: Separating Decisions from Generation in Multi-Agent Workflows

    Oct 15, 2026Kaixiang Wang, Zhaijiacheng Zhou, Jiong Lou +1Multi-Agent LLM SystemsLLM Agent Orchestration

  20. When Is Smaller Greener? Right-Sizing Language Models for Agentic AI

    Oct 14, 2026Sajid Siraj, Maria BarbatiLLM Agent Workflow OptimizationLLM Inference Efficiency