LLM Inference Efficiency

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

CardsList

352 papers40 papers in the last four weeks, down 30% on the four weeks before. 0.2% of all new papers.

  1. Graph-Based Self-Healing Tool Routing for Cost-Efficient LLM Agents

    Mar 2, 2026Neeraj BholaniLLM Agent OrchestrationLLM Agent Routing

  2. GroupGPT: A Token-efficient and Privacy-preserving Agentic Framework for Multi-User Chat Assistant

    Mar 1, 2026Zhuokang Shen, Yifan Wang, Hanyu Chen +6Conversational AgentsLLM Inference Efficiency

    East China Normal University Shanghai, China · University of Nottingham Ningbo Ningbo, China · Xiamen University Xiamen, China

  3. Tiny-Critic RAG: Empowering Agentic Fallback with Parameter-Efficient Small Language Models

    Mar 1, 2026Yichao Wu, Penghao Liang, Yafei Xiang +5Agentic RAGLLM Routing

    Northeastern University Boston, USA, 02115 · Independent Researcher Milpitas, USA, 95035 · Independent Researcher Austin, USA, 78613 +2

  4. Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

    Feb 27, 2026Ferran Agullo, Joan Oliveras, Chen Wang +5LLM Inference EfficiencyLLM Serving

    Barcelona Supercomputing Center (BSC), Barcelona, Spain · Universitat Politècnica de Catalunya - BarcelonaTech (UPC), Barcelona, Spain · IBM Research, New York, USA

  5. Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study

    Feb 27, 2026Athos GeorgiouLLM Inference EfficiencyLLM Inference

  6. Replacing Multi-Step Assembly of Data Preparation Pipelines with One-Step LLM Pipeline Generation for Table QA

    Feb 26, 2026Fengyu Li, Junhao Zhu, Kaishi Song +4Table QALLM Post-Training

    Zhejiang University Hangzhou, China · Northeastern University Shenyang, China · Aalborg University Aalborg, Denmark

  7. Sustainable LLM Inference using Context-Aware Model Switching

    Feb 25, 2026Yuvarani, Akashdeep Singh, Zahra Fathanah +3LLM Inference EfficiencyLLM Inference

    Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia

  8. To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

    Feb 23, 2026Zaifu Zhan, Min Zeng, Shuang Zhou +6CoT ReasoningEfficient Language Model Reasoning

    Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States · Department of Computer Science and Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States +1

  9. ISO-Bench: Can Coding Agents Optimize Real-World Inference Workloads?

    Feb 23, 2026Ayush Nangia, Shikhar Mishra, Aman Gokrani +1Coding Agent BenchmarksAI Coding Agents

  10. AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems

    Feb 22, 2026Emmanuel BamideleMemory Management for LLM AgentsAgent Memory Management

    Department of Computer Science, Georgia Institute of Technology

  11. RPU -- A Reasoning Processing Unit

    Feb 20, 2026Matthew Adiletta, Gu-Yeon Wei, David BrooksAI Accelerator InferenceLLM Inference Efficiency

    Harvard University

  12. Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

    Feb 17, 2026Yakov Pyotr ShkolnikovMemory Management for LLM AgentsLLM Agent Memory

  13. Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

    Feb 13, 2026Wei-Lin Chen, Liqian Peng, Tian Tan +5LLM Inference EfficiencyInference-Time Scaling

    University of Virginia · Google

  14. Characterizing WebGPU Dispatch Overhead for LLM Inference Across Four GPU Vendors, Three Backends, and Three Browsers

    Feb 9, 2026Jędrzej MaczanLLM Inference Efficiency

    Independent Researcher

  15. More Bang for the Buck: Improving the Inference of Large Language Models at a Fixed Budget using Reset and Discard (ReD)

    Jan 29, 2026Sagi Meir, Tommer D. Keidar, Noam Levi +2LLM Inference EfficiencyLLM Evaluation

    School of Chemistry, Tel Aviv University, Tel Aviv 6997801, Israel · The Center for Physics and Chemistry of Living Systems, Tel Aviv University, Tel Aviv 6997801, Israel · School of Physics and Astronomy, Tel Aviv University, Tel Aviv 6997801, Israel +1

  16. Nalar: Workflow-Aware Management of Agentic Applications

    Jan 8, 2026Saurabh Agarwal, Marco Laju, Donghyun Son +4LLM Inference EfficiencyLLM Agent Orchestration

    UT-Austin · Cisco-Research

  17. Step-Tagging: Toward controlling the generation of Language Reasoning Models through step monitoring

    Dec 16, 2025Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1Reasoning Trace AnalysisEfficient Language Model Reasoning

    1IBM Research Europe, Dublin, Ireland · 2Trinity College Dublin, Ireland · 3ADAPT Research Centre, Dublin, Ireland

  18. Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

    Nov 11, 2025Jon Saad-Falcon, Avanika Narayan, Hakki Orhun Akengin +12LLM Inference EfficiencyAI Accelerator Inference

    Stanford University · Together AI

  19. Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm

    Sep 28, 2025Kaisen Yang, Tinghe Zhang, Rushi Shah +4LLM ReasoningStructured Reasoning

    1Qizhi Institute · 2Dept. of Computer Science, Tsinghua University · College of AI, Tsinghua University +4

  20. EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning

    May 22, 2025Jiawei Liu, Qisi Chen, Jianshu Zhang +2LLM Inference EfficiencySemantic Textual Similarity

    School of Computer Science and Technology, University of Science and Technology of China · Department of Data Science, City University of Hong Kong · iFLYTEK Research

  21. Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Models

    Mar 31, 2025Rui Wang, Hongru Wang, Boyang Xue +9LLM Inference EfficiencyCost-Aware Inference

    University of Edinburgh, Edinburgh, UK · University of Macau, Macau SAR, China · University of Illinois Urbana-Champaign, USA

  22. Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies

    Nov 12, 2024Kyoungmin Kim, Jiacheng Li, Kijae Hong +3LLM Inference EfficiencyLLM Inference

    EPFL, Switzerland · NUS, Singapore · POSTECH, South Korea