LLM Inference Efficiency

LLM: Large Language Model

Ask about this topic

Overview

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

CardsList

352 papers40 papers in the last four weeks, down 30% on the four weeks before. 0.2% of all new papers.

  1. SLO-Guard: Crash-Aware, Budget-Consistent Autotuning for SLO-Constrained LLM Serving

    Apr 19, 2026Christian LysenstøenLLM Inference EfficiencyLLM Serving

    Inland Norway University of Applied Sciences · Visiting Student, University of California, Berkeley

  2. ONTO: A Token-Efficient Columnar Notation for LLM Input Optimization

    Apr 19, 2026Harshavardhanan DeekeswarLLM Inference EfficiencyLLM Input Optimization

    Independent Researcher · Chennai, India

  3. Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

    Apr 19, 2026Raman Saparkhan, Majd Hawasly, Md Rizwan Parvez +1LLM Inference EfficiencySelf-Consistency Decoding

    Carnegie Mellon University Qatar · Qatar Computing Research Institute

  4. Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling

    Apr 19, 2026Zizhang Luo, Yuhao Luo, Youwei Xiao +3LLM Inference EfficiencyMulti-Agent LLM Systems

    Peking University Beijing, China

  5. HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads

    Apr 18, 2026Justice Owusu Agyemang, Jerry John Kponyo, Obed Kwasi Somuah +3LLM Inference EfficiencyAI Coding Agents

    Sperix Labs · VIA Cybersecurity Lab, KNUST · Quantum and Assistive Technologies Lab, KNUST

  6. When Agents Go Quiet: Output Generation Capacity and Format-Cost Separation for LLM Document Synthesis

    Apr 17, 2026Justice Owusu Agyemang, Michael Agyare, Miriam Kobbinah +2LLM Inference EfficiencyLong-Form Document Generation

    Sperix Labs · GCTU, Ghana · NCA, Ghana +1

  7. Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents under Self-Consistency with Early Termination and Model Hotswap

    Apr 16, 2026Naryeong Kim, Shin YooLLM Inference Efficiency

    KAIST · Daejeon, Republic of Korea

  8. TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification

    Apr 16, 2026Adam RidaNeural Surrogate ModelingLLM Inference Efficiency

    Independent Researcher.

  9. PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Search

    Apr 15, 2026Tianhao Qian, Jiayu Chen, Zhenyu Sun +1LLM Inference EfficiencyInference-Time Search

    The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · TikTok

  10. Local-Splitter: A Measurement Study of Seven Tactics for Reducing Cloud LLM Token Usage on Coding-Agent Workloads

    Apr 14, 2026Justice Owusu Agyemang, Jerry John Kponyo, Elliot Amponsah +2LLM Agent Workflow OptimizationLLM Inference Efficiency

    Sperix Labs · VIA Cybersecurity Lab, KNUST · Quantum and Assistive Technologies Lab, KNUST

  11. PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Serving

    Apr 14, 2026Xu Bai, Muhammed Tawfiqul Islam, Chen Wang +1LLM Inference EfficiencyLLM Serving

    DisNet Lab University of Melbourne Melbourne, VIC, Australia · IBM Research Yorktown Heights, NY, USA

  12. NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

    Apr 13, 2026Shizheng Hou, Wenqi Pei, Nuo Chen +3Text-to-SQLBenchmark Design

    National University of Singapore · Zhejiang University

  13. Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures

    Apr 10, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezEnergy-Aware LLM InferenceLLM Inference Efficiency

    Universidad Politécnica de Madrid, C. de los Ciruelos, Boadilla del Monte, 28660, Madrid, Spain · Zurich University of Applied Sciences, Switzerland, Gertrudstrasse 15, Winterthur, 8400, Zurich, Switzerland

  14. ExecTune: Effective Steering of Black-Box LLMs with Guide Models

    Apr 9, 2026Vijay Lingam, Aditya Golatkar, Anwesan Pal +6Language Model SteeringLLM Inference Efficiency

    AWS AI

  15. Don't Overthink It: Inter-Rollout Action Agreement as a Free Adaptive-Compute Signal for LLM Agents

    Apr 9, 2026Khushal SethiInference-Time Compute AllocationLLM Inference Efficiency

    Stanford University

  16. Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving

    Apr 9, 2026Xunzhuo Liu, Bowei He, Xue Liu +3LLM ServingLLM Inference Efficiency

    vLLM Semantic Router Project · MBZUAI · McGill University +2

  17. Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey

    Apr 9, 2026Xiaojing Chen, Haiqi Yu, Wei Ni +5Energy-Efficient MLLLM Inference Efficiency

    Shanghai University, Shanghai, China · Nanyang Technological University, Singapore · Edith Cowan University, Perth, Australia +2

  18. ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving

    Apr 8, 2026Xiangchen Li, Saeid Ghafouri, Jiakun Fan +3LLM Inference EfficiencyDisaggregated LLM Serving

    Virginia Tech, Blacksburg, Virginia, USA · Queen’s University Belfast, Belfast, Northern Ireland, UK

  19. Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC

    Apr 8, 2026Mohammad Siavashi, Mariano Scazzariello, Gerald Q. Maguire +2LLM ServingLLM Inference

    KTH Royal Institute of Technology, Stockholm, Sweden · RISE, Stockholm, Sweden

  20. Agentic Compilation: Mitigating the LLM Rerun Crisis for Minimized-Inference-Cost Web Automation

    Apr 8, 2026Jagadeesh ChundruWeb AgentsLLM Agent Workflow Optimization

    Selfotix

  21. ReDAct: Uncertainty-Aware Deferral for LLM Agents

    Apr 8, 2026Dzianis Piatrashyn, Nikita Kotelevskii, Kirill Grishchenkov +7LLM AgentsLLM Agent Reliability

    Mohamed bin Zayed University of Artificial Intelligence · Ocapital · National University of Science and Technology (NUST) MISIS +3

  22. JTON: A Token-Efficient JSON Superset with Zen Grid Tabular Encoding for Large Language Models

    Apr 7, 2026Gowthamkumar NandakishoreToken EfficiencyLLM Inference Efficiency

  23. HYVE: Hybrid Views for LLM Context Engineering over Machine Data

    Apr 7, 2026Jian Tan, Fan Bu, Yuqing Gao +5LLM Context ManagementLLM Inference Efficiency

    Cisco Systems, Inc., San Jose, California, USA

  24. Chinese Language Is Not More Efficient Than English in Vibe Coding: A Preliminary Study on Token Cost and Problem-Solving Rate

    Apr 6, 2026Simiao Ren, Xingyu Shen, Yuchen Zhou +3LLM Inference EfficiencyMultilingual Code Generation

    Tsang · ∗,†Scam.ai

  25. Comparative Characterization of KV Cache Management Strategies for LLM Inference

    Apr 6, 2026Oteo Mamo, Olga Kogiou, Hyunjin Yi +1KV-Cache ManagementLLM Inference Efficiency

    Florida State University, Tallahassee, FL, USA

  26. Researchers waste 80% of LLM annotation costs by classifying one text at a time

    Apr 4, 2026Christian Pipal, Eva-Maria Vogel, Morgan Wack +1LLM-Assisted AnnotationLarge Language Model-Based Text Classification

    Department of Communication and Media Research (IKMZ), University of Zurich

  27. Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference

    Apr 3, 2026Cornelius Kummer, Lena Jurkschat, Michael Färber +1Prompt CompressionLLM Inference Efficiency

    ScaDS.AI Dresden/Leipzig, CIDS, TU Dresden

  28. Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy

    Apr 3, 2026Yihong Dong, Jianha Xiao, Xue Jiang +7LLM EvaluationEfficient Language Model Reasoning

    Peking University, China · Wuhan University, China

  29. Apriel-Reasoner: RL Post-Training for General-Purpose and Efficient Reasoning

    Apr 2, 2026Rafael Pardinas, Ehsan Kamalloo, David Vazquez +1LLM Inference EfficiencyRL for Language Model Reasoning

    ServiceNow AI Research

  30. Analysis of LLM Performance on AWS Bedrock: Receipt-item Categorisation Case Study

    Apr 2, 2026Gabby Sanchez, Sneha Oommen, Cassandra T. Britto +3LLM EvaluationClassification

    RMIT University, Melbourne, Australia