LLM Inference Efficiency

LLM: Large Language Model

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

352 papers40 papers in the last four weeks, down 30% on the four weeks before. 0.2% of all new papers.

The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Products
Mar 31, 2026Alicia Bao, Jiamian He, Angel Hsu +3LLM Inference Efficiency LLM Inference
James · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC 27516 · Arboretica, Rotterdam, The Netherlands +2
Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routing
Mar 30, 2026Alessio LangiuPrivacy Protection for LLM Agents Privacy Leakage in Language Models
National Research Council of Italy - Institute of Marine Sciences (CNR-ISMAR)
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference Efficiency LLM Evaluation
Huawei Switzerland · ACS Lab, Huawei Technologies
Runtime Burden Allocation for Structured LLM Routing in Agentic Expert Systems: A Full-Factorial Cross-Backend Methodology
Mar 26, 2026Zhou Hanlin, Chan Huah YongLLM Routing LLM Evaluation
School of Computer Sciences, Universiti Sains Malaysia (USM), Penang, Malaysia · Xiamen Software Vocational & Technical College, Xiamen, China
EcoThink: A Green Adaptive Inference Framework for Sustainable and Accessible Agents
Mar 26, 2026Linxiao Li, Zhixiang LuLLM Inference Efficiency Energy-Efficient ML
The University of Sydney Sydney, Australia · University of Liverpool Liverpool, United Kingdom
LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load
Mar 24, 2026Pranay Tummalapalli, Sahil Arayakandy, Ritam Pal +1On-Device Language Model Inference LLM Inference Efficiency
Conscious Engines
StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving
Mar 24, 2026Azam NouriLLM Serving LLM Inference Efficiency
Department of Science, Technology & Mathematics, Lincoln University, USA
Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
Mar 24, 2026Xunzhuo Liu, Bowei He, Xue Liu +3Memory-Augmented Language Model Agents Conversational Memory
vLLM Semantic Router Project · MBZUAI · McGill University +3
The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
Mar 22, 2026Huamin Chen, Xunzhuo Liu, Bowei He +5LLM Routing LLM Inference Serving
vLLM Semantic Router Project · MBZUAI · McGill University +3
ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models
Mar 22, 2026Haoyu Qiao, Hao Zhang, Shanwen Mao +2Edge-Cloud Collaborative Inference LLM Routing
the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China · the National Key Laboratory of Smart Farm Technologies and Systems, Harbin 150001, China · the Department of Computer Science and Technology, Harbin Institute of Technology (ShenZhen), ShenZhen 518055, China
Clinical Note Bloat Reduction for Efficient LLM Use
Mar 21, 2026Jordan L. Cahoon, Chloe Stanwyck, Asad Aali +6LLM Inference Efficiency Electronic Health Records
Department of Biomedical Data Science, Stanford University, Stanford, CA · Department of Pathology, Stanford University, Stanford, CA · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University, Stanford, CA +6
AE-LLM: Adaptive Efficiency Optimization for Large Language Models
Mar 20, 2026Kaito Tanaka, Masato Ito, Yuji Nishimura +2LLM Inference Efficiency Energy-Efficient ML
SANNO University
KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
Mar 20, 2026Yichun Xu, Navjot K. Khaira, Tejinder SinghKV-Cache Management LLM Inference Efficiency
Dell Technologies, Hopkinton, MA 01748, USA · Dell Technologies, Santa Clara, CA 95054, USA
PoC: Performance-oriented Context Compression for Large Language Models via Performance Prediction
Mar 20, 2026Runsong Zhao, Shilei Liu, Jiwei Tang +8Language Model Performance Prediction LLM Context Compression
Northeastern University, China · Future Living Lab of Alibaba · Tsinghua University
RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference
Mar 18, 2026Arpit Singh Gautam, Saurabh JhaLLM Quantization Neural Network Quantization
Resource Consumption Threats in Large Language Models
Mar 17, 2026Yuanhe Zhang, Xinyue Wang, Zhican Chen +8LLM Security LLM Inference Efficiency
Beijing University of Posts and Telecommunications · Hangzhou Dianzi University · Nanyang Technological University +1
Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
Mar 13, 2026Huamin Chen, Xunzhuo Liu, Junchen Jiang +2LLM Inference Efficiency LLM Routing
vLLM Semantic Router Project · Tensormesh Inc / UChicago · MBZUAI / McGill University
TaxBreak: Unmasking the Hidden Costs of LLM Inference Through Overhead Decomposition
Mar 12, 2026Prabhu Vellaisamy, Shreesh Tripathi, Vignesh Natarajan +3LLM Inference Efficiency LLM Inference
NeuroAI Computer Architecture Lab (NCAL), Electrical and Computer Engineering Department, Carnegie Mellon University
CacheSolidarity: Preventing Prefix Caching Side Channels in Multi-Tenant LLM Serving Systems
Mar 11, 2026Panagiotis Georgios Pennas, Konstantinos Papaioannou, Marco Guarnieri +1LLM Security Prefix Caching
IMDEA Software Institute Universidad Politécnica de Madrid · IMDEA Software Institute
Quantifying the Accuracy and Cost Impact of Design Decisions in Budget-Constrained Agentic LLM Search
Mar 9, 2026Kyle McCleary, James GhawalyAgentic Search Agentic RAG
Division of Computer Science & Engineering, Louisiana State University, Baton Rouge, Louisiana, USA
Ares: Adaptive Reasoning Effort Selection for Efficient LLM Agents
Mar 9, 2026Jingbo Yang, Bairu Hou, Wei Wei +2Efficient Language Model Reasoning LLM Inference Efficiency
Department of Computer Science, University of California, Santa Barbara · Center for Advanced AI, Accenture
The Compression Paradox in LLM Inference: Provider-Dependent Energy Effects of Prompt Compression
Mar 6, 2026Warren JohnsonLLM Inference Efficiency Energy-Aware LLM Inference
Plexor Labs Principal Researcher
Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Mar 6, 2026Warren JohnsonPrompt Compression LLM Inference Efficiency
Plexor Labs Principal Researcher
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Mar 5, 2026Di Zhang, Xun Wu, Shaohan Huang +9LLM Inference Efficiency LLM Quantization
Microsoft Research · Peking University · South China University of Technology
Beyond the Context Window: A Cost-Performance Analysis of Fact-Based Memory vs. Long-Context LLMs for Persistent Agents
Mar 5, 2026Natchanon Pollertlam, Witchayut KornsuwannawitPersistent Memory for Language Models Long-Term Conversational Memory
Bricks Technology
SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference
Mar 5, 2026Luchang Li, Dongfang Li, Bozhao Gong +1Disaggregated LLM Serving Prefill-Decode Disaggregation
Kingsoft Cloud
Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference
Mar 4, 2026Patrick Wilhelm, Thorsten Wittkopp, Odej KaoEnergy-Aware LLM Inference LLM Inference Efficiency
BIFOLD, Technische Universität Berlin, Berlin, Germany
Parallel Test-Time Scaling with Multi-Sequence Verifiers
Mar 3, 2026Yegon Kim, Seungyoo Lee, Chaeyun Jang +2LLM Answer Verification Test-Time Scaling
Graduate School of AI, KAIST, Seoul, South Korea
Same Pieces, Different Servers: A Tetris Benchmark for AI Agents as Served
Mar 2, 2026Haochuan WangLLM Inference Efficiency LLM Agent Evaluation
Massachusetts Institute of Technology

Mar 31, 2026J/K move · Enter open · S save

The Energy Footprint of LLM-Based Environmental Analysis: LLMs and Domain Products

Alicia Bao, Jiamian He, Angel Hsu, Diego Manya+2

James · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC 27516 · Arboretica, Rotterdam, The Netherlands+2

LLM Inference Efficiency LLM Inference Retrieval-Augmented Generation

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly used in domain-specific applications, including climate change and environmental research, understanding their energy footprint has become an important concern. The growing adoption of retrieval-augmented (RAG) systems for climate-domain specific analysis raises a key question: how does the energy consumption of domain-specific RAG workflows compare with that of direct generic LLM usage? Prior research has focused on standalone model calls or coarse token-based estimates, while leaving the energy implications of deployed application workflows insufficiently understood. In this paper, we assess the inference-time energy consumption of two LLM-based climate analysis chatbots (ChatNetZero and ChatNDC) compared to the generic GPT-4o-mini model. We estimate energy use under actual user queries by decomposing each workflow into retrieval, generation, and hallucination-checking components. We also test across different times of day and geographic access locations. Our results show that the energy consumption of domain-specific RAG systems depends strongly on their design. More agentic pipelines substantially increase inference-time energy use, particularly when used for additional accuracy or verification checks, although they may not yield proportional gains in response quality. While more research is needed to further test these initial findings more robustly across models, environments and prompting structures, this study provides a new understanding on how the design of domain-specific LLM products affects both the energy footprint and quality of output.