LLM Inference Efficiency

LLM: Large Language Model

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

352 papers40 papers in the last four weeks, down 30% on the four weeks before. 0.2% of all new papers.

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling
May 8, 2026Tong Zheng, Haolin Liu, Chengsong Huang +10LLM Inference Efficiency Automated Algorithm Discovery
UMD · UVA · WUSTL +3
VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection
May 8, 2026James Petullo, Sonny George, Dylan Cashman +1LLM Inference Efficiency Inference-Time Scaling
Computer Science Department, Brandeis University
Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation
May 8, 2026Joon Ha Kim, Geon-Woo Kim, Anoop Rachakonda +1LLM Inference Efficiency LLM Inference
University of Texas at Austin
Learning Agent Routing From Early Experience
May 8, 2026Yimin Wang, Jiahao Qiu, Xuan Qi +6LLM Inference Efficiency LLM Agent Routing
University of Michigan · Shanghai Jiao Tong University · AI Lab, Princeton University +3
Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management
May 7, 2026Haoyu Zheng, Fangcheng Fu, Jia Wu +6LLM Inference Efficiency KV Caching
Wuhan University · Shanghai Jiao Tong University · Macquarie University +2
Towards Generation-Efficient Uncertainty Estimation in Large Language Models
May 7, 2026Mingcheng Zhu, Yu Liu, Tingting ZhuLLM Inference Efficiency Uncertainty Quantification
University of Oxford
One Pool, Two Caches: Adaptive HBM Partitioning for Accelerating Generative Recommender Serving
May 6, 2026Wenjun Yu, Shuguang Han, Amelie Chi ZhouLLM Inference Efficiency GPU Acceleration
Hong Kong Baptist University · Hong Kong, China · Alibaba Inc +1
Adaptive Consensus in LLM Ensembles via Sequential Evidence Accumulation: Automatic Budget Identification and Calibrated Commit Signals
May 5, 2026Roberto E. MedinaLLM Inference Efficiency Multi-Agent Consensus
Independent Researcher
Tile-Level Activation Overlap for Efficient LLM Inference
May 5, 2026Abhinav Jangda, Tyler Sorensen, Sebastian Burckhardt +3LLM Inference Efficiency GPU Kernel Optimization
Microsoft Research · Microsoft
Human-Less LLM Serving: Quantifying the Human Tax on Throughput
May 3, 2026Jianhui Lian, Li Chen, Dan Li +1LLM Inference Efficiency LLM Serving
Tsinghua SIGS · Beijing Zhongguancun Laboratory · Tsinghua University
AgentStop: Terminating Local AI Agents Early to Save Energy in Consumer Devices
May 1, 2026Dzung Pham, Kleomenis Katevas, Ali Shahin Shamsabadi +1LLM Inference Efficiency On-Device Language Model Inference
University of Massachusetts Amherst Amherst, Massachusetts, USA · Brave Software London, UK · Brave Software, Imperial College London London, UK
SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusters
May 1, 2026Dongxin Guo, Jikun Wu, Siu Ming YiuLLM Inference Efficiency LLM Agent Orchestration
Silicon Showdown: Performance, Efficiency, and Ecosystem Barriers in Consumer-Grade LLM Inference
May 1, 2026Abdurrahman Javat, Allan KazakovLLM Inference Efficiency LLM Quantization
Bahcesehir University, Istanbul, Türkiye
Agent Capsules: Quality-Gated Granularity Control for Multi-Agent LLM Pipelines
May 1, 2026Aninda RayLLM Inference Efficiency Multi-Agent LLM Systems
Context Recycling for Long-Horizon LLM Inference
May 1, 2026Derek ThomasLLM Inference Efficiency Long-Context Language Model Inference
Independent Researcher · contextforge
LLM-Guided Runtime Parameter Optimization for Energy-Efficient Model Inference
Apr 29, 2026Katelyn Crumpacker, Dimitrios NikolopoulosLLM Inference Efficiency Energy-Efficient ML
Virginia Polytechnic and State University
Evergreen: Efficient Claim Verification for Semantic Aggregates
Apr 28, 2026Alexander W. Lee, Benjamin Han, Shayak Sen +3LLM Inference Efficiency Claim Verification
Brown University · Snowflake Inc.
Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving
Apr 28, 2026Shan Yu, Junyi Shu, Yuanjiang Ni +14LLM Inference Efficiency Multi-Agent LLM Systems
UCLA · Alibaba Cloud Computing · Alibaba Group +6
PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
Apr 27, 2026Ishan Patel, Ishan JoshiLLM Inference Efficiency Multi-Agent LLM Systems
Independent Researcher
GLM-5 Serving Parameter Tuning for OpenClaw: Single-Deployment MaaS Inference Optimization for Long-Context Agent Workloads
Apr 27, 2026Minjie Hua, Ning Wang, Peijun Yang +2LLM Inference Efficiency LLM Serving
China Unicom
Latency and Cost of Multi-Agent Intelligent Tutoring at Scale
Apr 27, 2026Iizalaarab Elhaimeur, Nikos ChrisochoidesLLM Inference Efficiency Multi-Agent LLM Systems
Center for Real-Time Computing · Computer Science Department · Old Dominion University +2
Scalable LLM-based Coding of Dialogue in Healthcare Simulation: Balancing Coding Performance, Processing Time, and Environmental Impact
Apr 25, 2026Kiyoshige Garces, Gloria Milena Fernandez-Nieto, Linxuan Zhao +4LLM Inference Efficiency LLM Prompting
RMIT University · Monash University · Adelaide University +2
PExA: Parallel Exploration Agent for Complex Text-to-SQL
Apr 24, 2026Tanmay Parekh, Ella Hofmann-Coyle, Shuyi Wang +3LLM Inference Efficiency Text-to-SQL
University of California Los Angeles · Bloomberg
EverydayGPT: Confidence-Gated Routing for Efficient and Safe Hybrid GPT-RAG Conversational QA
Apr 24, 2026Jaspreet Singh NahalLLM Inference Efficiency Retrieval-Augmented Generation
B.Tech Computer Science Engineering (Artificial Intelligence) Dr. A.P.J. Abdul Kalam Technical University, Delhi, India
Sub-Token Routing for KV Cache Compression
Apr 23, 2026Wei Jiang, Wei WangLLM Inference Efficiency KV Caching
Futurewei Technologies San Jose, CA 95131, USA
TRACES: Tagging Reasoning Steps for Adaptive Cost-Efficient Early-Stopping
Apr 22, 2026Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo +1Reasoning Trace Analysis Early Stopping
IBM Research Europe · Trinity College Dublin · Dublin, Ireland +1
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
Apr 22, 2026Fei Zuo, Zikang Zhou, Hao Cong +2LLM Inference Efficiency Mixed-Precision Quantization
1BA TechWorks (BMW Group) · 2National University of Singapore · 3Tsinghua University
Are Large Language Models Economically Viable for Industry Deployment?
Apr 21, 2026Abdullah Mohammad, Sushant Kumar Ray, Pushkar Arora +5LLM Inference Efficiency LLM Evaluation
DSEU-Okhla, New Delhi, India · University of Delhi, New Delhi, India · 6, 8Macquarie University, Sydney, Australia +1
How Much Thinking is Enough? Quantifying and Understanding Redundancy in LLM Reasoning
Apr 21, 2026Zhiyuan Zhai, Xinkai You, Wenjing Yan +1LLM Inference Efficiency RL for Language Model Reasoning
Fudan University · The Chinese University of Hong Kong
Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLM
Apr 20, 2026Hang Zeng, Xiangyu Liu, Yong Hu +5LLM Inference Efficiency Small Language Models
Shanghai Jiao Tong University, Shanghai, China · WeChat Tencent, Beijing, China · State University of New York at Buffalo, New York, United States

May 8, 2026J/K move · Enter open · S save

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Tong Zheng, Haolin Liu, Chengsong Huang, Huiwen Bao+9

UMD · UVA · WUSTL+3

LLM Inference Efficiency Automated Algorithm Discovery Test-Time Optimization Large Language Model-Guided Optimization Test-Time Scaling

Open paper AskPDF arXiv

Test-time scaling (TTS) has become an effective approach for improving large language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework, AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width--depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy--cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only $39.9 and 160 minutes. Our data, and code will be open-source at https://github.com/zhengkid/AutoTTS.