LLM Inference Efficiency

LLM: Large Language Model

The field studies how to serve large language models faster and more cheaply while preserving answer quality, especially for long contexts, reasoning, and multi-step agents. Systems approaches improve batching, scheduling, speculative decoding, and key-value cache storage through methods such as quantization, eviction, and reuse. Model-side approaches adapt reasoning budgets, route queries between models, and compress prompts or context to avoid unnecessary computation. A central challenge is that gains depend on the model, workload, hardware, and latency or energy goals, so evaluations increasingly compare quality, cost, latency, and energy together on tasks such as LongBench, GSM8K, and MATH.

Latest papers

352 papers40 papers in the last four weeks, down 30% on the four weeks before. 0.2% of all new papers.

Compute Has Shape: Self-Morphing the Internal Computation of Frozen LLMs
Oct 28, 2026Junxi Wu, Yiwen Guo, Shu-Tao XiaInference-Time Adaptation of Language Models Test-Time Scaling
School of Statistics and Data Science, Nankai University · Tsinghua Shenzhen International Graduate School, Tsinghua University · Independent Researcher
Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation
Oct 26, 2026Qiaomu Li, Ying XieLLM Reranking RAG Evaluation
College of Computing and Software Engineering, Kennesaw State University, Marietta, GA, USA
Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations
Oct 23, 2026Jiantang HuangStructured Output Generation Code Generation
Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?
Oct 23, 2026Yepeng Liu, Shengmai Chen, Xuandong Zhao +4Test-Time Adaptation Large Language Model-Guided Scientific Discovery
UC Santa Barbara · UC Berkeley
SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management
Oct 22, 2026Zihe Song, Wenting Zhao, Ye Liu +4Tool-Using Language Agents LLM Inference Efficiency
Department of Computer Science, University of Illinois Chicago · Salesforce Research
Amplifying Latency and Cost of LLM Inference via Rowhammer Attack
Oct 22, 2026Mrityunjay Shukla, Shubham Roy, Biswabandan PandaLLM Security LLM Inference Efficiency
Indian Institute of Technology Bombay, Mumbai, India
When Does Better Acceptance Pay? A Controlled Study of Speculative Decoding
Oct 18, 2026Yongchao Long, Chunrui Zheng, Yuxuan Zhou +10Speculative Decoding LLM Inference Efficiency
Department of Computer Science, Tianjin University of Technology · Baidu Inc. · School of Information Science and Engineering, Shandong University +3
Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits
Oct 16, 2026Shivam Mathur, Sri Harish Popuri, Priya Krishnan Sundararajan +2Text Classification LLM Inference Efficiency
Walmart Inc.
Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
Oct 16, 2026Chiatzen Wang, Bill Xu, Haochen Gu +1Speculative Decoding LLM Inference Efficiency
University of Waterloo
CodeHealth Predicts the Token Cost of Agentic Code Review: A Pre-Registered Study of 465 Agent-Authored Pull Requests
Oct 15, 2026Lo Gullstrand Heander, Markus Borg, Emma Söderberg +1Software Engineering Automated Code Review
Lund University, Lund, Sweden · CodeScene AB, Malmö, Sweden
JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs
Oct 15, 2026Kunlun Zhu, Tao Feng, Xu Han +5LLM Agent Harnesses LLM Agents
University of Illinois Urbana-Champaign · Princeton University · Google
Jev-LiteMAS: Separating Decisions from Generation in Multi-Agent Workflows
Oct 15, 2026Kaixiang Wang, Zhaijiacheng Zhou, Jiong Lou +1Multi-Agent LLM Systems LLM Agent Orchestration
Shanghai Jiao Tong University, Shanghai, China
Planning Commitment of LLM under Partial Observability
Oct 15, 2026Cassie Huang, Zhanwei Cao, Yeawon Lee +6LLM Planning LLM Agent Safety
Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference
Oct 14, 2026Danyang Zhao, Sendong Zhao, Yanrui Du +6Adaptive Retrieval Evidence Retrieval
Harbin Institute of Technology
When Is Smaller Greener? Right-Sizing Language Models for Agentic AI
Oct 14, 2026Sajid Siraj, Maria BarbatiLLM Agent Workflow Optimization LLM Inference Efficiency
Leeds University Business School, University of Leeds, Leeds, United Kingdom · Ca’Foscari University of Venice, Venice, Italy
Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion
Oct 13, 2026Zachary Broveak, Christian Kubicka, Thierry TambeMixture-of-Experts Language Models LLM Inference Efficiency
Stanford University
Towards Green AI: Evaluating Energy Efficiency for Quantized LLM Inference
Oct 13, 2026Petar BastaLLM Inference Efficiency LLM Quantization
University of Waterloo Computer Science Waterloo, Ontario, Canada
Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuning
Oct 11, 2026Xingke Yang, Sicong Li, Lingfeng Yao +4Personalized Language Models LLM Inference Efficiency
University of Houston, USA · Independent Researcher, USA
Measuring the Inference Energy Efficiency of Open-Weight LLMs across Benchmarks, Reasoning Modes, and Hardware
Oct 9, 2026Zeyu Yang, Wes ArmourEnergy-Aware LLM Inference LLM Inference Efficiency
Department of Engineering Science, University of Oxford
When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows
Oct 9, 2026Yuchen Ling, Shengcheng Yu, Zhenyu Chen +1LLM Agent Workflow Optimization LLM Agent Reliability
Nanjing University · Technical University of Munich
Attention via Black-Box Vector Search
Oct 7, 2026Stepan Zharkov, Krish Singal, Ashwin Padaki +1Attention Approximation Efficient Attention
University of Pennsylvania
ReFold: Training-Free Reversible Inter-Turn Context Folding for Long-Horizon Agents
Oct 6, 2026Yupeng Su, Jiayi Tian, Zheng Zhang +1LLM Inference Efficiency Long-Context Language Model Inference
University of California, Santa Barbara · Intel
RA-MoWE: Workflow-Affinity Embeddings for Query Clustering and Agentic Workflow Generation
Oct 6, 2026Qi Cheng, Shengyu Chen, Wei Cheng +4LLM Agent Workflow Optimization Agentic Workflows
Rutgers University New Brunswick, NJ · University of Pittsburgh Pittsburgh, PA · NEC Labs America Princeton, NJ +1
Persistent Memory in Multi-Agent LLM Inference: What It Costs, What It Buys, and When You Can Tell
Oct 6, 2026Hochan Son, Kyungdoe Han, Jaehan Koh +3LLM Inference Efficiency Multi-Agent LLM Systems
University of California, Los Angeles · University of Wisconsin · HCLTech America
Monte Carlo Estimation for KV Cache Eviction
Oct 6, 2026Ahsan Bilal, Muhammad Ahmed Mohsin, Muhammad Umer +5LLM Inference Efficiency KV-Cache Eviction
University of Oklahoma · Stanford University · University of Houston +3
Evaluating Inference Compute for Generative AI: A Framework for Enterprise Workloads
Oct 5, 2026Abbas Raza Ali, Muhammad Ajmal Siddiqui, Moona ZahidLLM Inference Efficiency Disaggregated LLM Serving
Citigroup Inc., London, United Kingdom · Ernst & Young LLP, London, United Kingdom · NVIDIA Corporation, London, United Kingdom
Expanding LLM Reasoning
Oct 4, 2026Rian Atri, Evan LuoLLM Inference Efficiency Inference-Time Scaling
Keiji AI · University of California, Berkeley
Engineering Sustainable Agents: A Systematic Comparison of Agentic LLMs for Developer Workflows
Oct 2, 2026Merve Astekin, Yan Naing Tun, Arda Goknil +5Software Engineering Agents LLM Agent Evaluation
SINTEF, Oslo, Norway · Singapore Management University, Singapore, Singapore · Ozyegin University, Istanbul, Turkiye
What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute
Oct 1, 2026Zhixu Du, Weijia Han, Hai Helen Li +1LLM Inference Efficiency LLM Inference Acceleration
Duke University · University of Washington
Beyond State-of-the-Art: Standardising Environmental Impact Metrics for AI Research
Oct 1, 2026Lachlan McGinness, Dan Pagendam, Robert OffnerLLM Inference Efficiency Energy-Efficient ML
School of Computing, Australian National University · Commonwealth Scientific and Industrial Research Organisation

Oct 28, 2026J/K move · Enter open · S save

Compute Has Shape: Self-Morphing the Internal Computation of Frozen LLMs

Junxi Wu, Yiwen Guo, Shu-Tao Xia

School of Statistics and Data Science, Nankai University · Tsinghua Shenzhen International Graduate School, Tsinghua University · Independent Researcher

Inference-Time Adaptation of Language Models Test-Time Scaling LLM Inference Efficiency

Open paper AskPDF arXiv

Test-time scaling improves large language models through additional computation, but how to adapt the topology of internal computation to individual inputs remains underexplored. Existing approaches largely fix recurrent or parallel architectures during training, or reconfigure frozen models along a single structural axis. We introduce SelfMorph, which makes computational shape an input-dependent inference-time decision. It unifies the original forward path (BASE), serial reuse of a layer span (REPEAT), and local branching and merging of perturbed hidden states (BRANCH) in a unified space parameterized by type, location, and intensity. Since they offer complementary benefits rather than uniform dominance, we use a lightweight selector that reads the backbone's depth-wise hidden state trajectory on the prompt to choose a morphology before response generation. We train the selector by distilling utility-based distribution from offline morphing that jointly account for task performance and computational cost. Evaluated on three LLMs spanning 4B-27B, SelfMorph achieves a 8.06% mean relative improvement on five in-distribution benchmarks and 4.45% on four unseen benchmarks. It achieves the strongest performance in 24 of 27 settings. These findings identify input-adaptive computational topology as a complementary axis of test-time scaling beyond generating more tokens. Our code is available at https://github.com/creator-xi/SelfMorph.