LLM Serving

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 143% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 142

All topics

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost
Sep 30, 2026Sietse SchelpeLLM Serving Long-Context Language Model Inference
Corbenic AI
Characterizing High Bandwidth Flash for LLM Serving
Sep 30, 2026Zack Yu, Chloe Wong, Coleman Hooper +7LLM Inference Efficiency LLM Serving
University of California, Berkeley Berkeley, California, USA · FuriosaAI Seoul, South Korea · University of California, Berkeley; ICSI; LBNL Berkeley, California, USA
Cascadia: A Control-Plane-Free Alternative to Hyperconverged AI Infrastructure
Sep 30, 2026Matias Parij, Pawan Paudel, Tate Berenbaum +1LLM Serving LLM Inference Scheduling
Not Community Labs Inc. · Intel Corporation
Decode-Latency Feedback Prefill: A Model-Free Controller and Its Generalization Limits
Sep 29, 2026Gaurav Agarwal, Ashish Garg, Isha SinghalLLM Serving LLM Inference
You Cannot Pick a Provider From the Price List: Market-Aware Routing for Open-Weight LLM Inference
Sep 29, 2026Liang He, Jingbo Wen, Yixiong Chen +4LLM Serving Cost-Aware Inference
University of Sydney · Johns Hopkins University · Stanford University +1
SPLASH: Switching Parallel Layouts of Attention with Seamless Handoff for LLM Serving
Sep 29, 2026Chuan Liu, Shuoming Zhang, Zhicheng Li +6LLM Serving Tensor Parallelism
State Key Laboratory of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing
Sep 29, 2026Guannan Lai, Gelin Bian, Hao-Xuan Ma +5LLM Serving Cost-Aware Inference
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · SinapisAI +1
Beyond Energy: When Sustainability Dimensions Reshape LLM Serving Decisions
Sep 28, 2026Tianyao Shi, Xipeng Shen, Yi DingLLM Inference Efficiency LLM Serving
Elmore Family School of Electrical and Computer Engineering, Purdue University, USA
WavePP: High-Throughput Pipeline Parallel LLM Prefill under Prefix Reuse
Sep 28, 2026Aaryam SharmaLLM Inference Efficiency LLM Serving
Baseten
DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Sep 28, 2026Xuan Truong Nguyen, Tien Son Pham, Tuan Duc Chu +2LLM Serving LLM Inference
Department of Next Generation Semiconductor Convergence and Open Sharing System (COSS), Seoul National University, Seoul 08826, South Korea · Van-Lang Institute of Semiconductor Technology (VIST), Hanoi, Vietnam · Efficient Computation Research Group, Moreh Vietnam +1
Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving
Sep 27, 2026Jiantong Jiang, Yue Yang, Peiyu Yang +1LLM Serving KV-Cache Management
University of Melbourne · Maincode
Efficient Iterative Retrieval with Heterogeneous Batching
Sep 21, 2026Dohyun Park, Hubertus Franke, Daniel G. Waddington +2Retrieval-Augmented Generation LLM Serving
University of Illinois Urbana-Champaign · IBM Research
Token Latency Fairness: Performance Isolation for Multi-Tenant LLM Serving
Sep 16, 2026Dev Bali, Soujanya Ponnapalli, Yichuan Wang +3LLM Serving LLM Inference Scheduling
University of California, Berkeley
JustFit: Just-in-Time State Management for Local LLM Serving
Sep 15, 2026Yuhua ChenLLM Serving On-Device Language Model Inference
Shared-Prefix KV Reuse Across Standard LoRA Adapters: Quality and Serving Tradeoffs
Sep 15, 2026Dushyant RajputLLM Inference Efficiency LLM Serving
AltSlate Labs LLP
HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving
Sep 8, 2026Han JinLLM Serving Adversarial Attacks on LLMs
Independent Researcher
Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
Sep 7, 2026Siyu Song, Qi Bai, Jinbo Hao +3LLM Serving LLM Inference Scheduling
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Sun Yat-sen University · School of Computer Engineering, Jiangsu Ocean University
Architecting Conversational Data Systems for Stateless LLM APIs: The Hydration Proxy Pattern
Sep 1, 2026Joseph AxisaLLM Grounding LLM Serving
Google Cloud · California, USA
From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix
Sep 1, 2026Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov +11LLM Serving Instruction Following
T-Tech
Prediction-Assisted Pricing and Admission for LLM APIs with Stochastic Token Consumption
Sep 1, 2026Patrick WongLLM Serving Learning-Augmented Algorithms
The Shadow Price of Intelligence: Quality Degradation in LLM Inference as a Supply Chain Problem
Aug 25, 2026Elioth SanabriaLLM Serving LLM Inference Scheduling
Department of Decision and Technology Analytics Lehigh University - College of Business
Keep, Customize, or Exit: Default Design and Token Pricing in LLM Reasoning Services
Aug 13, 2026Ahmet Bugra Gundogan, Yigit Turkmen, Melih BastopcuLLM Serving Inference-Time Compute Allocation
Department of Electrical and Electronics Engineering
Total Recall at What Cost? Benchmarking the Serving Cost of Agentic Memory Systems
Aug 12, 2026Natchanon Pollertlam, Witchayut KornsuwannawitLLM Serving LLM Agent Memory
Bricks Technology · Thailand
Lifecycle-Optimal Tokenization: Vocabulary Size as a Deployment-Regime-Dependent Infrastructure Parameter
Aug 11, 2026Rima Mittal, Ankit Gubrani, Satyanarayana KakolluLLM Inference Efficiency LLM Serving
LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
Aug 9, 2026Shuowei Jin, Xueshen Liu, Jiaxin Shan +4LLM Serving
University of Michigan · ByteDance
Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4LLM Serving LLM Inference Scheduling
The University of British Columbia · Microsoft Azure Research · NVIDIA
Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving
Aug 4, 2026Xiang Li, Pengcheng Wang, Huazheng Wang +1LLM Inference Efficiency LLM Serving
1Purdue University · 2Oregon State University
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
Aug 4, 2026Haiqiang Zhang, Yuanqing Lei, Wanting Li +2Retrieval-Augmented Generation LLM Serving
ETH Zurich · Columbia University · National University of Singapore
LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs
Aug 4, 2026Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh +1LLM Serving LLM Inference Acceleration
SWAT Lab., Polytechnique Montr´eal, Montr´eal, Canada
LaCache: Robust Semantic Caching for LLM Serving
Aug 3, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1LLM Serving LLM Security
Stony Brook University

Sep 30, 2026, cs.CLJ/K move · Enter open · S save

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

Sietse Schelpe

Corbenic AI

LLM Serving Long-Context Language Model Inference Persistent Memory for Language Models LLM Inference Acceleration KV-Cache Reuse

Open paper AskPDF arXiv

A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify (arXiv:2507.07505). We ask how much of the budget beneath that ceiling is spent on work the model has already done. Serving is stateless across requests: a model that answers a second question about a document recomputes the document's attention state from the first token. On seven real-world datasets, 98.7% of prompt tokens were text the model had already read. We present Galahad, a memory layer for vLLM, SGLang and llama.cpp that makes this reading a one-time cost. Taliesin saves the model's key-value (KV) state for a block of text and loads it on the next request that contains the same bytes, instead of recomputing it. Blaise keeps the documents themselves and passes the model only the section a question needs. On a recall test with 100 facts hidden in a 97,000-token corpus (Gemma 4 31B), Taliesin alone let the model attend to the whole corpus and answered 98 of 100 on llama.cpp at 3.0 s and 572 J per question, against 10 of 100, 9.3 s and 2,754 J for the same model without Galahad, which could hold only the last 12,000 tokens. With Blaise added, the model read about 668 tokens per question and answered 100 of 100 on all three runtimes at 0.59-0.64 s and 200-213 J; a tuned RAGFlow pipeline answered 77. Storing the corpus is a one-time cost of about 100 s and 28 kJ, whose energy is recovered after 13 questions. Restored state is bit-identical: all 262,144 output logits matched after restart, rehydration and hot-load. Galahad worked with all 30 models we tested under vLLM, and it fails closed: any load that does not pass its checks is recomputed. Together these results move LLM serving from stateless to stateful inference.

Momentum

Related topics

Latest papers 142