LLM Serving

LLM: Large Language Model

Overview

Large language model serving studies how to run inference efficiently for many concurrent requests, balancing response speed, throughput, cost, and service-level objectives. Common approaches include continuous batching and prefill–decode scheduling, request routing, and managing or compressing KV caches; newer systems also use disaggregated hardware, speculative decoding, and workload-aware power control. Evaluations use request traces and production-like workloads to measure time to first token, token latency, throughput, quality, and resource use across models and hardware. A central challenge is that workloads and output lengths vary, so policies that improve throughput or cost can worsen tail latency, fairness, or response quality.

Latest papers

CardsList

249 papers37 papers in the last four weeks, up 23% on the four weeks before. 0.3% of all new papers.

  1. LEDGE: An Efficient Simulator for Distributed LLM Serving on Heterogeneous Edge Devices

    Nov 18, 2026Zihao Wang, Ye Tian, Tajana Rosing +1LLM ServingParallel LLM Inference

  2. Measuring and Charging for Retained Context in Long-Context LLM Services

    Nov 18, 2026Yi Lu, Qianshi Lin, Yumeng Yang +1LLM ServingLLM Context Management

  3. Making Preemption Productive in Long-Output LLM Serving

    Nov 15, 2026Ziji Shi, Xinyi Wan, Chaoyi Ruan +3LLM ServingLLM Inference Scheduling

  4. VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling

    Nov 13, 2026Ishaan Datta, Alind Khare, Koushik Kumar Dey +2LLM ServingLLM Inference Scheduling

  5. Superfluid: Local AI Server for Distributed LLM Inference

    Nov 13, 2026Prabod Rathnayaka, Isura Manchanayake, Srikandabala Kogul +3Local LLM DeploymentLLM Serving

  6. Detent: Batch-Invariant LLM Serving and RL Rollouts at Near-Stock Speed

    Nov 7, 2026Peixu Xin, Runchuan Feng, Jiazhen Guo +2LLM ServingLLM Inference

  7. Intent-Aware Caching for Efficient LLM Serving

    Nov 6, 2026Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S. Nikolopoulos +3LLM Inference EfficiencyLLM Serving

  8. LoRazor: Trimming Adapter Execution for Efficient Multi-LoRA Serving

    Nov 5, 2026Xiang Li, Pengcheng Wang, Azam Ikram +1Multi-LoRA InferenceLLM Serving

  9. Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine

    Nov 5, 2026Xindan Zhang, Fei WangLLM Serving

  10. Create Your Own Work Stack: Deploying Self-hosted AI and Tooling in a Tech Startup

    Nov 5, 2026Saad Ahmed Bazaz, Talha Amjad, Abdurrehman Subhani +2Local LLM DeploymentCost-Aware Inference

  11. FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving

    Nov 5, 2026Xuan Wang, Zhuanhao Wu, Tingzhu Fang +17KV-Cache OffloadingLLM Serving

  12. ServerBench: How can your coding agents design specialized LLM serving systems?

    Nov 4, 2026Xinrui Zhong, Zhiming Wang, Yongqi Liu +5LLM ServingCoding Agent Benchmarks

  13. A Controlled Experimental Evaluation of Configuration and Cache Architecture in LLM Serving Engines

    Nov 4, 2026Muhammad Faizan Anwar, Sina KeshvadiLLM ServingPrefix Caching

  14. Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs

    Nov 3, 2026Smit Kadvani, Kriti Faujdar, Spurthi TallamLLM QuantizationLLM Serving

  15. ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration

    Nov 2, 2026Anshu Raina, Yao Fu, Zhang Jiejing +11LLM ServingLanguage Model Performance Prediction

  16. EpiLoRA: Fusing Many LoRA Adapters into Low-Precision GEMMs for Serving and RL

    Nov 2, 2026Hieu Tran, Hong YuMulti-LoRA InferenceLLM Serving

  17. Semantic Caches for LLM Web Services Leave Their Error Budget Unspent

    Nov 2, 2026Hanshuai Cui, Qianli Ma, Zhiqing TangSemantic CachingLLM Serving

  18. FactorSpec: Serving Exact Relational Constraints as Speculative Drafts

    Nov 1, 2026Jianchang Su, Dawei Xiang, Wei ZhangLLM ServingSpeculative Decoding

  19. LoRAForge: Co-Designing Post-Hoc Compression and Execution for Efficient Multi-LoRA Serving

    Oct 31, 2026Shenghui Li, Thiemo VoigtMulti-LoRA InferenceLLM Serving

  20. Capacity Planning for Resource-Constrained Clinical LLM Serving

    Oct 31, 2026Yi Tang, Geoffrey Martin, Yiye Zhang +1LLM ServingLLM Routing

  21. Multi-Turn Semantic Caching for Low-Cost LLM Serving

    Oct 31, 2026Baran Atalar, Selly Beyene, Xutong Liu +1Semantic CachingLLM Serving

  22. Inference First, Training Underneath: Layer-Granular LoRA Co-Serving with LayerLoom

    Oct 31, 2026Zongpu Zhang, Yishu Yin, Xuehai QianLLM ServingLLM Inference Scheduling

  23. TileWeave: Heterogeneous Tiling for Sparse Prefill on Mobile SoCs

    Oct 30, 2026Wentao Ni, Hong Cai, Mingu Lee +7LLM Inference AccelerationOn-Device Language Model Inference

  24. LEAP: Efficient LLM Inference Scheduling with Generalizable Length Prediction

    Oct 30, 2026Kwan Yan, Jialun Zhong, Yanzeng LiLLM Output Length PredictionLLM Inference Scheduling