Large Language Model Serving

Ask about this topic

Recent momentum

+9%

12 papers in the last 28 days · 0.2% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

4 new papers

A weekly snapshot of new work published in Large Language Model Serving.

Period ending 2026-09-14

5 new papers

A weekly snapshot of new work published in Large Language Model Serving.

Period ending 2026-09-07

4 new papers

A weekly snapshot of new work published in Large Language Model Serving.

104 papers

Latest in Large Language Model Serving

Open your feed →
CardsList

  1. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

    Sep 16, 2026Li ChenBenchmarking Large Language ModelsProgrambench

  2. CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization

    Sep 14, 2026Victor H. Chen, Hairui Yu, Stella K. Chung +1Edge ComputingLarge Language Model Serving

  3. MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

    Sep 14, 2026Tiancheng Zhang, Yulin Chen, Yunfeng Zhao +3Large Language Model ServingOptimal Scheduling

  4. AgentKV: Phase-Aware KV Eviction for Agentic LLMs

    Sep 14, 2026Taowen Tony Liu, Jeffrey T. H. Wong, Can Xiao +3Key-Value Cache EvictionKey-Value

  5. An Efficient and Modular Framework for Targeted Harm Mitigation in LLMS

    Sep 12, 2026Roberto Campbell, Momin Abbass, Muneeza Azmat +5Large Language Model AlignmentLarge Language Model Safety

  6. Composable CXL Memory as a Kubernetes-Native Shared Memory for LLM Serving

    Sep 9, 2026Hongjian Fan, Kevin Zhang, David Habinsky +1Kv-Cache BlocksKey-Value Cache

  7. Privacy-Preserving Split Learning for Federated LLM Fine-Tuning

    Sep 9, 2026Heng Jin, Chaoyu Zhang, Hexuan Yu +2Federated LearningLarge Language Model Fine-Tuning

  8. Combating Instruction Conflict via Energy-Driven Latent Conflict Detection

    Sep 8, 2026Mingyu Ma, Yuxin Wu, Jingbo Wang +3Large Language Model ServingKnowledge Conflicts

  9. HoneyRoute: Honeypot-Model Routing for Adversarial LLM Serving

    Sep 8, 2026Han JinHoneypotsLanguage Model Evasion Attacks

  10. Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

    Sep 7, 2026Siyu Song, Qi Bai, Jinbo Hao +3Large Language Model ServingPrefill

  11. From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix

    Sep 1, 2026Olga Tsymboi, Dmitrii Stoianov, Ramil Latypov +11Large Language Model ServingLLM Post-Training Methods

  12. Drift-Aware LLM Routing with Sparse Contexts and Shared Budgets

    Sep 1, 2026Cheung Hao Lee, Patrick WongLarge Language Model RoutingAdaptive Routing

  13. Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving

    Aug 6, 2026Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair +4Large Language Model ServingLow-Latency Systems

  14. Pin Once, Swap Light: Subspace-Aligned Centroid-Residual Training for Efficient Ultra-LoRA Serving

    Aug 4, 2026Xiang Li, Pengcheng Wang, Huazheng Wang +1Low-Rank Adaptation AdaptersLarge Language Model Serving

  15. LLM Serving in the Wild: An Empirical Study of Frameworks, Methods, and System Designs

    Aug 4, 2026Forough Majidi, Mohammad Mehdi Morovati, Foutse Khomh +1Large Language Model ServingEmpirical Study

  16. SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems via Deterministic Side Channels

    Aug 4, 2026Yongwan Jo, Jinyoung Park, Euihyun Lee +1Language Model Evasion AttacksSide-Channel Analysis

  17. LaCache: Robust Semantic Caching for LLM Serving

    Aug 3, 2026Jiacheng Liang, Yuhui Wang, Tanqiu Jiang +1CacheLanguage Model Evasion Attacks

  18. Diagnose Before You Compress: Prediction-Independent Bottleneck Witness Refinement for LLM Serving Traces

    Aug 1, 2026Liming Liu, Chao Hu, Mingfei Lu +7Large Language Model ServingWorkload

  19. Unleashing the Potential of Large Language Models: A Blueprint for Real-Time, Enterprise-Ready Deployments

    Aug 1, 2026Muhammad Faizan Raza, Shuo, Yang +2Large Language Model ServingReal-Time Systems

  20. TokTier: Exact Stateful CPU+GPU Tokenization for Agentic LLM Serving

    Jul 31, 2026Zhenyu Zhang, Zhichao CaoPer-Token LatencyLarge Language Model Serving

  21. DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs

    Jul 30, 2026Jiaxuan Chen, Jianshu She, Ye Yuan +5Large Language Model ServingInference Latency

  22. LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

    Jul 29, 2026Ming-Yen Lee, Hanchen Yang, Faaiq Waqar +4Large Language Model ServingLarge Language Model Memory

  23. Steering Instruction Hierarchies at Inference Time

    Jul 28, 2026Siqi Zeng, Sewoong Lee, Han Zhao +1Instruction-Tuned ModelsTraining-Free Inference Framework

  24. Kalypso: Relational LLM Serving

    Jul 26, 2026Hojae Son, Md Ashraful Islam, Huy Gia Cao +2Large Language Model ServingLarge Language

  25. WISERouter: LLM Routing with Workload Budget Constraint

    Jul 26, 2026Yifei Li, Zihui Gao, Laks V. S. LakshmananLarge Language Model RoutingLarge Language Model Serving

  26. TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI

    Jul 24, 2026Ritik Raj, Souvik Kundu, Sarbartha Banerjee +3Large Language Model RoutingAdaptive Routing

  27. Total Variation Distance Estimation in Autoregressive Models

    Jul 21, 2026Eric Price, Kevin Tian, Zhiyang Xun +1Autoregressive Language ModelsKullback-Leibler Divergence

  28. Talaria: Session-Aware Serverless Serving of Hundred-Billion-Parameter LLMs

    Jul 19, 2026Utopia Meng, Unicornt Zhao, Derek Li +2Large Language Model ServingSmall Language Models

  29. BlockServe: Block-Grained Continuous Batching for High-Throughput Diffusion LLM Serving

    Jul 9, 2026Yuanjie Zhu, Liangwei Yang, Ke Xu +4Large Language Model ServingWell-Tuned Static Batching Policy

  30. SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

    Jul 9, 2026Jiahao Wang, Kaizhan Lin, Kaixi Zhang +7Optimal SchedulingLarge Language Model Serving

  31. Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization

    Jul 9, 2026Jiantong Jiang, Peiyu Yang, Rui Zhang +1Key-Value CacheLarge Language Model Serving

  32. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    Jul 7, 2026Yihua LiuLarge Language Model ServingLow-Latency Systems

  33. Sangam: Efficiently Serving Diffusion LLMs with the AR Stack

    Jul 5, 2026Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1Large Language Model ServingDiffusion Language Models

  34. Online Linear Programming for Multi-Objective Routing in LLM Serving

    Jul 4, 2026Zixi Chen, Yinyu Ye, Zijie ZhouLarge Language Model RoutingLarge Language Model Serving

  35. Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

    Jul 2, 2026Jiaying Meng, Bojie LiLow-Latency SystemsLarge Language Model Serving

  36. Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

    Jul 2, 2026Shrikara Arun, Anjaly Parayil, Srikant Bharadwaj +2PrefillPer-Token Latency

  37. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    Jun 30, 2026Jie Li, Tongyang Wang, Yong ChenCacheLarge Language Model Serving

  38. TraceLab: Characterizing Coding Agent Workloads for LLM Serving

    Jun 29, 2026Kan Zhu, Mathew Jacob, Chenxi Ma +4Coding AgentsLarge Language Model Serving

  39. DiLaServe: High SLO Attainment Serving for Diffusion Language Models

    Jun 27, 2026Tzu-Tao Chang, Benjamin Yuanyang Hong, Kiet Pham +1Large Language Model ServingThroughput

  40. NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptation

    Jun 26, 2026Qiong Tang, Xiangkun Hu, Xiangyang Liu +2Sliding-Window AttentionEfficient Long-Context Inference

  41. Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

    Jun 25, 2026Yasmin Moslem, Magdalena Kacmajor, Vasudevan Nedumpozhimana +11Large Language Model ServingEscalation

  42. CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

    Jun 23, 2026Zhuoren Ye, Tianyu Wo, Dinghao Xue +4Kv-Cache BlocksLarge Language Model Serving

  43. RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving

    Jun 22, 2026Haifeng Wu, Srinivasan Manoharan, Fangbo Tu +2Large Language Model CallsLarge Language Model Serving

  44. Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

    Jun 21, 2026Li Kong, Qi Qi, Yinyu Ye +1Large Language Model ServingOptimal Scheduling

  45. An Empirical Study of OpenPangu Quantization on Ascend NPUs

    Jun 19, 2026Tong Shi, Jiacheng Wang, Hui Xie +4Weight-Only QuantizationNeural Processing Units

  46. FloatDoor: Platform-Triggered Backdoors in LLMs

    Jun 17, 2026Nils Loose, Jonas Sander, Felix Mächtle +1Language Model Evasion AttacksLarge Language Model Serving

  47. Beyond Prediction: Tail-Aware Scheduling for LLM Inference

    Jun 16, 2026Yueying Li, Yuanfan Chen, Jiayang Chen +6Large Language Model ServingOptimal Scheduling

  48. Service-Induced Congestion in Memory-Constrained LLM Serving

    Jun 14, 2026Ruicheng Ao, Jing Dong, Gan Luo +1Large Language Model ServingLLM Inference Efficiency

  49. CogGuard: Cognitive and Operational Profiling for Proactive Warning in Edge Intelligent Services

    Jun 13, 2026Zhi Yao, Weihao Chen, Zhiqing Tang +4GuardDanger Recognition

  50. Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees

    Jun 12, 2026Herbert Woisetschläger, Arastun Mammadli, Ryan Zhang +1Large Language Model RoutingLarge Language Model Serving

  51. Characterizing Software Aging in GPU-Based LLM Serving Systems

    Jun 10, 2026Domenico Cotroneo, Bojan CukicLarge Language Model ServingSoftware

  52. External Experience Serving in Production LLM Systems: A Deployment-Oriented Study of Quality-Cost Trade-offs

    Jun 10, 2026Lin Sun, Heming Zhang, Xiangzheng ZhangLarge Language Model ServingExperience

  53. FMplex: Model Virtualization for Serving Extensible Foundation Models

    Jun 8, 2026Hetvi Shastri, Pragya Sharma, Walid A. Hanafy +3Wireless Foundation ModelsFoundation Model

  54. Distilling Safe LLM Systems via Soft Prompts for On Device Settings

    Jun 8, 2026Motasem Alfarra, Cristina Pinneri, Dana Kianfar +2Large Language Model SafetySafety Alignment

  55. SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

    Jun 4, 2026Hongyuan Liu, Yawei Li, Zhiqiang Que +3Large Language Model QuantizationLarge Language Model Serving

  56. Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents

    Jun 4, 2026Zhuoming Chen, Xinrui Zhong, Qilong Feng +5Dynamic Sparse AttentionLarge Language Model Serving

  57. Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving

    Jun 4, 2026Hyungmin Kim, Minsoo Kim, Hongseok Kim +1Key-Value Cache CompressionLarge Language Model Serving

  58. RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention

    Jun 4, 2026Yang Liu, ZhaoKai Luo, HuaYi Jin +5Key-Value CacheLarge Language Model Serving

  59. Learning to Route LLMs from Implicit Cost-Performance Preferences via Meta-Learning

    Jun 4, 2026Jiahao Zeng, Ming Tang, Ningning DingLarge Language Model RoutingLarge Language Model Serving

  60. E2LLM: Towards Efficient LLM Serving in Heterogeneous Edge/Fog Environments

    Jun 2, 2026Truong-Thanh Le, Amir Taherkordi, Hoang-Loc La +3Large Language Model ServingLLM Inference Efficiency