GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving

Shakya Jayakody; Youpeng Zhao; Chinmay Dhanraj Nehate; Jun Wang

cs.DCMar 26, 2026

GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving

Authors: Shakya Jayakody, Youpeng Zhao, Chinmay Dhanraj Nehate, Jun Wang

Organizations: University of Central Florida, Orlando, FL, USA.

Abstract

The rise of million-token, agent-based applications has placed unprecedented demands on large language model (LLM) inference services. The long-running nature of these tasks increases their susceptibility to hardware and software faults, leading to costly job failures, wasted resources, and degraded user experience. The stateful key-value (KV) cache, which grows with the sequence length, presents a central challenge as it is a critical and vulnerable component in distributed serving systems. In this work, we propose GhostServe, a novel checkpointing solution to facilitate fault-tolerant LLM serving. Specifically, GhostServe protects the streaming KV cache in the shadow by applying erasure coding to generate and store the parity shards in host memory. In the event of device failures, GhostServe enables fast reconstruction of the lost KV cache, allowing the inference process to resume seamlessly without costly full recomputation or state replication. Evaluations demonstrate that GhostServe reduces checkpointing latency by up to 2.7x and recovery latency by 2.1x for a single batch, and 1.2x median response latency compared to existing methods, in the presence of system failures, paving the way for high-availability and cost-effective LLM serving at scale.

Figures & tables

	Checkpointing	Recomputation	Backup Overhead	Recovery Overhead	Overall Costs
SGLang Zheng et al. (2024)	$\times$	$\surd$	Low	High	High
SGLang + Replication Strati et al. (2024)	$\surd$	$\times$	High	Medium	High
GhostServe (Ours)	$\surd$	$\surd$	Low	Low	Low

Table 1: Comparison of prior works and GhostServe.

Figure 1: (a) Left: Prefill and decode stages in LLM inference. In the prefill stage, all queries are processed with keys and values computed and stored in memory. During the subsequent decode stage, only the attention for new queries is needed, significantly reducing attention computation. (b) Middle: Chunked prefill mechanism. During prefill, the input is divided into chunks of tokens, where the corresponding KV cache is written sequentially into contiguous chunks of the KV cache buffer. (c) Right: Illustrative example of erasure coding. The input data is first partitioned into

N

shards, which are then encoded to generate

K

parity shards. In the event of failures, the lost shards can be reconstructed from parity shards and the surviving data shards.

Figure 2: Comparison of checkpointing latency and memory overhead of our erasure coding (8:2) method and state replication Strati et al. (2024) during prefill. Results are profiled with LLaMA-3-70B using SGLang with a batch size of 16, and varying input sequence lengths (32/64K), a chunk size of 2K, in tensor parallelism (TP=8) across 8 $\times$ H200 GPUs.

Figure 3: (a) Left: System overview of GhostServe. (b) Right: Illustration of the chunk-level checkpointing and load-balancing strategy in terms of GPU execution timeline. As KV cache chunks are generated ( $T_{1},T_{2},T_{3},T_{4}$ ), a different GPU is assigned in a round-robin fashion to gather and compute the corresponding parity chunk, thereby distributing the checkpointing overhead. The computation for subsequent chunks then resumes after checkpointing ( $T_{1}^{{}^{\prime}},T_{2}^{{}^{\prime}},T_{3}^{{}^{\prime}},T_{4}^{{}^{\prime}}$ ).

Figure 4: Performance comparison of different fault-tolerant methods. The latency results are measured with a batch size of 16, chunk size of 2K, and an output length of 4K, using varying input lengths that range from 2K to 64K. I/O overhead represents the total I/O latency incurred during the checkpointing process. Recovery latency denotes the time required to restore 50% of the chunks to resume inference. All models are running with an 8:2 parity ratio for GhostServe.

Figure 5: Performance comparison of different fault-tolerant methods in online serving. Here, we measure P50/99 latency and effective-inference-time-ratio (EITR) under both failure-free and failure-induced environments. Faults are injected at random steps with a failure rate 15%.

Figure 6: Kernel Microbenchmark. All experiments are conducted using LLaMA-3-70B with a batch size of 16. (a) Left: Performance breakdown for erasure coding kernel during checkpointing and recovery for different chunk sizes. (b) Right: Impact of implementation method (PyTorch vs CUDA), kernel fusion, and CUDA graph on the erasure coding performance for different chunk sizes.

Figure 7: Cost-benefit analysis for serving LLaMA-3-70B model over the entire serving traces. Here, we compare the EITR and MTTR for different methods under varying failure rates (5%

\sim

15%).

Figure 8: Sensitivity Studies. All experiments are conducted using LLaMA-3-70B with a chunk size of 2K. Recovery latency is the time required to restore 50% of the KV cache. (a) Left: Performance comparison of GhostServe under different parity ratios. (b) Top Right: Performance comparison of different fault-recovery methods with varying batch sizes and TP sizes. (c) Bottom Right: Ablation studies on the impact of recomputation on the recovery latency.

Figure 9: Performance comparison of different methods when scaling to million-tokens. Results are reported using LLaMA-3-70B, with a batch size of 1, chunk size of 2K, and output length of 4K.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training
May 18, 2026Shujie Han, Feng Jiang, Patrick P. C. Lee +5Fault-Tolerant Distributed Training Distributed Training
Northwestern Polytechnical University · The Chinese University of Hong Kong · National University of Defense Technology
Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
Jun 22, 2026Yuhang Gan, Yiwei Yang, Yuyi Li +6LLM Inference
University of California Santa Cruz USA · UC Santa Cruz USA · University of California, Davis USA +2
InferScale: GPU-Native KV Injection for Personalized LLM Serving
Jul 29, 2026Peter Li, Prashant PandeyLLM Serving Memory-Augmented Language Models
Northeastern University Boston, Massachusetts, USA

May 18, 2026J/K move · Enter open · S save

TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training

Shujie Han, Feng Jiang, Patrick P. C. Lee, Xiao Zhang+4

Northwestern Polytechnical University · The Chinese University of Hong Kong · National University of Defense Technology

Fault-Tolerant Distributed Training Distributed Training LLM Training

Open paper AskPDF arXiv

Large Language Model (LLM) training is frequently interrupted by a heterogeneous spectrum of failures, from common GPU crashes to catastrophic cluster-wide outages. Existing checkpointing systems rely on monolithic, single-tier storage backend, forcing a trade-off between state-saving overhead and recovery speed. We propose TierCheck, a cluster-aware tiered checkpointing system that aligns storage placement with failure heterogeneity. TierCheck adopts a three-tier design that maintains lightweight differential checkpoints in local and peer memory for fast localized recovery, while asynchronously migrating heavyweight base checkpoints to remote persistent storage. It also ensures strict global consistency across tiers without stalling training, and achieves fast cluster-aware checkpoint restoration during recovery. Evaluations on models up to 40 billion parameters show that TierCheck achieves low training overhead, reduces end-to-end checkpointing time to under 10s, and supports high-frequency checkpointing, ultimately striking an optimal balance between low-overhead persistence and fast recovery.

Abstract

LLM Serving

Explore similar work

TierCheck: Tiered Checkpointing for Fault Tolerance in Large Language Model Training

Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

InferScale: GPU-Native KV Injection for Personalized LLM Serving