cs.DCApr 8, 2026

Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start

Authors: Xueshen Liu, Yongji Wu, Yuncheng Yao, Danyang Zhuo, Ion Stoica, Z. Morley Mao

Organizations: University of Michigan · UC Berkeley · Duke

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern LLM service providers increasingly rely on autoscaling and parallelism reconfiguration to respond to rapidly changing workloads, but cold-start latency remains a major bottleneck. While recent systems have reduced model weight loading to seconds, CUDA graph capture still takes tens of seconds to minutes and often dominates startup. Unfortunately, CUDA graphs cannot be naively serialized: beyond graph topology, they are tightly coupled to execution context, including device addresses embedded in kernel arguments and kernel code lazily loaded during warmup. Existing approaches either rely on brittle kernel-specific patching or heavyweight process-level checkpoint/restore that are inflexible to dynamic parallelism switching. We present Foundry, a template-based CUDA graph context materialization system that persists both graph topology and execution context during an offline processing stage, and reconstructs executable graphs online with negligible overhead. Foundry enforces deterministic memory layouts, automatically extracts and reloads kernel binaries required by captured graphs, and reduces online reconstruction costs through topology-based templating. For distributed serving, Foundry further enables a single-GPU offline capture to generate templates for multi-GPU deployments by patching only rank-dependent communication state. Across dense and MoE models up to 235B parameters, Foundry reduces cold-start latency by up to 99%, cutting the initialization time of Qwen3-235B-A22B from 10 minutes to 3.9 seconds while preserving the throughput gains of CUDA graphs.

Figures & tables

Figure 1. Breakdown of vLLM worker initialization of serving Qwen3-14B on 2xH200s.
Figure 2. Time per output token (TPOT) under different batch sizes for Qwen3-30B-A3B on 2xH200s with EP2, using vLLM with and without CUDA graphs.
Figure 3. Medusa materializes only the graph topology, while Foundry also materializes the execution context.
Figure 4. Overview of Foundry ’s workflow.
Figure 5. Foundry works as an interposition layer by implementing a CUDA driver hook.
Figure 6. CUDA graph topology and node parameter sets.
Figure 7. [Cold-Start Latency Reduction] Serving engine initialization latency across models and parallelism configurations on H200. Percentages indicate Foundry ’s reduction relative to vLLM with CUDA graphs.
Figure 8. [Cold-Start Latency Reduction] End-to-end engine initialization breakdown comparing vLLM, CUDA-checkpoint, and Foundry across four model configurations. Note: torch.compile only does bytecode transformation and cache loading.
Figure 9. [Serving Throughput Preservation] Mean time per output token (TPOT) for vLLM and Foundry across batch sizes 16–512.
Figure 10. [Effectiveness of Templating] Average per-graph cost of stream capture, template construction via driver APIs, and on-demand parameter update. All experiments capture 512 graphs.
Figure 11. [Effectiveness of Templating] Number of unique templates versus on-demand updated graphs out of 512 total. Percentages indicate the fraction of graphs served via on-demand update.
ModelL3-8BG3-12BQ3-14BQ3-235B
ParallelDP1DP1DP1EP8
Image size3.9 GB6.6 GB3.7 GB–
Archive size976 MB1.3 GB1.1 GB2.2 GB
Table 1. Storage cost comparison between CUDA-checkpoint and Foundry . L3, G3 and Q3 indicate Llama3, Gemma3 and Qwen3 respectively.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#ThreadsWall TimeAddNodeInstantiate
128.52 ms0.70 μ\mu s0.8 μ\mu s
229.32 ms1.44 μ\mu s1.6 μ\mu s
436.13 ms3.52 μ\mu s3.8 μ\mu s
837.54 ms7.31 μ\mu s6.6 μ\mu s
Appendix
Table 2. Durations of building 80 graphs each with 500 dummy nodes using different number of threads.
More in this topic

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Explore similar work

CardsList
Apr 25, 2026

Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference

Large Language Models (LLMs) have achieved strong performance across natural language and multimodal tasks, yet their practical deployment remains constrained by inference latency and kernel launch overhead, particularly in interactive, short-sequence settings. This paper presents a hybrid runtime framework that combines Just-In-Time (JIT) compilation with CUDA Graph execution to reduce launch overhead while preserving runtime flexibility during autoregressive decoding. The framework partitions transformer inference into static components executed via CUDA Graph replay and dynamic components handled through JIT-compiled kernels, enabling asynchronous graph capture and reuse across decoding steps. We evaluate the proposed approach on LLaMA-2 7B using single-GPU, batch-size-one inference across prompt lengths from 10 to 500 tokens. Experimental results show that the hybrid runtime reduces Time-to-First-Token (TTFT) by up to 66.0% and achieves lower P99 latency compared with TensorRT-LLM in this regime. These results indicate that hybrid JIT-CUDA Graph execution can effectively reduce inference latency and variance for short-sequence LLM workloads, making it a practical optimization strategy for latency-sensitive AI applications.

Divakar Kumar Yadav, Tian Zhao

Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA

LLM InferenceLLM Inference Acceleration
AskPDF
Jul 29, 2026

InferScale: GPU-Native KV Injection for Personalized LLM Serving

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

Peter Li, Prashant Pandey

Northeastern University Boston, Massachusetts, USA

LLM ServingMemory-Augmented Language ModelsKV CachingPersistent Memory for Language ModelsLLM Inference AccelerationKV-Cache Reuse
AskPDF
Jun 5, 2026

Breaking the Ice: Analyzing Cold Start Latency in vLLM

As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de-facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study on the startup latency of its engine. With major architectural innovations under it (e.g., the V1 API, introduction of torch.compile), in this paper, we present the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that this process is predominantly CPU-bound. Each step exhibits consistent and interpretable scaling trends with respect to model- and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM's startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All our benchmarking datasets, analysis tools, and prediction scripts are open-sourced at https://github.com/upb-cn/vllm-startup-profiler

Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin Wang

Paderborn University, Paderborn, Germany · IBM Research Europe, Zurich, Switzerland

LLM InferenceLLM Inference Serving
AskPDF