cs.DCApr 8, 2026

Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start

Authors: Xueshen Liu, Yongji Wu, Yuncheng Yao, Danyang Zhuo, Ion Stoica, Z. Morley Mao

Organizations: University of Michigan · UC Berkeley · Duke

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Modern LLM service providers increasingly rely on autoscaling and parallelism reconfiguration to respond to rapidly changing workloads, but cold-start latency remains a major bottleneck. While recent systems have reduced model weight loading to seconds, CUDA graph capture still takes tens of seconds to minutes and often dominates startup. Unfortunately, CUDA graphs cannot be naively serialized: beyond graph topology, they are tightly coupled to execution context, including device addresses embedded in kernel arguments and kernel code lazily loaded during warmup. Existing approaches either rely on brittle kernel-specific patching or heavyweight process-level checkpoint/restore that are inflexible to dynamic parallelism switching. We present Foundry, a template-based CUDA graph context materialization system that persists both graph topology and execution context during an offline processing stage, and reconstructs executable graphs online with negligible overhead. Foundry enforces deterministic memory layouts, automatically extracts and reloads kernel binaries required by captured graphs, and reduces online reconstruction costs through topology-based templating. For distributed serving, Foundry further enables a single-GPU offline capture to generate templates for multi-GPU deployments by patching only rank-dependent communication state. Across dense and MoE models up to 235B parameters, Foundry reduces cold-start latency by up to 99%, cutting the initialization time of Qwen3-235B-A22B from 10 minutes to 3.9 seconds while preserving the throughput gains of CUDA graphs.

Figures & tables

Figure 1. Breakdown of vLLM worker initialization of serving Qwen3-14B on 2xH200s.
Figure 2. Time per output token (TPOT) under different batch sizes for Qwen3-30B-A3B on 2xH200s with EP2, using vLLM with and without CUDA graphs.
Figure 3. Medusa materializes only the graph topology, while Foundry also materializes the execution context.
Figure 4. Overview of Foundry ’s workflow.
Figure 5. Foundry works as an interposition layer by implementing a CUDA driver hook.
Figure 6. CUDA graph topology and node parameter sets.
Figure 7. [Cold-Start Latency Reduction] Serving engine initialization latency across models and parallelism configurations on H200. Percentages indicate Foundry ’s reduction relative to vLLM with CUDA graphs.
Figure 8. [Cold-Start Latency Reduction] End-to-end engine initialization breakdown comparing vLLM, CUDA-checkpoint, and Foundry across four model configurations. Note: torch.compile only does bytecode transformation and cache loading.
Figure 9. [Serving Throughput Preservation] Mean time per output token (TPOT) for vLLM and Foundry across batch sizes 16–512.
Figure 10. [Effectiveness of Templating] Average per-graph cost of stream capture, template construction via driver APIs, and on-demand parameter update. All experiments capture 512 graphs.
Figure 11. [Effectiveness of Templating] Number of unique templates versus on-demand updated graphs out of 512 total. Percentages indicate the fraction of graphs served via on-demand update.
ModelL3-8BG3-12BQ3-14BQ3-235B
ParallelDP1DP1DP1EP8
Image size3.9 GB6.6 GB3.7 GB–
Archive size976 MB1.3 GB1.1 GB2.2 GB
Table 1. Storage cost comparison between CUDA-checkpoint and Foundry . L3, G3 and Q3 indicate Llama3, Gemma3 and Qwen3 respectively.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#ThreadsWall TimeAddNodeInstantiate
128.52 ms0.70 μ\mu s0.8 μ\mu s
229.32 ms1.44 μ\mu s1.6 μ\mu s
436.13 ms3.52 μ\mu s3.8 μ\mu s
837.54 ms7.31 μ\mu s6.6 μ\mu s
Appendix
Table 2. Durations of building 80 graphs each with 500 dummy nodes using different number of threads.
More in this topic

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Explore similar work

CardsList

  1. Hybrid JIT-CUDA Graph Optimization for Low-Latency Large Language Model Inference

    Apr 25, 2026Divakar Kumar Yadav, Tian ZhaoLLM InferenceLLM Inference Acceleration

    Department of Computer Science University of Wisconsin-Milwaukee Milwaukee, WI, USA

  2. InferScale: GPU-Native KV Injection for Personalized LLM Serving

    Jul 29, 2026Peter Li, Prashant PandeyLLM ServingMemory-Augmented Language Models

    Northeastern University Boston, Massachusetts, USA

  3. Breaking the Ice: Analyzing Cold Start Latency in vLLM

    Jun 5, 2026Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin WangLLM InferenceLLM Inference Serving

    Paderborn University, Paderborn, Germany · IBM Research Europe, Zurich, Switzerland