Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.
Figures & tables
Figure 1: Two opportunities for efficient hybrid-model serving: sharing cache capacity and exploiting idle decode-time bandwidth.
Figure 2: Illustration of a typical hybrid LLM architecture. Global-attention layers (full, DSA or QSA Qiu et al. (2026) ) hold a global-KV that grows with the context; bounded-state layers (sliding window, GDN, KDA) hold a window KV or recurrent state of fixed size per request. All cache components share GPU memory.
Platform (year)
CPU–GPU interconnect
GB/s
vs. A100
A100 (2020)
PCIe 4.0 × 16
64
1 ×
H100 (2022)
PCIe 5.0 × 16
128
2 ×
GH200 (2023)
NVLink-C2C
900
14 ×
GB300 (2025)
NVLink-C2C
900
14 ×
VR300 (2026)
NVLink-C2C
1,800
28 ×
Table 1: Bidirectional bandwidth of the CPU–GPU interconnect across NVIDIA generations: one PCIe × 16 slot per GPU, and one NVLink-C2C interface per Grace superchip (shared by the two GPUs of a GB300 superchip).
Figure 3: Loads that one GPU of Ring-2.6-1T (TP4) can run under SGLang’s default partition (state : global-KV = 0.9). The state pool caps the request count at S=61 ; the global-KV pool caps the product of request count and context length at G=1.09 M tokens.
SGLang Zheng et al. (2024)
vLLM Kwon et al. (2023)
TideCache
Technique
Unified Memory Pool Zhang and SGLang Team (2026)
Hybrid KV Manager vLLM Team (2026)
Transparent Cache Pool
Dynamic Sharing
Copy-Free Repartitioning
Transparency
Streaming Compatibility
Table 2: Comparison of hybrid cache management solutions.
Figure 4: TideCache overview: the Request Scheduler controls resident- and streaming-region admission; the Transparent Cache Pool manages cache backing, and the Cache Streamer feeds shared GPU slots from host memory.
Figure 5: One page move across epochs: plan and park on the scheduler thread, unmap and map on a worker after the step fence, apply at the next epoch in which every rank is idle.
Figure 6: The global-KV subpool (KV1, KV2, etc. denote successive layers’ caches): a resident region with its own pages per layer and a streaming region whose L layer ranges alias M=2 slots; HiCache restores into the former, streaming feeds the latter.
Figure 7: Double-buffered overlap with two slots: reading KV2 releases slot B for KV4’s transfer, which must finish before attention uses KV4. Bounded-state layers (BSL) and global attention provide computation to overlap with H2D.
Figure 8: Selective KV transfer with low launch overhead: (a) Graph Segmentation transfers selected KV but incurs repeated launches; (b) Fixed-Range Transfer retains low launch overhead but moves extra KV; (c) Graph-Compatible Handshake achieves both. Timelines are schematic.
Model
Hybrid combination
TP
Platform
Kimi K3 (48L)
MLA : KDA = 1 : 3
4
GB300
Ring-2.6-1T
MLA : LA = 1 : 7
4
GB300
GLM-5.3-Flash
DSA : KDA = 11 : 34
1
GB300
Qwen3.8-2.4T (36L)
GQA : GDN = 1 : 3
4
GB300
GLM-5.3-Flash
DSA : KDA = 11 : 34
4
GH200
Qwen3.8-27B
GQA : GDN = 1 : 3
1
GH200
Table 3: Hybrid layer combinations and hardware configurations of the evaluated models. MLA: multi-head latent attention; DSA: DeepSeek sparse attention; GQA: grouped-query attention; KDA: Kimi Delta Attention; GDN: Gated DeltaNet; LA: Lightning linear attention; SWA: sliding-window attention.
Figure 9: End-to-end performance across seven model/platform configurations. (a) P50 latency / token speedup across all requests; (b) turns/s and (c) P50 TTFT speedups for agent sessions. Both baselines use their best tested split ratios. Values are normalized to SGLang w/ HiCache (Tuned). All panels are higher-is-better. GH200 is on the left, GB300 on the right.
Figure 10: Qwen3.8-27B on GH200 under a shifting workload, with a burst at 300–600 s. (a) Overall P50 latency / token; (b) turns/s and (c) P50 TTFT for agent sessions. (d–e) Physical GPU capacity allocated to each cache component, including occupied and free space within its pool, alongside running and queueing request counts. Unassigned denotes pages outside both pools.
Figure 11: Dynamic cache partitioning on Kimi K3 and GLM-5.3-Flash. Turns/s is normalized to the better fixed partition; configurations denote req/s × concurrent sessions.
Figure 12: Sensitivity to the state : global-KV split ratio on Kimi K3. Horizontal lines show TideCache starting from ratio 3.
Figure 13: Median decode throughput versus batch size on GPT-OSS-120B / GH200 with 8K-token contexts.