Placing LLM Lookup Tables by Per-Layer Prefetch Deadlines
Organizations: Microsoft · Independent Researcher
Abstract
DeepSeek-V4.1's 203 GB of Engram n-gram tables sit in host DRAM today, taking 190 of the 223 GiB a 4-GPU serving process uses. Row indices depend only on input tokens, so reads can start with each step and overlap earlier layers. V4.1's two Engram layers have windows that differ by 30x (0.3 vs. 9.3 ms). We show that this deadline, together with session reuse and contention on a shared device, decides which tier each table can use and which layers need cache, and we give a model that predicts the resulting slowdown from measured read times. Agent and chat traffic reuse rows within sessions, and static hot sets transfer poorly. We build SlackTier, a table tier for SGLang that issues all lookups at step start. It reads the short-window layer first and splits a small GPU row cache across layers. The path runs inside the decode CUDA graph and is bit-exact with the original gather. Layer 14's table can go to NVMe uncached with no measurable decode cost. With both tables on four NVMe disks, mean time per output token on agent and chat replays rises at most 1.4% over unmodified SGLang, and host memory falls from 212 to 34 GiB. Both tables need no cache when read over RDMA (remote direct memory access) from another node's memory; one copy serves two instances. On a KV-constrained agent workload, using the freed memory as host KV cache at the same memory budget raises median throughput by 36% and cuts median p90 time to first token from 38 to 24 s.
Figures & tables
| Agent (OpenHands) | Chat (WildChat) | |||
|---|---|---|---|---|
| Policy | 1M rows | 5M rows | 1M rows | 5M rows |
| LRU | 74.0 | 81.0 | 34.8 | 41.7 |
| S3-FIFO-style | 74.5 | 78.7 | 32.7 | 39.2 |
| Static (same dist.) | 51.3 | 59.5 | 17.7 | 26.4 |
| Static (public list) | 29.2 | 39.0 | 8.0 | 12.3 |
| Configuration | Host memory | Cache | Agent | Chat |
|---|---|---|---|---|
| Host-DRAM table | 223 GiB | — | 29.41 ms (31.1) | 33.71 ms (35.1) |
| Layer 1 DRAM, layer 14 on 1 disk | 129 GiB | — | 0.0% (31.1) | 0.1% (35.1) |
| Both on 1 disk | 34 GiB | — | +6.4% (33.5) | +7.2% (37.7) |
| Both on 4 disks | 34 GiB | — | +2.6% (32.0) | +2.6% (36.1) |
| Both on 1 disk, cache on layer 14 only | 34 GiB | 0.5M | +5.6% (33.3) | +7.1% (37.7) |
| Both on 1 disk, cache split evenly | 34 GiB | 1M | +1.2% (31.8) | +4.8% (37.0) |
| L1 delay | L14 delay | Measured | Eq. 1 | Sum of excesses |
|---|---|---|---|---|
| 0.4 | 9 | +0.12 | 0.13 | 0.13 |
| 0.9 | 9 | +0.64 | 0.63 | 0.63 |
| 0.9 | 12 | +2.52 | 2.65 | 3.28 |
| 1.5 | 12 | +2.51 | 2.65 | 3.88 |
| 1.5 | 15 | +5.46 | 5.65 | 6.88 |
| 2.5 | 15 | +5.54 | 5.65 | 7.88 |
| Agent | Chat | |
|---|---|---|
| Full design (even split) | 29.62 | 35.30 |
| layer-1-first I/O | 29.86 | 36.09 |
| in-step dedup | 29.86 | 35.66 |
| L14-only cache | 30.85 | 36.09 |
| layer-1-first I/O | 31.34 | 36.69 |
| Tables | Host KV | RSS | Prefix hit | TTFT p50 | TTFT p90 | Tokens/s |
|---|---|---|---|---|---|---|
| DRAM | — | 223 GiB | 63% | 13.6 s | 38.1 s | 702 |
| 4 disks | — | 34 GiB | 62% | 14.5 s | 37.6 s | 700 |
| 4 disks | 185 GiB | 220 GiB | 76% | 10.1 s | 24.1 s | 956 |
| DRAM | 185 GiB | 408 GiB | 75% | 11.2 s | 25.5 s | 954 |
| Agent | Chat | |||
|---|---|---|---|---|
| SlackTier, tables in | TPOT | Total | TPOT | Total |
| Host DRAM | +1.0% [ 1.0, 3.0] | +0.1% | +0.5% [ 0.4, 0.7] | 0.2% |
| L1 DRAM, L14 NVMe | 0.8% [ 2.8, 0.3] | +0.1% | +2.7% [0.6, 8.1] | +3.1% |
| Both on 1 disk, split | +3.6% [2.5, 5.9] | +4.3% | +6.7% [6.1, 7.4] | +7.2% |
| Both on 4 disks, split | 0.5% [ 2.2, 0.5] | +0.0% | +1.4% [1.0, 1.9] | +0.6% |
| Both remote (RDMA) | +0.2% [ 1.2, 1.0] | 0.4% | 0.1% [ 0.4, 0.1] | 0.8% |
| PLE table location | Agent | Chat |
|---|---|---|
| 1 disk, no cache | +9.4% | +7.9% |
| 4 disks, 4 I/O threads, no cache | +5.8% | +3.2% |
| 1 disk, 1M-row cache | +4.5% | +4.8% |
| 4 disks, 4 I/O threads, cache | +3.4% | +2.4% |
| Work | Shows | This paper adds |
|---|---|---|
| Ma et al. (2026a) | Zero-stall condition; CXL pool near DRAM on a proxy model | Per-layer placement and cache split on released V4.1; NVMe and RDMA served end to end |
| MiaAI PR knapcio (2026) | NVMe row cache for both layers on GB10; layer-1 stalls larger than layer-14 | Measured windows that explain it; tier and cache chosen per layer by window |
| vLLM #59958 stecasta (2026) | Qwen PLE read from disk, compatible with CUDA graphs | Session locality; disk contention between layers |
| vLLM #57023 zxpdemonio (2026) | V4.1 tables in a Mooncake store, overlapped lookups | Placement by window; shared remote table measured with two instances |
| Han et al. (2026) | 2-bit tables; static hot-row cache | Static vs. session caches on interleaved agent traffic |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Agent (OpenHands) | Chat (WildChat) | |||||
|---|---|---|---|---|---|---|
| Configuration | Step | p99 | L1 hit | Step | p99 | L1 hit |
| Host-DRAM table | 29.40 (0.02) | 31.1 | — | 33.69 (0.08) | 35.1 | — |
| L1 DRAM, L14 1 disk, no cache | 29.39 (0.00) | 31.1 | — | 33.67 (0.11) | 35.1 | — |
| 1 disk, | 30.85 (0.08) | 33.0 | 0.00 | 36.09 (0.02) | 37.7 | 0.00 |
| 1 disk, | 29.89 (0.09) | 31.9 | 0.81 | 35.60 (0.04) | 37.2 | 0.33 |
| 1 disk, | 29.77 (0.13) | 31.6 | 0.85 | 35.42 (0.05) | 37.1 | 0.39 |
| Workload | 0.2% | 1% | 5% | Decode @5% |
|---|---|---|---|---|
| OpenHands agent, cold | 62 | 78 | 86 | 86 |
| OpenHands, warmed on other repos | 62 | 78 | 86 | 86 |
| SWE-agent (one trajectory per issue) | 59 | 85 | 88 | 89 |
| WildChat chat, cold | 30 | 45 | 53 | 47 |
LLM Inference Acceleration
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
See all 1,221 papers