Serving large language models at high concurrency with long contexts requires KV-cache capacity and attention bandwidth that grow with the decode pool. Existing parallelization strategies trade one against the other: data parallelism replicates model weights, limiting KV capacity; tensor parallelism stops scaling attention at the model's KV-head count and pushes dense collectives across nodes; and decode context parallelism (DCP) recovers capacity by sharding the sequence but adds cross-node aggregation of partial attention results. We observe that during decode, dense layers and attention scale differently: dense layers amortize weight reads across the batch and run efficiently within a single NVLink domain, whereas attention cost grows with every active request's KV cache. We present Decode Token Parallelism (TKNP), a decode-pool architecture that decouples weight placement from attention ownership. TKNP distributes a single model copy across nodes in balanced layer blocks. The active node holding a block executes its projections and MLP for the full batch using intra-node tensor parallelism, while every node computes attention for its own requests against locally resident KV cache. This extends request-local attention across nodes without replicating weights: only current-token activations cross the network, dense all-reduces remain within NVLink, and attention requires no cross-node merging of partial results. Implemented in vLLM and evaluated on Llama 3.1 405B and Nemotron-4 340B on up to 64 H100 GPUs, TKNP improves completion throughput for long-context, high-concurrency workloads by up to 1.7× over DCP and by up to 2.06× over data parallelism at matched GPU count.
Figures & tables
Figure 1: Decode-stage latency breakdown for a single decoder layer of Llama 3.1 405B on eight H100 GPUs (TP8) with 32K context. Attention grows with batch size, accounting for nearly 70% of summed component latency at batch 128.
Figure 2: Token Parallel architecture within a layer block. TP partitions KV heads within each node; TKNP partitions requests across nodes. The active root executes QKV and output projections and the MLP for the full batch, while all nodes compute attention for their own requests using local KV cache. Scatter/gather exchanges current-token activations over Ethernet or InfiniBand; dense all-reduces remain within NVLink. Four GPUs per node are shown for illustration. Weight storage on attention workers is omitted; the active root changes with the block.
Figure 3: Single-layer decode latency for Llama 3.1 405B on H100 GPUs. (a) TP8 sustains relatively stable MLP latency through batch 128, with larger increases at batches 256–512. (b) Attention grows with batch size; TP8, TP16, and TP64 nearly overlap. Both panels use a 0–2 ms latency range; TP1 values above this range are clipped.
Figure 4: Single-layer MLP compute and communication for Llama 3.1 405B at batch 128. Bars sum local compute measurements and one isolated all-reduce. The dotted line separates single-node and multi-node TP.
Figure 5: Usable KV capacity for Llama 3.1 405B. Each node contains eight GPUs; DP1 is TP8. TKNP, PP, and DCP provide approximately equal capacity, reaching 5.0 × that of DP on eight nodes.
Figure 6: Balanced weight placement and changing execution roles. Each node stores one stationary layer block and becomes the active root when that block executes. Other nodes compute attention for their assigned requests. The schematic shows equal-sized blocks; actual boundaries balance weight bytes, including endpoint parameters and quantization differences, to leave similar KV-cache budgets.
Figure 7: Dual-active-root microbatching. Shared workers alternate attention between two microbatches to overlap it with dense computation on the other root.
Figure 8: Llama 3.1 405B completion throughput on 32 H100 GPUs , including admission, capacity-induced waves, and drain, with 1,024 output tokens per request. While DP leads at low concurrency, TKNP achieves up to 2.06 × speedups on large batch, long sequence workloads.
Figure 10: Nemotron-4 340B completion throughput at 64K context on 32 H100 GPUs, with 1,024 output tokens per request, including all admission waves.
Figure 11: Eight-node decode throughput with microbatching for Llama 3.1 405B at 32K context on 64 H100 GPUs, with 1,024 output tokens per request. TKNP denotes the single-copy schedule; TKNP + microbatch uses two active roots.
Figure 12: TKNP sustains its throughput advantage under mixed input lengths. Llama 3.1 405B on 32 H100 GPUs. (a) SLO: arrival-to-completion time per output token ≤ 100 ms. (b,c) Completion throughput and p95 request completion time under Poisson arrivals and synchronized bursts.