cs.AIAug 25, 2026

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

Authors: Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal, Jay Iyer, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly, Raj Dandekar

Organizations: MIT and Vizuara

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight devices, buying memory headroom at the price of an all-reduce on every layer and a hardware bill that grows with the device count. The algorithms community shrinks the cache in place, with KV quantisation and eviction keeping a single GPU and spending a little quality instead. Compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost axis. We place tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16/8/4-bit, keep-ratios down to 0.25) on one costnormalised axis, cost per million tokens against latency, using a profiled simulator calibrated on A100, A40, and H100 hardware, and we go looking for the cost-equivalence crossover. We do not find one. Across two models (Llama-2 at 7B and 70B), three GPU types, and every level of memory relief we could construct, compression is cheaper by 1.20x to 2.00x. A 7B model on an 80 GB device cannot exhaust its KV budget within its own context window, and the boundary that decides between the strategies is model size relative to device memory, at roughly 36B parameters for an 80 GB card. Below that wall, compression dominates and extra GPUs are largely wasted spend; above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on one A100 at any KV setting, because the binding resource is weights, which KV compression does not touch. Tensor parallelism is the only lever that improves latency (compression makes per-token latency worse, by 8 to 93%, through batching contention), while compression is the only lever that multiplies capacity per dollar (16.5x, against 1.21x for an eightfold spend on GPUs).

Figures & tables

Fig. 1 : Study overview. (1) A memory-bound serving deployment can add GPUs (tensor parallelism, which splits weight and KV memory but adds all-reduce communication and multiplies cost) or compress the KV cache (quantisation and eviction, which keep one GPU but spend quality and dequantisation overhead). (2) The two are reported in separate literatures, on different axes, so a practitioner cannot tell which is cheaper. (3) We place both on one cost-normalised frontier (cost per million tokens against latency) at a fixed model, quality floor, and latency target, and search for the cost-equivalence crossover.
Fig. 2 : The cost-equivalence pipeline. A fixed model, quality floor, and latency target are held constant. On one side the tensor-parallel degree is swept (1, 2, 4, 8): each step divides per-GPU weight and KV memory, adds all-reduce communication, and multiplies cost by the device count. On the other side the KV-compression setting is swept (bit-width and keep-ratio). Both branches are evaluated through a profiled simulator and placed on a single cost-normalised frontier (cost per million tokens against latency), from which the dominant choice per regime, and the cost-equivalence crossover if one exists, are read. The schematic shows the pipeline as originally designed; the realised study substitutes Llama-2 models for the pictured Llama-3-8B (Sec. IV ) and inherits its calibration from the profiled backbone rather than from anchors of our own; the “measured anchors” and “crossover” the drawing depicts were design intent, and Sec. V reports that no crossover was found.
RegimeIntended sourceRealised here
Short chatShareGPTW1 (synthetic, 2048+256, unsaturated)
Long-contextLongBench / RULERnot reachable (profiling caps at 4k)
Large batchShareGPT batchedW2 (synthetic, 3840+256, 30 QPS)
Long + batchedRULER batchedW2 on Llama-2-70B (hardest bound)
TABLE I : Deployment regimes as designed, and how each was realised.
SettingValue
ModelsLlama-2-7B (primary); Llama-2-70B (above threshold)
GPUsA100-80GB, A40-48GB, H100
Prices (assumed)2.00,2.00, 1.00, $3.50 per GPU-hour
TP degree1, 2, 4, 8 (memory /p/p , measured all-reduce, cost ×p\times p )
KV compressionbit-width 16/8/4 ×\times keep-ratio 1.0/0.5/0.25
WorkloadsW1: 2048+256, 64 req; W2: 3840+256, 384 req, 30 QPS
TABLE II : Principal settings.
Fig. 3 : Cost-normalised frontier on W2 S . Tensor-parallel configurations (circles) and KV-compressed configurations (squares) on cost per million tokens against TTFT P99. The uncompressed single-GPU baseline is dominated : INT8 is both cheaper and faster. The three compressed points coincide exactly, because past INT8 the additional headroom is unused in this prefill-bound workload.
ConfigTTFT 99TPOT 50Thr.$/h$/Mtok
 (s)(ms)(tok/s)  
fp16 TP1234.972.79204.522.717
fp16 TP2133.948.85356.743.115
fp16 TP483.732.15569.983.899
fp16 TP852.421.58909.2164.888
INT8 TP1223.378.83214.422.591
TABLE III : W2 metric panel S . Llama-2-7B, A100-80GB, 3840+256 tokens, 30 QPS. fp16/TP =1=1 is the baseline; the three compressed rows are identical because past INT8 the memory constraint has stopped binding.
Fig. 4 : Cost at matched KV relief S . Compression is cheaper at every level and the ratio grows with relief: there is no cost-equivalence crossover.
Fig. 5 : Throughput gain against device-count price on Llama-2-7B S . The dashed line is break-even. Both workloads sit below it at every degree, so cost per token rises monotonically with tensor-parallel degree below the feasibility threshold . Sec. V-C shows this does not hold above it.
Fig. 6 : Concurrency capacity per GPU-dollar E , ctx 4096. Moving right along any curve (more GPUs) is nearly flat; moving between curves (more compression) is a step change.
Fig. 7 : Sensitivity of the result to unmodelled dequantisation overhead. Bars give the throughput loss compression would need to suffer before tensor parallelism becomes cheaper.
Fig. 8 : Feasibility at TP =1=1 across the profiled model set E . Above the device budget, no KV setting restores feasibility: the constraint is weights.
ModelWt. TP1fp16fp16INT4+k25INT4+k25
 (GB)TP1TP4TP1TP4
Llama-2-7b12.1291374792207
Llama-3-8B13.011855018888800
internlm-20b35.4753124862
CodeLlama-34b61.9133012164824
Llama-2-70b127.5012802054
TABLE IV : Concurrency capacity at ctx 4096 E ; 0 denotes OOM.
ConfigThr.KV mem$/h$/Mtok
 (tok/s)(%)  
fp16 TP1INFEASIBLE: 127.5 GB weights
fp16 TP253.399.1420.846
fp16 TP4123.825.8817.950
fp16 TP8184.59.21624.089
INT8 TP268.397.8416.268
TABLE V : W2 on Llama-2-70B S . Both arms start at TP =2=2 , which is the baseline here, because TP =1=1 is infeasible.
Fig. 9 : Matched relief on Llama-2-70B S . Compression remains cheaper above the threshold, where both arms start at TP =2=2 .
Fig. 10 : Cost per million tokens against parallel degree, normalised to each model’s cheapest feasible degree S . Monotonically increasing below the threshold; U-shaped above it.
Fig. 11 : Ablation A4 S : cheapest configuration per arm across GPU types (W2, 7B). Compression wins on every device; per-token cost orders opposite to hourly price.
RegimeChooseBecause
Weights exceed device memory ( ≳\gtrsim 36B at fp16, 80 GB)TP at pmin⁡p_{\min} ; compression cannot substituteKV compression does not shrink weights (Table IV )
pmin⁡p_{\min} runs memory-saturatedStep up one degree3.38×3.38\times throughput for 2×2\times price; cost falls 41% (W1) and 14% (W2), Sec. V-C
Weights fit; KV is bindingCompress first16.5×16.5\times capacity per dollar vs 1.21×1.21\times (Fig. 6 )
Weights fit; KV not bindingNeither; compression is inertCoincident points, Fig. 3
Latency target unmetTP, and pay for it4.5×4.5\times TTFT P99 cut; compression moves TPOT the wrong way ( +8+8 to 93%93\% )
Quality floor forbids compressionTPOnly remaining source of headroom
TABLE VI : Scale-out versus compress: the decision rule.
More in this topic

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Explore similar work

CardsList
Aug 30, 2026

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.

Hongyu Yu, Yifei Shen

Lenovo Research · University of Washington

LLM QuantizationLLM PruningLLM CompressionKV-Cache CompressionEfficient Language Model Inference
AskPDF
Jul 14, 2026

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

The key-value (KV) cache is the dominant memory bottleneck in long-context language model inference. Existing compression methods apply low-rank factorization or quantization independently, without jointly allocating rank and precision under a shared storage budget. We introduce JoLT, a training-free compressor that treats grouped prefill caches as fourth-order tensors and applies partial Tucker decomposition along the token and feature modes, the two axes that carry low-rank structure, while leaving the head and layer modes intact. A rotated low-bit quantizer captures the truncation residual, and a single Lagrangian dual allocates per-group Tucker ranks and residual bit-widths under a global byte constraint. FlashJoLT replaces the exact token-mode SVD with a randomized approximation that matches JoLT within the free zone at a fraction of the compression cost, and a fused Triton decode kernel evaluates attention directly over the stored factors without materializing dense KV tensors. Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2 - 3x compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3x and declines by only 0.90 and 2.40pp at 4x and 5x, respectively. JoLT demonstrates that tensor-aware low-rank decomposition and quantized residuals, unified under a single storage budget, achieve near-lossless KV-cache compression across diverse model architectures without retraining.

Rahul Krishnan, Volker Schulz

Universität Trier · Fachbereich IV, Mathematik, Universität Trier

Tucker DecompositionKV CachingLow-Rank Matrix DecompositionTensor DecompositionLow-Rank CompressionLow-Rank Approximation+2
AskPDF
Jun 4, 2026

Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving

Multi-turn LLM serving accumulates dialogue history whose Key-Value (KV) cache grows with every turn and every user, quickly exceeding the model weights themselves and making memory -- not compute -- the binding constraint on throughput. Non-uniform KV compression, which allocates heterogeneous budgets across attention heads, preserves accuracy far better than uniform schemes, yet remains impractical: modern serving stacks assume identical KV lengths across heads, so heterogeneity traps freed memory as page fragmentation, spends up to 25% of prefill time reclaiming scattered pages, and skews GPU workloads that inflate decode latency by up to 1.7×1.7\times or burn 15--20% of each decode step on re-planning. We observe that this heterogeneity need not be discovered at runtime: head-wise retention follows a two-level structural regularity -- an input-invariant head ranking with narrowly bounded per-head ratios -- that can be calibrated offline from as few as 50 samples. Building on this insight, we present Tangram, a serving framework that statically resolves what prior systems handle dynamically: Budget Reservation fixes each head's post-compression footprint at scheduling time, eliminating page reclamation; Ragged Paging clusters similar-budget heads into independent page tables, turning fragmentation into reclaimable memory; and Ahead-of-Time Load Balancing precomputes balanced GPU partitions with zero runtime planning. Implemented on vLLM, Tangram serves as a drop-in substrate for existing non-uniform compression methods, matching their accuracy while improving end-to-end throughput by up to 2.6×2.6\times over the full-KV baseline. Our implementation is publicly available at https://github.com/aiha-lab/TANGRAM.

Hyungmin Kim, Minsoo Kim, Hongseok Kim +1

Hanyang University Seoul, Republic of Korea · Apple · Rebellions Republic of Korea

LLM ServingKV CachingMemory-Efficient OptimizationLLM Inference AccelerationKV-Cache Compression
AskPDF