More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

Srikanta Datta Tumkur; Mehar Simhadri; Anshu Bansal; Jay Iyer; Sai Pavan Kumar; Sai Kapil Kumar; Ramesh Nampelly; Raj Dandekar

cs.AIAug 25, 2026

More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving

Authors: Srikanta Datta Tumkur, Mehar Simhadri, Anshu Bansal, Jay Iyer, Sai Pavan Kumar, Sai Kapil Kumar, Ramesh Nampelly, Raj Dandekar

Organizations: MIT and Vizuara

Abstract

When an LLM serving deployment runs out of KVcache room, there are two well-established ways out. Tensor parallelism shards the weights and the KV cache across two, four, or eight devices, buying memory headroom at the price of an all-reduce on every layer and a hardware bill that grows with the device count. The algorithms community shrinks the cache in place, with KV quantisation and eviction keeping a single GPU and spending a little quality instead. Compression papers report memory ratios, parallel-scaling papers report throughput curves, and almost nobody puts the two on the same cost axis. We place tensor-parallel configurations (degree 1 to 8) and KV-compressed configurations (16/8/4-bit, keep-ratios down to 0.25) on one costnormalised axis, cost per million tokens against latency, using a profiled simulator calibrated on A100, A40, and H100 hardware, and we go looking for the cost-equivalence crossover. We do not find one. Across two models (Llama-2 at 7B and 70B), three GPU types, and every level of memory relief we could construct, compression is cheaper by 1.20x to 2.00x. A 7B model on an 80 GB device cannot exhaust its KV budget within its own context window, and the boundary that decides between the strategies is model size relative to device memory, at roughly 36B parameters for an 80 GB card. Below that wall, compression dominates and extra GPUs are largely wasted spend; above it, tensor parallelism stops being a choice and becomes an entry ticket: Llama-2-70B is infeasible on one A100 at any KV setting, because the binding resource is weights, which KV compression does not touch. Tensor parallelism is the only lever that improves latency (compression makes per-token latency worse, by 8 to 93%, through batching contention), while compression is the only lever that multiplies capacity per dollar (16.5x, against 1.21x for an eightfold spend on GPUs).

Figures & tables

Fig. 1 : Study overview. (1) A memory-bound serving deployment can add GPUs (tensor parallelism, which splits weight and KV memory but adds all-reduce communication and multiplies cost) or compress the KV cache (quantisation and eviction, which keep one GPU but spend quality and dequantisation overhead). (2) The two are reported in separate literatures, on different axes, so a practitioner cannot tell which is cheaper. (3) We place both on one cost-normalised frontier (cost per million tokens against latency) at a fixed model, quality floor, and latency target, and search for the cost-equivalence crossover.

Fig. 2 : The cost-equivalence pipeline. A fixed model, quality floor, and latency target are held constant. On one side the tensor-parallel degree is swept (1, 2, 4, 8): each step divides per-GPU weight and KV memory, adds all-reduce communication, and multiplies cost by the device count. On the other side the KV-compression setting is swept (bit-width and keep-ratio). Both branches are evaluated through a profiled simulator and placed on a single cost-normalised frontier (cost per million tokens against latency), from which the dominant choice per regime, and the cost-equivalence crossover if one exists, are read. The schematic shows the pipeline as originally designed; the realised study substitutes Llama-2 models for the pictured Llama-3-8B (Sec. IV ) and inherits its calibration from the profiled backbone rather than from anchors of our own; the “measured anchors” and “crossover” the drawing depicts were design intent, and Sec. V reports that no crossover was found.

Regime	Intended source	Realised here
Short chat	ShareGPT	W1 (synthetic, 2048+256, unsaturated)
Long-context	LongBench / RULER	not reachable (profiling caps at 4k)
Large batch	ShareGPT batched	W2 (synthetic, 3840+256, 30 QPS)
Long + batched	RULER batched	W2 on Llama-2-70B (hardest bound)

TABLE I : Deployment regimes as designed, and how each was realised.

Setting	Value
Models	Llama-2-7B (primary); Llama-2-70B (above threshold)
GPUs	A100-80GB, A40-48GB, H100
Prices (assumed)	$2.00,$ 1.00, $3.50 per GPU-hour
TP degree	1, 2, 4, 8 (memory $/p$ , measured all-reduce, cost $\times p$ )
KV compression	bit-width 16/8/4 $\times$ keep-ratio 1.0/0.5/0.25
Workloads	W1: 2048+256, 64 req; W2: 3840+256, 384 req, 30 QPS

TABLE II : Principal settings.

Fig. 3 : Cost-normalised frontier on W2 S . Tensor-parallel configurations (circles) and KV-compressed configurations (squares) on cost per million tokens against TTFT P99. The uncompressed single-GPU baseline is dominated : INT8 is both cheaper and faster. The three compressed points coincide exactly, because past INT8 the additional headroom is unused in this prefill-bound workload.

Config	TTFT 99	TPOT 50	Thr.	$/h	$/Mtok
	(s)	(ms)	(tok/s)
fp16 TP1	234.9	72.79	204.5	2	2.717
fp16 TP2	133.9	48.85	356.7	4	3.115
fp16 TP4	83.7	32.15	569.9	8	3.899
fp16 TP8	52.4	21.58	909.2	16	4.888
INT8 TP1	223.3	78.83	214.4	2	2.591

TABLE III : W2 metric panel S . Llama-2-7B, A100-80GB, 3840+256 tokens, 30 QPS. fp16/TP

=1

is the baseline; the three compressed rows are identical because past INT8 the memory constraint has stopped binding.

Fig. 4 : Cost at matched KV relief S . Compression is cheaper at every level and the ratio grows with relief: there is no cost-equivalence crossover.

Fig. 5 : Throughput gain against device-count price on Llama-2-7B S . The dashed line is break-even. Both workloads sit below it at every degree, so cost per token rises monotonically with tensor-parallel degree below the feasibility threshold . Sec. V-C shows this does not hold above it.

Fig. 6 : Concurrency capacity per GPU-dollar E , ctx 4096. Moving right along any curve (more GPUs) is nearly flat; moving between curves (more compression) is a step change.

Fig. 7 : Sensitivity of the result to unmodelled dequantisation overhead. Bars give the throughput loss compression would need to suffer before tensor parallelism becomes cheaper.

Fig. 8 : Feasibility at TP $=1$ across the profiled model set E . Above the device budget, no KV setting restores feasibility: the constraint is weights.

Model	Wt. TP1	fp16	fp16	INT4+k25	INT4+k25
	(GB)	TP1	TP4	TP1	TP4
Llama-2-7b	12.1	29	137	479	2207
Llama-3-8B	13.0	118	550	1888	8800
internlm-20b	35.4	7	53	124	862
CodeLlama-34b	61.9	13	301	216	4824
Llama-2-70b	127.5	0	128	0	2054

TABLE IV : Concurrency capacity at ctx 4096 E ; 0 denotes OOM.

Config	Thr.	KV mem	$/h	$/Mtok
	(tok/s)	(%)
fp16 TP1	INFEASIBLE: 127.5 GB weights
fp16 TP2	53.3	99.1	4	20.846
fp16 TP4	123.8	25.8	8	17.950
fp16 TP8	184.5	9.2	16	24.089
INT8 TP2	68.3	97.8	4	16.268

TABLE V : W2 on Llama-2-70B S . Both arms start at TP

=2

, which is the baseline here, because TP

=1

is infeasible.

Fig. 9 : Matched relief on Llama-2-70B S . Compression remains cheaper above the threshold, where both arms start at TP $=2$ .

Fig. 10 : Cost per million tokens against parallel degree, normalised to each model’s cheapest feasible degree S . Monotonically increasing below the threshold; U-shaped above it.

Fig. 11 : Ablation A4 S : cheapest configuration per arm across GPU types (W2, 7B). Compression wins on every device; per-token cost orders opposite to hourly price.

Regime	Choose	Because
Weights exceed device memory ( $\gtrsim$ 36B at fp16, 80 GB)	TP at $p_{\min}$ ; compression cannot substitute	KV compression does not shrink weights (Table IV )
$p_{\min}$ runs memory-saturated	Step up one degree	$3.38\times$ throughput for $2\times$ price; cost falls 41% (W1) and 14% (W2), Sec. V-C
Weights fit; KV is binding	Compress first	$16.5\times$ capacity per dollar vs $1.21\times$ (Fig. 6 )
Weights fit; KV not binding	Neither; compression is inert	Coincident points, Fig. 3
Latency target unmet	TP, and pay for it	$4.5\times$ TTFT P99 cut; compression moves TPOT the wrong way ( $+8$ to $93\%$ )
Quality floor forbids compression	TP	Only remaining source of headroom

TABLE VI : Scale-out versus compress: the decision rule.

LLM Serving

Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.

See all 207 papers

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
Aug 30, 2026Hongyu Yu, Yifei ShenLLM Quantization LLM Pruning
Lenovo Research · University of Washington
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
Jul 14, 2026Rahul Krishnan, Volker SchulzTucker Decomposition KV Caching
Universität Trier · Fachbereich IV, Mathematik, Universität Trier
Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
Jun 4, 2026Hyungmin Kim, Minsoo Kim, Hongseok Kim +1LLM Serving KV Caching
Hanyang University Seoul, Republic of Korea · Apple · Rebellions Republic of Korea

Aug 30, 2026J/K move · Enter open · S save

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

Hongyu Yu, Yifei Shen

Lenovo Research · University of Washington

LLM Quantization LLM Pruning LLM Compression KV-Cache Compression Efficient Language Model Inference

Open paper AskPDF arXiv

Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.

Abstract

LLM Serving

Explore similar work

Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects

A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation

Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving