Benchmarking LLM Inference on Desktop-Scale NVIDIA Grace Blackwell Clusters
Organizations: University of Delaware, Newark, Delaware, USA
Abstract
Two NVIDIA DGX Spark GB10 systems connect directly over 200 GbE to serve larger language models and increase inference capacity, but the benefits depend on how they are deployed. We benchmark 17 open-weight checkpoints spanning dense and mixture-of-experts architectures, 8B to 235B parameters and four weight precisions on one and two RDMA-connected systems. Using vLLM, we compare tensor parallelism (TP=2), pipeline parallelism (PP=2) and a two-replica baseline at concurrency 1 to 64, with three independent launches per configuration. TP=2 delivers a median 1.59x peak-throughput gain over one Spark, versus 1.39x for PP=2 and has higher throughput in 93 of 96 matched comparisons with separated 95% Welch intervals. It reduces unloaded inter-token latency to 0.52x to 0.73x of one Spark, whereas PP=2 provides little reduction. TP=2 also achieves the highest goodput under a strict 100 ms inter-token latency objective wherever both strategies are supported and the objective is met. For models fitting on one Spark, TP=2 generally favors low-concurrency serving, while replication offers higher throughput at high concurrency. One Spark uses less GPU energy per output token at matched concurrency. Across three precision ladders, NVFP4 reduces unloaded latency more than adding a second Spark at BF16. NCCL microbenchmarks and comparisons with an earlier TCP campaign suggest that collective communication overhead contributes to the advantage of TP=2. These results provide deployment guidance for desktop-scale inference.
Figures & tables
| Quantity | Unit | Definition |
|---|---|---|
| Measurement window | s | Interval over which throughput, goodput, and energy are measured: at least 600 s, extended in 60 s steps until at least 20 requests have started inside it (3,600 s cap). |
| Concurrency | requests | Requests the client keeps in flight. A new request is issued the moment one completes (closed loop). |
| Replicate | launch | One fresh server launch followed by a complete ascending sweep over . Every point has three replicates; the order of the three deployments rotates across replicates. |
| Output throughput | tok/s | Output tokens received by the client during , from all in-flight requests, divided by the length of . |
| Peak throughput | tok/s | Largest replicate-mean output throughput of a deployment over ; on 43 of the 47 curves it is the value at . |
| TTFT | s | Time to first token: request submission to arrival of the first output token at the client. |
| Checkpoint | Type | Attention | Wts | KV | GiB | Runs | KV capacity (req.) | ||
|---|---|---|---|---|---|---|---|---|---|
| TP=2 | PP=2 | ||||||||
| Llama-3.1-8B BF16 | dense | full | BF16 | BF16 | 15.0 | ✓✓✓ | 72 | 163 | 161 |
| Llama-3.1-8B FP8 | dense | full | FP8 | FP8 | 8.5 | ✓✓✓ | 158 | 337 | 337 |
| Llama-3.1-8B NVFP4 | dense | full | NVFP4 | FP8 | 5.7 | ✓✓✓ | 164 | 343 | 344 |
| Qwen3.6-27B BF16 | dense | linear hybrid | BF16 | BF16 | 50.2 | ✓✓✓ | 56 | 187 | 181 |
| Qwen3.6-27B FP8 | dense | linear hybrid | FP8 | BF16 | 27.6 | ✓✓✓ | 85 | 218 | 212 |
| Peak tok/s | Peak ratio | ITL at (ms) | Max. strict goodput (req/s) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Checkpoint | TP | PP | TP/ | PP/ | PP/TP | TP | PP | TP | PP | |||
| Llama-3.1-8B BF16 | 131 | 218 | 204 | 1.66 | 1.55 | 0.93 | 72 | 38 | 73 | 0.18 | 0.73 | 0.32 |
| Llama-3.1-8B FP8 | 321 | 466 | 439 | 1.45 | 1.37 | 0.94 | 41 | 22 | 42 | 0.79 | 1.56 | 0.85 |
| Llama-3.1-8B NVFP4 | 390 | 529 | 522 | 1.36 | 1.34 | 0.99 | 26 | 15 | 26 | 1.33 | 1.85 | 1.62 |
| Qwen3.6-27B BF16 | 64.2 | 106 | 92.8 | 1.65 | 1.45 | 0.88 | 229 | 123 | 233 | 0.00 | 0.00 | 0.00 |
| Qwen3.6-27B FP8 | 81.6 | 137 | 135 | 1.68 | 1.65 | 0.98 | 130 | 70 | 131 | 0.00 | 0.13 | 0.00 |
| Metric | Median | 90th pct. | Max | |
|---|---|---|---|---|
| Output throughput | 0.38 | 2.13 | 10.2 | 7 |
| Median ITL | 0.39 | 2.04 | 10.4 | 9 |
| Median TTFT | 0.54 | 3.33 | 10.6 | 15 |
| Strict goodput | 0.50 | 3.67 | 86.6 | 12 |
| Loose goodput | 0.55 | 3.86 | 173.2 | 17 |
| Energy per token | 0.51 | 1.44 | 5.3 | 1 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Min. J/token | Ratio | GPU power at (W) | ||||||
|---|---|---|---|---|---|---|---|---|
| Checkpoint | TP | PP | TP/ | PP/ | TP | PP | ||
| Llama-3.1-8B BF16 | 0.47 | 0.53 | 0.55 | 1.12 | 1.16 | 63 | 115 | 111 |
| Llama-3.1-8B FP8 | 0.23 | 0.27 | 0.27 | 1.18 | 1.20 | 73 | 125 | 119 |
| Llama-3.1-8B NVFP4 | 0.17 | 0.20 | 0.21 | 1.18 | 1.19 | 67 | 107 | 107 |
| Qwen3.6-27B BF16 | 1.11 | 1.29 | 1.23 | 1.15 | 1.10 | 71 | 136 | 114 |
| Qwen3.6-27B FP8 | 0.77 | 0.87 | 0.83 | 1.12 | 1.07 | 62 | 121 | 111 |
| Peak tok/s ( ) | ITL at (ms) | ||||||
|---|---|---|---|---|---|---|---|
| Checkpoint | Deployment | TCP | RDMA | ratio | TCP | RDMA | ratio |
| Llama-3.1-8B BF16 | 1 | 130 | 131 | 1.01 | 72 | 72 | 1.01 |
| TP=2 | 130 | 218 | 1.68 | 48 | 38 | 0.78 | |
| PP=2 | 197 | 204 | 1.04 | 75 | 73 | 0.97 | |
| Llama-3.1-8B FP8 | 1 | 323 | 321 | 0.99 | 39 | 41 | 1.04 |
| TP=2 | 197 | 466 | 2.37 | 29 | 22 | 0.77 | |
| Item | Value |
|---|---|
| Container image | nvcr.io/nvidia/vllm (NGC release 26.09) on both nodes, digest sha256:557747337846eecf34486866f3d992383c6f8af869fdb0e365b5093fcaee02c2 |
| Engine | vLLM 0.29.0 (build 0.29.0+5013de39.nv26.9 ), PyTorch 2.14.0a0 (build nv26.9 ), CUDA runtime 13.4, Python 3.12.3 |
| Kernel libraries | FlashInfer 0.6.18.post1, flash-attn 2.8.3.post1 |
| Cluster software | Ray 2.56.1, NCCL 2.31.2 |
| GPU and driver | NVIDIA GB10, driver 580.178.04, compute capability 12.1, application clock 2,418 MHz on both nodes |
| Head operating system | Ubuntu 24.04.5 LTS, kernel 7.0.0-1019-nvidia (aarch64); the container runs Ubuntu 24.04.4 LTS |
| Checkpoint | Quantization | 4-bit linear kernel | FP8 linear kernel | MoE backend | Attention |
|---|---|---|---|---|---|
| Llama-3.1-8B BF16 | none | – | – | – | FlashAttention |
| Llama-3.1-8B FP8 | FP8 | – | FlashInfer scaled MM | – | FlashInfer |
| Llama-3.1-8B NVFP4 | NVFP4 | FlashInfer CUTLASS | – | – | FlashInfer |
| Qwen3.6-27B BF16 | none | – | – | – | FlashAttention |
| Qwen3.6-27B FP8 | block FP8 | – | CUTLASS block-scaled | – | FlashAttention |
| Qwen3.6-27B NVFP4 | NVFP4+W4A16+FP8 | Marlin, weight-only | FlashInfer scaled MM | – | FlashInfer |
LLM Inference
Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.
See all 319 papers