cs.PFOct 7, 2026
SaveReproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing
Organizations: Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland · ETH Zurich, Switzerland
Abstract
Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.
Figures & tables
| Version | Approach | Avg. CV | Methodology stage |
|---|---|---|---|
| V1 | Concurrent testing | 15.2% | Least controlled baseline |
| V2 | Parallel batches | 10.8% | Intermediate batching-control stage |
| V3 | Sequential + gaps | 8.3% | Sequential runs with inter-run spacing |
| V4 | Full isolation | 2.2% | Final controlled protocol |
Table 1: Cumulative Methodology Stages and Observed CV Reduction
Figure 1: Reproducibility across methodology stages and models. Left: average coefficient of variation (CV = / 100%) of median (P50) TTFT across five repetitions per fixed configuration, summarized for methodology stages V1–V4. Right: average configuration-level CV under the final protocol for each evaluated model. The 3% line indicates the predefined reference threshold used in the analysis. The 5% line labelled “Industry standard” is shown only as an additional visual reference and is not used in the statistical analysis or conclusions.
Figure 2: Queue saturation dynamics on the tested A100/vLLM stack. P50 TTFT is summarized across the evaluated models and context sizes at each target concurrency level, with five repetitions per fixed configuration. The plot shows the marked latency transition observed between 200 and 500 concurrent users.
Figure 3: Context-specific P99 TTFT comparison under the final protocol. Values summarize the tested concurrency levels for each model and context size, with five repetitions per fixed configuration. The lowest observed P99 TTFT at each context size is highlighted descriptively.
| Model | Avg CV | Best CV | Worst CV |
|---|---|---|---|
| Mistral-7B | 2.47% | 0.17% | 13.28% |
| Llama-3.1-8B | 2.14% | 0.36% | 5.96% |
| Qwen2.5-7B | 2.07% | 0.16% | 6.89% |
| Overall | 2.23% | 0.16% | 13.28% |
Supplementary Table 1: Reproducibility Results Across Models (Configuration-Level CV Statistics). Coefficient of Variation (CV = / 100%) is computed separately for each fixed model context-size concurrency configuration across its five repetition-level median (P50) TTFT values under the final Sequential Isolation protocol. Experiments use vLLM 0.9.1 on an Azure A100 80GB GPU with –dtype auto (resolving to bfloat16 for the evaluated models). There are 48 configurations per model and 144 configurations overall. Across all models, 113 of 144 configurations (78.5%) have CV below the predefined 3% reference threshold.
| Concurrent Users | Avg P50 TTFT (ms) | Degradation Factor |
|---|---|---|
| 200 | 91 | baseline (stable) |
| 500 | 12,068 | 133 (saturated) |
| 1000 | 18,542 | 204 (saturated) |
Supplementary Table 2: Queue Saturation Dynamics. P50 TTFT values are summarized across the three evaluated models and six context sizes at each concurrent-user level on the tested Azure A100 80GB/vLLM 0.9.1 stack. The degradation factor is computed relative to the 200-user measurement, the highest tested load preceding the marked latency transition observed between 200 and 500 users. The mechanism underlying this transition was not instrumented directly; increasing memory pressure and scheduler queueing are plausible contributors.
| Model | Avg P99 (ms) | vs Mistral | Load Wins |
|---|---|---|---|
| Qwen2.5-7B | 8,022 | -12.1% | 5/8 |
| Llama-3.1-8B | 8,496 | -6.9% | 1/8 |
| Mistral-7B | 9,124 | baseline | 2/8 |
Supplementary Table 3: Descriptive P99 Tail-Latency Comparison Across Models. Average P99 TTFT is summarized across 48 configurations per model (6 context sizes 8 user loads), with five repetitions per configuration, giving 240 runs per model. The “% vs Mistral” column reports the observed relative difference using Mistral-7B as the reference. The “Load Wins” column reports the number of the eight tested concurrency levels at which each model has the lowest observed P99 TTFT. These comparisons are descriptive; no inferential significance claim is made. Full per-load results are provided in Supplementary Table 4 .
| Users | Mistral P99 | Llama P99 | Qwen P99 | Winner |
|---|---|---|---|---|
| 5 | 314ms | 249ms | 238ms | Qwen |
| 10 | 93ms | 82ms | 118ms | Llama |
| 20 | 107ms | 147ms | 111ms | Mistral |
| 50 | 150ms | 148ms | 140ms | Qwen |
| 100 | 268ms | 272ms | 263ms | Qwen |
| 200 | 386ms | 372ms | 330ms | Qwen |
Supplementary Table 4: Complete P99 TTFT Results by User Load (All Models). P99 Time to First Token (TTFT) values averaged across all six context sizes (1k–32k tokens) for each user load level, computed across the five repetitions per configuration under the V4 Sequential Isolation Methodology. Loads 200 users represent the stable operating regime; loads 500 users fall in the saturated regime, where P99 values approach vLLM’s per-request timeout and should be interpreted as characterizing system failure behavior rather than normal operation.
| Context Size | Lowest observed model | P99 (ms) | Advantage |
|---|---|---|---|
| 1k–4k | Qwen | 7,852–8,667 | 12.8–18.9% |
| 8k | Llama | 6,987 | 23.7% |
| 16k | Mistral | 7,191 | 10.8% |
| 32k | Qwen | 6,051 | 29.4% |
Supplementary Table 5: Context-Specific Lowest Observed P99 and Performance Differences. Model with the lowest observed P99 TTFT at each context size based on P99 TTFT averaged across all eight user load levels. The “Advantage” column reports the percentage improvement of the model with the lowest observed P99 TTFT over the next-best model (for single-context entries) or the range of improvements across the 1k–4k contexts. This summary provides a descriptive comparison of context-specific model performance on the tested setup.
| Concurrent Users | Mistral-7B (tok/s) | Llama-3.1-8B (tok/s) | Qwen2.5-7B (tok/s) |
| 5 | 67.8 | 63.6 | 70.2 |
| 10 | 61.6 | 55.5 | 64.0 |
| 20 | 52.0 | 46.6 | 55.7 |
| 50 | 45.2 | 38.5 | 42.1 |
| 100 | 38.0 | 31.1 | 33.8 |
| 200 | 26.5 | 20.6 | 21.9 |
Supplementary Table 6: Per-Request Throughput (tok/s) by Concurrent User Load — Average Across Context Sizes ( 200 Users). Throughput values are derived from P50 inter-token latency (ITL) as 1000/ITL, then averaged across the six context sizes (1k–32k tokens) at each user load. Only stable operating loads ( 200 concurrent users) are reported; saturated loads ( 500 users) are excluded because ITL becomes dominated by queueing delay rather than generation speed. All measurements use the final Sequential Isolation protocol on an Azure A100 80GB GPU with vLLM 0.9.1 and –dtype auto, which resolves to bfloat16 for the evaluated models.
| Requirement | Best observed model | Rationale |
|---|---|---|
| Strict P99 SLA | Qwen2.5-7B | Best tail latency control (12.1% advantage) |
| 8k context workloads | Llama-3.1-8B | Context-specific optimization (23.7% P99 advantage) |
| Higher throughput across stable tested loads | Mistral-7B | Highest average per-request throughput across 200-user loads (48.5 tok/s) |
Supplementary Table 7: Descriptive Model Comparison by Workload Requirement. This table summarizes the lowest observed latency or throughput result for selected conditions on the tested single-A100/vLLM setup. The comparisons are specific to the present benchmark and should not be interpreted as general production recommendations; different hardware, serving frameworks and workloads may change the relative ordering.
| API price | Break-even tokens/month | Approx. Tokens/day |
|---|---|---|
| $0.15 / million | 14.9 billion | 498 million |
| $0.30 / million | 7.5 billion | 249 million |
| $0.60 / million | 3.7 billion | 124 million |
Supplementary Table 8: Cost Sensitivity and Break-Even Analysis. A continuously provisioned Azure Standard_NC24ads_A100_v4 instance at 2,241/month using 730 billed hours. Break-even volume is calculated as , where is the API price per token. Because API pricing varies, the table reports sensitivity across illustrative prices of 0.60 per million tokens. Measured single-instance throughput at 70% utilization is approximately 288 million tokens/month, below the full break-even range. Engineering, storage, networking and other operating overheads are not included, so the comparison is indicative rather than universal.