Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Arnold Olympio; Juan Manuel Servera Bondroit; Wael Abdelmalek; Guang Lu; João Carvalho

cs.PFOct 7, 2026

Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing

Authors: Arnold Olympio, Juan Manuel Servera Bondroit, Wael Abdelmalek, Guang Lu, João Carvalho

Organizations: Lucerne University of Applied Sciences and Arts, Switzerland · Microsoft, Switzerland · Uthereal AG, Switzerland · ETH Zurich, Switzerland

Abstract

Reproducible benchmarking of Large Language Model (LLM) inference is challenging because repeated measurements can vary with execution and system state. We present the Sequential Isolation Methodology, a controlled benchmarking and regression-testing protocol designed to reduce between-run measurement variance while deliberately varying workload concurrency. We evaluate three representative open-source LLMs on an NVIDIA A100 80GB GPU using vLLM 0.9.1 across six context sizes and eight concurrency levels, with five repetitions per configuration. The final protocol reduces average coefficient of variation (CV) from 15.2% in the least controlled methodology stage to 2.2% under the final protocol; using CV computed across the five repetition-level median (P50) TTFT values per configuration, 113 of 144 configurations (78.5%) achieve CV below 3%. The measurements also show a marked latency transition between 200 and 500 concurrent users on the tested stack and descriptive differences in P99 latency across the three models. We additionally provide an explicit cost break-even model with sensitivity to API pricing. The protocol is intended to provide a stable reference for reproducible comparison and regression testing rather than to predict absolute behavior under uncontrolled production traffic. Infrastructure-as-Code and benchmark scripts support replication of the experimental environment.

Figures & tables

Version	Approach	Avg. CV	Methodology stage
V1	Concurrent testing	15.2%	Least controlled baseline
V2	Parallel batches	10.8%	Intermediate batching-control stage
V3	Sequential + gaps	8.3%	Sequential runs with inter-run spacing
V4	Full isolation	2.2%	Final controlled protocol

Table 1: Cumulative Methodology Stages and Observed CV Reduction

Figure 1: Reproducibility across methodology stages and models. Left: average coefficient of variation (CV = $\sigma$ / $\mu$ $\times$ 100%) of median (P50) TTFT across five repetitions per fixed configuration, summarized for methodology stages V1–V4. Right: average configuration-level CV under the final protocol for each evaluated model. The 3% line indicates the predefined reference threshold used in the analysis. The 5% line labelled “Industry standard” is shown only as an additional visual reference and is not used in the statistical analysis or conclusions.

Figure 2: Queue saturation dynamics on the tested A100/vLLM stack. P50 TTFT is summarized across the evaluated models and context sizes at each target concurrency level, with five repetitions per fixed configuration. The plot shows the marked latency transition observed between 200 and 500 concurrent users.

Figure 3: Context-specific P99 TTFT comparison under the final protocol. Values summarize the tested concurrency levels for each model and context size, with five repetitions per fixed configuration. The lowest observed P99 TTFT at each context size is highlighted descriptively.

Model	Avg CV	Best CV	Worst CV
Mistral-7B	2.47%	0.17%	13.28%
Llama-3.1-8B	2.14%	0.36%	5.96%
Qwen2.5-7B	2.07%	0.16%	6.89%
Overall	2.23%	0.16%	13.28%

Supplementary Table 1: Reproducibility Results Across Models (Configuration-Level CV Statistics). Coefficient of Variation (CV =

\sigma

\mu

\times

100%) is computed separately for each fixed model

\times

context-size

\times

concurrency configuration across its five repetition-level median (P50) TTFT values under the final Sequential Isolation protocol. Experiments use vLLM 0.9.1 on an Azure A100 80GB GPU with –dtype auto (resolving to bfloat16 for the evaluated models). There are 48 configurations per model and 144 configurations overall. Across all models, 113 of 144 configurations (78.5%) have CV below the predefined 3% reference threshold.

Concurrent Users	Avg P50 TTFT (ms)	Degradation Factor
200	91	baseline (stable)
500	12,068	133 $\times$ (saturated)
1000	18,542	204 $\times$ (saturated)

Supplementary Table 2: Queue Saturation Dynamics. P50 TTFT values are summarized across the three evaluated models and six context sizes at each concurrent-user level on the tested Azure A100 80GB/vLLM 0.9.1 stack. The degradation factor is computed relative to the 200-user measurement, the highest tested load preceding the marked latency transition observed between 200 and 500 users. The mechanism underlying this transition was not instrumented directly; increasing memory pressure and scheduler queueing are plausible contributors.

Model	Avg P99 (ms)	vs Mistral	Load Wins
Qwen2.5-7B	8,022	-12.1%	5/8
Llama-3.1-8B	8,496	-6.9%	1/8
Mistral-7B	9,124	baseline	2/8

Supplementary Table 3: Descriptive P99 Tail-Latency Comparison Across Models. Average P99 TTFT is summarized across 48 configurations per model (6 context sizes

\times

8 user loads), with five repetitions per configuration, giving 240 runs per model. The “% vs Mistral” column reports the observed relative difference using Mistral-7B as the reference. The “Load Wins” column reports the number of the eight tested concurrency levels at which each model has the lowest observed P99 TTFT. These comparisons are descriptive; no inferential significance claim is made. Full per-load results are provided in Supplementary Table 4 .

Users	Mistral P99	Llama P99	Qwen P99	Winner
5	314ms	249ms	238ms	Qwen
10	93ms	82ms	118ms	Llama
20	107ms	147ms	111ms	Mistral
50	150ms	148ms	140ms	Qwen
100	268ms	272ms	263ms	Qwen
200	386ms	372ms	330ms	Qwen

Supplementary Table 4: Complete P99 TTFT Results by User Load (All Models). P99 Time to First Token (TTFT) values averaged across all six context sizes (1k–32k tokens) for each user load level, computed across the five repetitions per configuration under the V4 Sequential Isolation Methodology. Loads

\leq

200 users represent the stable operating regime; loads

\geq

500 users fall in the saturated regime, where P99 values approach vLLM’s per-request timeout and should be interpreted as characterizing system failure behavior rather than normal operation.

Context Size	Lowest observed model	P99 (ms)	Advantage
1k–4k	Qwen	7,852–8,667	12.8–18.9%
8k	Llama	6,987	23.7%
16k	Mistral	7,191	10.8%
32k	Qwen	6,051	29.4%

Supplementary Table 5: Context-Specific Lowest Observed P99 and Performance Differences. Model with the lowest observed P99 TTFT at each context size based on P99 TTFT averaged across all eight user load levels. The “Advantage” column reports the percentage improvement of the model with the lowest observed P99 TTFT over the next-best model (for single-context entries) or the range of improvements across the 1k–4k contexts. This summary provides a descriptive comparison of context-specific model performance on the tested setup.

Concurrent Users	Mistral-7B (tok/s)	Llama-3.1-8B (tok/s)	Qwen2.5-7B (tok/s)
5	67.8	63.6	70.2
10	61.6	55.5	64.0
20	52.0	46.6	55.7
50	45.2	38.5	42.1
100	38.0	31.1	33.8
200	26.5	20.6	21.9

Supplementary Table 6: Per-Request Throughput (tok/s) by Concurrent User Load — Average Across Context Sizes (

\leq

200 Users). Throughput values are derived from P50 inter-token latency (ITL) as 1000/ITL, then averaged across the six context sizes (1k–32k tokens) at each user load. Only stable operating loads (

\leq

200 concurrent users) are reported; saturated loads (

\geq

500 users) are excluded because ITL becomes dominated by queueing delay rather than generation speed. All measurements use the final Sequential Isolation protocol on an Azure A100 80GB GPU with vLLM 0.9.1 and –dtype auto, which resolves to bfloat16 for the evaluated models.

Requirement	Best observed model	Rationale
Strict P99 SLA	Qwen2.5-7B	Best tail latency control (12.1% advantage)
8k context workloads	Llama-3.1-8B	Context-specific optimization (23.7% P99 advantage)
Higher throughput across stable tested loads	Mistral-7B	Highest average per-request throughput across $\leq$ 200-user loads (48.5 tok/s)

Supplementary Table 7: Descriptive Model Comparison by Workload Requirement. This table summarizes the lowest observed latency or throughput result for selected conditions on the tested single-A100/vLLM setup. The comparisons are specific to the present benchmark and should not be interpreted as general production recommendations; different hardware, serving frameworks and workloads may change the relative ordering.

API price	Break-even tokens/month	Approx. Tokens/day
$0.15 / million	14.9 billion	498 million
$0.30 / million	7.5 billion	249 million
$0.60 / million	3.7 billion	124 million

Supplementary Table 8: Cost Sensitivity and Break-Even Analysis. A continuously provisioned Azure Standard_NC24ads_A100_v4 instance at

3.07/hour costs approximately

2,241/month using 730 billed hours. Break-even volume is calculated as

V^{*}=(c\times h)/p

, where

p

is the API price per token. Because API pricing varies, the table reports sensitivity across illustrative prices of

0.15–

0.60 per million tokens. Measured single-instance throughput at 70% utilization is approximately 288 million tokens/month, below the full break-even range. Engineering, storage, networking and other operating overheads are not included, so the comparison is indicative rather than universal.

Explore similar work

May 19, 2026cs.LG

The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility

Progress in LLMs is increasingly measured through standardized benchmarks, where state-of-the-art improvements are often separated by fractions of a percentage point. At the same time, the computational cost of evaluating modern LLMs has driven widespread adoption of specialized inference backends, software systems that execute trained models efficiently at inference time. While critical for scalability, system-level optimizations, such as custom CUDA kernels and reduced-precision arithmetic, can alter token probabilities and introduce non-determinism, possibly cascading into divergent generation. In this work, we first survey the inference landscape, identifying 200 distinct engines, and analyze 35,000 ML publications, finding that the specific inference stack is rarely reported despite this widespread diversity. We then present a systematic empirical study of how inference backends affect LLM benchmark results. Holding model weights, decoding parameters, and hardware constant, we evaluate five widely used inference engines, including vLLM, SGLang, and llama

.

cpp, across multiple open-weight models and established benchmarks. We show that the choice of backend alone can shift benchmark scores by up to 16.6 percentage points and induce high rates of output disagreement. By isolating backend optimizations and tracing the execution pipeline, we find this divergence is driven by system-level optimizations like prefix caching and CUDA graphs, custom kernels, and engine-specific defaults in logit processing. Our findings identify the inference backend as a previously unreported but consequential hyperparameter in the evaluation of LLM and advocate standardized reporting of inference stacks to improve the reproducibility and interpretability of benchmark comparisons.

David Pape, Jonathan Evertz, Lea Schönherr

CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

Large Language Model Benchmarks LLM Inference Optimization Hyperparameter Cuda Impact Analysis

Ask PDF

Sep 16, 2026cs.AI

AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall (2.4840 vs 2.4957,ms); and the KernelBench Level-1 suite admits 51% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail are released as open artifacts.

Li Chen

Agentic Benchmarks Large Language Model Benchmarks Tuning Graphics Processing Unit Kernels Large Language Model Serving Text Corpora

Ask PDF

Aug 5, 2026cs.SE

What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

Benchmark scores are reported as properties of a model, yet the inference framework used to produce them, such as HuggingFace, vLLM, or Ollama, are considered non-influential and their names and versions are almost never disclosed. In this work we investigate how much this choice can influence the model output. In a fully-crossed study (three instruction-tuned models x five inference frameworks x six benchmarks x four generation modes) we investigate how different tools (wrappers/backend) influence benchmark scores and how their score changes is influenced by generation hyper-parameters. We find backend to be a non-negligible factor where even under greedy, sampling-noise-free decoding, changing the backend can significantly alter models performance and this effect is structural and strongly model-dependent. Decomposing the variance according to generation mode reveal that considerable portion of the variability (roughly 39%) a practitioner sees out-of-the-box can stem from the backend, while the remaining stems from sampling noise and each framework's default generation parameters, both of which are avoidable by disclosing and matching the generation configuration. These divergences are more pronounced on factual than on social-bias benchmarks. Overall, benchmark numbers are not backend-agnostic therefore, we recommend disclosing the backend, its version, and the full generation configuration, also using deterministic decoding for cross-backend comparison.

Shahed Masoudian, Passant Shafaei, Monorama Swain +1

Johannes Kepler University Linz

Large Language Model Benchmarks LLM Inference Optimization

Ask PDF