Inference Validation for Regulated AI: A Governed Decision Pipeline for LLM Serving
Organizations: MasterControl, Inc.
Abstract
Deploying large language models in regulated industries requires more than picking the fastest inference engine: it requires evidence that a configuration meets contractual and legal obligations before it serves a single customer request. We describe Runway, an internal validation harness built at MasterControl that builds a governed decision pipeline around NVIDIA AIPerf. The key insight is separating concerns: AIPerf provides the trusted measurement, while Runway provides the experiment design that comes before it and the compliance, cost, and audit decisions that come after. We describe the architecture and the storage model that makes caching and audit one mechanism. We report three findings from Runway sweeps: TensorRT-LLM and vLLM rankings invert with concurrency; a newer, pricier GPU instance can be the cheaper one per request; and speculative decoding behaves differently across engines under load, so engine configuration can outweigh engine choice.
Figures & tables
| Concurrency | Throughput Winner | p95 Latency Winner |
|---|---|---|
| 1 | TensorRT-LLM (+35%) | TensorRT-LLM (12/12) |
| 5 | TensorRT-LLM (+13%) | TensorRT-LLM (12/12) |
| 10 | TensorRT-LLM (+10%) | TensorRT-LLM (12/12) |
| 50 | vLLM (+3%) | TensorRT-LLM (7/12) |
| 100 | vLLM (+19%) | vLLM (12/12) |
| Concurrency | Throughput | p50 Latency | Cost per Request |
|---|---|---|---|
| 1 | 2.13 | 0.47 | 0.85 |
| 5 | 1.82 | 0.55 | 0.99 |
| 10 | 1.85 | 0.54 | 0.98 |
| 50 | 1.94 | 0.52 | 0.93 |
| 100 | 2.04 | 0.49 | 0.88 |
| Concurrency | vLLM: MTP / baseline | TensorRT-LLM: MTP / baseline |
|---|---|---|
| 1 | 2.16 | 2.57 |
| 16 | 1.11 | 1.69 |
| 32 | 0.92 | 1.32 |
| 64 | 0.80 | 1.45 |
LLM Inference Efficiency
Large language model inference efficiency reduces the time, memory, energy, and cost of generating useful outputs.
See all 362 papers