Validation-driven Deployment of LLM Inference Platforms for Agentic Workloads
Organizations: IBM Research · Red Hat AI
Abstract
Enterprises are moving agentic workloads from closed APIs to their own open-weight model deployments. For this reason, lifecycle-wide validation, from small-scale testing through sizing and pre-production to production, becomes all the more important. Typically, rollouts are evaluated with ad hoc tools whose accuracy is rarely checked. We present an end-to-end evaluation pipeline for agentic LLM serving and empirically ground the design choices that connect its stages. Our experiments show that replaying captured traces reproduces the per-session load of the original agent sessions: at the 95th percentile replay stays within 1.18x of the recorded input tokens while live re-runs reach 4.98x, so a small set of recorded sessions yields a repeatable per-session token load. A code-coverage analysis of vLLM further shows that the evaluated live, replayed, and synthetic workloads are complementary in this configuration: each exercises Python serving paths not reached in the measured runs of the others. In a case study rolling out four large mixture-of-experts (MoE) models on llm-d, experiments show that the target models reached about half the optimal concurrency, and correspondingly about 60% of the throughput, on captured workloads compared with public traces. We open-source the relevant tools and release a trace dataset.
Figures & tables
| Type | Benchmark (Model) | Med. | P95 | Med. | P95 | Med. | P95 |
|---|---|---|---|---|---|---|---|
| Replay | HWE-Bench (GLM-5.3) | 0.91 | 1.12 | 0.74 | 1.28 | 0.89 | 1.14 |
| Re-run | HWE-Bench (GLM-5.3) | 1.01 | 4.98 | 1.09 | 3.84 | 1.01 | 1.85 |
| Replay | CyberGym (DeepSeek) | 1.00 | 1.18 | 1.05 | 2.04 | 1.02 | 1.30 |
| Re-run | CyberGym (DeepSeek) | 1.14 | 3.97 | 1.16 | 4.52 | 1.06 | 1.76 |
| Benchmark (Model) | TPOT CV | ||
|---|---|---|---|
| HWE-Bench (GLM-5.3) | 5.2% | 92% | 86.8% |
| CyberGym (DeepSeek) | 10.7% | 99.6% | 99.3% |
| Family | Runs | Cov. | Uniq. | Reached only by this family |
|---|---|---|---|---|
| live | 1 | 8,777 | 331 | Anthropic messages path; aborted KV transfer |
| weka | 3 | 8,431 | 54 | streaming tool-call parsing; deferred offload fences |
| exgentic-traces | 3 | 8,967 | 27 | sparse-attention short-context branch |
| smoke | 1 | 5,748 | 20 | greedy sampling |
| synthetic | 16 | 8,795 | 5 | sparse-SWA chunk cutoff |
| Union | 24 | 9,778 | 437 |
| Workload | Input tokens | Output tokens |
|---|---|---|
| HWE-Bench | 0.135 | 0.044 |
| CyberGym | 0.140 | 0.133 |
| HANDBOOK | 0.160 | 0.164 |
| DeepSWE | 0.161 | 0.081 |
| WEKA | 0.223 | 0.212 |
| Lesson | Evidence |
|---|---|
| Public traces suffice for initial configuration, not for sizing. | Tuning on WEKA and Moontrace gave a working configuration, but captured workloads reached the optimal concurrency and of the throughput (Sizing, step 1). |
| Record and replay on the target model. | Replay fidelity is highest when the original model is used, since its responses are closer to the recorded version, whereas equivalent fidelity cannot be expected from traces generated by other models. |
| No single benchmark matches the typical agentic workload. | Every captured trace was closer to the typical agentic workload than WEKA, yet none matched it alone (Table 4 ). |
| Router bugs depend on the prefix structure of the workload. | The routing imbalance appeared only with a constant shared prompt prefix, which WEKA lacks (Pre-production sizing, step 2). |
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Run | Family | Duration (min) | Cov. | Workload |
|---|---|---|---|---|
| smoke | smoke | 0.1 | 5,748 | Single non-streaming chat request (17 prompt tokens, 128 max tokens, temperature 0). Cold-start and single-call baseline. |
| cybergym | live | 136.8 | 8,777 | Live CyberGym agent benchmark on the Anthropic messages API—the only workload not using the OpenAI chat API. Large enough KV footprint to fill the CPU offload tier and force eviction. |
| weka-20m-c8 | weka | 20.0 | 8,245 | AIPerf streaming chat replay of WEKA multi-agent traces, concurrency 8; 291 requests. |
| weka-20m-c24 | weka | 20.0 | 8,331 | Same corpus and scenario at concurrency 24; 333 requests, 11 client timeouts (3.3% error rate). |
| weka-1h-c24 | weka | 60.0 | 8,332 | Same corpus at concurrency 24 over one hour; 822 requests, 40 timeouts. Long enough to outlive the metrics-window retention. |
| exgentic-2m-c10 | exgentic-traces | 2.0 | 8,738 | Inference Perf non-streaming replay of Exgentic/agent-llm-traces , 10 concurrent sessions; 49 requests. First coverage window after engine start, so it also captures lazy imports. |
| Class | Covered | Statements | Coverage |
|---|---|---|---|
| Reachable here | 74,562 | 239,319 | 31.16% |
| Features not enabled | 0 | 66,335 | 0% |
| Other architectures | 0 | 109,926 | 0% |
| Other accelerator | 0 | 4,091 | 0% |
| Total | 74,562 | 419,671 | 17.77% |
| Family | Runs | Duration | Coverage | Unique | What only this family reached |
|---|---|---|---|---|---|
| live | 1 | 136.8 | 8,777 | 331 | The only family to use the Anthropic messages API : its request handler, router, protocol models, and the token scanner that drives incremental parsing on that path. |
| weka | 3 | 100.0 | 8,431 | 54 | The only streaming family, which shifts tool-call parsing to different partial-token positions. In the KV offload scheduler , the only family to reconcile deferred fences : blocks still in flight to CPU as their requests finish, reconciled when those blocks come back for reuse. |
| exgentic-traces | 3 | 82.0 | 8,967 | 27 | The only family to enter the short-context branch of sparse attention , taken when a request offers fewer index candidates than the top- width so the indices are filled directly rather than selected. Also tool-call parser states reached only by the recorded trace mix: parser-engine transitions, the token scanner’s anchor-rescan path, and the tool-argument JSON wrapper’s non-finite-value fallback. |
| synthetic | 16 | 240.5 | 8,795 | 5 | Two tool-argument coercion branches, one parser fallback, and the workspace-area cutoff in sparse-SWA chunk planning , reached only by this family’s very long prompts. |
| smoke | 1 | 0.1 | 5,748 | 20 | Greedy sampling : at temperature 0 the sampling parameters take the zero-temperature branch that every other workload skips. Also the profiling-scope and first-request metrics paths that only a cold single request reaches. |
| Union | 24 | 559.4 | 9,778 | 437 |
| State, and the code that enters it | Stmts | smoke | synthetic | exgentic | weka | live | What the state is ( italics ), and what it takes to reach it |
|---|---|---|---|---|---|---|---|
| Queue backpressure: defer a waiting request that already holds KV blocks vllm/v1/core/sched/scheduler.py 878–883, 1345 req = from_queue.pop_request() self.deferred_waiting.add(req) if self._holds_kv_blocks(req): step_skipped_kv_holding.appendleft(req) | 7 | – | 7 | 7 | – | – | The scheduler cannot start every queued request in one step, and one it must postpone is already holding GPU cache blocks from a previous turn. It has to set that request aside without losing track of the blocks. Reaching it means offering more concurrent sessions than the token budget admits. Only the two generated and replayed agent families sustained enough queued work; the hour-long WEKA replays at concurrency 24 never filled the budget this way. |
| Prefix-cache hash collision: one block hash mapping to several blocks vllm/v1/core/block_pool.py 68–69, 91–97 elif isinstance(blocks, KVCacheBlock): self._cache[key] = {blocks.block_id: blocks, block.block_id: block} if isinstance(blocks, dict): return next(iter(blocks.values())) | 6 | – | 6 | 6 | 2 | – | Two different cache blocks turn out to have the same content hash, so the prefix index can no longer store one block per key and must keep a set of them. Reaching it means sending many sessions whose prompts share an identical prefix. Recursive agent fan-out over a common system prompt is what produces this. |
| LRU heap compaction while evicting from the CPU tier vllm/v1/kv_offload/cpu/policies/lru.py 55–70, 168 if len(self._heap) < max(64, 2 * len(self._evictable)): return self._heap = [(self._ranks[k], k) for k in self._evictable] heapq.heapify(self._heap) | 9 | – | 7 | 7 | 9 | 7 | The CPU tier is full and must evict. The heap that tracks least-recently-used blocks has accumulated so many stale entries that it is cheaper to rebuild it from scratch than keep sifting it. Reaching it means filling the CPU tier and then continuing to touch blocks already in it. Every sustained family reaches this; only weka drives the recency-update path that rebuilds the heap from the evictable set. |
| Deferred KV fence reconciled when a block is reused vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py 654–658, 1971–1973 for bid in job_status. deferred_fence_block_ids or (): self._block_id_to_pending_jobs .setdefault(bid, set()).add(job_id) | 8 | – | – | – | 8 | – | A request finished, but its cache blocks were still being copied to CPU. The server cannot free them yet, so it records a marker and settles the copy later, when something else asks for those same blocks. Reaching it means completing responses while transfers are still in flight. Reached only by streaming replay, where responses complete while transfers are still in flight. |
| Bulk cancellation across the waiting, KV-holding and deferred queues vllm/v1/core/sched/scheduler.py 2608–2630 if request.status == RequestStatus.RUNNING: running_requests_to_remove.add(request) self.waiting.remove_requests( waiting_requests_to_remove) | 6 | – | – | – | 6 | 6 | Requests are dropped while still waiting rather than while running, so the scheduler has to pull them out of several different queues at once instead of retiring them normally. Reaching it means clients going away before they are served. Client timeouts under replay and terminated agent sessions both produce it; no generated workload did. |
| Speculative draft tokens trimmed and padded after scheduling vllm/v1/core/sched/scheduler.py 2515–2547, 2917 del spec_token_ids[orig_num_spec_tokens:] num_invalid = orig_num_spec_tokens - len(spec_token_ids) spec_token_ids.extend([-1] * num_invalid) | 10 | – | 10 | 10 | – | – | Speculative decoding guesses several tokens ahead, but the scheduler admitted fewer tokens for this request than the drafter guessed, so the surplus guesses are cut and the gap padded out. Reaching it means batches large enough that drafting and admission disagree. Requires concurrent decode and chunked prefill in the same step. |
| Server metric (CPU GPU) | before | after |
|---|---|---|
| kv_offload_total_bytes_total | 0 | 108.1 GB |
| kv_offload_size_count | 0 | 2,816 ops |
| kv_offload_total_time_total | 0 | 3.33 s |
| GPU CPU bytes (for reference) | 0 | 193.8 GB |
| Coverage, gpu_worker.py | ||
| line 891, def | covered at import | |
| Workload | Input tokens | Output tokens |
|---|---|---|
| HWE-Bench | 0.135 | 0.044 |
| CyberGym | 0.140 | 0.133 |
| HANDBOOK | 0.160 | 0.164 |
| DeepSWE | 0.161 | 0.081 |
| WEKA | 0.223 | 0.212 |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 251 papers