Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, 4.87× the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.
Figures & tables
Figure 1: API cost and serving value relative to the strongest baseline. GPT 6.1 Sol offers lower-cost gains; Claude reaches higher value. Costs use OpenRouter rates ( Table 3 ).
Figure 2: Agents design the server and choose how many sessions it serves at once for a given model, GPU allocation, and workload. Replay preserves shared context and dependencies between requests; final checks assess model correctness and complete workload execution.
Figure 3: Stateful replay connects session progress to the agent’s design experiments (a). Final evaluation verifies the published system and measures request value against calibrated latency targets (b).
System
Gemma E4B ( 1× L40)
Qwen3.5 9B ( 2× L40)
Qwen3.8 27B INT4 ( 2× L40)
Best baseline
vLLM / = 0.61, c = 8
SGLang / = 1.62, c = 1
vLLM / = 1.47, c = 1
Other baseline
SGLang / = 0.16, c = 2
vLLM / = 1.49, c = 1
SGLang / = 1.42, c = 4
/ @ c
TTFT
TPOT
/ @ c
TTFT
TPOT
/ @ c
TTFT
TPOT
Claude Opus 5.5
2.47 @40
3.01
118.59
4.40 @16
2.25
28.16
2.64 @8
11.44
46.40
Claude Sonnet 5.5
2.97 @32
2.75
83.05
4.34 @20
12.34
26.76
2.01 @12
46.57
47.35
GPT 6 Astra
1.75 @32
4.77
94.19
3.18 @12
15.62
21.94
1.48 @4
12.96
42.83
Table 1: Serving performance across three deployments. Entries pair latency-qualified dollars per dollar with offered session concurrency c ; TTFT (s) and TPOT (ms) are p90 latencies. Shading marks the strongest baseline and largest agent value in each deployment.
Figure 4: Gross value G versus value-weighted SLO meet rate f . Large markers show final evaluations; open markers compare frozen systems at different loads. Arrows indicate increasing concurrency; dashed curves mark equal D=Gf .
Figure 5: Recorded concurrency paths and final outcomes. Colors mark value relative to the strongest baseline; labels count runs. Dots below panels mark first load declarations.
Figure 6: Retained changes more often show workload-relevant gains in the ≥2× group (a); representative trajectories show their follow-up (b). Groups use final value relative to the strongest baseline.
Figure 7: Workload-relevant checks on submitted implementations (a). Related symptoms precede submission in 8 of 11 not-passed runs (b); markers distinguish responses and recorded submission times.
Figure 8: Short windows yield higher average value (a); sampling sensitivity varies across systems (b). Colored curves average systems within each deployment.
Figure 9: Development trajectories connect published implementation changes to the optimizations identified in submitted systems. Filled cells mark latency-aware scheduling (T), speculative decoding (S), cross-turn reuse (R), host-memory state storage (K), and CUDA graphs (G); half-filled K cells indicate recurrent-state storage only. Groups use value relative to each deployment’s strongest reference; outcomes show dollars per dollar or the failed check.
Parameter
Qwen3.5 9B
Gemma E4B
Qwen3.8 27B INT4
Checkpoint source
( Qwen Team, 2026b )
( Google, 2026 )
( Red Hat AI, 2026 )
GPU allocation
2× L40
1× L40
2× L40
Context limit (tokens)
1,048,576
131,072
1,048,576
Input price (USD / million tokens)
0.1044
0.0200
0.0898
Output price (USD / million tokens)
0.1855
0.1000
2.6570
TTFT target (s)
4.10028
2.52693
10.2988
Table 2: Deployment parameters, service targets, and reference operating points.
Agent
Qwen 9B
Gemma E4B
Claude Opus 5.5
99.69
104.25
Claude Sonnet 5.5
78.60
30.72
GPT 6 Astra
110.81
119.79
GPT 6.1 Sol
13.01
13.67
Claude Opus 5
108.90
104.37
GPT 6 Sol
50.88
35.93
Table 3: Agent API cost estimates (USD) at OpenRouter rates for Figure 1 .
Figure 10: Individual concurrency trajectories underlying Figure 5 . Rows show declared concurrency, the maximum tested load, the number of changes, and the final outcome.
Figure 11: Higher-value runs establish end-to-end replay earlier, leaving more time for system development. Three failed runs never reach a usable replay.
Versions
Matched-seed value ratios
Opus 5.5 / Qwen 9B, c=16
v1 → v2
1.88, 1.32
Opus 5 / Qwen 9B, c=2
v3 → v4
1.90, 1.67, 1.44
Sonnet 5.5 / Gemma E4B, c=32
v3 → v4
1.24, 1.02, 1.05
Table 4: Selected qualified-value gains from successive versions at fixed concurrency. Each value corresponds to a scoreable matched-seed pair.
Figure 12: Workload coverage and latency-qualified value in an Opus 5.5 27B development trajectory. (a) Initially available streams across 128 additional workload seeds, with development and final draws marked. (b) Qualified and excluded token value for the same system at 16 sessions under the development and final replays.
Comparison
Seed
D before
D after
D ratio
G ratio
f ratio
Capacity
D
3.533
4.761
1.348
1.110
1.214
E
4.294
4.960
1.155
1.074
1.076
Draft length
D
2.240
2.308
1.030
1.030
1.000
E
2.450
2.544
1.039
1.029
1.009
Replay feedback
F
0.786
0.888
1.129
1.112
1.015
G
0.878
1.148
1.308
1.243
1.052
Table 5: Selected serving improvements from finding-guided changes on fresh common seeds. Capacity compares Qwen v1 to v6 at c=16 ; draft length compares Gemma K=3 to K=7 at c=40 ; replay feedback compares two resumed Opus 5 Gemma servers. Ratios are after/before; all arms pass verification.
Large language model (LLM)-based multi-agent systems (MAS) coordinate specialized agents to solve complex tasks, turning each user request into interdependent model and tool operations. Unlike independent requests, these operations become ready at different times and consume evolving context as agents coordinate and exchange results. This creates changing patterns of concurrency, computation, and memory demand that can expose different bottlenecks in LLM serving hardware. Yet benchmarking these effects requires capturing the collaboration rules that generate serving workloads, beyond the requests observed during execution. We present MAServeBench, a benchmark that treats multi-agent collaboration as a configurable dimension of serving workloads for LLM serving and hardware evaluation. Its composable stages capture how agents coordinate and exchange context in documented MAS workflows, enabling native executions to generate different workloads even when task inputs and tools remain unchanged. Hierarchical Semantic Traces preserve realized operations and dependencies for reproducible fixed-work replay and as inputs to architecture simulators. Task-linked measurements connect these workloads to serving execution and GPU/NPU hardware behavior, revealing why serving efficiency does not always translate into timely task completion. The benchmark code is available at https://github.com/ShengxuanQiu/MAServeBench.
For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engineer-years to support every model and workload. In this paper, we take the opposite bet and let coding agents automatically synthesize bespoke serving systems for different usage scenarios. We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end. VibeServe uses an outer loop to plan and track the search over system designs, and an inner loop to implement candidates, check correctness, and measure performance on the target benchmark. In standard serving settings, VibeServe matches vLLM and SGLang on Llama-3.1-8B, and delivers 2.3x the goodput of a tuned SGLang for a 397B MoE model on four AMD MI300As. In five non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss, spanning non-standard model architectures, workload knowledge, and hardware-specific optimizations. Together, these results argue that infrastructure software should favor generation-time specialization over runtime generality. Code is available at https://github.com/uw-syfi/vibesys.
Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real workload patterns, yet the data needed for such analysis is largely absent. Existing public traces and benchmarks do not capture real, day-to-day coding-agent usage across multiple agents and model families for serving-system analysis. To help fill this gap, we collect and release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls from our own day-to-day use of Claude Code and Codex. Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates. These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps. We release the dataset, trace collection pipeline, and analysis code at https://github.com/uw-syfi/TraceLab.git the project website is https://tracelab.cs.washington.edu.
Kan Zhu, Mathew Jacob, Chenxi Ma +4
University of Washington · 2Wuhan University of Technology · *Work done while interning at the University of Washington. +1