Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, 4.87× the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.
Figures & tables
Figure 1: API cost and serving value relative to the strongest baseline. GPT 6.1 Sol offers lower-cost gains; Claude reaches higher value. Costs use OpenRouter rates ( Table 3 ).
Figure 2: Agents design the server and choose how many sessions it serves at once for a given model, GPU allocation, and workload. Replay preserves shared context and dependencies between requests; final checks assess model correctness and complete workload execution.
Figure 3: Stateful replay connects session progress to the agent’s design experiments (a). Final evaluation verifies the published system and measures request value against calibrated latency targets (b).
System
Gemma E4B ( 1× L40)
Qwen3.5 9B ( 2× L40)
Qwen3.8 27B INT4 ( 2× L40)
Best baseline
vLLM / = 0.61, c = 8
SGLang / = 1.62, c = 1
vLLM / = 1.47, c = 1
Other baseline
SGLang / = 0.16, c = 2
vLLM / = 1.49, c = 1
SGLang / = 1.42, c = 4
/ @ c
TTFT
TPOT
/ @ c
TTFT
TPOT
/ @ c
TTFT
TPOT
Claude Opus 5.5
2.47 @40
3.01
118.59
4.40 @16
2.25
28.16
2.64 @8
11.44
46.40
Claude Sonnet 5.5
2.97 @32
2.75
83.05
4.34 @20
12.34
26.76
2.01 @12
46.57
47.35
GPT 6 Astra
1.75 @32
4.77
94.19
3.18 @12
15.62
21.94
1.48 @4
12.96
42.83
Table 1: Serving performance across three deployments. Entries pair latency-qualified dollars per dollar with offered session concurrency c ; TTFT (s) and TPOT (ms) are p90 latencies. Shading marks the strongest baseline and largest agent value in each deployment.
Figure 4: Gross value G versus value-weighted SLO meet rate f . Large markers show final evaluations; open markers compare frozen systems at different loads. Arrows indicate increasing concurrency; dashed curves mark equal D=Gf .
Figure 5: Recorded concurrency paths and final outcomes. Colors mark value relative to the strongest baseline; labels count runs. Dots below panels mark first load declarations.
Figure 6: Retained changes more often show workload-relevant gains in the ≥2× group (a); representative trajectories show their follow-up (b). Groups use final value relative to the strongest baseline.
Figure 7: Workload-relevant checks on submitted implementations (a). Related symptoms precede submission in 8 of 11 not-passed runs (b); markers distinguish responses and recorded submission times.
Figure 8: Short windows yield higher average value (a); sampling sensitivity varies across systems (b). Colored curves average systems within each deployment.
Figure 9: Development trajectories connect published implementation changes to the optimizations identified in submitted systems. Filled cells mark latency-aware scheduling (T), speculative decoding (S), cross-turn reuse (R), host-memory state storage (K), and CUDA graphs (G); half-filled K cells indicate recurrent-state storage only. Groups use value relative to each deployment’s strongest reference; outcomes show dollars per dollar or the failed check.
Parameter
Qwen3.5 9B
Gemma E4B
Qwen3.8 27B INT4
Checkpoint source
( Qwen Team, 2026b )
( Google, 2026 )
( Red Hat AI, 2026 )
GPU allocation
2× L40
1× L40
2× L40
Context limit (tokens)
1,048,576
131,072
1,048,576
Input price (USD / million tokens)
0.1044
0.0200
0.0898
Output price (USD / million tokens)
0.1855
0.1000
2.6570
TTFT target (s)
4.10028
2.52693
10.2988
Table 2: Deployment parameters, service targets, and reference operating points.
Agent
Qwen 9B
Gemma E4B
Claude Opus 5.5
99.69
104.25
Claude Sonnet 5.5
78.60
30.72
GPT 6 Astra
110.81
119.79
GPT 6.1 Sol
13.01
13.67
Claude Opus 5
108.90
104.37
GPT 6 Sol
50.88
35.93
Table 3: Agent API cost estimates (USD) at OpenRouter rates for Figure 1 .
Figure 10: Individual concurrency trajectories underlying Figure 5 . Rows show declared concurrency, the maximum tested load, the number of changes, and the final outcome.
Figure 11: Higher-value runs establish end-to-end replay earlier, leaving more time for system development. Three failed runs never reach a usable replay.
Versions
Matched-seed value ratios
Opus 5.5 / Qwen 9B, c=16
v1 → v2
1.88, 1.32
Opus 5 / Qwen 9B, c=2
v3 → v4
1.90, 1.67, 1.44
Sonnet 5.5 / Gemma E4B, c=32
v3 → v4
1.24, 1.02, 1.05
Table 4: Selected qualified-value gains from successive versions at fixed concurrency. Each value corresponds to a scoreable matched-seed pair.
Figure 12: Workload coverage and latency-qualified value in an Opus 5.5 27B development trajectory. (a) Initially available streams across 128 additional workload seeds, with development and final draws marked. (b) Qualified and excluded token value for the same system at 16 sessions under the development and final replays.
Comparison
Seed
D before
D after
D ratio
G ratio
f ratio
Capacity
D
3.533
4.761
1.348
1.110
1.214
E
4.294
4.960
1.155
1.074
1.076
Draft length
D
2.240
2.308
1.030
1.030
1.000
E
2.450
2.544
1.039
1.029
1.009
Replay feedback
F
0.786
0.888
1.129
1.112
1.015
G
0.878
1.148
1.308
1.243
1.052
Table 5: Selected serving improvements from finding-guided changes on fresh common seeds. Capacity compares Qwen v1 to v6 at c=16 ; draft length compares Gemma K=3 to K=7 at c=40 ; replay feedback compares two resumed Opus 5 Gemma servers. Ratios are after/before; all arms pass verification.