Deploying large language models (LLMs) under a fixed accelerator budget requires jointly deciding how computational capacity is allocated across model sizes and how incoming requests are routed across that capacity. Although prior work has extensively studied LLM routing and serving efficiency, the relative importance of fleet composition and routing under resource constraints remains less understood. We investigate this question using a controlled four-GPU serving environment with heterogeneous clinical NLP workloads spanning short factual verification, biomedical semantic matching, and long-context clinical summarization. We systematically vary fleet composition, routing policy, and offered load while holding the workload trace and hardware budget fixed. Across 24 configurations, descriptive main-effect shares attribute more variation to fleet composition than to the evaluated routing policies in time to first token, latency, and long-context coverage. All-small fleets provide the best responsiveness and throughput but substantially lower long-context coverage, whereas larger-model capacity improves coverage at the cost of serving efficiency. Heterogeneous fleets provide intermediate operating points but can introduce strong-model bottlenecks when large-model capacity is scarce. In a focused evaluation of this bottleneck, capacity-aware routing reduces mean latency and increases throughput while incurring a small reduction in long-context coverage. Within this measured regime, the results support prioritizing capacity allocation, with routing managing demand within the provisioned capacity.
Figures & tables
Component
Realization
Role in evaluation
Workload
One fixed trace spanning three task families and request-cost profiles
Creates simultaneous demand for fast and strong tiers
Arrival
Nominal and overload regimes
When queueing and tail latency emerge
Fleet
4S , 3S+1L , 2S+2L , 4L
How a fixed GPU budget should be divided
Control
Three main routers plus focused gates
What routing adds after provisioning
Outcomes
Serving, GPU, and quality metrics
Which configurations lie on the operational frontier
TABLE I: Experimental factors for mixed-workload inference under a fixed resource envelope. Fleet composition, routing policy, and load are varied independently while the workload trace remains fixed.
Fig. 1: System diagram for the four-GPU study. The displayed 84/60/96 request counts are nominal counts implied by the intended 0.35/0.25/0.40 mixture, not the realized trace; Table III reports the actual 92/59/89 counts. Routing boxes denote alternative policies, not a sequential cascade. Model-quality labels refer to benchmark coverage proxies, not clinical validation.
Policy
Primary signal
Implemented rule
RoundRobin
Instance index
Cycle over all instances without task or model-tier semantics.
GenericTier
Length
Select the strong tier when prompt length ≥1700 tokens or maximum output length ≥224 tokens; otherwise select the fast tier.
ServiceClass
Metadata and risk
Select the strong tier for standard-generation, long-context, background, or summarization requests, or when risk ≥0.60 .
QualityRiskGate
Risk and length
Use risk thresholds 0.62/0.42, length thresholds 1700/224, and clinical-keyword density 0.08 to gate strong-tier escalation.
BottleneckAwareTier
Fleet composition
Treat strong capacity as scarce when its fleet fraction is ≤0.34 ; then use stricter risk thresholds 0.72/0.52 together with 1700/224 length thresholds.
TTFTBudgetGate
Risk and prefill budget
Use risk thresholds 0.72/0.50, prefill budgets 1400/2200, output threshold 224, and strong-fleet scarcity 0.34 to protect first-token responsiveness.
TABLE II: Routing rules used in the main study and focused follow-up. “Risk” is the deterministic profile score produced from task and request metadata; token thresholds refer to estimated prompt or maximum output length.
Dataset family
Requests
Prompt pattern
Serving role
PubHealth
92
Short public-health claim verification with label-style output
Fast, latency-sensitive factual requests
BioSSES
59
Short sentence-pair similarity
Moderate semantic reasoning
MIMIC-BHC
89
Long-context EHR summarization
Expensive, quality-sensitive generation
TABLE III: Workload composition.
Fig. 2: Primary serving metrics for the four-GPU fleet study. Across TTFT, p95 latency, and throughput, fleet composition shifts the operating regime much more than routing-policy swaps do. Marker shape denotes policy, and overload points are marked with an x.
Fleet
Load
Policy
TTFT (s)
Latency (s)
p95 (s)
Throughput (requests/s)
MIMIC recall
4S
N
RR
0.112
2.640
12.37
0.885
0.190
4S
O
RR
0.110
2.693
12.69
1.693
0.192
3S+1L
N
GT
0.175
4.918
23.55
0.586
0.270
3S+1L
O
GT
0.183
5.681
26.74
1.002
0.266
2S+2L
N
GT
0.170
4.685
22.40
0.613
0.264
2S+2L
O
GT
0.169
5.007
23.56
1.126
0.263
TABLE IV: Representative operating points from the GPU fleet matrix.
Fig. 3: Observed TTFT–coverage trade-offs in the original single-run matrix. The vertical axis is MIMIC token recall, not clinical factuality or safety; small coverage gaps have no reported confidence intervals.
Load
Policy
TTFT (s)
Latency (s)
p95 (s)
Throughput (requests/s)
Queue (s)
N
GT
0.173
4.995
23.77
0.578
47.77
N
QRG
0.170
4.981
23.74
0.579
47.50
N
BAT
0.165
4.429
20.89
0.655
34.06
N
TBG
0.157
4.512
22.86
0.634
35.77
O
GT
0.181
5.661
26.74
1.005
30.60
O
QRG
0.175
5.635
26.47
1.010
30.40
TABLE V: Original single-run 3S+1L follow-up using capacity-aware routing variants.
Fleet
Load
Policy
n
TTFT (s)
Latency (s)
p95 (s)
Throughput
Queue (s)
4S
N
GT
4
0.119 ± 0.004
2.691 ± 0.040
12.551 ± 0.349
0.881 ± 0.004
2.680 ± 0.198
4S
O
GT
4
0.112 ± 0.003
2.755 ± 0.031
12.907 ± 0.223
1.685 ± 0.007
1.091 ± 0.097
3S+1L
N
GT
4
0.178 ± 0.002
5.011 ± 0.047
23.743 ± 0.178
0.576 ± 0.005
48.523 ± 1.789
3S+1L
N
BAT
5
0.167 ± 0.002
4.467 ± 0.043
20.985 ± 0.192
0.650 ± 0.005
35.314 ± 1.667
3S+1L
O
GT
4
0.187 ± 0.002
5.680 ± 0.083
27.475 ± 0.700
0.998 ± 0.019
30.851 ± 1.586
3S+1L
O
BAT
5
0.167 ± 0.003
4.940 ± 0.036
23.678 ± 0.235
1.169 ± 0.008
22.265 ± 0.553
TABLE VI: Fresh process-level reliability runs using the fixed 240-request trace.