Capacity Planning for Resource-Constrained Clinical LLM Serving
Organizations: Population Health Sciences, Weill Cornell Medicine, New York, USA · Systems Engineering, Cornell University, Ithaca, USA
Abstract
Deploying large language models (LLMs) under a fixed accelerator budget requires jointly deciding how computational capacity is allocated across model sizes and how incoming requests are routed across that capacity. Although prior work has extensively studied LLM routing and serving efficiency, the relative importance of fleet composition and routing under resource constraints remains less understood. We investigate this question using a controlled four-GPU serving environment with heterogeneous clinical NLP workloads spanning short factual verification, biomedical semantic matching, and long-context clinical summarization. We systematically vary fleet composition, routing policy, and offered load while holding the workload trace and hardware budget fixed. Across 24 configurations, descriptive main-effect shares attribute more variation to fleet composition than to the evaluated routing policies in time to first token, latency, and long-context coverage. All-small fleets provide the best responsiveness and throughput but substantially lower long-context coverage, whereas larger-model capacity improves coverage at the cost of serving efficiency. Heterogeneous fleets provide intermediate operating points but can introduce strong-model bottlenecks when large-model capacity is scarce. In a focused evaluation of this bottleneck, capacity-aware routing reduces mean latency and increases throughput while incurring a small reduction in long-context coverage. Within this measured regime, the results support prioritizing capacity allocation, with routing managing demand within the provisioned capacity.
Figures & tables
| Component | Realization | Role in evaluation |
| Workload | One fixed trace spanning three task families and request-cost profiles | Creates simultaneous demand for fast and strong tiers |
| Arrival | Nominal and overload regimes | When queueing and tail latency emerge |
| Fleet | 4S , 3S+1L , 2S+2L , 4L | How a fixed GPU budget should be divided |
| Control | Three main routers plus focused gates | What routing adds after provisioning |
| Outcomes | Serving, GPU, and quality metrics | Which configurations lie on the operational frontier |
| Policy | Primary signal | Implemented rule |
| RoundRobin | Instance index | Cycle over all instances without task or model-tier semantics. |
| GenericTier | Length | Select the strong tier when prompt length tokens or maximum output length tokens; otherwise select the fast tier. |
| ServiceClass | Metadata and risk | Select the strong tier for standard-generation, long-context, background, or summarization requests, or when risk . |
| QualityRiskGate | Risk and length | Use risk thresholds 0.62/0.42, length thresholds 1700/224, and clinical-keyword density 0.08 to gate strong-tier escalation. |
| BottleneckAwareTier | Fleet composition | Treat strong capacity as scarce when its fleet fraction is ; then use stricter risk thresholds 0.72/0.52 together with 1700/224 length thresholds. |
| TTFTBudgetGate | Risk and prefill budget | Use risk thresholds 0.72/0.50, prefill budgets 1400/2200, output threshold 224, and strong-fleet scarcity 0.34 to protect first-token responsiveness. |
| Dataset family | Requests | Prompt pattern | Serving role |
| PubHealth | 92 | Short public-health claim verification with label-style output | Fast, latency-sensitive factual requests |
| BioSSES | 59 | Short sentence-pair similarity | Moderate semantic reasoning |
| MIMIC-BHC | 89 | Long-context EHR summarization | Expensive, quality-sensitive generation |
| Fleet | Load | Policy | TTFT (s) | Latency (s) | p95 (s) | Throughput (requests/s) | MIMIC recall |
| 4S | N | RR | 0.112 | 2.640 | 12.37 | 0.885 | 0.190 |
| 4S | O | RR | 0.110 | 2.693 | 12.69 | 1.693 | 0.192 |
| 3S+1L | N | GT | 0.175 | 4.918 | 23.55 | 0.586 | 0.270 |
| 3S+1L | O | GT | 0.183 | 5.681 | 26.74 | 1.002 | 0.266 |
| 2S+2L | N | GT | 0.170 | 4.685 | 22.40 | 0.613 | 0.264 |
| 2S+2L | O | GT | 0.169 | 5.007 | 23.56 | 1.126 | 0.263 |
| Load | Policy | TTFT (s) | Latency (s) | p95 (s) | Throughput (requests/s) | Queue (s) |
| N | GT | 0.173 | 4.995 | 23.77 | 0.578 | 47.77 |
| N | QRG | 0.170 | 4.981 | 23.74 | 0.579 | 47.50 |
| N | BAT | 0.165 | 4.429 | 20.89 | 0.655 | 34.06 |
| N | TBG | 0.157 | 4.512 | 22.86 | 0.634 | 35.77 |
| O | GT | 0.181 | 5.661 | 26.74 | 1.005 | 30.60 |
| O | QRG | 0.175 | 5.635 | 26.47 | 1.010 | 30.40 |
| Fleet | Load | Policy | TTFT (s) | Latency (s) | p95 (s) | Throughput | Queue (s) | |
| 4S | N | GT | 4 | 0.119 0.004 | 2.691 0.040 | 12.551 0.349 | 0.881 0.004 | 2.680 0.198 |
| 4S | O | GT | 4 | 0.112 0.003 | 2.755 0.031 | 12.907 0.223 | 1.685 0.007 | 1.091 0.097 |
| 3S+1L | N | GT | 4 | 0.178 0.002 | 5.011 0.047 | 23.743 0.178 | 0.576 0.005 | 48.523 1.789 |
| 3S+1L | N | BAT | 5 | 0.167 0.002 | 4.467 0.043 | 20.985 0.192 | 0.650 0.005 | 35.314 1.667 |
| 3S+1L | O | GT | 4 | 0.187 0.002 | 5.680 0.083 | 27.475 0.700 | 0.998 0.019 | 30.851 1.586 |
| 3S+1L | O | BAT | 5 | 0.167 0.003 | 4.940 0.036 | 23.678 0.235 | 1.169 0.008 | 22.265 0.553 |
| Fleet | Load | Policy | TTFT (s) | Latency (s) | p95 (s) | Throughput (requests/s) | Queue | GPU util. (%) | PubHealth acc. | MIMIC recall |
| 4S | N | RR | 0.112 | 2.640 | 12.37 | 0.885 | 2.412 | 51.68 | 0.6196 | 0.1900 |
| 4S | N | GT | 0.112 | 2.628 | 12.38 | 0.885 | 2.401 | 51.05 | 0.6196 | 0.1889 |
| 4S | N | SC | 0.112 | 2.635 | 12.38 | 0.885 | 2.409 | 51.58 | 0.6196 | 0.1893 |
| 4S | O | RR | 0.110 | 2.693 | 12.69 | 1.693 | 0.972 | 74.51 | 0.6196 | 0.1924 |
| 4S | O | GT | 0.109 | 2.703 | 12.68 | 1.693 | 0.979 | 73.93 | 0.6196 | 0.1927 |
| 4S | O | SC | 0.110 | 2.689 | 12.34 | 1.698 | 0.943 | 74.25 | 0.6196 | 0.1901 |
| Primary objective | Starting fleet | Operational interpretation |
| Responsiveness | 4S | Lowest TTFT and highest throughput; weaker long-context coverage |
| Balanced service | 2S+2L | Reduces dependence on one strong server while retaining measured coverage |
| Scarce strong tier | 3S+1L | Use capacity-aware escalation to avoid concentrating the queue |
| Coverage-heavy | 4L | Strong-model capacity throughout, with a slower serving regime |
LLM Serving
Large language model serving studies how to deliver model responses with low latency, high throughput, and efficient use of compute and memory.
See all 249 papers