Deploying a large language model (LLM) requires choosing the right accelerator, parallelization strategy, engine configuration, and replica count amongst ~140 other variables across clouds, regions, and markets. These decisions are intertwined, and the best cluster-wide plan continually changes as jobs contend for shared capacity on shifting workloads. We present Koi, a self-calibrating cluster planner for heterogeneous, multi-cloud fleets. Koi uses a Recursive Language Model (RLM) hierarchy to propose hypotheses and reason over deployment while deterministic components predict serving behavior, score candidates, and control placement. It represents inference behavior as a fixed causal graph linking decisions (e.g., accelerator, parallelism), runtime mediators (e.g., KV-cache pressure, communication overhead), and outcomes (e.g., latency, throughput). Each deployment strengthens or weakens scoped causal relationships, building a comprehensive playbook for inference that the planner employs to guide future decisions. Job-wise configurations are composed into cluster-wide plans and evaluated against serving objectives, resource constraints, SLO risks, and cost. Across benchmarks, Koi achieves 1.18x greater goodput than the strongest baseline while reducing $/token by 20% in a real heterogeneous cluster. Ablations isolate the contribution of Koi's architecture: a pure language model attains a composite score of 35.3, the same model equipped with Koi's deterministic tools reaches 51.6, and the full Koi planner reaches 70.8.
Figures & tables
Figure 1. Parallelism configurations expose conflicting performance, cost, and utilization trade-offs. It used Llama 3 70B on AWS A10G GPUs with 2,048/512 input/output tokens. Each axis is normalized independently.
Figure 2. Parallelism substantially changes inference performance and cost. Done on AWS A10G instances with Llama 3 70B model. The workload has 2,048 input and 512 output tokens per request.
Figure 3. End-to-end Koi architecture. Each planning tick snapshots cluster state, observes runtime telemetry, updates causal confidence, proposes deployment candidates, and deterministically evaluates and selects a cluster-wide plan. Full end-to-end \systemsystem architecture.
Method
Hrs
Batch prog. (%)
Admit online (%)
Admit batch (%)
/token(10^{-4}$ )
TTFT met (%)
TPOT met (%)
Goodput (tok/s)
Overall score
Non-learning baselines
FCFS + first-fit
3
57.71
77.78
57.14
2.95
3.90
6.89
1830.84
49.13
12
38.69
78.26
50.00
6.09
2.45
6.27
1772.01
31.46
Best fixed configuration
3
57.71
66.67
57.14
2.91
5.24
13.61
1857.31
50.24
12
24.54
69.57
57.14
29.65
7.63
13.55
364.03
17.59
EDF + first-fit
3
57.71
77.78
57.14
2.94
4.05
7.27
1833.16
49.27
Table 1. Simulated benchmark results for 3-hour (400-GPU) and 12-hour (800-GPU) heterogeneous workloads. Bold marks the best value and underline the second-best; ties are marked for all matching methods
Method
Batch prog. (%)
Admit online (%)
Admit batch (%)
/token(10^{-4}$ )
TTFT met (%)
TPOT met (%)
Goodput (tok/s)
Overall score
Contextual bandit
50.00
53.33
100.00
0.45
22.23
33.66
2291.03
51.05
Rolling EvoX
50.00
80.00
100.00
0.44
28.08
35.48
2348.15
54.90
OpenEvolve
75.00
100.00
100.00
0.30
65.66
91.41
3497.43
84.44
Koi
100.00
100.00
100.00
0.24
79.05
99.11
4119.51
94.50
Δ vs. best baseline
+25.00pp
+0.00pp
+0.00pp
+20.00%
+13.39pp
+7.70pp
+17.79%
+11.91%
Table 2. Real-hardware results for the 3-hour, 104-GPU heterogeneous benchmark. Bold marks the best value and underline the second-best.
Figure 4. Real-cluster allocation, SLO attainment, and adaptation over three hours. (a) Plan-declared logical GPU allocation for each policy, with Koi’s realized swap marked; (b) equal-job mean TTFT (solid) and TPOT (dashed) attainment among served online jobs; (c) Koi’s calibrated DRO radius; and (d) active/applicable and cumulatively discovered mechanisms. \gmnew{Four time-series panels from the three-hour real-cluster runs compare logical GPU allocation and served-job SLO attainment across five policies, and show \system's DRO radius and active and cumulative mechanism counts.}
Figure 5. Online workload scheduling across methods. Per-job GPU assignments for nine workloads over 12 hours under Koi, EvoX, OpenEvolve, the contextual bandit, and GPT-only. Colors denote GPU families; markers indicate placement, swap, and rank relaunch events. Five aligned timelines compare GPU assignments for the same nine online workloads over 12 hours. Rows identify workloads, columns represent time, colors identify GPU families, and markers identify placement, swap, and rank relaunch events.
Figure 6. Ablations over the 3-hour heterogeneous simulation. Overall score, TTFT and TPOT SLO attainment, and goodput across five planner variants. Three bar-chart panels compare five planner variants on overall score, TTFT and TPOT SLO attainment, and goodput. Full \systemachieves the highest value across all reported metrics, while GPT-only and \systemablations perform worse.
Method
Batch progress (%)
Admit (%) online / batch
/token(10^{-4}$ )
SLO met (%) TTFT/TPOT
Goodput (tok/s)
Overall score
Best Fixed Configuration
37.58
60.87/42.86
3.56
24.38/31.09
2147.77
50.69
Contextual Bandit
1.89
43.48/7.14
14.04
10.32/10.95
544.34
14.59
Rolling EvoX
14.92
65.22/14.29
4.76
29.61/37.10
1606.39
40.37
Openevolve
14.92
65.22/14.29
6.76
20.26/32.46
1130.77
33.18
GPT-Only (non agentic)
40.06
65.22/ 57.14
4.29
18.15/33.62
1783.61
49.17
Koi
36.62
78.26 / 57.14
3.51
31.37 / 39.29
2175.63
56.68
Table 3. Mixed online/batch homogeneous results. All runs use the 12-hour workload horizon on the simulator.
Variant
Batch progress (%)
Admit (%) online / batch
/token(10^{-4}$ )
SLO met (%) TTFT/TPOT
Goodput (tok/s)
Overall score
GPT-only (non agentic)
52.04
77.78/71.43
6.31
3.98/18.98
855.60
35.28
Opencode SDK (GPT-5.6-Luna)
54.77
88.89/57.14
2.95
25.74/32.62
1826.64
51.55
Koi without DAG/mechanisms
59.42
94.44/100.00
3.05
21.66/38.33
1770.03
58.34
Koi without online learning
63.01
94.44/85.71
3.33
16.49/37.74
1621.48
54.79
Full Koi
84.55
94.44 /71.43
1.85
28.79 / 41.80
2910.64
70.83
Δ vs. best baseline
+21.54pp
−8.00 pp
+37.29%
+3.05pp
+59.34%
+21.41%
Table 4. Agentic and Koi ablations. All runs use the 3-hour workload horizon on the simulator.
Base model
Batch progress (%)
Admit (%) online / batch
Planner $/tick
Overall score
GPT-5.6 Sol
67.31
94.44/85.71
0.75
62.12
DeepSeek-V4 Pro
48.76
94.44/71.43
0.30
50.44
Cohere Command-A
67.37
94.44/85.71
0.44
68.84
Koi (default, GPT-5.6 Luna)
84.55
94.44 /71.43
0.057
70.83
Table 5. Koi backbone LLM ablation. All runs use the 3-hour workload horizon on the simulator.