Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane
Abstract
Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Reported performance, energy, carbon, and cost improvements remain the results of the cited papers and systems. I group the literature into recurring design patterns and use those patterns to sketch a Sustainable Inference Control Plane (SICP) for llm-d. The proposed control plane would consider latency, energy, carbon, cache reuse, serving cost, and quality when routing and scaling, while keeping TTFT/TPOT SLOs as hard constraints. I also outline an evaluation framework based on SLO-satisfied goodput per joule and per gram CO2e, together with a reproducible experimental plan. The main observation from connecting the literature is that sustainable LLM inference is unlikely to come from one "green" model or one accelerator; it is more naturally treated as a control problem across model, phase, cache, hardware, replica, region, and time.
Figures & tables
| Work | Primary lever | Representative finding | Implication for sustainable distributed inference |
|---|---|---|---|
| vLLM / PagedAttention ( Kwon et al., 2023 ) | KV memory | 2–4 throughput over prior serving baselines at similar latency in reported experiments | Memory fragmentation and cache duplication are sustainability issues because they reduce useful work per accelerator-hour. |
| Splitwise ( Patel et al., 2024 ) | P/D phase split | Up to 1.4 throughput at 20% lower cost; alternative design points increase throughput under fixed power/cost | Match accelerator class and power state to phase behavior. |
| DistServe ( Zhong et al., 2024 ) | P/D disaggregation | Large goodput gains under TTFT/TPOT constraints in reported workloads | Optimize sustainability subject to phase-specific SLOs, not average latency. |
| QServe ( Lin et al., 2024 ) | W4A8KV4 co-design | Higher serving throughput and lower dollar cost than compared baselines | Precision must be selected with hardware/kernel awareness; KV precision is a first-class lever. |
| Energy Considerations ( Fernandez et al., 2025 ) | Workload-aware optimization | Up to 73% lower energy than an unoptimized baseline; optimization efficacy varies by workload/hardware | Sustainability policies must be conditional and learned from telemetry. |
| DynamoLLM ( Stojkovic et al., 2025b ) | Dynamic cluster management | Reports 53% energy and 38% operational-carbon reduction while meeting SLOs | Energy/carbon belong in the cluster controller’s objective. |