cs.DCOct 10, 2026

Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving

Authors: Burak Topcu, Musa Cim, Zuhal Ozturk, Mahmut Taylan Kandemir

Organizations: Department of Computer Science and Engineering, The Pennsylvania State University, State College, PA 16801, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large Language Model (LLM) inference is driving a growing share of datacenter carbon emissions. Although carbon-aware scheduling can reduce these emissions in geo-distributed datacenters by exploiting the spatiotemporal variations in grid carbon intensity, the latency-sensitive nature of LLM serving constrains this design space. While prior studies investigate the carbon savings achievable through spatial scheduling, they oversimplify the dynamics of modern inference engines, such as queueing and continuous batching, which strongly shape inference latency. We address this issue by designing and evaluating a latency-centric, geo-distributed inference simulator, calibrated with hardware-grounded latency and power measurements, that reproduces these dynamics and enables the study of carbon savings under strict service-level objectives (SLOs). We then formulate a series of SLO- and carbon-aware schedulers of increasing sophistication, from per-request heuristics to batch-level optimization: (i) a greedy heuristic, (ii) an integer linear programming (ILP)-based scheduler jointly assigning batches of requests, (iii) an iterative ILP-based solution projecting its own assignments back into the load estimates to capture load coupling, and (iv) a facility mixed-integer linear programming (MILP) scheduler pricing the energy of batched requests collectively rather than in isolation. Across three LLMs under MLPerf SLOs and varying system loads, the MILP-based option offers the best carbon-SLO balance, reducing carbon emissions by up to 60% relative to the round-robin policy, with an average SLO violation of 3%. Our characterization further shows that savings grow with the spread of carbon intensity across datacenters and shrink as system load increases, while SLO attainment depends on how frequently schedulers receive datacenters' load updates.

Figures & tables

Study(i) Formulation(ii) Batch-aware(iii) OverheadsReported latency
PlacementLatency targetLatencyPowerQueueTransfer
SLIT [ 38 ] (2025)Fixed 15-min planMin. TTFT, unbounded✗✗✗✓Only mean TTFT
Green-LLM [ 39 ] (2025)Fixed 1-h planMean E2E ≤\leq 5 s✗✗✗✓None ($ penalty)
FREESH [ 40 ] (2025)Fixed 30-min planE2E ≤5×\leq 5\times profiled✗✗✗✗E2E viol., p99
GCOS [ 41 ] (2026)Per job (GPU-h)Job wait ≤\leq 10–20% runtime–✗✗✓Wait violation
MARLIN [ 42 ] (2026)Fixed 15-min planMin. TTFT, unbounded✗✗✗✓Only mean TTFT
TABLE I: Carbon-aware geo-distributed LLM scheduling approaches (✓ modeled, ✗ not modeled, ∘\circ partial, – not applicable).
Fig. 1: Per-engine latency model across models and TP degrees: (a, b) prefill and decode step time across prefill lengths and batch sizes, respectively; (c) prediction error against measured step time, with per-setup MAPE.
Fig. 2: Per-GPU power during (a) prefill and (b) decode across models and TP degrees.
Fig. 3: Overview of the simulator architecture on an example topology.
ParameterValue
Geo-distributed topology
Schedulers / DCs per scheduler4 / 4
Nodes per DC / GPUs per node4 / 8 ×\times A100-40GB
Inter-region RTT85–280 ms, AWS p50 [ 47 ]
PUE\mathrm{PUE} across DCs1.1 uniform (real per DC in sensitivity)
Serving configuration
TABLE II: Topology, serving, and scheduler parameters.
Fig. 4: Carbon–SLO trade-off and TTFT tail behavior across scheduling policies, loads, and models. G denotes greedy policies, with α=0\alpha{=}0 corresponding to the MOER [ 43 ] policy.
Fig. 5: Carbon saving and SLO violation at high load under uniform and real per-DC PUE [ 72 ] (a) . Facility MILP savings at the base hour and three lowest- and highest-CV hours (CV of CI across DCs) from 2021 to 2024, under varying loads (b) .
Fig. 6: Per-request scheduling time at high load, spanning p50 to p99 ( a ). SLO violation at moderate load across polling periods, under synchronous and asynchronous polling (b) . Carbon saving and SLO violation when serving Llama-3.1-8B at high load across various topologies (c) .
Fig. 7: Carbon saving and SLO violation at high load with oracle and predicted OSLs (a) . Facility MILP on Mixtral-8x7B with long and mixed prompts as the per-engine batch capacity grows in steps of 4 and 8, respectively (b) .
More in this topic

LLM Inference Scheduling

Large language model inference scheduling allocates requests, batches, compute, and KV-cache memory to balance latency, throughput, and service-level objectives.

See all 160 papers

Explore similar work

CardsList
Sep 3, 2026

Toward Sustainable Distributed LLM Inference: A Systems Synthesis and Research Agenda for an Energy-, Carbon-, and Cache-Aware llm-d Control Plane

Large language model (LLM) sustainability is increasingly a serving-systems problem, not only a training problem. In production, energy and carbon impact depend on more than model size: workload shape, batching, key-value (KV) cache reuse, prefill/decode placement, model and accelerator choice, power state, geographic carbon intensity, and service-level objectives (SLOs) all matter. Recent systems papers study many of these factors separately. This paper connects those results and asks a practical engineering question: what do they imply when the decision point is a distributed inference control plane such as llm-d? The contribution here is synthesis, not a new set of benchmark results. Reported performance, energy, carbon, and cost improvements remain the results of the cited papers and systems. I group the literature into recurring design patterns and use those patterns to sketch a Sustainable Inference Control Plane (SICP) for llm-d. The proposed control plane would consider latency, energy, carbon, cache reuse, serving cost, and quality when routing and scaling, while keeping TTFT/TPOT SLOs as hard constraints. I also outline an evaluation framework based on SLO-satisfied goodput per joule and per gram CO2e, together with a reproducible experimental plan. The main observation from connecting the literature is that sustainable LLM inference is unlikely to come from one "green" model or one accelerator; it is more naturally treated as a control problem across model, phase, cache, hardware, replica, region, and time.

Twinkll Sisodia

Energy-Aware LLM InferenceLLM InferenceLLM Inference ServingLLM Inference EfficiencyCarbon-Aware ComputingLLM Inference Scheduling
AskPDF
May 12, 2026

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

The growing deployment of large language models (LLMs) makes per-request routing essential for balancing response quality and computational cost across heterogeneous model pools. Current routing methods rarely consider sustainable energy use and CO2 emissions as optimization objectives, despite grid carbon intensity varying by time and region, and models differing significantly in energy consumption. To address this gap, we introduce Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes per-request CO2 emissions subject to explicit accuracy floors and p95-latency service-level objectives (SLOs). GAR employs adaptive constraint optimization through per-dataset floor tuning and incorporates lightweight estimators for correctness, tail latency, and carbon emissions, enabling real-time routing decisions without additional inference passes. We present GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants that achieve high feasibility coverage while limiting accuracy degradation. Comprehensive experiments across standard NLP benchmarks with heterogeneous LLM pools (7B-70B) demonstrate that GAR achieves substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees, providing a practical, theoretically grounded approach to sustainable LLM inference.

Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha +1

Manipal University Jaipur · TCS Research

LLM Inference EfficiencyConstrained OptimizationLLM RoutingCarbon-Aware Computing
AskPDF
Aug 13, 2026

InFactPlanner: Planning Sustainable Geo-Distributed LLM Data Centers

The rapid growth of LLM inference is shifting sustainability concerns from one-time training to continuous serving, where infrastructure decisions shape energy use, carbon emissions, water consumption, and service quality. Yet operators often need to compare deployment alternatives before large-scale infrastructure is built, making direct measurement costly, slow, and sometimes infeasible. We present InFactPlanner, a trace-driven decision-support framework for what-if analysis of sustainable AI data center deployment for LLM inference across single and geo-distributed sites. InFactPlanner combines query traces, hardware-model profiles, candidate site configurations, PUE/WUE parameters, renewable generation models, and time-varying grid carbon intensity to estimate power, energy, carbon emissions, water use, latency, and server utilization. The framework abstracts low-level serving effects into configurable hardware-model profiles, enabling rapid comparison of site selection, capacity placement, hardware, model, renewable integration, and routing choices. We validate the energy accounting pipeline by reproducing reference LLM inference energy estimates with less than 10% deviation, evaluate scalability across multiple data centers and server counts, and demonstrate scenario-driven decision analyses for hardware selection, renewable placement, geographic deployment, and carbon-aware routing. Our results show that sustainability-optimal choices can differ from latency-optimal ones, and that the carbon value of deployment depends strongly on the local grid mix.

Nicoletta Tsiopani, Moysis Symeonides, George Pallis +1

Department of Computer Science University of Cyprus

Energy-Efficient MLLLM Inference ServingCloud ComputingData Center Energy ManagementCarbon-Aware Computing
AskPDF