Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

Xiaoyang Zhang; Yuejun Yan; Qingang Zhang; Yonggang Wen

cs.DCOct 26, 2026

Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

Authors: Xiaoyang Zhang, Yuejun Yan, Qingang Zhang, Yonggang Wen

Organizations: Nanyang Technological University · Alibaba Group

Abstract

The sustainability of the Web has drawn growing attention, as large language model (LLM)-based web services leave a substantial carbon footprint. These services typically run in geo-distributed data centers whose grid carbon intensity varies across regions and over time, so routing each query to a cleaner region or a smaller model can reduce emissions. A conversation, however, is a stateful web session: each turn reuses the key--value (KV) cache of its history, which resides only where earlier turns were served. Serving a turn on another deployment requires recomputing this history, whose carbon cost grows with the conversation and can outweigh the saving of the move. Existing carbon-aware LLM routers treat a conversation's queries as independent and ignore this cost. We present CarbonRouter, an online framework that jointly selects the model, GPU type, and region for each query to minimize carbon emissions. CarbonRouter introduces a cache-aware carbon cost model, which charges history recomputation to every candidate deployment using cached-token counts from the serving backends, and uses a lightweight gate that keeps a session on its cached deployment when it would likely move back at the next turn and rebuild its history again. We evaluate CarbonRouter in trace-driven experiments on two real-world conversation datasets with five LLMs, three GPU types, and four regions, where it achieves the lowest carbon of all policies. CarbonRouter reduces CO $_2$ by 16.1% on average relative to FairCarbon, a state-of-the-art carbon-aware router with the same action space, while also improving predicted answer quality and estimated latency. Moreover, the saving rises further to 23.1% for sessions longer than 30 turns.

Figures & tables

Figure 1. Carbon intensity. (a) 2024 annual mean of 21 countries. (b) Five-minute intensity of our four regions (strip: cleanest region). Left: bar chart of annual carbon intensity for 21 countries, from India at about 705 gCO2e/kWh to Switzerland at about 35, a 20.4-fold spread. Right: four time series (AT, GB, BE, US-CAL-CISO) over four days varying between about 60 and 330 gCO2e/kWh, with a colored strip marking the greenest region.

Figure 2. Carbon cost of a KV-cache miss on ShareGPT. (a) History length. (b) Extra carbon of a miss (bars: median, error bars: interquartile range) and extra prefill latency (line). (c) Break-even ratio

CI_{B}/CI_{A}

for leaving cached region

A

, with 128 new and 256 output tokens.

H^{\star}

: break-even history at a ratio of 0.5. Left: bar chart of query shares by history length, with 24 percent of queries above 8K tokens. Middle: extra carbon of a cache miss grows from about 3 percent for short histories to about 630 percent above 8K tokens, and extra latency grows to about 2.5 seconds. Right: the break-even ratio falls from 1 at zero history to 0.14 at 16K tokens; at a ratio of 0.5 the break-even history is about 1.8K tokens.

Figure 3. Overview of CarbonRouter: each query goes to the lowest-carbon model, GPU, and region, given its cached history. Four colored panels from left to right. Web LLM Users: three users at laptops, two example turns of a chat session, and a gateway. Request Processing: a quality predictor with a bar chart of predicted quality for 1B to 70B models against a dashed quality floor, and a token estimator. Above the CCAR Routing Engine, two independent modules, a carbon intensity API and energy-and-latency profiles, feed its carbon estimate. CCAR Routing Engine: a pipeline of filter feasible, estimate carbon, and greedy proposal; a session-state box holding the last deployment and the cached tokens feeds the carbon estimate and a gate, drawn as a diamond, that asks whether staying saves carbon, with a green yes-branch staying on the cached deployment in GB and a red no-branch switching to the greedy proposal. Geo-distributed Data Centers: AT, GB, BE, and US-CAL-CISO, each with A10, A100, and H100 GPUs, GB holding the KV cache, plus accounting and logs. An arrow along the bottom carries the cached tokens per deployment, reported by the backends, to the session state.

Method	CO ${}_{2}\downarrow$	Quality $\uparrow$	P95 TTFT (s) $\downarrow$	CO 2 saving (%)	CO ${}_{2}\downarrow$	Quality $\uparrow$	P95 TTFT (s) $\downarrow$	CO 2 saving (%)	CO 2 saving (%)
	ShareChat				ShareGPT				Average
Session-Affinity	1.000	1.000	0.05	53.7 [52.3, 55.1]	1.000	1.000	0.04	54.9 [52.8, 56.9]	54.3 [53.1, 55.6]
Carbon-Affinity	0.586	0.999	0.09	21.0 [20.0, 22.0]	0.575	1.000	0.05	21.6 [19.7, 23.5]	21.3 [20.3, 22.4]
GAR †	0.630	0.973	1.72	26.5 [24.2, 28.8]	0.660	0.976	0.56	31.6 [28.5, 34.8]	29.1 [27.1, 30.9]
FairCarbon	0.545	0.973	1.67	15.1 [13.4, 16.9]	0.544	0.976	0.53	17.0 [14.3, 19.7]	16.1 [14.4, 17.6]
CarbonRouter (ours)	0.463	0.979	0.52	–	0.451	0.979	0.20	–	–

Table 1. Overall performance, normalized to Session-Affinity

=1

. P95 TTFT is estimated from prefill time only (Equation ( 6 )). CO 2 saving: CarbonRouter’s saving over each row, with 95% confidence intervals. † Adapted to our setting.

Figure 4. CarbonRouter versus FairCarbon on ShareChat by session length, with 95% confidence intervals in (a). Four bar-chart panels over four nested session-length groups (more than 5, 10, 20, and 30 turns) on ShareChat. CarbonRouter's carbon saving over FairCarbon is 15.3, 16.6, 19.8, and 23.1 percent. FairCarbon's cache hit rate stays near 59 to 61 percent, whereas CarbonRouter's rises from 67.9 to 73.7 percent. FairCarbon's switches per session rise from 7.8 to 22.4 and its recomputation energy per query from 129 to 205 mWh, while CarbonRouter's switches rise from 6.1 to 14.9 and its recomputation energy stays between 54 and 61 mWh.

Variant	CO ${}_{2}\downarrow$	Energy $\downarrow$	TTFT (s) $\downarrow$	$\Delta$ CO 2 (%)	$\Delta$ TTFT (%)
CarbonRouter	0.463	0.826	0.52	–	–
w/o Gate	0.479	0.854	0.89	3.5 [2.6, 4.6]	71.4 [60.3, 81.9]
w/o Pricing	0.504	0.907	0.74	8.9 [7.8, 10.2]	42.4 [35.7, 49.6]

Table 2. Ablation on ShareChat, normalized to Session-Affinity

=1

. TTFT: estimated P95, as in Table 1 .

\Delta

: increase over CarbonRouter, with 95% confidence intervals.

Figure 5. Ablation on ShareChat by session length. (a) CO 2 increase over the full model, with 95% confidence intervals. (b) 95th-percentile estimated TTFT. (c) Switches blocked by the gate per session. Three bar-chart panels over four session-length groups (all sessions and sessions longer than 10, 20, and 30 turns) on ShareChat. (a) The CO2 increase without the gate rises from 3.5 to 7.3 percent, and without pricing from 8.9 to 11.7 percent. (b) CarbonRouter's estimated P95 TTFT stays between 0.52 and 0.61 seconds, against 0.89 to 1.47 seconds without the gate and 0.74 to 1.10 seconds without pricing. (c) The gate blocks 0.7 switches per session over all sessions and 4.1 for sessions longer than 30 turns.

$\gamma$	$\kappa=5$	$\kappa=6$	$\kappa=8$
0.025	18.8 [16.7, 21.0]	16.7 [14.7, 18.8]	13.6 [11.9, 15.4]
0.05	16.8 [14.9, 18.7]	15.1 [13.4, 16.9]	12.6 [11.1, 14.1]
0.1	12.9 [11.2, 14.7]	12.7 [11.1, 14.1]	9.7 [8.4, 11.0]

Table 3. CO 2 saving (%) of CarbonRouter over FairCarbon on ShareChat by

\gamma

and

\kappa

, with 95% confidence intervals.

$c$	$\beta_{Q}=0.8$	$\beta_{Q}=0.85$	$\beta_{Q}=0.9$
0.5	17.8 [16.1, 19.4]	15.1 [13.4, 16.9]	13.2 [11.9, 14.6]
0.35	11.4 [9.7, 13.1]	11.2 [9.4, 12.8]	11.4 [9.7, 13.2]
0.25	10.7 [9.0, 12.5]	9.4 [7.7, 11.0]	10.5 [8.6, 12.3]

Table 4. CO 2 saving (%) of CarbonRouter over FairCarbon on ShareChat by

c

and

\beta_{Q}

, with 95% confidence intervals.

Carbon-aware computing
Method	Model	GPU	Region	KV cache	Interactive workload
Temporal shifting ( Radovanović et al., 2023 ; Wiesner et al., 2021 ; Hanafy et al., 2023 )	✗	✗	✗	✗	✗
Geo. load balancing ( Liu et al., 2011 ; Li et al., 2024 )	✗	✗	✓	✗	✓
Carbon-aware LLM inference
Bernhard et al. ( Bernhard and Yardimci, 2026 )	✗	✗	✓	✗	✓
AgentDecarbonizer ( Yan et al., 2026 )	✗	✗	✓	✓	✗

Table 5. Comparison with existing carbon-aware methods.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	GPU	Relative energy	Prefill	MFU	Decode
Gemma-3-1B	A10	0.048	8,725	0.14	300.0
	A100	0.033	33,348	0.21	777.5
	H100	0.027	72,950	0.15	1,675.0
Llama-3.2-3B	A10	0.115	3,623	0.17	100.0
	A100	0.080	13,848	0.27	259.2
	H100	0.064	30,292	0.18	558.3

Appendix

Table 6. Analytical model–GPU profiles. Energy is relative to Llama-3.1-70B on A100, and throughputs are in tokens/s.

Carbon-Aware Computing

See all 21 papers

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization
May 12, 2026Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha +1LLM Inference Efficiency Constrained Optimization
Manipal University Jaipur · TCS Research
Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems
Sep 11, 2026Aikaterini Maria Panteleaki, Varatheepan Paramanayakam, Spyros Tragoudas +1LLM Routing Carbon-Aware Computing
School of Electrical, Computer and Biomedical Engineering Southern Illinois University Carbondale
Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving
Oct 10, 2026Burak Topcu, Musa Cim, Zuhal Ozturk +1LLM Inference Scheduling Carbon-Aware Computing
Department of Computer Science and Engineering, The Pennsylvania State University, State College, PA 16801, USA

May 12, 2026J/K move · Enter open · S save

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar Dasgupta

Manipal University Jaipur · TCS Research

LLM Inference Efficiency Constrained Optimization LLM Routing Carbon-Aware Computing

Open paper AskPDF arXiv

The growing deployment of large language models (LLMs) makes per-request routing essential for balancing response quality and computational cost across heterogeneous model pools. Current routing methods rarely consider sustainable energy use and CO2 emissions as optimization objectives, despite grid carbon intensity varying by time and region, and models differing significantly in energy consumption. To address this gap, we introduce Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes per-request CO2 emissions subject to explicit accuracy floors and p95-latency service-level objectives (SLOs). GAR employs adaptive constraint optimization through per-dataset floor tuning and incorporates lightweight estimators for correctness, tail latency, and carbon emissions, enabling real-time routing decisions without additional inference passes. We present GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants that achieve high feasibility coverage while limiting accuracy degradation. Comprehensive experiments across standard NLP benchmarks with heterogeneous LLM pools (7B-70B) demonstrate that GAR achieves substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees, providing a practical, theoretically grounded approach to sustainable LLM inference.

Abstract

Carbon-Aware Computing

Explore similar work

GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

Carbon-Aware Routing for Function Calling in Edge-Cloud LLM Systems

Dissecting the Carbon-SLO Tension in Geo-Distributed LLM Serving